Paper deep dive
H-Node Attack and Defense in Large Language Models
Eric Yocam, Varghese Vaidyan, Yong Wang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/31/2026, 1:37:17 AM
Summary
The paper introduces H-Node Adversarial Noise Cancellation (H-Node ANC), a mechanistic framework for identifying and mitigating hallucination in LLMs by targeting specific hidden-state dimensions called Hallucination Nodes (H-Nodes). The authors demonstrate that these nodes can be localized using logistic regression probes on last-token hidden states. They propose a white-box adversarial attack that amplifies these nodes to induce hallucinations and an adaptive defense mechanism that uses confidence-weighted cancellation to suppress them, achieving significant robustness improvements across four transformer architectures (OPT, Phi-3, LLaMA-3, Mistral) without degrading general reasoning capabilities.
Entities (6)
Relation Signals (2)
H-Node ANC → validatedon → OPT-125M
confidence 99% · All contributions are validated on OPT-125M, Phi-3-mini-4k-instruct, LLaMA-3-8B-Instruct, and Mistral-7B-Instruct-v0.3
H-Node ANC → defends → LLMs
confidence 95% · We present H-Node Adversarial Noise Cancellation (H-Node ANC), a mechanistic framework that identifies, exploits, and defends hallucination representations in transformer-based large language models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present H-Node Adversarial Noise Cancellation (H-Node ANC), a mechanistic framework that identifies, exploits, and defends hallucination representations in transformer-based large language models (LLMs) at the level of individual hidden-state dimensions. A logistic regression probe trained on last-token hidden states localizes hallucination signal to a small set of high-variance dimensions -- termed Hallucination Nodes (H-Nodes) -- with probe AUC reaching 0.90 across four architectures. A white-box adversarial attack amplifies these dimensions at inference time via a real-time forward hook, achieving a selectivity of 3.02x with less than 10% visibility to the defender. Adaptive ANC defense suppresses H-Node excess in-pass using confidence-weighted cancellation, reducing grounded activation drift by 33-42% over static cancellation. A dynamic iterative extension that re-ranks cancellation targets across successive passes recovers up to 0.69 robustness from a single-pass baseline of 8%. All contributions are validated on OPT-125M, Phi-3-mini-4k-instruct, LLaMA-3-8B-Instruct, and Mistral-7B-Instruct-v0.3 (125M-8B parameters). Perplexity impact is surgical (<5%) and MMLU degradation is at most 3%, confirming that the defense does not impair general reasoning capability.
Tags
Links
- Source: https://arxiv.org/abs/2603.26045v1
- Canonical: https://arxiv.org/abs/2603.26045v1
Trouble viewing inline? Open PDF directly →
Full Text
63,497 characters extracted from source content.
Expand or collapse full text
H-Node Attack and Defense in Large Language Models Eric Yocam1, Varghese Vaidyan2, Yong Wang3 Abstract. We present H-Node Adversarial Noise Cancellation (H-Node ANC), a mechanistic framework that identifies, exploits, and defends hallucination representations in transformer-based large language models (LLMs) at the level of individual hidden-state dimensions. A logistic regression probe trained on last-token hidden states localizes hallucination signal to a small set of high-variance dimensions—termed Hallucination Nodes (H-Nodes)—with probe AUC reaching 0.90 across four architectures. A white-box adversarial attack amplifies these dimensions at inference time via a real-time forward hook, achieving a selectivity of 3.02× with less than 10% visibility to the defender. Adaptive ANC defense suppresses H-Node excess in-pass using confidence-weighted cancellation, reducing grounded activation drift by 33–42% over static cancellation. A dynamic iterative extension that re-ranks cancellation targets across successive passes recovers up to 0.69 robustness from a single-pass baseline of 8%. All contributions are validated on OPT-125M, Phi-3-mini-4k-instruct, LLaMA-3-8B-Instruct, and Mistral-7B-Instruct-v0.3 (125M–8B parameters). Perplexity impact is surgical (<<5%) and MMLU degradation is at most 3%, confirming that the defense does not impair general reasoning capability. Key words and phrases: hallucination detection, adversarial machine learning, large language models, mechanistic interpretability, activation engineering, inference-time defense, truthfulness, transformer probing 1Department of Computer Science and Software Engineering, California Polytechnic State University, San Luis Obispo, CA 93407, USA 2Beacom College of Computer and Cyber Sciences, Dakota State University, Madison, SD 57042, USA 3Department of Computer Science, University of Idaho, Moscow, ID 83844, USA 1. Introduction Hallucination in large language models (LLMs), the generation of factually incorrect content stated with apparent confidence, has emerged as a critical safety and reliability barrier to deployment in high-stakes domains [18, 4]. Rapid scaling of LLMs [6] has accelerated deployment in high-stakes settings while simultaneously amplifying the consequences of hallucination. Emergent capabilities that appear on scale [38] make it increasingly difficult to anticipate failure modes from the behavior of a small-model alone. While behavioral interventions such as retrieval augmentation [21] and reinforcement learning from human feedback [31] can reduce hallucination rates at the output level, they do not address the underlying representational mechanisms that produce them. A growing body of research on mechanistic interpretability suggests that factual and hallucinated completions produce measurably distinct patterns in hidden states of transformers [27, 40], yet no prior work has simultaneously formalized this distinction as an adversarial attack surface and constructed a principled real-time defense that operates within the same mechanistic framework. This paper closes that gap. We make four primary contributions. (1) H-Node Localization. We demonstrate that logistic regression probes applied to last-token hidden states—rather than mean-pooled representations—identify a small set of dimensions per layer, which we term Hallucination Nodes (H-Nodes), that reliably separate hallucinated from grounded completions with AUC up to 0.90. We show that hallucination signal peaks consistently at approximately 50% transformer depth across all four tested architectures, an architectural regularity not previously reported. (2) White-Box Mechanistic Attack. We construct a targeted adversarial attack that amplifies H-Node activations toward the hallucination distribution at inference time using a real-time forward hook. The attack is trained on a held-out data split with an independent random seed from the defender, modeling a realistic scenario where attacker and defender derive partially overlapping but non-identical node sets from the same open-weight model. The attack selectivity reaches 3.02×, and less than 10% of the injected signal is visible to the defender probe. (3) Adaptive ANC Defense. We present Adaptive Adversarial Node Cancellation (ANC), a confidence-weighted cancellation scheme that suppresses H-Node excess in-pass. A static ablation establishes the baseline, and the adaptive variant—which scales cancellation strength by the probe’s confidence score for each sample—reduces grounded drift by 33–42% while maintaining higher selectivity than Inference-Time Intervention (ITI) [23] and Decoding by Contrasting Layers (DoLA) [9] across all models. (4) Dynamic Iterative Extension and Cross-Architecture Validation. We extend the single-pass defense to a multi-pass dynamic scheme that re-ranks cancellation targets by residual excess after each pass, enabling the defender to discover and suppress attacker-only nodes that were invisible in the initial pass. We validate all contributions on four models spanning two architectural lineages (OPT and LLaMA/Mistral families) from 125M to 8B parameters. The experimental pipeline operates in three sequential phases. Phase 1 establishes the model’s hallucination geometry through probe training, layer sweep, and H-Node identification. Phase 2 deploys a white-box adversarial attack by injecting an activation signal through a real-time forward hook. Phase 3 responds with Adaptive ANC, iteratively re-ranking cancellation targets across successive passes to recover robustness. The detailed breakdown of the component-level appears in Fig. 1. Phase 1BaselinePhase 2AttackPhase 3Defense Probe training, layer sweep, H-Node ID Forward-hook injection Adaptive ANC + re-ranking Figure 1. Three-phase experimental pipeline overview. The paper is organized as follows: Section 2 surveys related work and identifies the research gap. Sections 3–6 formalize the threat model, probe architecture, attack construction, and ANC defense. Section 7 reports experimental results. Sections 8 and 9 discuss findings and conclude. 2. Related Work This section surveys the five bodies of previous work most directly relevant to H-Node ANC—hallucination detection, mechanistic interpretability, probing classifiers, inference-time intervention, and adversarial attacks—and consolidates the resulting research gap in a structured eight-method comparison table (Table 1). 2.1. Hallucination in Language Models Ji et al. [18] provide a comprehensive taxonomy of LLM hallucination, distinguishing intrinsic contradictions from extrinsic fabrications, and surveying mitigation strategies across the training, decoding, and post-hoc correction stages. TruthfulQA [24] established the standard benchmark for truthfulness evaluation, demonstrating that larger models do not necessarily become more truthful. HaluEval [22] extended the evaluation to domain-specific hallucinations in question-answering tasks. Our work treats hallucination not as a behavioral phenomenon that is measured at the output, but as a representational state that is detected and modified at the hidden-state level [3, 26]. 2.2. Mechanistic Interpretability The circuit framework [12] formalized the analysis of transformer components as computational mechanisms. Meng et al. [27] localized factual associations to specific layers of MLP through causal tracking. Zou et al. [40] demonstrated that high-level concepts including honesty are linearly represented in the residual stream and can be extracted by contrastive probing. Our H-Node probe extends this line of work to the adversarial setting: rather than reading the representation, we simultaneously attack and defend it. Geva et al. [13] showed that feed-forward sublayers function as key-value memory stores, providing a complementary view of factual storage at the component level. Petroni et al. [32] demonstrated that pretrained language models implicitly store relational knowledge in their parameters, establishing factual recall as a native capability of the transformer architecture rather than an emergent fine-tuning artifact. Dai et al. [11] identified individual “knowledge neurons” in pretrained transformers whose activation correlates with specific factual expressions, providing neuron-level evidence that complements our H-Node localization at the hidden-state dimension level. Hernandez et al. [17] demonstrated that relational knowledge in LLMs is encoded through linear transformations of subject representations, further supporting the linear structure assumption underlying the design of the H-Node probe. 2.3. Probing Representations Belinkov [5] reviews the probing paradigm for extracting structural information from neural representations. Alain and Bengio [2] established that linear probes trained on intermediate representations serve as reliable indicators of the information encoded at each layer, providing the theoretical foundation for our logistic regression H-Node probe. Tenney et al. [33] further demonstrated that transformer layers process the linguistic structure in an ordered progression, supporting the use of layer-sweep AUC as a principled method to identify the depth at which the hallucination signal is maximally concentrated. The key methodological distinction in our work is the use of last-token rather than mean-pooled activations, which we show provides 0.04–0.24 AUC improvement across all models. This is not merely a technical detail; it reflects the semantic role of the final answer token as the representational locus of the model’s committed response. Burns et al. [7] demonstrated that latent knowledge can be extracted from hidden states fully unsupervised, motivating our use of probe coefficients as the primary signal for the identification of H-Nodes. 2.4. Inference-Time Intervention Li et al. [23] proposed ITI, which shifts hidden states along a probing direction at inference time to improve truthfulness. Our work differs from ITI in three respects: we operate on individual dimensions (H-Nodes) rather than a single pooled direction; we introduce an adversarial attacker using the same mechanism in reverse; and we demonstrate that adaptive confidence weighting provides a selectivity advantage of 1.54×–4.53× over ITI across all tested models. 2.5. Decoding-Based Approaches DoLA [9] contrasts late-layer and early-layer logit distributions to amplify factual content during decoding. Unlike DoLA, our approach operates on the hidden state rather than the output distribution, enabling real-time cancellation before downstream layers propagate the hallucination signal. Our experimental comparison shows that DoLA reduces the accuracy of MC1 in three of four models, while H-Node ANC maintains MC1 with zero performance degradation. 2.6. Adversarial Attacks on LLMs Goodfellow et al. [14] established the adversarial perturbation framework for neural networks. Madry et al. [25] formalized adversarial robustness as a min-max optimization problem, establishing the theoretical foundation on which the activation-space attack construction is built. Wallace et al. [37] demonstrated that universal adversarial triggers transferable between inputs and models can be found by gradient-based search, motivating our use of a model-independent forward-hook architecture rather than input-level perturbation. Carlini et al. [8] showed that memorized training data can be extracted from LLMs through targeted querying, underscoring that open-weight models expose internal representations to adversarial exploitation beyond prompt-level attacks. Recent work has also extended adversarial attacks to LLM prompts [41] and fine-tuning procedures. Our attack operates in a distinct modality—activation space—and assumes white-box access to model weights, which is the default threat model for open-weight models available from public repositories. 2.7. Self-Knowledge and Uncertainty in LLMs Kadavath et al. [20] demonstrated that large language models possess calibrated self-knowledge: when asked whether a stated claim is true, model confidence correlates with empirical accuracy. Azaria and Mitchell [3] showed that internal activation patterns at specific layers reliably distinguish true from false statements, providing direct activation-level evidence for the H-Node hypothesis. Marks and Tegmark [26] revealed that truth values are linearly encoded in transformer representations, exhibiting a consistent geometric structure across layers and model families. Together, these findings establish that the representational basis for truthfulness exists and is structurally accessible—the contribution of this work is to simultaneously attack and defend that basis. 2.8. Research Gap Table 1 situates our contribution against eight prior representative methods in five dimensions. The table reveals that no existing method simultaneously addresses adversarial attack, real-time defense, cross-architecture validation, mechanistic localization at the node level, and adaptive confidence-weighted intervention. The comparison reveals a consistent pattern across the literature: prior work addresses either the detection problem (probing, representation engineering) or the mitigation problem (ITI, DoLA, RLHF) but not both within a unified adversarial framework. Methods that address both, such as activation addition [35], do not model an independent adversary or evaluate robustness under sequential attack-then-defend ordering. The absence of cross-architecture validation at scale is equally notable: most mechanistic results are demonstrated on a single model. The H-Node ANC fills this gap by providing matched experimental conditions across four models spanning two architectural lineages and the 64× parameter scale. Table 1. Comparison of Related Methods Against H-Node ANC Method Mechanistic Adversarial Real-Time Adaptive Multi-Model Attack+Defense Gap Addressed Node-Level Attack Defense Weighting Validation Unified ITI [23] ✓ × ✓ × Limited × Detection+intervention DoLA [9] × × ✓ × Limited × Decoding contrast Repr. Eng. [40] ✓ × ✓ × × × Concept steering ROME [27] ✓ × × × × × Fact localization Activation Add. [35] ✓ × ✓ × × × Behavior steering Universal Adv. [41] × ✓ × × Partial × Prompt attacks RLHF [31] × × ✓ × × × Alignment training RAG [21] × × ✓ × × × External grounding H-Node ANC (Ours) ✓ ✓ ✓ ✓ ✓ ✓ All dimensions 3. Threat Model and System Architecture This section formalizes the white-box threat model governing attacker and defender capabilities for open-weight LLMs, defines the probe independence assumption that creates the structural asymmetry at the center of this work, and presents the three-phase experimental pipeline as a process flow diagram (Fig. 2). 3.1. Process Flow The three-phase experimental process—baseline characterization, adversarial attack, and iterative defense—is illustrated in Fig. 2. The baseline phase establishes the model’s unmodified hallucination profile through probe training and layer sweep. The attack phase deploys an independent attacker probe to inject hallucination signals via a real-time forward hook. The defense phase responds with the ANC hook operating on the already-attacked activation state, iterating dynamically to discover and suppress attacker-only nodes. TruthfulQA HaluEval Dataset Activation Extraction (Last Token) Layer Sweep AUC Probe Training H-Node Identification (Top-50) Baseline Confidence Profile Attacker Probe (Seed 99) Activation Injection (6 Methods) RT Attack Forward Hook Attacked LLM State Defender Probe (Seed 42) Adaptive ANC Defense Dynamic Iterative Passes Robustness Measurement ρ PHASE 1: BASELINEPHASE 2: ATTACKPHASE 3: DEFENSE Figure 2. Three-phase experimental process flow. 3.2. Threat Model We assume a white-box threat model appropriate for open-weight LLMs. Both the attacker and the defender have full access to the model weights, tokenizer, and architecture. This assumption reflects the deployment reality of models available via public repositories such as HuggingFace: any party with a downloaded model can extract activations locally, train probes offline, and prepare injection hooks before any interaction with a deployment endpoint. We explicitly scope our attack to deployments where the adversary controls or can intercept the forward pass—self-hosted endpoints, fine-tuned model providers, or compromised inference infrastructure. API-only deployments where hidden states are never exposed are out of scope, as the hook mechanism requires access to intermediate layer activations. The key asymmetry in our model is probe independence: attacker and defender derive their H-Node sets independently, using separate training data splits and different random seeds. This models the realistic scenario where two parties both possess the model weights but train on different datasets or use different methodology. The resulting overlap is an empirical property of the model’s representation geometry, not an experimental parameter. 3.3. System Overview The system comprises five stages: (1) activation extraction with last-token pooling at all layers, (2) independent probe training for defender and attacker with separate data splits, (3) H-Node identification via signed probe coefficients, (4) adversarial injection using a real-time forward hook at the best layer, and (5) ANC defense via a combined hook that fires after injection. The architecture is model-agnostic: all components interface with the model through standard HuggingFace AutoModelForCausalLM APIs and forward hooks that do not require modification of model weights. Figure 3. Probe coefficient distributions and H-Node set overlap (OPT-125M). Figure 4. Defender vs. attacker probe coefficient scatter and activation shift by node category. The pipeline architecture reveals the fundamental adversarial asymmetry at the heart of this work. The defender and attacker derive their respective H-Node sets from the same model but via independent training procedures. Fig. 3 shows the probe coefficient distributions and H-Node set overlap: of 50 nodes per probe, only 18 overlap, leaving 32 attacker-only dimensions that bypass single-pass cancellation entirely. Fig. 4 shows the full coefficient scatter across all hidden dimensions and the mean activation shift by node category—attacker-only and overlap nodes exhibit equal amplification (Δ=0.0135 =0.0135), while defender-only nodes show only residual suppression (Δ=0.0031 =0.0031). The resulting overlap rate ranges from 14% (Phi-3-mini) to 36% (OPT-125M, Mistral-7B) across all four models (Table 5), establishing a structural ceiling on single-pass robustness that motivates the iterative dynamic extension. 4. H-Node Probe Architecture This section describes last-token activation extraction, the layer sweep procedure for best-layer selection, and the percentile-baseline H-Node identification algorithm that converts probe coefficients into a targeted set of hallucination-sensitive hidden-state dimensions. 4.1. Activation Extraction For a transformer model [36] with L layers and hidden dimension d, we extract hidden states at every layer for each input sequence. The key methodological contribution is the use of last-token rather than mean-pooled activations. For a prompt of the form Q: [question] : [answer], the last non-padding token represents the model’s committed answer state: (1) l=l[b,t∗,:]h_l=H_l[b,t^*,:] where t∗=maxt:token[t]≠padt^*= \t:token[t] \ and l∈ℝB×T×dH_l ^B× T× d is the hidden state tensor at layer l. 4.2. Layer Sweep and Best-Layer Selection We train a logistic regression probe on each layer independently and select the best layer by AUC on a held-out evaluation set. The ensemble representation concatenates the top-4 layers by AUC: (2) ens=[l1;l2;l3;l4]x_ens=[h_l_1;h_l_2;h_l_3;h_l_4] where l1,l2,l3,l4l_1,l_2,l_3,l_4 are selected by descending single-layer AUC. 4.3. H-Node Identification Given a trained probe with coefficient vector ∈ℝdw ^d, H-Nodes are the top-N dimensions by magnitude of the positive coefficients: (3) ℋ=argsort()desc[:N]H=argsort(w)_desc[:N] where N=50N=50 in all experiments. The baseline activation for each H-Node j∈ℋj is computed as the P-th percentile of grounded sample activations: (4) bj=PctP(hl,j(i):y(i)=0)b_j=Pct_P (\h_l,j^(i):y^(i)=0\ ) with P=80P=80 selected via sweep over 50,60,70,75,80,85,90,95,99\50,60,70,75,80,85,90,95,99\. 4.4. Probe Quality Across Models Fig. 5 shows the layer-wise AUC trajectory for all four models. The figure demonstrates a consistent architectural pattern: hallucination signal emerges in early layers and peaks at approximately 50% transformer depth before declining in the final layers. This pattern holds across the 12-layer OPT-125M and all three 32-layer models, suggesting a universal computational structure in which the model commits to factual vs. fabricated content during mid-layer processing. Figure 5. Probe AUC by transformer layer: last-token vs. mean-pool pooling (OPT-125M). The trajectory analysis shows that last-token pooling provides consistent improvement over mean pooling across all four models, with gains ranging from ++0.04 to ++0.24 AUC points. The improvement is largest for Phi-3-mini (++0.24) and OPT-125M (++0.13), confirming that answer-position representations carry significantly more hallucination signal than sequence-averaged representations. All four models exhibit a clear peak followed by gradual decline, with AUC plateau values of 0.75 (OPT), 0.89 (Phi-3), 0.90 (LLaMA), and 0.90 (Mistral), demonstrating that probe quality scales with model capacity. 5. Adversarial Attack Construction This section constructs six adversarial injection variants of increasing sophistication—from mean injection through a real-time Fourier forward hook—and defines the selectivity metric used throughout to quantify whether attack signal is concentrated on hallucination dimensions or bleeds into grounded representations. 5.1. Attack Methodology The attacker trains an independent probe on a disjoint data split with a different random seed, identifying attacker H-Nodes ℋatkH_atk that partially overlap with the defender set ℋdefH_def. The attack amplifies activations at H-Nodes toward the hallucination distribution by adding scaled excess above the attacker’s baseline: (5) h~l,j=hl,j+αatk⋅catk⋅max(0,bjatk−hl,j) h_l,j=h_l,j+ _atk· c_atk· (0,b_j^atk-h_l,j) for each node j∈ℋatkj _atk, where catkc_atk is the confidence score of the attacker probe and αatk _atk is the attack scaling factor. 5.2. Attack Variants We implement six variants of attack of increasing sophistication: Mean inject: Amplifies toward the mean activation of hallucinated training samples. Percentile-80 inject: Amplifies towards the 80th percentile of hallucinated activations, targeting the high-intensity region. Dual inject: Combines amplification of pro-hallucination nodes with suppression of anti-hallucination nodes simultaneously. Fourier inject: Applies FFT to the excess signal, zeroes the top-k frequency components, and re-injects the modified signal, creating a structured perturbation with a specific frequency-domain signature. Zero inject: Clamps target nodes to the attacker’s baseline, removing factual representation. Real-time hook: Implements the Fourier attack as a live forward hook, firing at the best layer during inference. The Fourier injection variant is particularly significant, as it bridges digital signal processing [10] and generative AI: by representing the excess hallucination signal in the frequency domain and targeting dominant frequency components, it creates a perturbation with a structured, detectable signature that can be tuned to evade threshold-based defenses. 5.3. Attack Selectivity Attack selectivity is defined as the ratio of hallucination amplification to grounded drift: (6) Selatk=Δc¯hallΔc¯grnd+ϵSel_atk= c_hall c_grnd+ε A selectivity greater than 1.0 indicates that the attack moves hallucinated samples toward higher probe confidence more than it moves grounded samples, confirming that the attack targets the hallucination representation specifically. 6. Adaptive ANC Defense This section presents the single-pass ANC formulation, an ablation comparing static and confidence-weighted cancellation, and the dynamic iterative extension that discovers and suppresses attacker-only nodes across successive passes via a robustness-based stopping criterion. 6.1. Single-Pass ANC The ANC defense operates as a forward hook at the defender’s best layer. For each token in the forward pass, the defender probe computes a confidence score cdefc_def. If cdef≥τc_def≥τ (confidence threshold), the hook cancels excess activation at each defender H-Node: (7) h~l,j=hl,j−αdef⋅cdef⋅max(0,hl,j−bjdef) h_l,j=h_l,j- _def· c_def· (0,h_l,j-b_j^def) The key distinction from static cancellation is the multiplicative factor cdefc_def: samples that the probe classifies as weakly hallucinated receive proportionally weaker cancellation, reducing over-correction on borderline cases. This confidence-weighting scheme is grounded in the neural network calibration literature [15], which establishes that the probe output probabilities serve as reliable confidence signals when the classifier is properly regularized. 6.2. Static vs. Adaptive Ablation The static ANC variant uses cdef=1.0c_def=1.0 for all samples above the threshold. The adaptive variant uses the actual probability of the probe. The selectivity metric for the defense is the following: (8) Seldef=Δc¯hallΔc¯grnd+ϵSel_def= c_hall c_grnd+ε where Δc¯hall c_hall is the reduction in hallucination confidence and Δc¯grnd c_grnd is the drift of grounded confidence (collateral damage). 6.3. Dynamic Iterative Extension The dynamic iterative defense addresses the structural limitation that attacker-only nodes (ℋatk∖ℋdefH_atk _def) are invisible to single-pass cancellation. After each pass, the defense re-ranks all dimensions by current excess above the baseline and targets the top-N by this residual signal: (9) ℋ(t+1)=argsort(∑imax(0,h~l,⋅(t)−def))desc[:N]H^(t+1)=argsort ( _i (0, h^(t)_l,·-b^def) )_desc[:N] The max(0,⋅) (0,·) operator acts as a ReLU-like rectifier, ensuring that only dimensions exhibiting excess activation above the grounded baseline contribute to the re-ranking score. This prevents the defense from inadvertently amplifying dimensions where the attacked hidden state falls below the baseline—a condition corresponding to anti-hallucination suppression rather than hallucination injection, and which should not be treated as a cancellation target. This allows the defender to discover attacker-only nodes organically: once the known overlap nodes are suppressed in pass 1, the attacker’s uncontested dimensions become the highest-excess dimensions in the residual and are automatically selected in pass 2 onward. The defense halts when the improvement in the robustness of the attacker probe falls below a tolerance ϵ=10−4ε=10^-4, or when the selectivity per-pass drops below 1.0 (indicating that the defense begins to suppress grounded activations more than the attack signal). This stopping criterion was a key correction from an initial implementation that used defender probe improvement as the stopping signal, a criterion that fired prematurely because the defender probe could not see attacker-only nodes being suppressed in later passes. 6.4. Robustness Metric Defense robustness is defined as the fractional neutralization of attack amplification: (10) ρ=1−AdefendedAundefendedρ=1- A_defendedA_undefended where A=c¯atk,hall−c¯atk,grndA= c_atk,hall- c_atk,grnd is the attacker probe’s measure of hallucination amplification. A robustness of 1.0 indicates complete neutralization; 0.0 indicates that there is no defense effect. 7. Experimental Results This section reports results across five experimental components: probe quality and layer trajectory, cancellation selectivity and static-versus-adaptive ablation, SOTA comparison against ITI [23] and DoLA [9], the complete adversarial pipeline with overlap analysis and iterative robustness, and preservation of capability under perplexity and MMLU evaluation. 7.1. Experimental Setup All experiments use 300 samples from TruthfulQA (multiple-choice format) [24] and 300 samples from HaluEval (QA split) [22]. Data are divided into three equal splits: defender training (seed 42), attacker training (seed 99), and shared evaluation. Four models are evaluated: OPT-125M [39], Phi-3-mini-4k-instruct [1], LLaMA-3-8B-Instruct [29] (building on the LLaMA 2 lineage [34]) and Mistral-7B-Instruct-v0.3 [19]. The models are loaded in bfloat16 (OPT: float16) with device map auto. The top-50 H-Nodes are used for all experiments. Cancellation α=0.9α=0.9, confidence threshold τ=0.45τ=0.45, baseline percentile P=80P=80. Generation benchmarks use MC1 (shuffled-choice) and MC2 (normalized probability mass over all true answers) scoring with answer-only conditional log-probability, ensuring the model is scored on the answer token sequence alone rather than the full prompt. MMLU evaluation uses a 100-question diverse subset [16]. WikiText-103 perplexity uses 80 sentences [28]. The evaluation of factual precision at the level of sentences via FActScore [30] is reserved for future work on the 70B scale, where the prompting of the chat-format makes the generation deltas interpretable. 7.2. Probe Quality and Layer Trajectory Table 2 reports probe quality results across all four models. The ensemble AUC exceeds the single-layer AUC in all cases, which justifies the four-layer concatenation strategy. The consistent last-token advantage over mean-pool activations—with gains of ++0.04 to ++0.24 AUC points—confirms that the answer token position carries a disproportionate hallucination signal. LLaMA-3-8B and Mistral-7B both achieve probe AUC of 0.90 at their best layers (15 and 16, respectively), with comparable single-layer and ensemble performance suggesting that the hallucination representation is well-concentrated in a small layer window for these larger models. The activation trajectory analysis reveals that all four models peak at approximately 50% depth: layer 6 of 12 for OPT (50%), layer 17 of 32 for Phi-3 (53%), layer 15 of 32 for LLaMA (47%), and layer 16 of 32 for Mistral (50%). This depth universality—spanning 125M to 8B parameters and two architectural lineages—suggests that mid-layer commitment to factual versus fabricated content is a structural property of auto-regressive transformers, not an artifact of any particular model family. Table 2. Probe Quality and Layer Analysis Metric OPT-125M Phi-3-mini LLaMA-3-8B Mistral-7B Best layer 6 17 15 16 Peak depth (%) 50 53 47 50 AUC last-token 0.754 0.888 0.898 0.905 AUC mean-pool 0.627 0.648 0.862 0.798 Last-token gain +0.126 +0.240 +0.036 +0.106 Ensemble AUC 0.753 0.890 0.899 0.901 7.3. Cancellation and Defense Selectivity Table 3 presents cancellation performance and the static-vs.-adaptive ablation. Across all four models, the adaptive variant reduces grounded drift by 33–42% relative to static cancellation while maintaining comparable or higher reduction in hallucinations. Selectivity (reduction/drift ratio) consistently favors the adaptive variant, reaching 5.88× on Mistral-7B. The larger models (LLaMA, Mistral) show higher selectivity despite lower absolute reduction values, indicating that the hallucination representation becomes more distinct from the grounded representation at scale—the ANC defense can be more surgical precisely because the signal is better separated. Table 3. Cancellation Performance and Static vs. Adaptive Ablation Metric OPT-125M Phi-3-mini LLaMA-3-8B Mistral-7B Hall. reduction (pct80) 0.026 0.008 0.006 0.015 Grounded drift 0.008 0.002 0.001 0.003 Selectivity (pct80) 3.39× 3.33× 5.14× 5.88× Best pct sweep sel. 8.73× 3.43× 5.28× 8.64× Static ANC sel. 3.10× 3.09× 4.72× 4.69× Adaptive ANC sel. 3.39× 3.33× 5.14× 5.88× Drift reduction (%) 41.5 35.4 33.3 38.1 7.4. SOTA Comparison: ITI and DoLA Table 4 compares the ANC of the H-Node with ITI [23] and DoLA [9] across all four models. The H-Node ANC achieves selectivity advantages over the ITI of ++1.72× to ++4.53×, with the advantage growing with model scale. This scaling behavior is significant: as models improve, the hallucination signal becomes more structured, and the H-Node ANC becomes proportionally more effective relative to direction-based methods. DoLA degrades the precision of MC1 in three of four models (delta of −-0.04 to −-0.03), while H-Node ANC maintains MC1 with a near-zero delta across all models, confirming that frequency-domain cancellation is more surgical than contrastive decoding to preserve the ability to select answers. Table 4. Comparison Against SOTA: ITI and DoLA Metric OPT-125M Phi-3-mini LLaMA-3-8B Mistral-7B ITI best sel. 1.67× 1.79× 1.84× 1.35× H-Node adv. over ITI ++1.72× ++1.54× ++3.30× ++4.53× DoLA MC1 accuracy 0.210 0.230 0.340 0.220 DoLA MC1 delta −-0.040 −-0.030 ++0.060 −-0.020 H-Node MC1 delta −-0.003 ≈ 0 ≈ 0 −-0.002 7.5. Adversarial Pipeline Results Table 5 presents the full results of the adversarial pipeline, including the novel overlap analysis and the iterative defense robustness. Table 5. Adversarial Pipeline: Attack, Overlap, and Defense Robustness Metric OPT-125M Phi-3-mini LLaMA-3-8B Mistral-7B Attack (undefended, best method) Atk. amplitude 0.084 0.201 — — Atk. selectivity 2.87× 3.02× — — Def. visibility 0.072 0.060 — — Overlap Analysis Overlap rate (%) 36.0 14.0 26.0 36.0 Transfer rate (%) 64.0 86.0 74.0 64.0 Defense Robustness Single-pass ρ 0.082 0.078 0.035 0.083 Dynamic iterative ρ 0.689 0.371 0.125 0.339 Phase 4c ablation best — 0.445 0.159 0.380 The results of the adversarial pipeline demonstrate several key findings. First, single-pass robustness is uniformly low (3–8%) across all models because the defender’s fixed 50-node set covers at most 36% of the attacker’s nodes, leaving the majority of attack signal completely unaddressed. This is not a failure of the cancellation mechanism—the per-node cancellation is effective—but a consequence of structural geometry: the attacker operates primarily in dimensions the defender did not identify. Second, dynamic iterative defense transforms this landscape substantially. By re-ranking cancellation targets after each pass, the defender discovers attacker-only nodes that become the highest-excess dimensions once the overlap nodes are suppressed. OPT achieves robustness of 0.689 in 5 passes, an improvement of ++0.607 over a single-pass. Mistral reaches 0.339 (++0.256) and Phi-3 reaches 0.371 (++0.293). LLaMA shows a more modest improvement (0.125), consistent with its intermediate overlap rate (26%) and more diffuse hallucination geometry. Third, Phase 4c ablation confirms that 15-iteration runs with thresh=0.45 and robustness-based stopping (Variant A) produce the best results, reaching 0.445 in Phi-3 and 0.380 in Mistral. The key methodological finding from this ablation is that the stopping criterion must track attacker probe robustness improvement, not defender probe confidence—the latter fires prematurely because the defender probe cannot detect improvements in attacker-node suppression. Fig. 6 visualizes the per-iteration robustness trajectory for the Fourier attack across both fixed-node and dynamic-node variants. Fig. 7 shows the final robustness values across all attack methods for single-pass and dynamic defense conditions. Figure 6. Iterative defense robustness and selectivity per pass (Fourier attack). Figure 7. Single-pass vs. dynamic iterative robustness across attack methods. The defense trajectory in Fig. 6 illustrates the mechanism of dynamic node expansion visually. The fixed-node variant plateaus after 1–2 passes because all targeted nodes have been suppressed and no new signal is reachable. The dynamic-node variant continues to improve each pass as residual excess in attacker-only dimensions enters the top-N sorted list. Per-pass selectivity stays above 1.0 through all iterations shown, confirming that the defense remains targeted rather than becoming a broad suppression that degrades grounded performance. Fig. 7 confirms that dynamic re-ranking produces the largest gains across all attack variants, with the dynamic Fourier method reaching 0.689 robustness and approaching the 50% target threshold. 7.6. Capability Preservation Table 6 reports perplexity and MMLU results under attack-defended condition. All four models show an impact of surgical perplexity (<<5%), confirming that the ANC hook does not affect fluency. OPT-125M shows the highest increase in PPL at 1.8%, while LLaMA-3-8B is effectively unchanged at 0.0%. The impact of MMLU ranges from −-3% (LLaMA) to ++2% (Mistral), falling within the preserved-to-minor range. The absence of significant MMLU degradation is particularly important: it confirms that the ANC defense suppresses hallucination-specific signal without disrupting the general reasoning representations that govern multi-domain question answering. Table 6. Capability Preservation: Perplexity and MMLU Under ANC Metric OPT-125M Phi-3-mini LLaMA-3-8B Mistral-7B PPL baseline 65.42 11.66 21.07 14.46 PPL (atk→ ) 66.60 12.04 21.08 14.55 PPL delta (%) ++1.8 ++3.3 ++0.0 ++0.6 PPL verdict Surgical Surgical Surgical Surgical MMLU baseline 0.19 0.38 0.37 0.44 MMLU (atk→ ) 0.18 0.39 0.34 0.46 MMLU delta −-0.01 ++0.01 −-0.03 ++0.02 MMLU verdict Preserved Preserved Minor Minor 7.7. Generation Benchmarks: Capability Sanity Check The generation benchmarks in this work serve a single purpose: to confirm that the ANC intervention does not collapse observable output quality. They are not a measure of practical hallucination reduction. This distinction matters because bare Q:/A: prompt formatting is retained throughout to preserve a clean mechanistic signal—using chat-format prompting would inflate MC1/MC2 scores, but would simultaneously confound the activation-space measurements that constitute the primary contribution. The near-chance baselines are therefore an expected consequence of a deliberate methodological choice, not a reflection of defense efficacy. Behavioral mitigation evaluation at the generation level is reserved for chat-formatted models on the 70B scale, where instruction-following formatting produces interpretable and meaningful deltas. Under these conditions, the accuracy of MC1 ranges from 0.24 (Mistral) to 0.28 (LLaMA) at baseline across all four models, against a random chance level of 0.25 on 4-choice questions. The ANC defense shifts these values by at most 0.01 in either direction. MC2 truthfulness scores range from 0.38 to 0.43, with defense deltas within ± 0.003. Both results confirm the intended claim: the intervention is inert with respect to output distribution, neither improving nor degrading generation quality at this scale and format. 8. Discussion This section interprets the key empirical findings—the Hydra effect, the nonlinear relationship between overlap rate and dynamic robustness, and the white-box realism assumption—and frames remaining scope boundaries as deliberate design trade-offs that motivate the architectural extensions identified for future work. 8.1. The Hydra Effect, Signal Redundancy, and Orthogonal Projection A key finding from the adversarial pipeline is the “Hydra effect”: when the primary H-Nodes identified by the defender are suppressed, the hallucination signal redistributes through secondary dimensions that were not identified as primary nodes. This is most pronounced in the Dual and Zero attack variants, which create broad-spectrum activation shifts that fixed-node cancellation cannot fully neutralize. The dynamic iterative defense addresses this directly by tracking residual excess across all dimensions, but the fundamental challenge remains: LLMs are highly redundant, and hallucination may have multiple representational pathways. The structural solution to the Hydra effect is the layer-wise projection onto the orthogonal complement of the hallucination subspace. Let ∈ℝdv ^d be the unit-norm hallucination direction extracted from the probe (e.g., the principal component of the top sign-in coefficient vector w). The orthogonal projection matrix (11) ⟂=−⊤P_ =I-vv applying the hidden state lh_l at the best layer yields a representation ⟂lP_ h_l from which all components along the hallucination direction have been removed, regardless of which specific dimensions carry the signal. Unlike node-level cancellation in ANC, which suppresses a discrete set ℋH of 50 dimensions, Eq. (11) neutralizes the entire one-dimensional hallucination subspace simultaneously. This eliminates the attacker’s ability to exploit dimensions outside the defender’s identified node set, removing the structural bypass that produces the Hydra effect. Extension to a rank-k subspace uses ⟂=−kk⊤P_ =I-V_kV_k where k∈ℝd×kV_k ^d× k contains the top-k hallucination directions. The current H-Node ANC architecture is deliberately retained at node-level granularity to preserve per-coefficient interpretability and auditable cancellation; subspace projection is identified as the natural successor architecture for deployments where robustness takes priority over interpretability. 8.2. Transfer Rate as a Structural Ceiling The relationship between overlap rate and dynamic robustness is not linear: Phi-3 at 14% overlap achieves 0.371 dynamic robustness, while OPT at 36% overlap achieves 0.689. This suggests that single-pass robustness is bounded by the transfer rate, but dynamic iteration can partially overcome this bound by discovering attacker-only dimensions. The theoretical ceiling for dynamic defense is not the single-pass overlap rate, but rather the point at which the residual hallucination signal is indistinguishable from grounded signal noise—a different and generally higher threshold. 8.3. White-Box Realism for Open-Weight Models The white-box assumption is not a limitation of this work, but a correct characterization of the threat environment for open-weight LLMs. For all four models tested, full weights are publicly available. Any attacker can download the model, train probes locally, and prepare injection hooks offline prior to any deployment interaction. The probe independence experiment—using different data splits and random seeds—models the realistic scenario where attacker and defender both possess the weights but derive their node sets independently. The resulting 14–36% overlap is an empirical property of hallucination geometry in each model, not an experimental parameter. This is consistent with the findings of Kadavath et al. [20] that LLMs possess calibrated internal uncertainty estimates, suggesting that the probe signal taps a genuine representational property rather than a surface artifact. 8.4. Design Trade-offs and Scope Boundaries The H-Node ANC framework embodies three deliberate architectural choices that extend current results and motivate the extensions outlined above. Each choice reflects a trade-off between evaluation cleanliness, simplicity of deployment, and mechanistic interpretability. Evaluation Protocol Scoping. Bare Q:/A: prompt formatting is used throughout to isolate activation-space effects from prompt-engineering artifacts; this choice is what makes the probe-confidence and cancellation-selectivity metrics interpretable as pure mechanistic measurements. The consequence is that instruction-tuned models, designed for chat interaction, produce near-chance MC1/MC2 baselines. Inflating generation scores via chat-format prompting would confound the activation-space signal that is the primary contribution, so the bare-format constraint is a deliberate scope boundary, not a quality gap. Evaluation under chat formatting at 70B scale, where generation deltas are expected to become meaningful, is identified as a tractable future extension. Stateless Per-Pass Context Scope. The ANC hook operates on each forward pass independently, without session state or access to the KV cache from prior turns. This design enables real-time deployment as a drop-in forward hook with no modifications to the model architecture or inference infrastructure, incurring O(1)O(1) overhead per token relative to the unmodified forward pass. The resulting constraint is that once a hallucinated token has been committed to the KV cache, it persists as context for subsequent tokens outside the hook’s reach. A stateful multi-turn extension would proceed as follows. At each generation step t, before computing the next token’s hidden state, a cache scrubber applies a decay matrix =−βℋD=I- _H to the key and value tensors stored in the KV cache, where ℋP_H is the projection onto the H-Node subspace and β∈(0,1]β∈(0,1] is a decay rate. This retroactively attenuates hallucination-aligned content that has already entered the cache without recomputing prior hidden states. The computational overhead scales as O(Ncache⋅|ℋ|)O(N_cache·|H|) per step—linear in cache depth—compared to the O(1)O(1) cost of the stateless hook. The engineering trade-off is therefore explicit: stateless cancellation is preferable for single-turn or latency-sensitive deployments; stateful cache scrubbing is appropriate when multi-turn coherence and sustained hallucination suppression justify the added per-step cost. The stateless architecture is retained in this foundational paper as a deliberate scope boundary, with the cache scrubber identified as the direct successor for multi-turn deployment scenarios. Node-Level Granularity Versus Subspace Architecture. The H-Node ANC operates at individual hidden-state dimensions identified by probe coefficients rather than learned subspaces or orthogonal projections. This design decision preserves direct mechanistic interpretability: each cancelled node corresponds to a specific signed coefficient in the probe, making the defense auditable and the attack measurable. The trade-off is a structural coverage ceiling—nodes outside the defender’s 50-node set are not addressed in a single pass—which the dynamic iterative extension partially overcomes by discovering attacker-only nodes through residual re-ranking. The absolute robustness ceiling observed near 0.444 reflects this dimension-level granularity; layer-wise projection onto the orthogonal complement of the hallucination direction is the natural successor architecture that would remove the ceiling while sacrificing per-node interpretability. 8.5. Limitations Three boundaries constrain the current results. First, all four models are evaluated in bare Q:/A: format, which produces near-chance MC1/MC2 baselines for instruction-tuned models; generation-level hallucination reduction cannot be quantified at this scale and format (see Section 7, “Generation Benchmarks: Capability Sanity Check”). Second, the empirical evaluation is bounded at 8B parameters; whether the 50% depth universality and H-Node localization properties hold at 70B scale remains an open empirical question (see Section 8, “Design Trade-offs and Scope Boundaries”). Third, the adversarial pipeline assumes a white-box attacker with full weight access, and the robustness results reported here do not extend to gray-box or black-box threat models where the attacker cannot extract hidden-state activations directly (see Section 3, “Threat Model”). Each of these boundaries is a consequence of the design trade-offs described above, rather than a fundamental constraint on the H-Node ANC framework, and each identifies a concrete axis for future experimental validation. 8.6. Future Work Several directions extend naturally from this work. Layer-wise projection onto the orthogonal complement of the hallucination direction would eliminate the attacker’s ability to exploit dimensions outside the defender’s node set, addressing the structural bypass problem directly. Training of the ensemble probe across multiple seeds and data splits would reduce overlap variance and produce a more stable H-Node set with broader attacker coverage. Extension to 70B-scale models would enable meaningful MC1/MC2 generation deltas and validate whether the 50% depth universality holds at extreme scale. Multi-turn generation scenarios require a modified defense that remains effective after hallucinated tokens have entered the KV cache context. Finally, applying the H-Node framework to domain-specific fine-tuned models—where hallucination patterns may concentrate differently across layers—represents both a validation opportunity and a deployment-relevant extension. 9. Conclusion We presented H-Node Adversarial Noise Cancellation (H-Node ANC), a mechanistic framework for attacking and defending hallucination representations in transformer LLMs. Our key findings are: (1) the hallucination signal localizes to the H-Nodes at approximately 50% transformer depth consistently across architectures; (2) last-token pooling outperforms mean pooling by 0.04–0.24 AUC; (3) adaptive confidence-weighted cancellation reduces grounded drift by 33–42% versus static cancellation; (4) the ANC of the H-Node achieves a selectivity advantage of 1.54×–4.53× over ITI; (5) the dynamic iterative defense recovers up to 0.689 robustness from an 8% single-pass baseline by discovering attacker-only nodes across passes; and (6) the impact of perplexity is surgical (<<5%) and the degradation of MMLU is minor (≤ 3%) across all four models. These results, replicated across OPT-125M, Phi-3-mini, LLaMA-3-8B, and Mistral-7B, establish H-Node ANC as a principled, architecture-agnostic, and scalable framework for real-time hallucination defense. Future work will extend to layer-wise subspace projection, multi-turn generation scenarios, and ensemble probe training for improved overlap coverage. Of these directions, stateful integration of the ANC hook into the KV cache—enabling hallucination suppression to persist across tokens within a generation—represents the most direct path toward deployment-grade, context-aware defense and the most impactful open problem for the field. References [1] M. Abdin, J. Aneja, H. Awadalla, A. Awasthi, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, and H. Behl (2024) Phi-3 technical report: a highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Cited by: §7.1. [2] G. Alain and Y. Bengio (2017) Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations Workshop, Cited by: §2.3. [3] A. Azaria and T. Mitchell (2023) The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 967–976. Cited by: §2.1, §2.7. [4] Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, Q. V. Do, Y. Xu, and P. Fung (2023) A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, p. 675–718. Cited by: §1. [5] Y. Belinkov (2022) Probing classifiers: promises, shortcomings, and advances. Computational Linguistics 48 (1), p. 207–219. External Links: Document Cited by: §2.3. [6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, p. 1877–1901. Cited by: §1. [7] C. Burns, H. Ye, D. Klein, and J. Steinhardt (2023) Discovering latent knowledge in language models without supervision. In International Conference on Learning Representations, Cited by: §2.3. [8] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel (2021) Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium, p. 2633–2650. Cited by: §2.6. [9] Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, and P. He (2024) DoLA: decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, Cited by: §1, §2.5, Table 1, §7.4, §7. [10] J. W. Cooley and J. W. Tukey (1965) An algorithm for the machine calculation of complex Fourier series. Mathematics of Computation 19 (90), p. 297–301. External Links: Document Cited by: §5.2. [11] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei (2022) Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8493–8502. External Links: Document Cited by: §2.2. [12] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah (2021) A mathematical framework for transformer circuits. In Proceedings of the Transformer Circuits Workshop, Cited by: §2.2. [13] M. Geva, R. Schuster, J. Berant, and O. Levy (2021) Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 9556–9571. External Links: Document Cited by: §2.2. [14] I. J. Goodfellow, J. Shlens, and C. Szegedy (2015) Explaining and harnessing adversarial examples. In International Conference on Learning Representations, Cited by: §2.6. [15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, p. 1321–1330. Cited by: §6.1. [16] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021) Measuring massive multitask language understanding. In International Conference on Learning Representations, Cited by: §7.1. [17] E. Hernandez, A. S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y. Belinkov, and D. Bau (2024) Linearity of relation decoding in transformer language models. In International Conference on Learning Representations, Cited by: §2.2. [18] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung (2023) Survey of hallucination in natural language generation. ACM Computing Surveys 55 (12), p. 1–38. External Links: Document Cited by: §1, §2.1. [19] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, J. Wang, T. Lacroix, and W. El Sayed (2023) Mistral 7B. arXiv preprint arXiv:2310.06825. Cited by: §7.1. [20] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and Anthropic (2022) Language models (mostly) know what they don’t know. arXiv preprint arXiv:2207.05221. Cited by: §2.7, §8.3. [21] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, Vol. 33, p. 9459–9474. Cited by: §1, Table 1. [22] J. Li, X. Cheng, W. X. Zhao, J. Nie, and J. Wen (2023) HaluEval: a large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747. Cited by: §2.1, §7.1. [23] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg (2023) Inference-time intervention: eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §1, §2.4, Table 1, §7.4, §7. [24] S. Lin, J. Hilton, and O. Evans (2022) TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252. External Links: Document Cited by: §2.1, §7.1. [25] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Cited by: §2.6. [26] S. Marks and M. Tegmark (2023) The geometry of truth: emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824. Cited by: §2.1, §2.7. [27] K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2022) Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems, Vol. 35, p. 17359–17372. Cited by: §1, §2.2, Table 1. [28] S. Merity, C. Xiong, J. Bradbury, and R. Socher (2017) Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Cited by: §7.1. [29] Meta AI (2024) Introducing Meta Llama 3: the most capable openly available LLM to date. Meta AI Blog. External Links: Link Cited by: §7.1. [30] S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi (2023) FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 12076–12100. External Links: Document Cited by: §7.1. [31] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. Cited by: §1, Table 1. [32] F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller (2019) Language models as knowledge bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, p. 2463–2473. External Links: Document Cited by: §2.2. [33] I. Tenney, D. Das, and E. Pavlick (2019) BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4593–4601. External Links: Document Cited by: §2.3. [34] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Biber, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §7.1. [35] A. M. Turner, L. Thiergart, G. Leech, D. Udell, U. Mini, and M. MacDiarmid (2024) Activation addition: steering language models without optimization. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, Cited by: §2.8, Table 1. [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, p. 5998–6008. Cited by: §4.1. [37] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh (2019) Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, p. 2153–2162. External Links: Document Cited by: §2.6. [38] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus (2022) Emergent abilities of large language models. Transactions on Machine Learning Research. Cited by: §1. [39] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simber, P. S. Su, S. Torabi Ziaee, L. Zettlemoyer, and V. Stoyanov (2022) OPT: open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. Cited by: §7.1. [40] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks (2023) Representation engineering: a top-down approach to AI transparency. In arXiv preprint arXiv:2310.01405, Cited by: §1, §2.2, Table 1. [41] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. In arXiv preprint arXiv:2307.15043, Cited by: §2.6, Table 1.