Paper deep dive
Adversarial Entropy Inflation Against Gumbel-Based Inference Verification
Nikita Kezins
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Gumbel-based inference verification bounds LLM weight exfiltration by only forgiving token choices that plausibly arise from honest GPU nondeterminism, reporting a >200x slowdown for a steganographic adversary under benign prompt traffic. This bound assumes a passive attacker; we show it degrades sharply against an adversary who instead controls the prompt distribution. Because the verifier's admissible-token-set size is driven by the model's own output entropy, prompts engineered to break grammatical and sub-word structure -- rather than benign conversational traffic -- widen that set and open a materially larger covert channel. Across six instruction-tuned models spanning 1B to 32B parameters and three random seeds, our strongest attack (character- and script-level disruption) roughly doubles bits leaked per token relative to benign prompts, cutting the slowdown factor to 60x - 118x. These results indicate that static, benign-traffic-calibrated thresholds are insufficient for this defense, and that jitter-forgiveness thresholds should instead be calibrated dynamically against local token entropy.
Tags
Links
- Source: https://arxiv.org/abs/2608.23375v1
- Canonical: https://arxiv.org/abs/2608.23375v1
Trouble viewing inline? Open PDF directly â
Full Text
10,641 characters extracted from source content.
Expand or collapse full text
NeurIPS 2026 Workshop FLMSec Adversarial Entropy Inflation Against Gumbel-Based Inference Verification Nikita Kezins Affiliation: Delft University of Technology Email: nikitakezins@gmail.com Abstract Gumbel-based inference verification bounds LLM weight exfiltration by only forgiving token choices that plausibly arise from honest GPU nondeterminism, reporting a >200Ă>200Ă slowdown for a steganographic adversary under benign prompt traffic. This bound assumes a passive attacker; we show it degrades sharply against an adversary who instead controls the prompt distribution. Because the verifierâs admissible-token-set size is driven by the modelâs own output entropy, prompts engineered to break grammatical and sub-word structure â rather than benign conversational traffic â widen that set and open a materially larger covert channel. Across six instruction-tuned models spanning 1B to 32B parameters and three random seeds, our strongest attack (character- and script-level disruption) roughly doubles bits leaked per token relative to benign prompts, cutting the slowdown factor to 6060â118Ă118Ă. These results indicate that static, benign-traffic-calibrated thresholds are insufficient for this defense, and that jitter-forgiveness thresholds should instead be calibrated dynamically against local token entropy. 1 Introduction As organizations scale their artificial intelligence capabilities, proprietary models become highly valuable assets, raising significant concerns regarding model weight exfiltration. One vector for such exfiltration is through standard API inference, where malicious users can encode and leak model weights via deliberate token selection. This technique effectively establishes a covert subliminal channel 4 within the modelâs standard output stream. As a defense against this, 3 proposed verifying sampled tokens under a predefined seed selection, building on the same Gumbel-margin statistic used more broadly for detecting inference misconfiguration and tampering 1. By utilizing Gumbel sampling, their method ensures nearly identical token selection under an argmax policy for both the verifier and the prover. While theoretically sound, this approach is vulnerable to GPU non-determinism (jitter) caused by non-associative floating-point operations. When multiple tokens have near-identical likelihoods, this jitter can flip the argmax token selection. The necessity for verifiers to forgive these jitter-induced discrepancies inadvertently creates an attack gap that an active exfiltrator can exploit. Prior work demonstrated that under benign input conditions (e.g., a subset of the LMSYS Chatbot Arena dataset 5), only a small fraction of tokens exhibit branching â where multiple tokens share similar likelihoods, allowing GPU jitter to alter the output â with the rest exactly determined (see Methodology for the formal admissible-set definition). This underlies the reported 200x slowdown in exfiltration rates relative to the theoretical maximum. Because the capacity of this covert channel is strictly bound by the entropy of the token distribution, widening the attack gap requires an exfiltrator to maximize output randomness. In this work, we demonstrate that by deliberately breaking token dependencies and maximizing output entropy via adversarial prompting, an attacker can double the exfiltration capacity, reducing the defenseâs slowdown factor from 200x to approximately 100x. 2 Methodology Preliminaries. The GLS verifier of 3 reconstructs, per output position, the Gumbel-perturbed logits the proverâs sampler would use under a shared seed, then defines the admissible set AtA_t as tokens within tolerance Ï of the top Gumbel score â i.e. tokens a compliant prover could plausibly emit under GPU nondeterminism. Verification passes iff the observed token lies in AtA_t; residual exfiltration bandwidth at that position is bounded by log2âĄ|At| _2|A_t| bits. On benign traffic AtA_t is a singleton (âforcedâ) almost everywhere, giving the reported >200Ă>200Ă slowdown. Since |At||A_t| tracks the modelâs own output entropy, an adversary who raises that entropy through prompt design directly widens AtA_t â the attack surface we study. We build upon the experimental framework provided by 3. To establish a baseline, we evaluate a subset of 150 benign prompts from the LMSYS dataset 5, matching the original study. We then design three tiers of adversarial attacks, each progressively escalating the attempt to induce high-entropy distributions. We generate 150 adversarial prompts for each attack type. TERSE (Low-Context Generation): 722 combinatorially generated one-word fill-in-the-blank and low-context prompts (e.g. single-word mood/preference/description completions, terse conversational openers). The hypothesis is that genuine uncertainty concentrates on the first token of a response; isolating it rather than diluting it across a longer sequence should widen the set of admissible tokens. SCRAMBLE (Grammatical Dependency Disruption): 233 prompts across four categories: unrelated-item lists with explicit anti-correlation instructions, pseudo-random password/hash/PIN generation, gibberish-syllable continuation, and short surrealist continuations. This targets local word-to-word grammatical dependencies, fundamentally undermining the structure that makes fluent prose low-entropy. CIPHER (Sub-Word and Token-Level Disruption): 157 prompts across four categories: unbroken alphanumeric blocks, base64/hex/base32 dumps, gibberish-prefix âcontext bombsâ the model must continue verbatim, and multi-script character collisions (Latin/Cyrillic/Katakana/Arabic). Because even âunrelatedâ word lists rely on standard vocabulary, this attack targets character-level distributions directly to maximize entropy at the token boundary. We executed these four prompt sets (benign, plus three attacks; 150 prompts each) across three distinct random seeds. To ensure generalizability across different architectures and scales, we evaluated six instruction-tuned models: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Qwen2.5-14B, Qwen3-30B-MoE, Qwen3-32B. 3 Experiments and Results Figure 1: Top Left: Extractable info (vocab-size-corrected) at verifier tolerance Ï=1.0Ï=1.0. Top Right: Relative exfiltration gain vs. benign baseline. Bottom: FPR vs. Extractable Info Pareto curves. CIPHER maximizes covert bandwidth; TERSE collapses on smaller models. Table 1: Impact of adversarial prompts on exfiltration bandwidth and defense slowdown. CIPHER significantly increases bits extracted per token and drastically reduces the verifierâs slowdown multiplier compared to the benign baseline. Bits Extracted per Token (â ) Slowdown Multiplier (â ) Model Benign TERSE SCRAMBLE CIPHER Benign TERSE SCRAMBLE CIPHER 1B 0.085± 0.007 0.073± 0.004 0.132± 0.009 0.160± 0.007 201± 17Ă 232± 12Ă 129± 9Ă 106± 5Ă 3B 0.067± 0.006 0.048± 0.002 0.103± 0.007 0.146± 0.011 254± 22Ă 352± 10Ă 166± 11Ă 117± 8Ă 8B 0.068± 0.002 0.058± 0.011 0.112± 0.010 0.183± 0.011 252± 7Ă 304± 68Ă 153± 14Ă 93± 6Ă 14B 0.093± 0.002 0.103± 0.007 0.167± 0.012 0.167± 0.016 186± 4Ă 168± 11Ă 103± 7Ă 104± 9Ă 30B-MoE 0.119± 0.011 0.145± 0.011 0.258± 0.015 0.286± 0.013 146± 15Ă 120± 9Ă 67± 4Ă 60± 3Ă 32B-dense 0.070± 0.001 0.072± 0.008 0.099± 0.005 0.145± 0.003 248± 4Ă 241± 24Ă 174± 8Ă 118± 3Ă As shown in Figure 1 and Table 1, adversarial prompt design significantly impacts the efficacy of Gumbel-sampling-based verification. We highlight three core findings: 1. Shattering the Slowdown Multiplier: Table 1 validates the claimed >200Ă>200Ă benign-case slowdown (e.g., 248Ă248Ă on 32B-dense), but CIPHER systematically dismantles it, dropping the slowdown to 118Ă118Ă on 32B-dense and 60Ă60Ă on 30B-MoE. 2. Sub-word Disruption Maximizes Leakage: CIPHER (context-bomb/multi-script) consistently achieves the highest exfiltration capacity. By attacking character-level priors, it prevents models from collapsing onto common vocabulary, yielding up to a 170% relative increase over benign prompts and peaking at 0.286 bits extracted per token on Qwen3-30B-MoE. 3. Low-Context Prompts Induce Collapse: TERSE underperforms the benign baseline on sub-14B models. Restricting context causes smaller models to default to high-probability tokens, ironically collapsing the output distribution into a near-deterministic state and exacerbating the defenseâs slowdown (peaking at 352Ă352Ă on the 3B model). 4 Conclusion Gumbel-based verificationâs security guarantees are heavily contingent on output entropy: adversarial prompts that disrupt syntactic and sub-word priors widen the jitter-induced attack gap, slashing the purported 200Ă200Ă slowdown to 60Ă60Ăâ118Ă118Ă depending on architecture. Verifier designs cannot rely on static, benign-calibrated baselines; jitter forgiveness thresholds must instead be calibrated dynamically against local token entropy. This is not unique to the unilateral weight-exfiltration setting â broader bilateral compute-verification architectures built on the same unexplained-information bound 2 inherit an identical entropy-dependent attack surface. Acknowledgments and Disclosure of Funding This work was conducted during the Hardware Assurance Program at the Cambridge AI Safety Hub (CAISH). References Karvonen et al. (2025) A. Karvonen, D. Reuter, R. Rinberg, L. Marks, A. Garriga-Alonso, and K. Warr DiFR: inference verification despite nondeterminism. External Links: 2511.20621, Link Cited by: §1. Petrie and MĂŒhlhĂ€user (2026) J. Petrie and Y. MĂŒhlhĂ€user Verifying AI compute by bounding unexplained information exfiltration. In ICML Workshop on Technical AI Governance Research, External Links: Link Cited by: §4. Rinberg et al. (2026) R. Rinberg, A. Karvonen, A. Hoover, D. Reuter, and K. Warr Verifying llm inference to detect model weight exfiltration. External Links: 2511.02620, Link Cited by: §1, §2, §2. Simmons (1984) G. J. Simmons The prisonersâ problem and the subliminal channel. In Advances in Cryptology: Proceedings of Crypto 83, p. 51â67. Cited by: §1. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, Eric. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang LMSYS-chat-1m: a large-scale real-world llm conversation dataset. External Links: 2309.11998 Cited by: §1, §2.