Paper deep dive
IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation
Yanpei Guo, Wenjie Qu, Linyu Wu, Shengfang Zhai, Lionel Z. Wang, Ming Xu, Yue Liu, Binhang Yuan, Dawn Song, Jiaheng Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/20/2026, 10:05:51 AM
Summary
The paper introduces IMMACULATE, a practical auditing framework for black-box Large Language Model (LLM) API services. It addresses trust asymmetry by detecting economically motivated deviations like model substitution, quantization abuse, and token overbilling without requiring trusted hardware or access to model internals. IMMACULATE uses verifiable computation on a small, random subset of requests and introduces the Logit Distance Distribution (LDD) as a fidelity metric to verify execution correctness under numerical non-determinism, achieving strong detection guarantees with under 1% throughput overhead.
Entities (8)
Relation Signals (7)
IMMACULATE → detects → Token Overbilling
confidence 95% · IMMACULATE ... detects economically motivated deviations-such as ... token overbilling
IMMACULATE → detects → Model Substitution
confidence 95% · IMMACULATE ... detects economically motivated deviations-such as model substitution
IMMACULATE → uses → Logit Distance Distribution
confidence 95% · we introduce the logit distance distribution (LDD) as a quantitative indicator ... IMMACULATE ... verifies that the execution remains within an admissible numerical deviation
IMMACULATE → detects → Quantization Abuse
confidence 90% · IMMACULATE ... detects economically motivated deviations-such as ... quantization abuse
Logit Distance Distribution → measures → Approximation Fidelity
confidence 90% · LDD as a quantitative indicator of approximation fidelity between a returned response and a claimed model
IMMACULATE → uses → Verifiable Computation
confidence 90% · IMMACULATE selectively audits a small fraction of requests using verifiable computation
IMMACULATE → implementson → vLLM
confidence 85% · Our prototype implementation built on vLLM incurs only ~1% overhead.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.22700v1
- Canonical: https://arxiv.org/abs/2602.22700v1
Trouble viewing inline? Open PDF directly →
Full Text
64,724 characters extracted from source content.
Expand or collapse full text
IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation Yanpei Guo Wenjie Qu Linyu Wu Shengfang Zhai Lionel Z. WANG Ming Xu Yue Liu Binhang Yuan Dawn Song Jiaheng Zhang Abstract Commercial large language models are typically deployed as black-box API services, requiring users to trust providers to execute inference correctly and report token usage honestly. We present IMMACULATE, a practical auditing framework that detects economically motivated deviations-such as model substitution, quantization abuse, and token overbilling-without trusted hardware or access to model internals. IMMACULATE selectively audits a small fraction of requests using verifiable computation, achieving strong detection guarantees while amortizing cryptographic overhead. Experiments on dense and MoE models show that IMMACULATE reliably distinguishes benign and malicious executions with under 1% throughput overhead. Our code is published at https://github.com/guo-yanpei/Immaculate. Machine Learning, ICML 1 Introduction Large language models (LLMs) have become a critical computational substrate across a wide range of applications, enabling high-quality text generation (Brown et al., 2020; Chowdhery et al., 2023), conversational agents (Adamopoulou and Moussiades, 2020), code synthesis (Chen, 2021), and reasoning-intensive workloads (Wei et al., 2022). Today, most frontier LLMs including OpenAI (OpenAI, 2023), Google (Google Cloud, 2023), and Anthropic (Anthropic, 2023) are deployed through API-based inference services, where users interact with proprietary models via black-box endpoints and are charged based on reported token usage and model specifications. Despite its success, the black-box API model introduces a fundamental trust asymmetry. Users lack any visibility into the deployed model or its internal execution, making it hard for users to audit whether the providers faithfully executed the claimed inference procedure. From an economic perspective, providers may have incentives to silently deviate from the advertised protocol—for example, by substituting a distilled model or applying aggressive quantization to reduce computational cost. Indeed, anecdotal reports of quality degradation under unchanged model claims have appeared in public forums, highlighting an emerging accountability gap in LLM service provision (webtailken, 2024; theevildays, 2024). Compared to model substitution, token overbilling poses a more severe integrity threat (Sun et al., 2025a). For reasoning-intensive workloads, API providers (e.g., OpenAI) hide intermediate chain-of-thought tokens while still billing for full internal token usage (OpenAI, 2023). As token usage is a dominant cost factor for downstream applications and autonomous agents (Chen et al., 2023; Mohammadi et al., 2025), the absence of verifiable billing guarantees undermines consumer trust. Unlike model substitution, token misreporting leaves no observable signal in the output, making detection impossible under standard black-box access. These challenges highlight the growing importance of auditing black-box LLM services. As LLM APIs become critical infrastructure for downstream applications and autonomous agents, users must be able to verify that providers execute inference honestly—without gaining access to proprietary models or internal states. However, auditing black-box LLM services is inherently difficult due to competing system requirements: • Economy: the auditing mechanism should introduce minimal computational and monetary overhead for benign LLM service providers. • Proprietorship: the LLM vendor’s model architecture, parameters, and internal inference states should remain confidential. • Robustness: the framework should offer provable guarantees for detecting protocol deviations, without falsely flagging honest servers. A representative approach (Cai et al., 2025) enforces runtime integrity using trusted execution environments (TEEs), particularly GPU-based TEEs (NVIDIA, 2023). While TEEs provide strong integrity guarantees with modest overhead (≈20%≈ 20\% on NVIDIA H100 (Tan et al., 2025)), their economic viability is limited: GPU-TEE support is restricted to specific NVIDIA hardware, and large-scale adoption of GPU-TEEs would require costly redesign of existing inference infrastructure (e.g., Google TPUs (Jouppi et al., 2017), AWS Inferentia (AWS, 2022), Ascend GPUs (Zhou et al., 2025)). An alternative line of work relies on cryptographic proofs to attest correct inference execution (Chen et al., 2024a). However, such approaches incur substantial prover overhead and are fundamentally limited to deterministic computations. Existing systems (e.g., zkLLM (Sun et al., 2024), zkGPT (Qu et al., 2025)) therefore restrict inference to integer-only arithmetic, which is inefficient on modern GPUs, rendering these approaches impractical for real-world LLM services (Cai et al., 2025; Gao et al., 2024). Our approach. In this work, we present Immaculate, a practical auditing framework for black-box LLM services that following the line of proof-based approaches. Compared to prior proof-based schemes, Immaculate eliminates the need for integer-only inference and remains compatible with existing inference infrastructures, while incurring only minor proving overhead. The key insight behind cost-saving is economic: a rational service provider can only benefit from misbehavior by deviating from the claimed inference or billing protocol on a non-negligible fraction of requests. Therefore, instead of verifying every request, the server can be considered benign if it can successfully prove its execution on a small, random subset of requests. As a result, an auditor can detect large-scale deviations with overwhelming probability by issuing a small number of random queries that mimic normal user requests and requiring the server to prove their execution. Consequently, the relative overhead of proof generation is negligible compared to the cost of serving billions of inferences per day. This idea is visualized in Figure 1. To address the inherent non-determinism of floating-point execution on modern accelerators, we introduce the logit distance distribution (LDD) as a quantitative indicator of approximation fidelity between a returned response and a claimed model. Rather than proving exact LLM execution—which is infeasible under numerical non-determinism—we require the server to prove the computation of the LDD. The resulting distribution serves as a verifiable audit footprint that captures systematic deviations from the claimed inference procedure while remaining stable under benign numerical noise. When aggregated over a large number of requests, LDDs induced by benign execution and malicious deviation become discriminative. This design enables sound verification of LLM execution without requiring bitwise reproducibility. Figure 1: The auditor sends random requests like other users and requires the model owner to prove the responses after they are received. By combining probabilistic auditing and proving LDD on audited requests, Immaculate provides economical, strong, provable guarantees for LLM service integrity. The contributions can be summarized as follows: • We propose Immaculate, a practical auditing framework for black-box LLM services that provides strong integrity guarantees without relying on GPU-based trusted execution environments. • We introduce the logit distance distribution (LDD), a fidelity metric that quantifies the approximation gap between a deployed execution and a claimed full-precision model, enabling verification under numerical non-determinism. • We demonstrate that Immaculate is economical and compatible with existing large-scale LLM deployments. Our prototype implementation built on vLLM incurs only ∼ 1% overhead. Our code is published at https://github.com/guo-yanpei/Immaculate. 2 Preliminary and Related Work Notation. For a function f(⋅)f(·), the assignment y:=f(x)y:=f(x) denotes a deterministic map whose output is uniquely determined by its input. The assignment y←g(x)y← g(x) denotes a non-deterministic map, typically reflecting hardware-dependent variability such as GPU parallelism. We use [n][n] to denote set 1,2,⋯,n \1,2,·s,n \. 2.1 Cryptographic Commitment Cryptographic commitment schemes (Merkle, 2019) allow a party to commit to a message and later reveal it, while guaranteeing binding and hiding. In practice, commitments can be instantiated using cryptographic hash functions, e.g., c=(m||r)c= Hash(m||r). 2.2 Verifiable Computation Verifiable computation (VC) (Gennaro et al., 2013) enables a powerful prover to convince a lightweight verifier that a deterministic public function f(,)=f(x,w)=y was executed correctly, without revealing the private witness w. Compared to re-execution, VC provides sublinear (often polylogarithmic) verification cost and witness privacy. Formally, VC consists of a prover–verifier pair (,)( Prove, Vrfy) satisfying completeness, soundness, and zero-knowledge. VC can be instantiated via Trusted Execution Environments (TEEs) (Sabt et al., 2015) or zero-knowledge proofs (ZKPs) (Goldwasser et al., 2019): TEEs offer near-native performance under hardware trust assumptions, while ZKPs provide stronger cryptographic guarantees at substantially higher cost. 2.3 Related Works Table 1: The comparison between our work and prior works. Robustness Infra-agnostic Efficiency GPU TEE ✓ ✗ ✓ Cryptography ✓ ✓ ✗ Empirical ✗ ✓ ✓ Our Approach ✓ ✓ ✓ Existing LLM auditing approaches can be broadly categorized into three paradigms: empirical methods, GPU-TEE–based verification, and cryptographic verification. Table 1 summarizes their trade-offs in terms of robustness, hardware requirements, and efficiency. GPU-TEE–based approaches provide strong robustness guarantees but rely on specialized infrastructure, which introduces considerable hardware procurement costs. Cryptographic approaches offer strong security guarantees without trusted hardware; however, they incur substantial proof-generation cost and are typically restricted to integer-only model executions, making them inefficient on modern AI-accelerated hardware. Empirical auditing methods, which operate without trusted hardware or cryptographic primitives, are computationally efficient and easily deployable. Nevertheless, they generally lack formal guarantees of completeness and soundness, and are typically vulnerable to adaptive attacks such as aggressive quantization (Cai et al., 2025) or token overbilling (Sun et al., 2025a). In contrast, our approach achieves a favorable balance across all three dimensions, combining robustness with hardware compatibility and practical efficiency. A more detailed discussion of existing auditing schemes is provided in Appendix A. 3 Threat Model System setting. Our auditing framework involves two parties: a cloud-based LLM inference server Srv and a trusted auditor Adt. The server claims to perform inference using a specified model ℳθM_θ with a certain precision. For each query x→ x, Srv returns an output sequence y→ y along with a reported token count T. The goal of Adt is to act as a normal user by submitting random queries to Srv and verifying whether Srv processes the requests faithfully. Threat model. We assume Srv is rational and economically motivated, and may deviate from the claimed protocol to reduce computation cost or increase billing. We exclude attacks that increase computational cost and manage to serve users with an alternative model. A malicious server is assumed to misbehave on at least 10%10\% of queries. We assume Adt always behaves honestly, and Srv cannot distinguish the requests from Adt and from normal user. Auditing goal. An auditing scheme must distinguish between the following two types of servers: 1. Honest execution: (x→)=ℳθ(x→) Srv( x)=M_θ( x) for all x→ x. 2. α-dishonest execution: Prx→[(x→)≠ℳθ(x→)]≥α. _ x[ Srv( x) _θ( x)]≥α. The auditing framework outputs ACCEPT or REJECT and satisfies: • Completeness: An honest server is accepted with overwhelming probability. • Soundness: An α-dishonest server is rejected with high probability. • Efficiency: Auditing introduces only marginal overhead beyond standard inference. • Privacy: The model’s parameters and architecture remain hidden from all parties except the model server. • Generality: The scheme applies across various LLM architectures and hardware platforms. In our paper, following prior works (Cai et al., 2025; Sun et al., 2025a), we consider the following integrity deviation attacks: • Model substitution. The server replaces the claimed model with a lower-cost alternative. • Aggressive quantization. The server executes the claimed model architecture but uses a lower-precision arithmetic format than advertised (e.g., FP8 instead of BF16). • Token overreporting. The server executes all recurrent hybrid steps faithfully but manipulates the output reconstruction function D to inflate the reported token count T beyond actual usage. 4 Logit Distance Distribution A central challenge in auditing LLM inference is the absence of a well-defined “ground truth” execution. Due to GPU-level non-determinism and numerical instability in finite-precision arithmetic, benign executions can even produce different outputs. Our key insight is that, for each LLM inference task, one can define an idealized and deterministic reference execution under full-precision arithmetic. Faithfulness can then be assessed by measuring the approximation fidelity between the observed execution and this ideal reference. Intuitively, if the discrepancy between claimed model and executed model arise solely from numerical error, their distance would exhibit a stable and predictable distribution. In contrast, malicious behaviors—such as model substitution or aggressive quantization—introduce structured deviations that generate significantly stronger and easily distinguishable error signals. We hence try to use such distance distribution to quantize the approximation fidelity. However, a central challenge lies in the fact that the discrete steps in LLMs can significantly amplify numerical errors. For example, since LLMs generate outputs auto-regressively, even a small numerical error that leads to a different token selection can cause the subsequent inference sequence to fully diverge from the reference. Therefore, a mechanism is needed to align the observed inference with the reference. In Section 4.1, we introduce an abstraction of general LLMs, which serves as the foundation for our alignment technique. In Section 4.2, we present the logit distance distribution (LDD) as a metric for quantifying approximation fidelity, which relies on our alignment technique. 4.1 Hybrid Computation Model: LLM Abstraction Let ⊆ℝ+X ^+ denote the continuous hidden-state space, D a finite discrete decision set (e.g., vocabulary symbols or expert indices), θ the model parameter space, and Γ the token vocabulary. Given a prompt x→∈Γ+ x∈ ^+, a large language model (LLM) autoregressively produces an output sequence y→∈Γ+ y∈ ^+. Hybrid computation model. We abstract LLM inference as a hybrid continuous–discrete process ℳθ:=(Eθ,Fθ,S,Gθ,D),M_θ:=(E_θ,F_θ,S,G_θ,D), which evolves through a sequence of transitions combining differentiable neural computation and discrete control-flow decisions. Concretely, inference proceeds as follows. • Initial embedding. The input prompt x→ x is mapped into an initial continuous hidden state: h0←Eθ(x→),Eθ:Γ+→.h_0← E_θ( x), E_θ: ^+ . • Recurrent hybrid step. For each generation step i=1,2,…i=1,2,…, the model executes: • Continuous transformation. Differentiable neural operators (e.g., attention, MLPs, normalization) map the current state hi−1h_i-1 to an intermediate state h~i h_i and a decision vector ℓi _i (e.g., logits): (h~i,ℓi)←Fθ(hi−1),Fθ:→×.( h_i, _i)← F_θ(h_i-1), F_θ:X ×X. • Discrete decision. A discrete choice di∈d_i (such as token sampling or MoE routing) is selected according to ℓi _i and a randomness source rir_i: di:=S(ℓi,ri),S:×ℛ→.d_i:=S( _i,r_i), S:X×R . • State update. The discrete decision is injected back into the continuous computation stream: hi←Gθ(h~i,di),Gθ:×→.h_i← G_θ( h_i,d_i), G_θ:X×D . • Output reconstruction. After N recurrent steps, the output sequence and reported token count are reconstructed: (y→,T):=D(d1,d2,…,dN).( y,T):=D(\d_1,d_2,…,d_N\). The hybrid computation model distinguishes continuous transformations from discrete decisions in two key respects: • Continuous transformations are subject to runtime non-determinism, whereas discrete decisions are fully determined by their inputs. • Continuous outputs vary smoothly under small input perturbations, while discrete decisions can change abruptly in response to the same perturbations, leading to different control-flow paths. Ideal and approximate execution. We define the full-precision model ℳ⋆M as the idealized execution of the above hybrid process where all continuous transformations are evaluated over the real numbers ℝR. In contrast, a deployed implementation ℳM operates using finite-precision arithmetic (e.g., FP16, BF16, FP8), yielding a numerical approximation of ℳ⋆M . A central goal is to quantify the fidelity of ℳM relative to ℳ⋆M . Control-flow alignment. A crucial challenge in defining and measuring approximation fidelity is that ℳM and ℳ⋆M may follow completely different control flows due to tiny numerical differences. This situation can arise when the two models make different discrete decisions. One way to address this issue is to enforce ℳ⋆M to choose the same discrete decisions as ℳM at every step. This approach is visualized in Figure 2. Figure 2: When fixing discrete selections did_i, the entire inference (green workflow) can be viewed as a continuous computation, as discrete selection no longer introduces branching uncertainty. 4.2 Logit Distance Distribution However, this enforcement introduces a new issue. For each enforced decision did_i, we must evaluate the “proximity” between the logits ℓi⋆ _i and the decision did_i. If a decision is far inconsistent with the underlying state (i.e., the distance is too large), such enforcement should be flagged as misbehavior. Measuring the fidelity between logits ℓi⋆ _i and the discrete choice did_i is challenging, as it involves comparing two inherently different entities: continuous values and discrete outcomes. Instead, we compare the distance between the inputs to the discrete selection function, namely the logits ℓi _i and ℓi⋆ _i . Since the discrete selection function is deterministic, recording the logits is also sufficient to reproduce the decision did_i. Therefore, validating the proximity of a discrete selection can be transformed into checking distance between ℓi _i and ℓi⋆ _i. Any misbehavior tends to make ℓi _i and ℓi⋆ _i far. Hence, the approximation fidelity can be measured by logit distance distribution, formally defined below. Definition 4.1 (Logit Distance Distribution). Conditioned on an identical discrete decisions, let ℓi\ _i\ denote the logits produced by a deployed model execution, and let ℓi⋆\ _i \ denote the corresponding logits produced by the full-precision model. The logit distance distribution (LDD) is defined as the distribution of (ℓi,ℓi⋆)\ Dis( _i, _i )\, where (⋅,⋅) Dis(·,·) denotes a distance metric between two logit vectors, like KL divergence or total variance (TV) distance. The following propositions characterizes the statistical signatures of three existing attacks studied in previous works (Cai et al., 2025; Sun et al., 2025a): Proposition 4.2. Model substitution introduces a systematic bias in logit outputs. As a result, model substitution typically yields substantially larger LDDs. Proposition 4.3. Reducing numerical precision manifests as increased variance in logit deviations. Thus, precision reduction causes a progressive broadening of the LDD distribution. Both propositions are proved in Appendix C.1. Proposition 4.4. Token overreporting is a special case of model substitution under the hybrid computation model. A proof sketch is token overreporting can be viewed as adding some dummy recurrent hybrid steps to the model. A complete proof is presented at Appendix C.2. 5 Immaculate: An Auditing Framework for LLM Execution In this section, we present Immaculate, a practical auditing framework for detecting α-dishonest execution of large language models (LLMs) by an untrusted service provider. At a high level, Immaculate adopts randomized auditing to reduce proving cost, and combines LDD with verifiable computation to solve intrinsic numerical non-determinism. 5.1 Randomized Auditing Our key observation for significantly reducing the proving cost is that it is unnecessary to prove every query. Instead, the auditor can adopt randomized auditing by proving only a small random subset of queries. If a server deviates on an α-fraction of requests, such behavior can be detected with overwhelming probability from this subset. Importantly, the required number of audited queries depends only on α and the desired confidence level, rather than on the total service volume. Example. Assume a method can detect malicious responses with zero false positive and only 1%1\% detection rate. If the server cheats on a fraction α=0.1α=0.1 of requests, then achieving an overall detection probability of 95%95\% (i.e., evasion probability η=5%η=5\%) requires only N=logηlog(1−α⋅1%)≈ 3,000N= η (1-α· 1\%)\;≈\;3,000 audited queries. Since a production LLM service may process billions of requests per day, the cost of proving only a few thousand queries is amortized to negligible overhead. 5.2 Reproducibility via Discrete-State Commitments Exact reproduction of LLM execution is infeasible due to unavoidable numerical non-determinism. Instead of proving bitwise equivalence, Immaculate verifies that the execution remains within an admissible numerical deviation from a reference full-precision model, using the LDD metric. At initialization, the model owner publishes a cryptographic commitment to the claimed full-precision model ℳθ⋆M_θ . During inference, the server runs finite-precision model ℳθM_θ, recording and committing to ℓi\ _i\ at each discrete decision. Importantly, these committed logits enables reproducing every discrete selection result. For an audited query, server proves the LDD via VC, using the committed runtime logits ℓi \ _i \. Concretely, the VC proof establishes that: 1. discrete decisions are derived from the committed logits ℓi\ _i\; 2. continuous transformations are computed using ℳθ⋆M_θ , obtaining ℓi⋆ \ _i \; 3. the final output (y→,T)( y,T) is consistent with the discrete decisions; 4. distribution of distance between ℓi\ _i\ and ℓi⋆ \ _i \ is output. Optimization: Top-K distance. As top-K selection is the most common discrete selection method in LLMs, we design a specialized top-K distance metric, allowing the server to cache and commit only the K selected indices instead of the full logits vector. The detailed method is provided in Appendix D. Figure 3: Logit TV-distance distribution of LLaMA3-70B. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior. 5.3 Utilizing LDD in Practice. A central goal of our scheme is to produce observable and verifiable auditing footprints—concrete statistical evidence that can be independently examined to assess service integrity. The LDD serves as such an auditing footprint. There are many possible ways to utilize LDD for integrity auditing. In this work, we present LDD as a clear metric to distinguish between benign and malicious responses, along with some simple yet effective approaches for leveraging LDD in practice. Empirically, we test LDD over thousands of requests, and present a representative result in Figure 4. It aggregates benign executions and malicious executions over thousands of requests, and find they induce well-separated distance distributions. We can observe that a particularly discriminative statistic is the tail probability, e.g. Pr[(ℓi,ℓi⋆)>0.1] [ TV( _i, _i)>0.1]. Based on this observation, we can detect malicious requests by evaluating the density of “tailing logits”. For a request involving T discrete steps, we define p(t1):=1T|i∈[T]:(ℓi,ℓi⋆)>t1|.p(t_1):= 1T | \i∈[T]: TV( _i, _i)>t_1 \ |. A response is flagged as malicious (e.g. computed vis FP8) if p(t1)>t2p(t_1)>t_2, where thresholds t1t_1 and t2t_2 are auditing parameters. The choosing strategy of t1t_1 and t2t_2 is provided in Append E. 5.4 End-to-end System We now describe the end-to-end execution flow of Immaculate, showing how randomized auditing, logit commitments, and verifiable computation are integrated into a practical LLM inference service. Setup: model commitment. At deployment time, the model provider commits to the claimed full-precision reference model ℳθ⋆M_θ by publishing a cryptographic hash commitment. This commitment binds the server to a specific reference model while preserving full model confidentiality. Online inference and logging. For each incoming query x→ x, the server performs standard LLM inference. During execution, the server additionally records the logits ℓi\ _i\ at each discrete decision point (e.g., token selection or expert routing). Along with the response (y→,T)( y,T), Srv also returns a commitment to ℓi\ _i\ to the user. We estimate logging cost of commercial open-source LLMs Even the largest open-source models require caching only about 1 KB of data per token, so this logging and commitment step incurs negligible overhead. The cached messages can be deleted after a prescribed threshold if the user does not request auditing, and therefore this mechanism does not introduce significant storage costs. Randomized auditing. Adt submits random queries to Srv while behaving indistinguishably from a normal user. After receiving responses, Adt immediately reveals its auditor identity and requests an audit proof from Srv. In response, Srv invokes a verifiable computation (VC) procedure to compute and prove the corresponding LDD. These LDDs serve as observable auditing footprints, providing statistical evidence of server-side execution integrity. Formal pseudo-code description is given in Appendix B. In Appendix F, we also analyze a strategic attacks from Srv, and present the best strategy for a cost-incentive server is honestly commit runtime logits of its deployed model. 6 Experiments We evaluate Immaculate along two complementary dimensions: (1) its completeness and detection effectiveness, and (2) the system’s imposed overhead. Concretely, our experiments answer the following questions: 1. Does the proposed logit distance distribution (LDD) meaningfully characterize approximation fidelity between a deployed execution and the claimed full-precision model? 2. Can LDD reliably distinguish benign executions from malicious behaviors such as aggressive quantization and model substitution? 3. What per-request detection probability and false positive rate can be achieved using LDD-based decision rules? 4. When combined with randomized auditing, does Immaculate provide strong completeness and soundness guarantees against an α-dishonest server? 5. What is the end-to-end performance overhead introduced by Immaculate for model providers? 6.1 Experimental Setup System implementation. We build our inference system atop vLLM (Kwon et al., 2023), a widely used high-performance LLM inference framework. Verifiable computation (VC) is implemented using HuggingFace Transformers (Wolf et al., 2019) and executed in FP32 arithmetic within a TDX enclave, serving as the reference full-precision execution. For all experiments, we adopt Top-20 token sampling strategy. Hardware. All experiments are conducted on NVIDIA RTX 6000 Pro GPUs with 96 GB of GPU memory. We perform inference using tensor parallelism with a degree of 2 across GPUs. Datasets. We evaluate on GSM8K, TriviaQA, and WebQuestions, representing mathematical reasoning, factual QA, and open-domain QA, respectively. For scalability, we subsample 500 prompts per dataset, resulting in a total of 1,500 evaluation queries. An initial setup phase is conducted using the first 200 examples from each of GSM8K, TriviaQA, and WebQuestions. We randomly sample 200 prompts from each dataset to form an independent setup set for calibrating auditing thresholds; all remaining prompts are held out and used exclusively for evaluation. Models. We evaluate both dense and mixture-of-experts (MoE) architectures, including dense models LLaMA3-70B, Qwen3-32B and MoE models Qwen3-30B-A3B, DeepSeek-V2-Lite. Attacks. We evaluate Immaculate against model substitution and aggressive quantization. We omit token overreporting from separate empirical evaluation because we have proved in Section 4.1, it can be reduced to a special case of model substitution. For model substitution attacks, we evaluates 1) Substitute LLaMA3-70B with LLaMA3-8B; 2) Substitute Qwen3-32B with Qwen3-14B. We do not explore model substitution attacks for MoE models because larger versions are too costly to run. For quantization attacks, we assume all benign deployments are under BF16, and consider FP8 as quantization attacks, following the setting in (Cai et al., 2025). 6.2 Global Logit Discrepancy Distribution We first evaluate whether the global LDD reflects approximation fidelity under different deployment regimes. Specifically, we compare three settings against the same full-precision reference model: 1) benign BF16 execution; 2) FP8 quantized execution; 3) model substitution (for dense models only). Table 2: Proportion of TV distance larger than certain threshold Model Deploy >0.1>0.1 >0.2>0.2 >0.3>0.3 LLaMA3 BF16 0.017%0.017\% 1.7×10−51.7× 10^-5 1.9×10−61.9× 10^-6 FP8 1.7%1.7\% 0.23%0.23\% 0.054%0.054\% Sub. 11%11\% 8.2%8.2\% 6.2%6.2\% Qwen3 BF16 0.17%0.17\% 0.025%0.025\% 6.0×10−56.0× 10^-5 FP8 4.8%4.8\% 1.1%1.1\% 0.42%0.42\% Sub. 28%28\% 20%20\% 15%15\% Qwen3MoE BF16 0.074%0.074\% 2.6×10−52.6× 10^-5 4.7×10−64.7× 10^-6 FP8 5.5%5.5\% 1.0%1.0\% 0.23%0.23\% Deepseek BF16 6.6×10−56.6× 10^-5 6.0×10−66.0× 10^-6 8.5×10−78.5× 10^-7 FP8 1.6%1.6\% 0.19%0.19\% 0.053%0.053\% Figure 4 has visualized the aggregated logit TV-distance distributions across all evaluated prompts. Across all evaluated dense and MoE models, benign BF16 execution induces sharply concentrated distributions with rapidly decaying tails. In contrast, FP8 quantization substantially increases the tail mass at moderate and large logit distances, while model substitution causes a pronounced right-shift, yielding orders-of-magnitude higher probability mass in the extreme tail. To quantify the separation, we further report in Table 2 the probability that the total-variation (TV) distance between deployed and reference logits exceeds different thresholds. It shows that FP8 quantization increases these tail probabilities by several orders of magnitude, while model substitution amplifies them further. These results confirm that tail events in the LDD provide a highly discriminative signal for distinguishing benign execution from economically motivated deviations, motivating the threshold-based detection rule used in subsequent experiments. In Appendix G, we also present global LDDs beyond TV distances. All results show distinct tail-probability across different deployments. 6.3 Per-request Decision Based on LDD Table 3: Malicious request detection rate (the rate of malicious requests that can be detected under FP=10−5FP=10^-5). As shown in Section 5.1, 1% per-request detection probability is sufficient for effective randomized auditing. Model Deploy GSM8K TriviaQA WebQuestions LLaMA3 FP8 9.0%9.0\% 9.7%9.7\% 5.3%5.3\% Sub. 42%42\% 95%95\% 99%99\% Qwen3 FP8 1.3%1.3\% 3.6%3.6\% 2.0%2.0\% Sub. 96%96\% 96%96\% 97%97\% Qwen3MoE FP8 2.0%2.0\% 2.1%2.1\% 5.0%5.0\% Deepseek FP8 10.3%10.3\% 3.0%3.0\% 3.7%3.7\% While global LDDs exhibit clear separation, auditing decisions must ultimately be made at the per-request level. In practice, a malicious LLM provider may apply heterogeneous deployment strategies across requests, thereby blurring the global distribution and reducing its discriminative power. Following Section 5.3, we define a per-request statistic p(t1)p(t_1) based on the fraction of logit distance exceeding a threshold t1t_1. The request is flagged as malicious if p(t1)>t2p(t_1)>t_2. The auditing parameters t1,t2t_1,t_2 is computed via method given in Section 5.3. Table 3 present the resulting detection rates. For quantization attacks, a request is detected with probability at least 1.3%1.3\%, whereas model substitution is detected with probability larger than 40%40\%. Accurately estimating the per-request false positive rate under benign execution is challenging due to the extreme rarity of tail events. Owing to computational constraints, our experiments evaluate only on thousands of requests; consequently, all benign requests observed in practice are correctly classified, making direct empirical estimation of the false positive rate infeasible. To address this limitation, we adopt an extreme value theory (EVT)–based approach to model the tail behavior of benign LDDs. Specifically, we fit an EVT model to the extreme tail of the benign distribution p(t1)p(t_1) and estimate the probability that a benign request produces an LDD exceeding t2t_2. Results show that among all datasets and models, the false positive rate is smaller than 10−510^-5. Completeness and soundness. Assume Immaculate audits approximately 3,000 requests per day. To achieve overwhelming completeness, among 3,000 requests, Adt allows at most 3 of them classified to be malicious. Consider the at most 10−510^-5 per-request false positive rate, the probability of rejecting a benign server is 1−∑k=03(3000k)(10−5)k(1−10−5)3000−k≤10−7,1- _k=0^3 3000k(10^-5)^k(1-10^-5)^3000-k≤ 10^-7, representing overwhelming completeness. For an α=0.1α=0.1 dishonest server, each audited request is detected with probability at least 10−310^-3. The probability of observing 4 or more detections in a day is therefore 1−∑k=03(3000k)(10−3)k(1−10−3)3000−k≥0.3.1- _k=0^3 3000k(10^-3)^k(1-10^-3)^3000-k≥ 0.3. Over a month-long horizon, the probability of persistent evasion becomes negligible. 6.4 Auditing Parameter Study Table 4: Detection rate and false positive rate under different hyperparameter Hyper. LLaMA3 Qwen3 Qwen3MoE Deepseek 0.02 Detect 2.1%2.1\% 0.44%0.44\% 0.93%0.93\% 1.3%1.3\% FP 2×10−92× 10^-9 2×10−82× 10^-8 5×10−105× 10^-10 8×10−108× 10^-10 0.03 Detect 6.2%6.2\% 0.67%0.67\% 2.1%2.1\% 3.0%3.0\% FP 1×10−81× 10^-8 3×10−73× 10^-7 1×10−81× 10^-8 2×10−82× 10^-8 0.04 Detect 6.2%6.2\% 1.2%1.2\% 2.1%2.1\% 3.0%3.0\% FP 1×10−81× 10^-8 1×10−61× 10^-6 1×10−81× 10^-8 2×10−82× 10^-8 0.05 Detect 6.2%6.2\% 2.3%2.3\% 3.6%3.6\% 5.6%5.6\% FP 1×10−81× 10^-8 4×10−64× 10^-6 9×10−79× 10^-7 8×10−88× 10^-8 0.06 Detect 9.4%9.4\% 3.3%3.3\% 3.6%3.6\% 8.8%8.8\% FP 4×10−74× 10^-7 2×10−52× 10^-5 9×10−79× 10^-7 5×10−75× 10^-7 0.07 Detect 9.4%9.4\% 5.1%5.1\% 5.2%5.2\% 8.8%8.8\% FP 4×10−74× 10^-7 6×10−56× 10^-5 4×10−64× 10^-6 5×10−75× 10^-7 The derivation of the auditing parameters is largely heuristic, so we explore a wider range of parameter choices to demonstrate the robustness of our scheme. When computing t1t_1 and t2t_2 via the ceremony, we employ an ad-hoc hyperparameter that accepts only (t1,t2)(t_1,t_2) pairs achieving a 5% detection rate on the ceremony dataset. Here, we vary this hyperparameter around the 5% threshold to derive different auditing parameters t1t_1 and t2t_2, and report their corresponding false positive rates and detection rates in Table 3. Due to space constraints, we aggregate the results from three datasets to present the overall false positive rate and detection rate. Results show that for all derived auditing parameters, the FP rate and detection rate are always acceptable. 6.5 Overhead to Benign Execution We finally evaluate the efficiency impact of Immaculate. Since verifiable computation is triggered only on a small audited subset, the dominant overhead arises from storing and committing lightweight runtime values. Across all evaluated models, the end-to-end throughput overhead for a benign server is below 1%1\%, demonstrating practical deployability. The VC deployed in CPU TEE is hundreds of times slower than inference. Consider that the proportion of audited request is smaller than 10−510^-5, the overall cost is negligible. Table 5: Overhead Evaluation. ‘Thp.’ denotes the inference throughput loss. ‘Request VC’ denotes the runtime of VC on CPU-TEE compared with GPU inference Thp. Request VC LLaMA3-70B 0.3%0.3\% 400×400× Qwen3-32B 0.3%0.3\% 400×400× Qwen3-30B-A3B 0.9%0.9\% 900×900× DeepSeek-V2-Lite 1.0%1.0\% 800×800× 7 Conclusion We have introduced Immaculate, a practical and robust auditing framework for LLM inference services that operate as black-box APIs. By combining selective verifiable computation with a novel Logit Distance Distribution (LDD) metric, Immaculate enables auditors to detect economically motivated deviations—such as model substitution, aggressive quantization, and token overreporting—without requiring access to model internals or relying on trusted hardware. Looking ahead, we believe that Immaculate lays the foundation for a new class of auditing frameworks. Future work may explore additional optimizations, particularly in leveraging LDD to more effectively identify malicious requests. Ultimately, our framework represents a step toward greater transparency, accountability, and trust in commercial LLM services. References E. Adamopoulou and L. Moussiades (2020) An overview of chatbot technology. In IFIP international conference on artificial intelligence applications and innovations, p. 373–383. Cited by: §1. Anthropic (2023) Anthropic api documentation. Note: https://docs.anthropic.com Cited by: §1. AWS (2022) AWS inferentia. External Links: Link Cited by: §1. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. W. Cai, T. Shi, X. Zhao, and D. Song (2025) Are you getting what you pay for? auditing model substitution in llm apis. arXiv preprint arXiv:2504.04715. Cited by: §1, §1, §2.3, §3, §4.2, §6.1. B. Chen, S. Waiwitlikhit, I. Stoica, and D. Kang (2024a) Zkml: an optimizing system for ml inference in zero-knowledge proofs. In Proceedings of the Nineteenth European Conference on Computer Systems, p. 560–574. Cited by: §1. L. Chen, M. Zaharia, and J. Zou (2023) Frugalgpt: how to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. Cited by: §1. L. Chen, M. Zaharia, and J. Zou (2024b) How is chatgpt’s behavior changing over time?. Harvard Data Science Review 6 (2). Cited by: Appendix A. M. Chen (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. (2023) Palm: scaling language modeling with pathways. Journal of Machine Learning Research 24 (240), p. 1–113. Cited by: §1. S. Eyuboglu, K. Goel, A. Desai, L. Chen, M. Monfort, C. Ré, and J. Zou (2024) Model changelists: characterizing updates to ml models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 2432–2453. Cited by: Appendix A. I. Gao, P. Liang, and C. Guestrin (2024) Model equality testing: which model is this api serving?. arXiv preprint arXiv:2410.20247. Cited by: Appendix A, §1. R. Gennaro, C. Gentry, B. Parno, and M. Raykova (2013) Quadratic span programs and succinct nizks without pcps. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, p. 626–645. Cited by: §2.2. S. Goldwasser, S. Micali, and C. Rackoff (2019) The knowledge complexity of interactive proof-systems. In Providing sound foundations for cryptography: On the work of shafi goldwasser and silvio micali, p. 203–225. Cited by: §2.2. Google Cloud (2023) Vertex ai generative ai. Note: https://cloud.google.com/vertex-ai Cited by: §1. Y. Huang, M. Nasr, A. Angelopoulos, N. Carlini, W. Chiang, C. A. Choquette-Choo, D. Ippolito, M. Jagielski, K. Lee, K. Z. Liu, et al. (2025) Exploring and mitigating adversarial manipulation of voting-based leaderboards. arXiv preprint arXiv:2501.07493. Cited by: Appendix A. N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, et al. (2017) In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, p. 1–12. Cited by: §1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: §6.1. R. C. Merkle (2019) Protocols for public key cryptosystems. In Secure communications and asymmetric cryptosystems, p. 73–104. Cited by: §2.1. M. Mohammadi, Y. Li, J. Lo, and W. Yip (2025) Evaluation and benchmarking of llm agents: a survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, p. 6129–6139. Cited by: §1. NVIDIA (2023) Confidential compute on nvidia hopper h100. Technical report NVIDIA. Cited by: §1. J. M. Ong, M. Di Ferrante, A. Pazdera, R. Garner, S. Jaghouar, M. Basra, M. Ryabinin, and J. Hagemann (2025) Toploc: a locality sensitive hashing scheme for trustless verifiable inference. arXiv preprint arXiv:2501.16007. Cited by: Appendix A. OpenAI (2023) OpenAI api documentation. Note: https://platform.openai.com/docs Cited by: §1, §1. W. Qu, Y. Sun, X. Liu, T. Lu, Y. Guo, K. Chen, and J. Zhang (2025) ZkGPT: an efficient non-interactive zero-knowledge proof framework for llm inference. In 34st USENIX Security Symposium (USENIX Security 25), Cited by: §1. M. Sabt, M. Achemlal, and A. Bouabdallah (2015) Trusted execution environment: what it is, and what it is not. In 2015 IEEE Trustcom/BigDataSE/Ispa, Vol. 1, p. 57–64. Cited by: §2.2. G. Sun, Z. Wang, B. Tian, M. Liu, Z. Shen, S. He, Y. He, W. Ye, Y. Wang, and A. Li (2025a) CoIn: counting the invisible reasoning tokens in commercial opaque llm apis. arXiv preprint arXiv:2505.13778. Cited by: Appendix A, §1, §2.3, §3, §4.2. H. Sun, J. Li, and H. Zhang (2024) Zkllm: zero knowledge proofs for large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 4405–4419. Cited by: §1. M. Sun, Y. Yin, Z. Xu, J. Z. Kolter, and Z. Liu (2025b) Idiosyncrasies in large language models. arXiv preprint arXiv:2502.12150. Cited by: Appendix A. Y. Tan, C. Tan, Z. Mi, and H. Chen (2025) Pipellm: fast and confidential large language model services with speculative pipelined encryption. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, p. 843–857. Cited by: §1. theevildays (2024) OpenAI did made gpt3.5 more stupid?. External Links: Link Cited by: §1. A. A. Velasco, S. Tsirtsis, N. Okati, and M. Gomez-Rodriguez (2025) Is your llm overcharging you? tokenization, transparency, and incentives. arXiv preprint arXiv:2505.21627. Cited by: Appendix A. webtailken (2024) GPT4-turbo more “stupid/lazy” - it’s not a gpt4. External Links: Link Cited by: §1. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: §1. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. (2019) Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. Cited by: §6.1. J. Yao, H. Su, T. Liao, Z. Cheng, H. Zhang, X. Wang, and P. Viswanath (2025) Nondeterminism-aware optimistic verification for floating-point neural networks. arXiv preprint arXiv:2510.16028. Cited by: Appendix A. Y. Zhou, Z. Wang, Z. Wang, R. Zhang, C. Tian, X. Wang, W. Dou, G. Chen, B. Wang, Y. Tian, et al. (2025) Accelerating model training on ascend chips: an industrial system for profiling, analysis and optimization. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), p. 1387–1408. Cited by: §1. X. Zhu, Y. Ye, T. Qiu, H. Zhu, S. Tan, A. Mannan, J. Michala, R. A. Popa, and W. Neiswanger (2025) Auditing black-box llm apis with a rank-based uniformity test. arXiv preprint arXiv:2506.06975. Cited by: Appendix A. Appendix A Extended Related Work Existing LLM auditing frameworks struggle to simultaneously achieve economy, proprietorship, and robustness. Section 2.3 discusses the limitations of cryptography-based approaches and GPU TEEs in terms of deployment cost and infrastructure requirements. In this section, we review additional auditing paradigms and analyze their limitations under the black-box service setting. Direct Verification. When the model is fully public, auditing becomes straightforward. The most direct approach is to re-execute inference and compare the outputs with those returned by the service. TopLoc (Ong et al., 2025) accelerates verification by transforming decoding operations into prefilling, reducing the computational overhead of re-execution. Yao et al. (Yao et al., 2025) propose verifying only a subset of randomly selected layers and integrate the auditing process with a blockchain-based logging mechanism. However, these methods require the auditor to access model weights and execution details, making them unsuitable for proprietary commercial models. Detecting LLM-generated text. Another line of work attempts to audit black-box services by analyzing their output behavior. Prior studies track performance drift and behavioral changes over time (Chen et al., 2024b; Eyuboglu et al., 2024). Other methods aim to identify the deployed model by examining output distributions, including classifier-based fingerprinting (Sun et al., 2025b), statistical discrepancy tests (Gao et al., 2024; Zhu et al., 2025), and identity-style prompting (Huang et al., 2025). However, these approaches typically require large query budgets and are vulnerable to randomized model substitution or decoding variability. Moreover, since they rely solely on observable text outputs, they cannot detect attacks such as token overreporting, which do not affect the returned content. Token Overreport Detection. CoIn (Sun et al., 2025a) addresses the token overreporting issue by committing embeddings of hidden reasoning tokens in a Merkle hash tree for quantity verification, and by performing embedding-based relevance checks to detect low-effort or fabricated content. Nevertheless, a provider may still generate hidden tokens using a cheaper or simplified model while preserving semantic consistency, allowing the inflated sequence to pass such semantic validation. Complementary work by Velasco et al. (Velasco et al., 2025) analyzes the problem from an economic perspective, showing that the pay-per-token pricing mechanism inherently incentivizes overreporting under information asymmetry. They argue that, since provider revenue scales with reported token length while users cannot observe the internal generation process, strategic tokenization or count manipulation becomes financially attractive, and propose pay-per-character pricing as an incentive-compatible alternative. However, such pricing changes do not provide technical guarantees on faithful execution. Summary. Existing approaches each address only part of the auditing challenge. Direct re-execution methods provide strong guarantees but require full model access, violating proprietorship. Output-based statistical methods are economical but lack robustness against adaptive or invisible deviations. Economic or billing-based solutions address incentive misalignment but do not verify execution correctness. In contrast, Immaculate aims to achieve a balanced design that simultaneously ensures economy, preserves model confidentiality, and provides provable detection guarantees for execution deviations under black-box access. Appendix B Algorithms The pseudo-code is presented at Algorithm 1, Algorithm 2 and Algorithm 3. Algorithm 1 Srv: LLM Server’s algorithm 0: LLM model ℳθ:=(Eθ,Fθ,S,Gθ,D)M_θ:=(E_θ,F_θ,S,G_θ,D) 1: ψM:=(ℳθ⋆) _M:= Hash(M_θ ), where ℳθ⋆M_θ is the full-precision version of ℳθM_θ. 2: Publish ψM _M 3: upon receiving ⟨,x→⟩ Request, x from Usr do 4: r←$0,1λr _ \0,1 \^λ 5: h0←Eθ(x→)h_0← E_θ( x). 6: for i=1,2,⋯,Ni=1,2,·s,N do 7: h~i,ℓi←Fθ(hi−1) h_i, _i← F_θ(h_i-1) 8: di:=S(ℓi,r)d_i:=S( _i,r) 9: hi←Gθ(h~i,di)h_i← G_θ( h_i,d_i) 10: end for 11: y→,T:=D(d1,⋯,dN) y,T:=D( \d_1,·s,d_N \) 12: ψ:=(r,ℓ1,⋯,ℓN)ψ:= Hash(r, \ _1,·s, _N \) 13: send ⟨,y→,T,ψ⟩ Response, y,T,ψ to Usr. 14: upon receiving ⟨⟩ Audit from Adt do 15: w:=(ℳθ⋆,r,ℓ1,⋯,ℓN)w:= (M_θ ,r, \ _1,·s, _N \ ) 16: ϕ,π:=.((ψM,ψ,x→,y→,T);w)φ,π:= VC. prove (( _M,ψ, x, y,T);w ) 17: send ⟨,ϕ,π⟩ Proof,φ,π to Adt. 18: end upon 19: end upon Algorithm 2 Adt: LLM Auditor’s algorithm 0: Auditing parameter ap 0: b∈0,1,⊥b∈ \0,1, \ 1: Generate random prompt x→ x 2: send ⟨,x→⟩ Request, x to Srv 3: upon receiving ⟨,y→,T,ψ⟩ Response, y,T,ψ do 4: send ⟨⟩ Audit to Srv. 5: end upon 6: upon receiving ⟨,ϕ,π⟩ Proof,φ,π do 7: if .((ψM,ψ,x→,y→,T),π) VC. prove (( _M,ψ, x, y,T),π ) then 8: Return ⊥ . 9: end if 10: Return 0/1 based on ϕφ and ap. 11: end upon Algorithm 3 VC: Verifiable computation function 0: Public input: model commitment ψM _M, logits commitment ψ, prompt x→ x, response y→ y, token usage T; Private input: model ℳθ⋆=(Eθ⋆,Fθ⋆,S,Gθ⋆,D)M_θ =(E_θ ,F_θ ,S,G_θ ,D), seed r, logits ℓ1,⋯,ℓN \ _1,·s, _N \. 0: Logits distance distribution 1: if ψ≠(r,ℓ1,⋯,ℓN)ψ≠ Hash (r, \ _1,·s, _N \ ) or ψM≠(ℳθ⋆) ψ_M≠ Hash(M_θ ) then 2: abort 3: end if 4: h0⋆←Eθ⋆(x→)h _0← E_θ ( x). 5: for i=1,2,⋯,Ni=1,2,·s,N do 6: h~i⋆,ℓi⋆←Fθ⋆(hi−1⋆) h _i, _i← F_θ (h _i-1) 7: δi=(ℓi,ℓi⋆) _i= Dis( _i, _i) 8: di:=S(ℓi,r)d_i:=S( _i,r) 9: hi⋆←Gθ⋆(h~i⋆,di)h _i← G_θ ( h _i,d_i) 10: end for 11: if y→,T≠D(d1,⋯,dN) y,T≠ D( \d_1,·s,d_N \) then 12: abort 13: end if 14: return distribution of δ1,⋯,δN \ _1,·s, _N \ Appendix C Proof of Propositions C.1 Proposition 4.2 and Proposition 4.3 Proof. Fix a prompt x→ x and a sequence of discrete decisions d→=(d1,…,dN) d=(d_1,…,d_N), thereby aligning the control flow of two executions. Let ℓi⋆(θ) _i (θ) denote the deterministic logits produced by the full-precision model at step i. For an approximate execution ℳ(A)M^(A), the observed logits can be written as ℓi(A)=ℓi⋆(θ(A))+ηi(A), _i^(A)= _i (θ^(A))+ _i^(A), where ηi(A) _i^(A) captures run-dependent numerical error due to finite precision or kernel non-determinism. Comparing ℳ(A)M^(A) with the ideal model ℳ(B)=ℳ⋆M^(B)=M yields ℓi(A)−ℓi(B)=(ℓi⋆(θ(A))−ℓi⋆(θ(B)))⏟Δimodel+ηi(A)⏟Δiprec, _i^(A)- _i^(B)= ( _i (θ^(A))- _i (θ^(B)) )_ _i^model+ _i^(A)_ _i^prec, where ηi(B)=0 _i^(B)=0 by definition. Model substitution. If θ(A)≠θ(B)θ^(A)≠θ^(B), then Δimodel≠0 _i^model≠ 0, producing a systematic (biased) deviation in logits. Precision reduction. If θ(A)=θ(B)θ^(A)=θ^(B), then Δimodel=0 _i^model=0 and discrepancies arise solely from numerical noise. Lower-precision formats induce higher variance: Var(ηiFP8)>Var(ηiFP16),Var( _i^FP8)>Var( _i^FP16), resulting in stochastic, zero-mean deviations rather than persistent bias. ∎ C.2 Proposition 4.4 Proof. Let the claimed model be ℳ:=(Eθ,Fθ,S,Gθ,D),M:=(E_θ,F_θ,S,G_θ,D), and consider a fixed prompt x→ x with discrete decision sequence d→=(d1,…,dN) d=(d_1,…,d_N) generated by ℳM. The corresponding output is (y→,T)=D(d→),T=N.( y,T)=D( d), T=N. We construct an alternative model ℳ′:=(Eθ,Fθ′,S,Gθ′,D′),M :=(E_θ,F _θ,S,G _θ,D ), which differs from ℳM only in the recurrent hybrid step and output reconstruction. Define K>0K>0 and extend the discrete decision sequence as d→′:=(d1,…,dN,d¯,…,d¯⏟K), d :=(d_1,…,d_N, d,…, d_K), where d¯∈ d is a fixed dummy decision. Define the modified state update such that Gθ′(h~,d¯)=handFθ′(h)=(h,ℓ),G _θ( h, d)=h F _θ(h)=(h, ), i.e., the dummy steps leave the hidden state invariant and do not affect subsequent decisions. Finally, define the reconstruction function D′(d→′):=(y→,N+K),D ( d ):=( y,N+K), where y→ y is recovered solely from the prefix (d1,…,dN)(d_1,…,d_N). By construction, ℳ′(x→)=(y→,T+K),ℳ(x→)=(y→,T),M ( x)=( y,T+K), ( x)=( y,T), while the observable output sequence y→ y is identical. Hence, token overreporting corresponds to replacing ℳM with a functionally distinct model ℳ′M that modifies the hybrid transition structure. Since ℳ′≠ℳM but preserves output semantics while altering reported resource usage, token overreporting is equivalent to a form of model substitution, and therefore constitutes a special case of model downsizing. ∎ Appendix D Optimization: Verification of Top-K. Among all discrete operations in LLM inference, top-K selection is the most dominant, appearing in both token sampling and mixture-of-experts routing. Formally, top-K is a deterministic operator K:ℝn→[n]K, Top_K:R^n→[n]^K, which returns the indices of the K largest entries in a logits vector. In practice, n is typically much larger than K. For example, in token sampling, n corresponds to the size of the entire vocabulary, often exceeding 10510^5, while K is typically around 20. Due to the large discrepancy between n and K, storing and committing to the full logits vector can be prohibitively expensive. Instead, Immaculate allows the LLM to store only the runtime top-K indices rather than the entire logits vector, by introducing a minimal-perturbation distance criterion. Top-K distance. Given logits ℓ∈ℝn ^n and a claimed index set I⊆[n]I [n] with |I|=K|I|=K, we define ΔTopK(ℓ,I):=minℓ′:K(ℓ′)=I‖ℓ′−ℓ‖1. _TopK( ,I):= _ : Top_K( )=I\| - \|_1. By construction, ΔTopK(ℓ,I)=0 _TopK( ,I)=0 if and only if I is a valid top-K set. Otherwise, it quantifies the minimal numerical perturbation required to make I the exact top-K. The defined distance is 11-Lipschitz with respect to ℓ , ensuring robustness to small numerical noise. Efficient computation For each given logit vector ℓ and index set I, ΔTopK(ℓ,I) _TopK( ,I) can be evaluated efficiently. Since I denotes the indices of the largest K entries in the logits vector, there must exist a scalar threshold t∈ℝt such that ℓi′≥t∀i∈I,ℓj′≤t∀j∉I. _i≥ t ∀ i∈ I, _j≤ t ∀ j∉ I. Using a greedy algorithm, for a fixed threshold t, the optimal perturbed logits ℓ′ satisfy ℓi′=max(ℓi,t)∀i∈I,ℓj′=min(ℓj,t)∀j∉I. _i= ( _i,t) ∀ i∈ I, _j= ( _j,t) ∀ j∉ I. Intuitively, entries in I are minimally raised to t, while entries outside I are minimally lowered. Assume the logits are sorted as ℓ1≥ℓ2≥⋯≥ℓn _1≥ _2≥·s≥ _n, and let t lie in the interval ℓu≥t≥ℓu+1 _u≥ t≥ _u+1. The resulting distance is given by ∑i≤u,i∉I(ℓi−t)+∑i≥u+1,i∈I(t−ℓi). _i≤ u,\,i∉ I( _i-t)+ _i≥ u+1,\,i∈ I(t- _i). When t∈[ℓu,ℓu+1]t∈[ _u, _u+1], this expression is a linear function of t, and thus monotonic. The distance reaches its minimum when t is either ℓu _u or ℓu+1 _u+1. Therefore, we only need to consider n candidate thresholds, where t equals ℓ1,ℓ2,…,ℓn _1, _2,…, _n. Utilizing this observation, the top-K distance can be computed in linear time. Integration with Immaculate. During inference, the server stores only the top-K indices. During proving, the VC recomputes logits ℓ using the full-precision model and verifies the recorded top-K indices I by evaluating ΔTopK(ℓ,I) _TopK( ,I). The resulting distance distribution is then used for statistical verification. Appendix E Ceremony: choice of t1t_1, t2t_2 To derive the desirable hyperparameters t1t_1 and t2t_2, an initial setup is required: the model provider runs both their deployed LLM and the quantized version they intend to prevent on a prescribed, sufficiently large dataset within a TEE to obtain the empirical per-request LDD as a reference. To search for the optimal hyperparameters, the auditor enumerates each t1t_1 and identifies the largest corresponding t2t_2 such that the detection rate exceeds 5%. The (t1,t2)(t_1,t_2) pair yielding the lowest false positive rate is selected as the final parameter setting. Appendix F Adaptive Adversary Analysis The preceding discussion assumes that the adversary commits to logits generated by a simplified or quantized execution. A more subtle concern is whether a rational adversary could strategically commit to fabricated logits to increase the probability of evasion. Here, we prove that as long as the LLM server is rational (though not necessarily benign), no such strategy can offer additional benefit. Rationality assumption. We assume the adversary is rational: under a fixed computational budget T, it always adopts the highest-quality approximation of the claimed model achievable within that budget. Proposition F.1 (Logit Commitment Optimality). Let ℳoptM_opt be the best approximation of the claimed model ℳ⋆M achievable within budget T. Then the adversary’s dominant strategy is to truthfully commit to the logits produced by ℳoptM_opt. Proof. Let ℳTM_T denote the set of all inference procedures executable within budget T, and let Lℳ(x→)L_M( x) be the logits produced by model ℳM on input x→ x. Let D(⋅,⋅)D(·,·) be a distance metric measuring deviation from the claimed logits Lℳ⋆(x→)L_M ( x). By definition, ℳopt=argminℳ∈ℳTD(Lℳ(x→),Lℳ⋆(x→)).M_opt= _M _TD\! (L_M( x),\,L_M ( x) ). Suppose the adversary generates a committed logit vector LfakeL_fake such that D(Lfake,Lℳ⋆(x→))<D(Lℳopt(x→),Lℳ⋆(x→)).D\! (L_fake,\,L_M ( x) )<D\! (L_M_opt( x),\,L_M ( x) ). Producing LfakeL_fake requires some computational process within budget T, implying the existence of a model in ℳTM_T that approximates ℳ⋆M better than ℳoptM_opt. This contradicts the optimality of ℳoptM_opt. ∎ Appendix G Other Global LDDs We present visualization of global logit TV distance in Figure 4, logit KL divergence in Figure 5, token sampling top-K distance in Figure 6 and expert top-K distance in Figure 7. (a) LLaMA3-70B (b) Qwen3-32B (c) Qwen3-30B-A3B (d) DeepSeek-V2-Lite Figure 4: Global logit TV-distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior. (a) LLaMA3-70B (b) Qwen3-32B (c) Qwen3-30B-A3B (d) DeepSeek-V2-Lite Figure 5: Global logit KL divergence distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior. (a) LLaMA3-70B (b) Qwen3-32B (c) Qwen3-30B-A3B (d) DeepSeek-V2-Lite Figure 6: Global logit Top-K distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior. (a) Qwen3-30B-A3B (b) DeepSeek-V2-Lite Figure 7: Global logit Top-K distance distribution. Probabilities are displayed on a logarithmic y-axis to better capture the tail behavior.