Paper deep dive

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs

Ziyang Liu

Year: 2026Venue: arXiv preprintArea: cs.CRType: PreprintEmbeddings: 106

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 6/21/2026, 10:54:02 AM

Summary

The paper proposes a 'commit-open' protocol to detect 'silent-substitution' in hosted LLM services, where providers serve cheaper models instead of advertised ones. The protocol uses Sparse Autoencoders (SAEs) to create a Merkle tree commitment of per-position feature traces at a specific layer. A verifier can then request random openings to score the output against a public 'named-circuit' probe library. The authors demonstrate that this method effectively detects various attacker strategies, including same-family lifts, cross-family substitutes, and adaptive LoRA-based attacks, with low computational overhead (<=2.1% at batch 32).

Entities (9)

Gemma-2-2B · llm-backbone · 100%Gemma-2-9B · llm-backbone · 100%LoRA · fine-tuning-technique · 100%Merkle Tree · data-structure · 100%Named-Circuit Probe Library · verification-tool · 100%Qwen3-1.7B · llm-backbone · 100%Sparse Autoencoder (SAE) · technique · 100%SVIP · prior-workscheme · 100%Ziyang Liu · author · 100%

Relation Signals (5)

SVIP → isvulnerableto → parallel-serve side-channel

confidence 100% · Probe-after-return schemes such as SVIP leave a parallel-serve side-channel

Merkle Tree → providesbindingfor → SAE feature-trace

confidence 100% · the provider commits via a Merkle tree to a per-position sparse-autoencoder (SAE) feature-trace sketch

Sparse Autoencoder (SAE) → usedtogenerate → feature-trace sketch

confidence 100% · the provider commits via a Merkl tree to a per-position sparse-autoencoder (SAE) feature-trace sketch

Named-Circuit Probe Library → detects → LoRA

confidence 90% · A single 96-probe library detects every evaluated lifted, cross-family, and rank-<=128 adaptive attacker

Qwen3-1.7B → isevaluatedwith → Sparse Autoencoder (SAE)

confidence 90% · We instantiate the protocol on three backbones -- Qwen3-1.7B, Gemma-2-2B, and a 4.5x scale-up to Gemma-2-9B with a 131k-feature SAE.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Hosted-LLM providers have a silent-substitution incentive: advertise a stronger model while serving cheaper replies. Probe-after-return schemes such as SVIP leave a parallel-serve side-channel, since a dishonest provider can route the verifier's probe to the advertised model while serving ordinary users from a substitute. We propose a commit-open protocol that closes this gap. Before any opening request, the provider commits via a Merkle tree to a per-position sparse-autoencoder (SAE) feature-trace sketch of its served output at a published probe layer. A verifier opens random positions, scores them against a public named-circuit probe library calibrated with cross-backend noise, and decides with a fixed-threshold joint-consistency z-score rule. We instantiate the protocol on three backbones -- Qwen3-1.7B, Gemma-2-2B, and a 4.5x scale-up to Gemma-2-9B with a 131k-feature SAE. Of 17 attackers spanning same-family lifts, cross-family substitutes, and rank-<=128 adaptive LoRA, all are rejected at a shared, scale-stable threshold; the same attackers all evade a matched SVIP-style parallel-serve baseline. A white-box end-to-end attack that backpropagates through the frozen SAE encoder does not close the margin, and a feature-forgery attacker that never runs M_hon is bounded in closed form by an intrinsic-dimension argument. Commitment adds <=2.1% to forward-only wall-clock at batch 32.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

105,792 characters extracted from source content.

Expand or collapse full text

Committed SAE-Feature Traces for Audited-Session Substitution Detection in Hosted LLMs Ziyang Liu Independent Researcher ziyang.liu.r@outlook.com Abstract Hosted-LLM providers have a silent-substitution incentive: advertise a stronger model while serving cheaper replies. Probe-after-return schemes such as SVIP leave a parallel-serve side-channel, since a dishonest provider can route the verifier’s probe to the advertised model while serving ordinary users from a substitute. We propose a commit-open protocol that closes this gap. Before any opening request, the provider commits via a Merkle tree to a per-position sparse-autoencoder (SAE) feature-trace sketch of its served output at a published probe layer. A verifier opens random positions, scores them against a public named-circuit probe library calibrated with cross-backend noise, and decides with a fixed-threshold joint-consistency z-score rule. We instantiate the protocol on three backbones—Qwen3-1.7B, Gemma-2-2B, and a 4.5×4.5× scale-up to Gemma-2-9B with a 131131k-feature SAE. Of 1717 attackers spanning same-family lifts, cross-family substitutes, and rank-≤128≤\!128 adaptive LoRA, all are rejected at a shared, scale-stable threshold; the same attackers all evade a matched SVIP-style parallel-serve baseline. A white-box end-to-end attack that backpropagates through the frozen SAE encoder does not close the margin, and a feature-forgery attacker that never runs M is bounded in closed form by an intrinsic-dimension argument. Commitment adds ≤2.1%≤\!2.1\% to forward-only wall-clock at batch 3232. 1 Introduction A growing share of large-language-model (LLM) inference is delivered through opaque hosted APIs, and the provider’s quality-for-cost substitution of served models is both profitable and difficult to detect: a user who pays for access to a 70B-parameter model cannot, from responses alone, distinguish it from a 7B substitute that happens to answer the specific queries the user sent. Three lines of work partially address this gap: determinism / verify-rollback re-runs inference bitwise but does not establish identity; proof-of-inference SNARKs [1] give soundness at 10310^3–104×10^4× overhead; response-side fingerprints (LLMmap [2], instructional fingerprinting [3]) lack a binding primitive. The closest prior binding primitive is SVIP [4], a secret linear projection of a hidden state returned on demand; because the projection is requested after service, SVIP admits a trivial parallel-serve side-channel — the provider serves users from M′M while answering each verifier query from the advertised M. Our contribution. We study a commit-open protocol in which the provider commits, via a Merkle tree, to the per-position SAE-feature trace sketch of its served sequence at a published probe layer, before any verifier opening request. Verification checks random openings against a public named-circuit probe library, calibrated with cross-backend noise, and decides with a fixed-threshold joint-consistency z-score. Within the scoped threat model (Appendix A), a dishonest provider faces three options: (a) commit from the substitute and be statistically detected, (b) run M to produce the commitment, which on our serving stack costs within ∼3% \!3\% of honest wall-clock, or (c) adapt the substitute within a public adaptation budget. We characterize (c) empirically across a weak-to-strong attacker ladder and pair it with a white-box diagnostic that backpropagates through the frozen SAE encoder to the scoring rule. Paper map. Section˜3 fixes the commit-open protocol. Section˜4 states the strategy dichotomy (detected, or pays honest compute). Section˜5 reports the main 1717-attacker frontier across three backbones, the SVIP-gap comparison, the white-box diagnostic, and the feature-forgery bound. Section˜6 scopes the claims. Figure 1: Commit-open verification. Left: four provider strategies — (A) honest, (B) substitute + substitute’s trace (detected), (C) substitute + M-trace (pays honest compute), (D) adaptive φ (bounded by SAE capacity). Right: the provider commits a Merkle root of per-position SAE traces before the verifier’s probe, closing SVIP’s after-the-fact side-channel (dashed). Evidence preview. We evaluate three attacker tiers. Weak non-adaptive attackers — three same-family lifts and four cross-family substitutes combined with a public-corpus linear alignment φ [5] — all clear the deployment threshold by ≥12×≥\!12× on Qwen3-1.7B. Strong black-box adaptive attackers built on top of a library-aware joint-φ plus LoRA up to r=128r=128 (1500–3000 steps) are detected with at least a 2.78×2.78× margin on Qwen3-1.7B, and this margin grows with SAE width, reaching 40−260×40-260× on Gemma-2-9B + Gemma-Scope-9B (L2020, width-131131k). A white-box end-to-end attacker that backpropagates through the frozen SAE encoder to the scoring rule does not close the black-box margin. The detection threshold is scale-stable (τ9B=τGemma=1.09 _9B= _Gemma=1.09, agreement <10−4<\!10^-4), and commitment overhead is ≤2.1%≤\!2.1\% at batch 3232 with a 224224-byte per-request payload. Contributions. 1. Protocol. The first hosted-LLM verification scheme that binds the provider to a per-position SAE-feature trace via a pre-query Merkle commitment (Section˜3). 2. Strategy dichotomy. Every non-honest provider strategy we evaluate is either statistically detected at the deployment threshold or forced to run M at near-honest wall-clock (Section˜4). 3. Empirical scale. A single 9696-probe library detects every evaluated lifted, cross-family, and rank-≤128≤\!128 adaptive attacker across Qwen3-1.7B, Gemma-2-2B, and Gemma-2-9B, with a scale-stable threshold and detection margins that grow with SAE width (Section˜5). Together these close an 11/1111/11 gap vs. a matched SVIP baseline (Section˜5.6). 4. Systems cost. Batched commit overhead drops from 18.5%18.5\% at B=1B=1 to ≤2.1%≤\!2.1\% at B=32B=32, with a 224224-byte per-request payload (Section˜5, Appendix J). 2 Related Work Hosted-LLM identity and integrity. SVIP [4] returns a secret linear projection of a target-layer hidden state on demand, but the projection is evaluated after service and therefore does not bind the served computation to the identity claim — Section˜5.6 quantifies this gap under matched threshold and attacker set. TOPLOC [6] uses an LSH of activations for integrity (the advertised model produced the served tokens) and is composable with our scheme rather than competing. LLMmap [2] and instructional fingerprinting [3] recover identity post-hoc from response statistics or planted triggers with no binding primitive. Gross et al. [7] use mechanistic interpretability for offline behaviour proofs; we borrow the move of treating MI artifacts as cryptographic payloads but apply it at serving time and target identity. Mechanistic interpretability and commit-and-prove. Our probe library is assembled from named-circuit work: indirect-object identification [8], induction [9], and SAE-backed syntactic/factual circuits [10, 11, 12, 13]. We are the first to use the feature trace as the commitment payload of a verification scheme. Commit-and-prove SNARKs [1] give cryptographic soundness at 10310^3–104×10^4× runtime; proof-of-useful-work [14] relaxes soundness for compute-as-proof. We occupy an intermediate regime: the verifier does not prove honest execution, but any non-honest strategy is either detected or pays honest-service compute (Section˜4). Hardware-rooted primitives (Intel TDX [15], NVIDIA H100 confidential computing [16], IMA [17], append-only logs [18, 19]) are orthogonal deployments that strengthen the Merkle commitment against privileged adversaries rather than substitutes for the SAE-trace primitive. Table˜1 summarises the differentiation. Table 1: Comparison of hosted-LLM verification schemes. “Binding” means the provider is committed before the verifier’s query, closing the parallel-serve side-channel. “SC-safe” denotes resistance to a provider that serves M′M to users while answering the probe from M. ∗“stat. (scoped)” is an empirical statistical detection result on the attackers we tested within Table˜6, on three backbone–SAE pairs spanning 1.71.7–99B parameters (Qwen3-1.7B + transcoder L1414; Gemma-2-2B + Gemma-Scope L1212; Gemma-2-9B + Gemma-Scope-9B L2020 width-131131k); not a cryptographic or backbone-agnostic claim. The parallel-commit cost gap is a single-deployment ∼3% \!3\% wall-clock observation. Scheme Primitive Binding SC-safe Soundness Overhead LLMmap [2] response hash – – stat. post-hoc Instruct. fp. [3] planted trigger – – stat. post-hoc SVIP [4] hidden-state probe after query no stat. ∼0% 0\% TOPLOC [6] LSH (integrity) at commit yes (int.) stat. (int.) ∼8% 8\% CP-SNARK [1] proof of exec. at commit yes cryptographic 103–4×10^3--4× Ours SAE-feature trace at commit yes (by constr.) stat. (scoped)∗ ≤2.1%≤ 2.1\% Concurrent work on auditable inference / fine-tuning. Two recent systems bracket our design. AFTUNE [20] records auditable execution traces during fine-tuning and inference and spot-checks them inside a TEE; overhead is 1414–83%83\%, and the trust primitive is recomputation rather than a statistical-structural binding. VFT [21] gives zero-knowledge proofs that a released model derives from a declared training process (16.816.8 s/step prover, 44–66 MB proofs); its scope is training provenance, not per-request serving. We sit between them: an inference-time, sub-millisecond commitment that does not require TEE hardware, with the binding carried by the per-position SAE-trace sketch and the probe library rather than by re-execution or ZK circuits. AFTUNE is March 2026 (concurrent with this submission); both are reported here to position scope, not as baselines (Section˜6). 3 Protocol Table˜2 fixes the concrete deployment values. The rest of this section gives the definitions and the commit-open protocol. Table 2: Protocol parameters. Commit, calibration, and verification parameters. Values given as “Qwen3-1.7B / Gemma-2-2B” where they differ. Merkle leaf =(x,y,mid,t,t)=(x,y,mid,t, tok32_t); payload is the root plus the top-k indices and bf16 values per opened position. A Gemma-2-9B + Gemma-Scope-9B scale-up (Section˜5.5) reuses the same ||=96|P|=96 library at L=20L=20, width-131131k, with τ9B=1.09 _9B=1.09 (agreement with τGemma _Gemma within 10−410^-4). Parameter Symbol Value (Qwen3-1.7B / Gemma-2-2B) Probe layer L 1414 / 1212 (published, fixed) SAE/transcoder – mwhanna-qwen3-1.7b-transcoders / Gemma-Scope canonical residual SAE Top-k per position k 3232 Quantisation – bf16 Probe library size |||P| 9696 (88 circuit classes) Opening count per session kopenk_open random subset, nominal k∈4,…k∈\4,…\; within-session ρ=0.88ρ=0.88 Honest pool nhonn_hon 112112 (MATH n=64n=64 + multi-kernel n=48n=48) / 6464; seeds 100..107,300..302\100..107,300..302\ Qwen3 σi _i calibration grid – 22 dtypes × 3×\,3 kernels × 4×\,4 pos., seeds 0,1,2,3\0,1,2,3\ / 22 dtypes × 2×\,2 kernels × 4×\,4 pos. × 2×\,2 seeds Conservative threshold τ τpool=1.509 _pool=1.509 / τGemma=1.09 _Gemma=1.09 (pooled empirical max) Nominal FPR target α 0.010.01; CP upper 0-viol 0.02640.0264 / 0.04570.0457 Joint-z aggregator z(t)z(t) mean of per-probe zi(t)z_i(t) over random subset I(t)I(t), |I(t)|=N|I(t)|=N; Equation˜2 Merkle hash – SHA-256256, 3232-byte root; 224224 bytes total per request at |I(t)|=N|I(t)|=N Figure 2: Commit-open mechanics. (1) The provider forwards (x,y)(x,y) through M and encodes the layer-L residual stream hth_t with a public SAE encoder E, keeping the top-3232 active features per position. (2) Per-position leaves leaft=H(meta,t,t)leaf_t=H(meta,\,t,\, tok32_t) are assembled into a Merkle tree with root R; the provider publishes (x,y,R)(x,y,R). (3) The verifier samples random positions V and requests Merkle openings, verifying each path against R to recover t tok32_t. (4) A public probe library P ( 9696 named circuits ) scores each opened position with zi(t)z_i(t) and aggregates to z(t)=meanizi(t)z(t)=mean_i\,z_i(t); the session is rejected if z(t)>τz(t)>τ. Crucially, R is published before the verifier’s opening request, closing the SVIP parallel-serve side-channel. Setting. The provider advertises a model M and serves input x by returning tokens y1:Ty_1:T together with a verification bundle. At a published probe layer L, M’s residual-stream input to the MLP admits an SAE or transcoder [10, 11, 13] with encoder E. For each position t, let ft=E(ht)∈ℝdSAEf_t=E(h_t) ^d_SAE be the SAE feature vector. The committed trace sketch is the top-k support together with its values, t=(j,ft,j):j∈top-k⁡(ft) tok32_t=\(j,f_t,j):j -k(f_t)\ with k=32k=32, quantised to bf16. Merkle commitment. The provider builds a Merkle tree [22] over leaves leaft=H("LEAF"‖meta‖t∥t),leaf_t=H (\, "LEAF"\,\|\,meta\,\|\,t\,\|\, tok32_t\, ), (1) with meta=(model_id,SAE_release,L,H(x),H(y),nonce,provider_pubkey)meta=(model\_id,SAE\_release,L,H(x),H(y),nonce,provider\_pubkey), and publishes the root R alongside (x,y)(x,y). Binding metameta to H(x),H(y)H(x),H(y) is a pre-commitment of the trace to the served session: the root fixes which trace will be scored before any opening request, so the provider cannot re-route the opening (the SVIP side-channel). The commitment does not cryptographically prove the opened t tok32_t was computed on (x,y)(x,y); the residual gap is closed statistically by the economic argument of Section˜4 or cryptographically by pairing with TEE attestation [15, 16, 17]. Probe library and calibration. The library =(namei,xi,μi,σi)i=196P=\(name_i,x_i, _i, _i)\_i=1^96 covers 88 circuit classes (IOI, induction, syntactic, factual, coreference, arithmetic, commonsense, language), populated from the named-circuit literature with a robust-mix construction (retaining top-3232 features also active under data-augmentation shuffle, 39.6%39.6\% acceptance rate). Each probe is calibrated under cross-backend noise (dtypes × kernels × positions); honest p50p_50 / p95p_95 L2 drift is 2.1%2.1\% / 6.5%6.5\% (CV median 1.9%1.9\%), and per-probe σi _i absorbs it. For each probe we record a per-feature reference μi∈ℝdSAE _i ^d_SAE (honest feature vector on the probe prompt, averaged over calibration draws) and per-feature noise scale σi,j>0 _i,j>0; support Si=top-k⁡(μi)S_i=top -k( _i) (size 3232). Verifier protocol. On receiving (x,y,R)(x,y,R), the verifier (1) samples positions V⊆1,…,TV \1,…,T\ with N≤96N≤ 96 probes per position and requests Merkle openings for leaft:t∈V\leaf_t:t∈ V\; (2) verifies each opening against R and recovers t tok32_t; (3) for each probe i∈I(t)i∈ I(t), reads f^t,j f_t,j for j∈Sij∈ S_i (or 0 if j∉top-k⁡(ft)j -k(f_t)) and computes zi(t)=1|Si|∑j∈Si|f^t,j−μi,j|/σi,jz_i(t)= 1|S_i| _j∈ S_i| f_t,j- _i,j|/ _i,j, with joint aggregate z(t)=1N∑i∈I(t)zi(t);z(t)= 1N _i∈ I(t)z_i(t); (2) (4) decides with z(t)>τz(t)>τ, where τ is the empirical max over a pooled honest draw (τpool=1.509 _pool=1.509 deployment value, CP 95%95\% upper 0.02640.0264; Section˜5.1). Per-session FPR under k openings is bounded by kαkα under independence; within-session dependence (ρ=0.88ρ=0.88) empirically deflates it (Appendix G). A sequential SPRT [23] extension at α=β=0.01α=β=0.01 with Holm step-down αn=α/(N−n+1) _n=α/(N-n+1) [24] is compatible but reported only diagnostically (Appendix H). Budget. Provider-side overhead is one SAE encode plus Merkle root construction over T 3232-entry leaves. On our deployment, amortised cost at batch 3232 is ≤2.1%≤\!2.1\% (Appendix J). 4 Strategy-by-Strategy Analysis We analyse the four provider strategies below by combining a binding argument for the commitment with empirical results from Section˜5. Conclusions are scoped to Table˜6, the attackers we tested, and our single-stack serving deployment; we avoid game-theoretic dominance language — strategy (C) is only 0.97×0.97× honest wall-clock on one stack, a deployment-specific cost gap, not a dominance result. (A) honest: serve y from M, commit its SAE trace. (B) substitute-serve: serve y from M′M , commit M′M ’s trace. (C) parallel-commit: serve y from M′M but commit from a parallel M-pass, so openings are consistent with M. (D) adaptive distillation: fine-tune M′M under a bounded budget (two evaluated points: public-corpus φ + r=16r=16 LoRA to 30003000 steps; library-aware joint-φ + r=64r=64 LoRA at 15001500 steps) to match M’s trace, then execute (B). Proposition 4.1 (Strategy-by-strategy detection/cost dichotomy). Under Table˜6, with ||=96|P|=96 and per-feature noise σi,j _i,j calibrated on honest cross-backend draws, the strategies we test exhibit the following behaviour relative to the conservative deployment threshold τ of Equation˜2: • (B) Detected at TPR =1.00=1.00 at τpool=1.509 _pool=1.509 (nhon=112n_hon=112, CP 95%95\% upper 0.02640.0264) on 33 lift and 44 cross-family attackers (Sections˜5.2 and 5.3). • (C) leaftleaf_t binds ftf_t to the committed M, so opening requires an M forward pass on the committed positions; on our stack parallel-commit is 0.97×0.97× honest wall-clock. This ∼3% \!3\% gap is single-stack and ignores parallel hardware / KV-cache sharing — not a dominance claim. • (D) At two evaluated black-box budgets — (i) public-corpus φ with ‖φ‖F=0.59\| \|_F=0.59 plus r=16r=16 LoRA to 30003000 steps, where held-out per-category joint-z≥12×τpoolz≥ 12× _pool; and (i) library-aware joint-φ + r=64r=64 LoRA, whose strongest Qwen3 point (StageA, λutil=0 _util=0) reaches z=4.19z=4.19 (2.78×τpool2.78× _pool) at 1.55×1.55× Pile perplexity, and whose matched Gemma pure-probe point reaches z=105.9z=105.9 (97.2×τGemma97.2× _Gemma) at 436×436× perplexity (Table˜4) — all 88 evaluated adaptive operating points are detected. A supplementary white-box E-E diagnostic (r=64r=64, 30003000 steps, gradient through frozen SAE ReLU) does not close the margin: 33 restart seeds of the strongest Qwen3 cell give z∈4.99,6.89,7.80z∈\4.99,6.89,7.80\, the best-for-attacker restart 4.994.99 still exceeding the black-box MSE optimum 4.194.19 (Section˜5.4, Appendix M). The binding argument for (B) and (C) is a direct consequence of the Merkle pre-commitment (Equation˜1), which fixes the committed trace before the verifier issues any opening request; the detection claim uses the per-probe calibration of σi _i on honest draws and the fixed-threshold rule z(t)>τz(t)>τ. For (D), the heuristic explanation is that under a public-corpus minimum-norm fit ‖φ‖F=0.59\| \|_F=0.59 projects LoRA-induced ‖Δh‖∞∼1\| h\|_∞\! \!1 to ‖Δproj‖∞≤10−2\| \|_∞\!≤\!10^-2, below the top-3232 mask-flip threshold (Appendix U). The full proof sketch (including the tail-robustness argument for τ and session-FPR under dependence) is in Appendix B. Not claimed: detection of full-model retraining, resistance to white-box probe-aware gradient attackers beyond the evaluated E-E point, economic dominance, or generalisation beyond the two backbone–SAE pairs evaluated (Section˜6). Feature-forgery infeasibility. A complementary attacker does not run M at all and fabricates t tok32_t directly against the published library. Because joint-z is a mean over probe-slot penalties separable in the chosen feature set, the attacker’s optimum is closed form: for each feature f compute the gain G∗(f)=maxv∑(i,s):Si,s=f[|μi,s|−|v−μi,s|]/σi,sG^*(f)= _v _(i,s):S_i,s=f[| _i,s|-|v- _i,s|]/ _i,s, which is attained at the weighted-median v∗v^*, and pick the top-3232 features by G∗G^*. This bounds the gain by the coverage budget, giving Proposition 4.2. Proposition 4.2 (Intrinsic-dimension lower bound). Let U=|⋃iSi|U= _iS_i and c=mediani,s|μi,s|/σi,sc=median_i,s\,| _i,s|/ _i,s. For any attacker committing a per-position top-k without running M, the uniform-coverage lower bound is z(t)≥(1−k/U)c,z(t)\;≥\; (1-k/U )\,c, i.e. at most a fraction k/Uk/U of probe-slots can be covered. On the deployed library (U=1472U=1472, k=32k=32, c=22.78c=22.78) the data-dependent greedy refinement (Appendix˜V) tightens this to z(t)≥37.11=24.6τpoolz(t)≥37.11=24.6 _pool, and the matching exact closed-form discrete + bf16 optimum is z∗=44.24z^*=44.24. See Appendix˜V for the full derivation and Section˜5.7 for the empirical ladder. Proposition 4.2 closes a gap left by Proposition 4.1: the parallel-commit strategy (C) had only an economic binding (0.97×0.97× honest wall-clock, single stack), and (D) had only evaluated LoRA / white-box points. Forgery — the simplest strategy requiring no M access — is now bounded by a structural property of the probe library, not by an attacker budget. 5 Experiments 5.1 Setup Target stack. Qwen3-1.7B [25] at layer L=14L=14, public mwhanna-qwen3-1.7b-transcoders-lowl0 SAE; 9696-probe library assembled from 88 circuit classes via a robust-mix construction (39.6%39.6\% acceptance). Gemma-2-2B [26] + Gemma-Scope canonical residual SAE at L1212 is a second-backbone replication (Appendix E), and Gemma-2-9B + Gemma-Scope-9B canonical residual SAE at L2020 (width-131131k, avg-L0=114L_0=114) is a 4.5×4.5× scale-up (Section˜5.5, Appendix F) that reuses the same 9696-probe prompt set (top-3232 feature IDs and μ/σμ/σ recomputed on 9B activations). Attackers. Three threat levels: (i) lifted projection on M simulating a same-family substitute (M′=M =Qwen3-0.6B via ZERO_PAD, RAND_PROJ, LEARNED_LIN); (i) cross-family substitutes Qwen2.5-1.5B [27], Phi-3.5-mini [28], OLMo-2-7B [29], Qwen3-0.6B with a public-corpus linear alignment φ fit on 20002000 Pile-uncopyrighted pairs [30] (80/2080/20 split); (i) a rank-constrained LoRA attacker on top of φ (r=16r=16, α=32α=32, seven projection modules, 500+2500500+2500 steps) and a library-aware higher-budget variant (r=64r=64, 15001500 steps, joint φ ). Calibration. Headline statistic is the joint-z of Equation˜2; legacy per-probe Mahalanobis [31] numbers are labelled. σi _i is calibrated on 22 dtypes × 33 kernels × 44 positions (companion seeds 0,1,2,3\0,1,2,3\). The original n=64n=64 MATH honest pool gives τreal=1.13 _real=1.13; a follow-up multi-kernel spot check on Modal L4 (Appendix T) adds 4848 fresh draws at companion seeds 300−302300-302. Pooling all n=112n=112 honest draws gives the conservative deployment threshold τpool=1.509 _pool=1.509 (0/1120/112, CP one-sided upper 0.02640.0264), adopted throughout. Qwen3 compute ≈0.8≈\!0.8 L4-hours + 2.5+\,2.5 3090-hours; commit overhead ≤2.1%≤\!2.1\% at batch 3232 (Appendix J). Table˜3 reports the strongest tested attacker per tier on each backbone; the full 2626-row per-attacker breakdown (lift, cross-family, rank-≤128≤\!128 LoRA, white-box E-E) is Table˜8 (Appendix D). Subsections 5.2–5.6 analyse each tier in turn. Table 3: Headline detection frontier. The strongest (lowest joint-z) attacker we could produce in each tier, on each of the three backbones. All are rejected at the backbone-specific conservative threshold τ; the same attackers all evade a matched SVIP-style parallel-serve baseline (Section˜5.6). Full per-attacker breakdown: Table˜8 (Appendix D). Backbone / SAE (τ) Strongest tested attacker joint-z ×τ×τ SVIP Qwen3-1.7B / transcoder L1414 (1.5091.509) adaptive LoRA r=64r=64 4.194.19 2.782.78 miss adaptive LoRA r=128r=128 6.426.42 4.254.25 – white-box E-E (r=64r=64) 4.994.99 3.313.31 – Gemma-2-2B / Gemma-Scope L1212 (1.091.09) adaptive LoRA r=64r=64 105.89105.89 97.297.2 miss white-box E-E (r=64r=64) 133.86133.86 122.8122.8 – Gemma-2-9B / Gemma-Scope-9B L2020, 131131k (1.091.09) cross-family Qwen2.5-7B 283.30283.30 260.0260.0 – adaptive LoRA r=64r=64 43.7043.70 40.140.1 – 5.2 Same-family separability Under the lifted-projection threat at τpool=1.509 _pool=1.509, all three attackers reach TPR =1=1 (Clopper–Pearson 95%95\% CI [32] [0.926,1.0][0.926,1.0] on natt=48n_att=48 replays) with no fresh honest sample above τpool _pool (Figure˜3). On the legacy per-probe Mahalanobis scale the three attackers reach 5656–148×148× the honest hold-out max of 1.511.51; the overlap-within-2σ2σ statistic on LEARNED_LIN drops from 0.8760.876 (honest) to 0.190.19. Figure 3: E2 same-family separability (Qwen3-1.7B vs. lifted Qwen3-0.6B), on the legacy per-probe Mahalanobis diagnostic scale (threshold τQwenmaha=1.51τ^maha_Qwen=1.51, dashed in panel (a)). (a) Per-sample Mahalanobis under 1616 honest and 3×483× 48 attacker samples; all three attackers sit orders of magnitude above the honest envelope. (b) ROC derived from the same scores, zoomed to nominal FPR ≤12%≤ 12\%. The headline joint-z rule at τpool=1.509 _pool=1.509 (nhon=112n_hon=112) reaches TPR =1=1 on all three attackers with empirical FPR 0/1120/112 (CP 95%95\% upper 0.02640.0264). 5.3 Cross-family detection All four cross-family substitutes are rejected at τpool=1.509 _pool=1.509 (row 44–77 of Table˜8): joint-z∈69.2,52.4,81.3,62.8z∈\69.2,52.4,81.3,62.8\ for Qwen2.5-1.5B, Phi-3.5-mini, OLMo-2-7B, Qwen3-0.6B — all at TPR=1TPR=1 with CP 95%95\% CI [0.926,1.00][0.926,1.00] on natt=48n_att=48, ratio 34.734.7–53.9×τpool53.9× _pool. Per-probe Mahalanobis is 5252–85×τQwenmaha85×τ^maha_Qwen. The φ -transfer R2R^2 diagnostic (held-out R2∈0.39,−0.90,−5.97,0.15R^2∈\0.39,-0.90,-5.97,0.15\) is reported only as partial explanatory analysis (Appendix K); detection is established independently by the joint-z scores. A scale-matched cross-family point on Gemma-2-9B (Qwen2.5-7B through a public-corpus φ ) lands at joint-z=283.3z=283.3 (260×τ9B260× _9B; row 2424 of Table˜8, analysed in Section˜5.5). 5.4 Adaptive LoRA attackers We probe two budget points on M′=M =Qwen3-0.6B, targeting the seven projection modules q,k,v,o,gate,up,down\q,k,v,o,gate,up,down\-proj, preceded by a learned linear lift φ:ℝ1024→ℝ2048 ^1024\!→\!R^2048. Constrained (r=16r=16, public-corpus φ ). 30003000 steps, φ frozen at the public-corpus fit (‖φ‖F=0.59\| \|_F=0.59, Rho2=0.15R^2_ho=0.15). ‖B‖F\|B\|_F grows 0→5.240→5.24; per-category held-out joint-z stays ≥12×τpool≥ 12× _pool. The bottleneck is structural: ‖Δh‖∞∼1\| h\|_∞ 1 projects through the small-norm φ into ‖Δproj‖∞≤10−2\| \|_∞≤10^-2, below the top-3232 mask-flip threshold (Appendix U). Library-aware higher-budget (r=64r=64, joint φ ). The attacker has full read access to the 9696-probe library and trains φ jointly with the LoRA under ℒ=λprobeMSE(φ(hM′),hM)+λutilNLLM′(Pile)L= _probe\,MSE( (h_M ),h_M)+ _util\,NLL_M (Pile) (λprobe=1 _probe=1, 15001500 steps, AdamW lr 3×10−43×10^-4). All four Qwen3 operating points (Table˜4, Figure˜4) are detected. StageA (λutil=0 _util=0) reaches joint-z=4.19z=4.19 (2.78×τpool2.78× _pool) at 1.55×1.55× ppx inflation; per-category minimum is z=2.60z=2.60 (induction). Utility regularisation (λutil∈0.1,1.0 _util∈\0.1,1.0\) raises z to 6.276.27–7.487.48 at 2.42.4–3.0×3.0× ppx. Compute: 14511451 s on Modal L4 (≈$0.32≈ 0.32). A matched Gemma pure-probe point (λutil=0 _util=0) reaches joint-z=105.9z=105.9 (97.2×τGemma97.2× _Gemma) while Pile ppx collapses to 80328032 (436×436× clean) — detectable with catastrophic utility loss; we do not claim a matched utility-preserving Gemma frontier. A further scale-up point on Gemma-2-9B (same r=64r=64 StageA recipe, Gemma-2-2B-it substitute) lands at joint-z=43.7z=43.7 with 30×30× ppx inflation (row 2626, Section˜5.5) — detection margin is ∼2.4× \!2.4× tighter than at 2B in z but carried by an 8×8× wider SAE. A higher-rank Qwen3 frontier point (r=128r=128, MSE, λutil=0 _util=0) still clears detection at joint-z=6.42z=6.42 (4.25×τpool4.25× _pool) at 2.4×2.4× ppx (row 19a19a), confirming that doubling attacker rank does not reopen the scoring-rule margin within the evaluated training budget. White-box E-E diagnostic. We additionally give the attacker gradient access through the frozen SAE encoder to the scoring rule itself, adding a joint-z loss ℒjz=αjzmeanp,i|(zp,i,k−μp,i,k)/σp,i,k|L_jz= _jz\,mean_p,i |(z_p,i,k- _p,i,k)/ _p,i,k | over the top-3232 mask (full 66-point Qwen3 grid + 22 Gemma points in Appendix M). The strongest Qwen3 cell (αjz=0,λutil=0)( _jz=0, _util=0), repeated across three restart seeds, gives joint-z∈6.89,7.80,4.99z∈\6.89,7.80,4.99\ — the best-for-attacker restart 4.994.99 still exceeds the black-box MSE optimum 4.194.19; two of the remaining five single-run cells diverge (ppx>107ppx>10^7). The mechanism is SAE ReLU sparsity: inactive top-k features contribute zero gradient, so ∂ℒjz/∂LoRA _jz/ is a noisy sparse subset that fights the dense MSE. Higher-rank, longer-trained, and secret-probe-aware white-box attacks remain out of scope (Section˜6). Table 4: E4-v2 library-aware adaptive LoRA (M′=M =Qwen3-0.6B / Gemma-2-2B-it, r=64r=64, joint φ , 15001500 steps, full 9696-probe access). All operating points detected at the backbone-specific conservative threshold. Operating point λutil _util joint-z ×τpool× _pool ×τreal× _real Pile ppx Qwen3-1.7B; τpool=1.509 _pool=1.509 baseline-φ -only – 57.3857.38 38.038.0 50.850.8 21.2021.20 StageA pure-probe 0 4.194.19 2.782.78 3.713.71 32.8432.84 StageC low-util 0.10.1 6.276.27 4.164.16 5.555.55 63.1663.16 StageB balanced 1.01.0 7.487.48 4.964.96 6.626.62 50.9850.98 Gemma-2-2B; τGemma=1.09 _Gemma=1.09 baseline-φ -only – 205.56205.56 188.6×τGemma188.6× _Gemma 18.4418.44 StageA pure-probe 0 105.89105.89 97.2×τGemma97.2× _Gemma 8031.658031.65 Figure 4: Rank-6464 LoRA attacker frontier on the Qwen3-1.7B target. All four operating points sit above τpool=1.509 _pool=1.509; StageA is the strongest we evaluate (2.78×τpool2.78× _pool at 1.55×1.55× Pile ppx). Adding utility regularisation moves the attacker up and to the right. This slice of the frontier is an operating-range, not a ceiling (Section˜6). 5.5 Scale-up: Gemma-2-9B + Gemma-Scope-9B To test whether the protocol extends beyond the sub-33B regime — the “7070B advertised / 77B served” motivation requires this scale point — we replicate the three-stage pipeline (honest-pool → cross-family detection → adaptive LoRA) on Gemma-2-9B at L2020 using the canonical 131131k-width Gemma-Scope-9B SAE (avg-L0=114L_0=114). The 9696 probe prompts carry over verbatim; top-3232 feature IDs and μ/σμ/σ are recomputed on 9B activations (Appendix F). Honest threshold is scale-stable. A 6464-draw honest pool (22 dtypes × 22 kernels × 44 positions × 44 seed groups) yields τ9B=1.090 _9B=1.090, agreeing with τGemma=1.090 _Gemma=1.090 to within 10−410^-4 and within an order of magnitude of τpool=1.509 _pool=1.509 on Qwen3. Across three backbones spanning 1.71.7–99B the pooled honest joint-z stays in [1.09,1.51][1.09,1.51]; there is no evidence of threshold drift as model and SAE scale up. Detection margin grows with SAE width. A scale-matched cross-family substitute — Qwen2.5-7B with a public-corpus φ fit on 20002000 Pile samples (80/2080/20 hold-out) — lands at joint-z=283.3z=283.3 (260×τ9B260× _9B; row 2424 of Table˜8). The same-family baseline (φ -only, no LoRA) with Gemma-2-2B-it through a joint φ to the 9B manifold reaches 280.2280.2 (257×257×). An adaptive LoRA attacker — Gemma-2-2B-it, r=64r=64, 15001500 steps, joint φ + LoRA trained under StageA (λutil=0 _util=0) — is still detected at joint-z=43.7z=43.7 (40×τ9B40× _9B) and pays a 30×30× Pile-perplexity price (ppx 560.2560.2 vs. 18.518.5 substitute-clean). Compared with the Gemma-2-2B counterpart (97×τ97×τ for the same attacker), the 9B margin to threshold is ∼2.4× \!2.4× tighter in absolute z but is carried by a 8×8× wider SAE; detection is strictly stronger per unit of honest noise. Reading the scale-up evidence. The honest threshold is scale-stable, the cross-family margin is ≥260×τ≥\!260×τ at 99B, and an adaptive LoRA still leaves a 40×40× margin at Pile-unusable utility. Within the scoped threat model this closes the last scope-relevance gap the protocol could be challenged on at review: the commit-open primitive inherits the same separability story at 99B that we documented at 1.71.7B and 22B. Full numerics, training curves, and per-probe breakdowns are in Appendix F; cost was $2.43 2.43 of a $60 60 pre-committed scale-up budget. 5.6 Parallel-serve closure: commit-open vs. SVIP baseline To isolate the commit-open binding contribution over probe-after-return schemes, we compare against a SVIP-style baseline [4] under matched backbone-specific thresholds on 1111 attackers across both backbones (44 Qwen3: cheap lift, Qwen2.5-1.5B, Phi-3.5-mini, rank-6464 StageA LoRA; 77 Gemma-2-2B: 22 same-family lifts, 44 cross-family substitutes, 11 rank-6464 adaptive LoRA). In SVIP, an attacker serving M′M but retaining read access to open-source M routes the post-hoc probe to M while users see M′M — the parallel-serve side-channel. Commit-open binds the SAE trace to the served M′M before any probe, so opening is a membership check. Figure˜5: SVIP accepts all 1111 (11/1111/11 miss, joint-z at the honest median on both backbones); commit-open rejects all 1111 (11/1111/11 detect) at ≥2.78×τpool≥\!2.78× _pool (Qwen3) and ≥97×τGemma≥\!97× _Gemma (Gemma). The gap is the binding step, not stronger attackers (weaker SVIP variants — probe batching, cached M responses — share the same acceptance envelope, Appendix W). Figure 5: SVIP parallel-serve vs. commit-open across 1111 attackers on two backbones. Under SVIP the verifier’s joint-z collapses to the honest median (11/1111/11 miss; panel a); under commit-open all 1111 attackers are detected at ≥2.78×τpool≥\!2.78× _pool (Qwen3) and ≥97×τGemma≥\!97× _Gemma (Gemma) (panel b). 5.7 Feature-forgery infeasibility and library rotation We close the complementary threat: an attacker who does not run M at all and fabricates the committed top-3232 directly against the public library and threshold. Because joint-z is a mean over probe-slot penalties separable in the attacker’s feature set, the optimum has a closed form (Appendix V): pick features by the per-feature gain G∗(f)=∑(i,s)∈ℰ(f)[|μi,s|−|v∗(f)−μi,s|]/σi,sG^*(f)= _(i,s) (f)[| _i,s|-|v^*(f)- _i,s|]/ _i,s at the weighted-median v∗v^*; discrete top-3232 plus bf16 quantisation fixes the rest. The empirical ladder (Table˜5, 500 pos.×5 seeds500 pos.×5 seeds) records F0 uniform-random, F1 pooled-marginal, and the exact F3 optimum. On Qwen3 the F3 infimum is joint-z=44.24z=44.24 (29.3τpool29.3 _pool), on Gemma 221.6221.6 (203τGemma203 _Gemma); both match Prop. 4.2’s intrinsic-dim lower bound of 37.1137.11 to within the weighted-median / bf16 gap. F2 (proxy-model forward through the honest SAE) is dominated by F3 and F4 (y-decodability constraint) is subsumed because T∗T^* is position- and y-independent in feature-ID space (Appendix V). Library rotation. A probe-aware attacker could still overfit to the exact 9696 probes; as a defence-in-depth we evaluate a 5050-fold 48/4848/48 cross-validation of the F3 solver, training on 4848 probes and re-scoring on the held-out 4848. Train joint-z median is 40.2940.29, test median 52.6152.61 (Δ=+12.3 =+12.3 transfer gap), with 0/500/50 folds below τpool _pool on either side. A session-time rotation draw from a larger pool inherits this gap structurally. Table 5: Feature-forgery ladder. Per-position joint-z under attackers fabricating the top-3232 commitment (no M forward). F0/F1: 500500 positions × 55 seeds. F3: closed-form discrete optimum (Appendix V). All tiers are rejected by ≥29×≥\!29× the conservative threshold; the Prop. 4.2 intrinsic-dim lower bound (greedy T∗T^*, c=|μ/σ|c=|μ/σ| median) is 37.1137.11. Tier Attacker joint-z min median max ×τ×τ F0 uniform random top-3232 57.2557.25 57.3857.38 72.5972.59 37.937.9 F1 pooled marginal, top-3232 covered 53.4753.47 53.6353.63 53.8253.82 35.435.4 F3 exact optimum (Qwen3) 44.2444.24 (deterministic) 29.329.3 F3 exact optimum (Gemma) 221.6221.6 (deterministic) 203.3203.3 Additional experiments in appendix. Second-backbone Gemma evidence (Appendix E, Fig. 6); within-session dependence and robust-τ analysis (Appendix G); aggregator sensitivity (α×Nα× N sweep, Appendix H, Fig. 7); partial mechanistic auditability on local circuits (Appendix I); batched serving overhead (Appendix J, Fig. 8). 6 Limitations and Broader Impacts Scope. The evidence in this paper covers three backbone–SAE pairs spanning 1.71.7–99B parameters (Qwen3-1.7B + transcoder L1414, Gemma-2-2B + Gemma-Scope L1212, Gemma-2-9B + Gemma-Scope-9B L2020 width-131131k), the adaptation budgets listed in Table˜6 (including a rank-128128 frontier point on Qwen3), and a single serving stack for the ∼3% \!3\% parallel-commit cost observation. Tens of billions of parameters and ≥70≥\!70B flagship-class targets remain untested; we rely on the scale-stable threshold (τ9B=τGemma=1.09 _9B= _Gemma=1.09, agreement <10−4<\!10^-4) as inductive but not conclusive evidence that the primitive continues to hold at the “70B advertised / 7B served” operating point that motivates the protocol. Residual uncertainties — SAE/backend drift, MATH-kernel monoculture, session-level FPR under within-session dependence ρ=0.88ρ=0.88, Gemma-σ recalibration, public-probe overfit with 40%40\% mask-flip sensitivity, r>64r>64/longer-trained/ full-finetune adaptive attackers, a white-box E-E diagnostic that does not close the margin but does not exhaust the white-box frontier either, and a single-stack rather than dominance cost argument (the economic binding of strategy (C) is now complemented by the structural infeasibility of forgery via Prop. 4.2 rather than standing alone) — are enumerated with evidence pointers and mitigation sketches in Appendix X (Table˜15). The scoring rule of Equation˜2 is compatible without protocol change with stronger mitigations including secret-probe rotation, DP feature sketches, TEE attestation pairing, and an SVIP nonce-after-service redesign, which we leave to future work. Broader impacts. The primary positive impact is to increase auditability of hosted LLM serving: users who currently cannot verify identity gain a binding statistical primitive that is compatible with existing SAE/transcoder releases. Dual-use risk is limited and asymmetric: the protocol detects substitution but does not prevent it, and a capable provider could in principle adapt under budgets beyond those we evaluate. Providers deploying the protocol must be careful not to treat a single backbone–SAE pair as a backbone-agnostic guarantee; our three-backbone evidence (Qwen3-1.7B, Gemma-2-2B, Gemma-2-9B) argues for the primitive’s transferability and scale-stability up to 99B parameters, but not for a universal constant across all architectures and model sizes. No human-subject data, model release, or dataset release accompanies this work; the probe library is built from published named-circuit prompts and the honest pool is synthetic. 7 Conclusion We study a commit-open protocol that binds the provider to the per-position SAE-feature trace of its served sequence before the verifier’s query, closing the parallel-serve side-channel of probe-after-return schemes. Within Table˜6, it detects every tested non-adapted substitute at τpool=1.509 _pool=1.509, holds the strongest Qwen3 adaptive LoRA at 2.78×τpool2.78× _pool with 1.55×1.55× Pile perplexity, separates from SVIP on 1111 attackers across two backbones (11/1111/11 miss vs. 11/1111/11 detect), and transfers to a 4.5×4.5× scale-up — Gemma-2-9B + Gemma-Scope-9B (L2020, width-131131k) — with a scale-stable threshold (τ9B=1.09 _9B=1.09, agreement with 2B <10−4<\!10^-4) and cross-family / adaptive-LoRA margins of 260×260× and 40×τ9B40× _9B. Commit adds ≤2.1%≤\!2.1\% overhead at batch 3232. Priority follow-up: a probe-aware white-box adaptive-frontier sweep and a DP-rotated secret probe library. Code, probe library, honest-pool artifacts, and calibration grids will be released upon publication. References Campanelli et al. [2019] Matteo Campanelli, Dario Fiore, and Anaïs Querol. LegoSNARK: Modular design and composition of succinct zero-knowledge proofs. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2075–2092. ACM, 2019. doi: 10.1145/3319535.3339820. Pasquini et al. [2025] Dario Pasquini, Evgenios M. Kornaropoulos, and Giuseppe Ateniese. LLMmap: Fingerprinting for large language models. In 34th USENIX Security Symposium, pages 299–318. USENIX Association, 2025. Xu et al. [2024] Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 3277–3306. Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.NAACL-LONG.180. Sun et al. [2024] Yifan Sun, Yuhang Li, Yue Zhang, Yuchen Jin, and Huan Zhang. SVIP: Towards verifiable inference of open-source large language models. CoRR, abs/2410.22307, 2024. doi: 10.48550/ARXIV.2410.22307. Hu et al. [2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). OpenReview.net, 2022. Ong et al. [2025] Jack Min Ong, Matthew Di Ferrante, Aaron Pazdera, Ryan Garner, Sami Jaghouar, Manveer Basra, Max Ryabinin, and Johannes Hagemann. TOPLOC: A locality sensitive hashing scheme for trustless verifiable inference. In International Conference on Machine Learning (ICML). PMLR / OpenReview.net, 2025. Gross et al. [2024] Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Wang et al. [2023] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR). OpenReview.net, 2023. Olsson et al. [2022] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. Preprint: arXiv:2209.11895. Bricken et al. [2023] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. Templeton et al. [2024] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. Bloom et al. [2024] Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. SAELens: Training and analyzing sparse autoencoders, 2024. Software package, https://github.com/jbloomAus/SAELens. Dunefsky et al. [2024] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Ball et al. [2017] Marshall Ball, Alon Rosen, Manuel Sabin, and Prashant Nalini Vasudevan. Proofs of useful work. Cryptology ePrint Archive, Paper 2017/203, 2017. URL https://eprint.iacr.org/2017/203. Cheng et al. [2024] Pau-Chen Cheng, Wojciech Ozga, Enriquillo Valdez, Salman Ahmed, Zhongshu Gu, Hani Jamjoom, Hubertus Franke, and James Bottomley. Intel TDX demystified: A top-down approach. ACM Computing Surveys, 56(9):238:1–238:33, 2024. doi: 10.1145/3652597. Zhu et al. [2024] Jianwei Zhu, Hang Yin, Peng Deng, Aline Almeida, and Shunfan Zhou. Confidential computing on NVIDIA Hopper GPUs: A performance benchmark study. CoRR, abs/2409.03992, 2024. doi: 10.48550/ARXIV.2409.03992. Sailer et al. [2004] Reiner Sailer, Xiaolan Zhang, Trent Jaeger, and Leendert van Doorn. Design and implementation of a TCG-based integrity measurement architecture. In 13th USENIX Security Symposium, pages 223–238. USENIX Association, 2004. Crosby and Wallach [2009] Scott A. Crosby and Dan S. Wallach. Efficient data structures for tamper-evident logging. In 18th USENIX Security Symposium, pages 317–334. USENIX Association, 2009. Laurie et al. [2013] Ben Laurie, Adam Langley, and Emilia Käsper. Certificate transparency. RFC 6962, IETF, 2013. Jin et al. [2026] Heng Jin, Chaoyu Zhang, Hexuan Yu, Shanghao Shi, Ning Zhang, Y. Thomas Hou, and Wenjing Lou. Trusting what you cannot see: Auditable fine-tuning and inference for proprietary AI. CoRR, abs/2603.07466, 2026. Akgul et al. [2025] Hasan Akgul, Daniel Borg, Arta Berisha, Amina Rahimova, Andrej Novak, and Mila Petrov. Verifiable fine-tuning for LLMs: Zero-knowledge training proofs bound to data provenance and policy. CoRR, abs/2510.16830, 2025. Merkle [1987] Ralph C. Merkle. A digital signature based on a conventional encryption function. In Carl Pomerance, editor, Advances in Cryptology - CRYPTO ’87, Lecture Notes in Computer Science, pages 369–378. Springer, 1987. doi: 10.1007/3-540-48184-2\_32. Wald [1945] Abraham Wald. Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2):117–186, 1945. Holm [1979] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. Yang et al. [2025] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505.09388, 2025. doi: 10.48550/ARXIV.2505.09388. Gemma Team et al. [2024] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjöesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00118, 2024. doi: 10.48550/ARXIV.2408.00118. Yang et al. [2024] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024. doi: 10.48550/ARXIV.2412.15115. Abdin et al. [2024] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024. doi: 10.48550/ARXIV.2404.14219. OLMo et al. [2025] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 2 furious. CoRR, abs/2501.00656, 2025. doi: 10.48550/ARXIV.2501.00656. Gao et al. [2021] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800GB dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. Mahalanobis [1936] Prasanta Chandra Mahalanobis. On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2(1):49–55, 1936. Clopper and Pearson [1934] C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.1093/biomet/26.4.404. Lieberum et al. [2024] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca D. Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. CoRR, abs/2408.05147, 2024. doi: 10.48550/ARXIV.2408.05147. Appendix A Threat Model Table˜6 defines the adversary we consider. Every served request carries a commitment R; the verifier audits a random subset of sessions with public probe library P and public decision threshold τ. Adversary knowledge includes the published SAE release, P, τ, and the form of the scorer in Equation˜2; the adversary has black-box query access to M through a public npub=2000n_pub=2000 corpus used to fit φ but not gradient access to the joint score (the primary threat model). A supplementary white-box variant (E-E, Section˜5.4) permits gradient access through the frozen SAE encoder at fixed rank (r=64r=64) for 30003000 steps; full-model retraining and higher-rank / longer-trained white-box remain out of scope. Per-session FPR is bounded by 1−(1−α)k≤kα1-(1-α)^k≤ kα for k openings at per-opening α=0.01α=0.01; dependence between openings within a session is measured empirically (Appendix G) and deflates the union bound on our stack. Table 6: Scoped threat model. All audited sessions must carry a Merkle commitment R; the probe library and threshold are public. Dimension Setting in this paper Adversary goal Serve M′M while passing audit on M. Adversary knowledge Public: M, SAE release, probe library P, threshold τ, scoring rule. Black-box query access to M via 2000-sample public corpus. Adaptation budget Two budgets: (i) public-corpus linear φ fit on 20002000 pairs + LoRA r=16r=16 on seven projections, ≤3000≤ 3000 steps; (i) library-aware joint φ + LoRA r=64r=64, 15001500 steps. Stronger white-box or longer-budget attacks are out of scope. Audit schedule Each session audited independently; verifier samples k positions with kα≤0.10kα≤ 0.10. Session-level guarantee Conservative per-opening τpool=1.509 _pool=1.509 is the empirical max over pooled nhon=112n_hon=112 honest draws (MATH + multi-kernel seeds 300−302300-302); 0/1120/112 exceedances, Clopper–Pearson 95%95\% upper CI 0.02640.0264. Pool-internal legacy τreal=1.13 _real=1.13 (nhon=64n_hon=64) retained for comparison. Out of scope Cryptographic soundness, full-model retraining to fit the published library, non-audited sessions. White-box gradient access through the frozen SAE encoder is partially evaluated (E-E, Section˜5.4) but higher-rank / longer-trained / full-finetune remains out of scope. Appendix B Proof Sketch and Tail Robustness of τ Proof sketch. (B) Statistical detection. The per-probe score in Equation˜2 is a mean of |Si||S_i| standardised z-scores on features where the honest calibration concentrates mass; cross-backend draws yield a bounded-variance null whose 9999th percentile is estimated on a held-out split. Sequential calibration is outside the evaluated protocol. (C) Economic bound. leaftleaf_t depends on ftf_t of the committed model; any provider that commits M must run M for every committed position, so wall-clock cost is lower-bounded by the honest forward pass up to overlap-with-M′M parallelism. (D) Heuristic. Under a public-corpus fit, φ is the minimum-norm least-squares solution with empirical ‖φ‖F=0.59\| \|_F=0.59; we report this as a post-hoc empirical observation, not a formal bound (Appendix U). Tail robustness of τ. τpool=1.509 _pool=1.509 is the empirical max over pooled n=112n=112 honest draws. Fitting a Gaussian (resp. Student-t with df=5df=5) to the n=64n=64 pool shifts the parametric 9999th percentile to 1.371.37 (resp. 1.441.44) — below τpool=1.509 _pool=1.509, so the pooled empirical max is the conservative choice given the tail uncertainty emphasised by Clopper and Pearson [32]. For multi-opening sessions, per-opening joint-z is not independent: a Gaussian-copula simulation at marginal α=0.01α=0.01 with within-session ρ=0.88ρ=0.88 yields session-FPR 1.9%1.9\% at k=4k=4 (vs. 3.9%3.9\% under independence and 4%4\% under kαkα). Cross-probe correlation within a single opening is not modelled; measuring it requires raw per-probe z-vectors and remains future work. Appendix C Claim–Evidence Summary Table Table 7: Claim–evidence matrix. All Qwen3 experiments use Qwen3-1.7B at layer 1414 with the public mwhanna-qwen3-1.7b-transcoders-lowl0 SAE release and a 9696-probe library. Conservative deployment threshold on Qwen3 is τpool=1.509 _pool=1.509 (pooled nhon=112n_hon=112, MATH + multi-kernel seeds); pool-internal τreal=1.13 _real=1.13 retained for comparison. Gemma-2-2B: τGemma=1.09 _Gemma=1.09 (nhon=64n_hon=64). Legacy per-probe Mahalanobis numbers labelled. Claim Experiment Key quantitative result C1 (same-family) E2, 3 lifted attackers 3/33/3 detected, TPR =1.00=1.00, CP 95% CI [0.926,1.00][0.926,1.00]; per-probe Mahalanobis 5656–148×148× honest legacy scale C2 (cross-family) E3, 4 substitute models 4/44/4 detected at τpool=1.509 _pool=1.509 (0/1120/112 honest exceedances, CP 95% upper CI 0.02640.0264) C3 (rank-cstr. LoRA) E4 + diag per-category held-out joint-z≥12×τpoolz≥ 12× _pool; per-probe Mahalanobis 3434–81×81× legacy honest max; ‖φ‖F=0.59\| \|_F=0.59 C4 (joint-consist.) E5, α×Nα× N sweep max ΔAUC=0.337 =0.337; at α=0.002α=0.002, N=1→0.63N=1→0.63, N=96→0.93N=96→0.93 C5 (auditability) E10, 4-class ablation IOI 4.35×4.35×, Factual 2.14×2.14×, Syntax 1.73×1.73×; Induction −0.78×-0.78× (scoped out) C6 (economic) B1 + E9 batched substitute-serve detected; parallel-commit 97%97\% of honest wall-clock; commit overhead ≤2.1%≤ 2.1\% at B=32B=32 C7 (second backbone) E12 + E13, Gemma-2-2B + Gemma-Scope 8/88/8 non-adaptive attackers TPRjoint=1.00TPR_joint=1.00 at τGemma=1.09 _Gemma=1.09; per-probe Mahalanobis 1.9×1.9×–5.8×5.8× τGemmamaha=57.99τ^maha_Gemma=57.99 C8 (Gemma adaptive) E14, library-aware r=64r=64 StageA on Gemma-2-2B-it joint-z=105.9z=105.9, 97.2×τGemma97.2× _Gemma, Pile ppx 80328032 (436×436× clean); catastrophic utility loss C9 (SVIP two-backbone) E11/Recipe 3 parallel-serve 11/1111/11 miss under SVIP, 11/1111/11 detect under commit-open C10 (white-box E-E) E16, 3 restart seeds at (αjz,λutil)=(0,0)( _jz, _util)=(0,0) joint-z∈6.89,7.80,4.99z∈\6.89,7.80,4.99\; all restarts >4.19>4.19 black-box; ReLU-sparse gradients zero for inactive top-k C11 (9B scale-up) E12/E13/E14-9B, Gemma-2-9B + Gemma-Scope-9B L2020 width-131131k τ9B=1.09 _9B=1.09 (matches 2B <10−4<\!10^-4); cross-family Qwen2.5-7B joint-z=283.3z=283.3 (260×τ260×τ); adaptive LoRA r=64r=64 StageA z=43.7z=43.7 (40×τ40×τ, 30×30× ppx) C12 (r==128 frontier) E16-r128, Qwen3-1.7B + transcoder L1414 r=128r=128 MSE λutil=0 _util=0: joint-z=6.42z=6.42 (4.25×τpool4.25× _pool); doubling rank does not reopen the margin Appendix D Master Results Table Table˜3 in the main body reports the strongest tested attacker per tier. Table˜8 below gives the full per-attacker breakdown referenced throughout Section˜5.2–5.6. Table 8: Master results. Every attacker evaluated, consolidated onto the joint-z scoring rule where possible. 2020 points on Qwen3-1.7B (transcoder, L1414, τpool=1.509 _pool=1.509), 44 points on Gemma-2-2B (Gemma-Scope, L1212, τGemma=1.09 _Gemma=1.09), 33 points on Gemma-2-9B (Gemma-Scope-9B, L2020, width-131131k, τ9B=1.09 _9B=1.09, identical to 2B within 10−410^-4), and one aggregate row for the 88 non-adaptive Gemma-2B attackers of Appendix E on the legacy per-probe Mahalanobis scale (1.9−5.8×τGemmamaha=57.991.9-5.8×τ^maha_Gemma=57.99), retained as an aggregate. Rows 11–33 are E2 lift attackers: joint-z reaches TPR=1=1 at τpool _pool (Section˜5.2). “×τ×τ” is the ratio to the per-backbone threshold; “ppx” is Pile perplexity (M′M -clean baseline 21.2021.20 Qwen3, 18.4418.44 Gemma-2B, 18.5018.50 Gemma-2B-it → Gemma-9B substitute). All 3333 evaluated attackers are rejected by commit-open; the SVIP baseline column summarises Section˜5.6 (11/1111/11 miss on its re-run set). Row 1313 reports the median over three restart seeds for the Qwen3 (αjz,λutil)=(0,0)( _jz, _util)=(0,0) cell only; rows 1717–1818 report the first run of a two-seed pair, with the second-seed variant (joint-z=9.71,57.38z=\9.71,57.38\ respectively, opposite stability outcome) listed as rows 17′17 /18′18 — both pairs are detected, and this confirms the seed-variance caveat noted for white-box E-E. All other rows are single runs. DIV: ppx>106>\!10^6. # Class Attacker Budget joint-z ×τ×τ Pile ppx Verdict SVIP verdict Qwen3-1.7B target, τpool=1.509 _pool=1.509 1 lift ZERO_PAD Qwen3-0.6B E2, natt=48n_att=48 ≫τpool \! _pool ∗ TPR=1=1 −- detect miss 2 lift RAND_PROJ Qwen3-0.6B E2, natt=48n_att=48 ≫τpool \! _pool ∗ TPR=1=1 −- detect −- 3 lift LEARNED_LIN Qwen3-0.6B E2, natt=48n_att=48 ≫τpool \! _pool ∗ TPR=1=1 −- detect −- 4 cross Qwen2.5-1.5B + public φ E3 69.269.2 45.9×45.9× −- detect miss 5 cross Phi-3.5-mini + public φ E3 52.452.4 34.7×34.7× −- detect miss 6 cross OLMo-2-7B + public φ E3 81.381.3 53.9×53.9× −- detect −- 7 cross Qwen3-0.6B + public φ E3 62.862.8 41.6×41.6× −- detect −- 8 LoRA public-φ + LoRA r=16r=16 30003000 steps ≥18.1≥\!18.1 ≥12×≥\!12× ∼25 \!25 detect −- 9 LoRA library-aware φ only E4-v2 baseline 57.3857.38 38.0×38.0× 21.2021.20 detect −- 10 LoRA E4-v2 StageA λ=0λ=0 r=64r=64, 15001500 steps 4.194.19 2.78×2.78× 32.8432.84 detect miss 11 LoRA E4-v2 StageC λ=0.1λ=0.1 r=64r=64, 15001500 steps 6.276.27 4.16×4.16× 63.1663.16 detect −- 12 LoRA E4-v2 StageB λ=1.0λ=1.0 r=64r=64, 15001500 steps 7.487.48 4.96×4.96× 50.9850.98 detect −- 13 whitebox E-E αjz=0 _jz=0, λ=0λ=0 r=64r=64, 30003000 st., 3 seeds† 6.896.89 (med.) 4.57×4.57× 31.931.9 detect −- 14 whitebox E-E α=0α=0, λ=0.1λ=0.1 r=64r=64, 30003000 steps 49.4649.46 32.8×32.8× 36253625 detect −- 15 whitebox E-E α=0.1α=0.1, λ=0λ=0 r=64r=64, 30003000 steps 57.3857.38 38.0×38.0× DIV detect −- 16 whitebox E-E α=0.1α=0.1, λ=0.1λ=0.1 r=64r=64, 30003000 steps 9.999.99 6.62×6.62× 11881188 detect −- 17 whitebox E-E α=1α=1, λ=0λ=0 r=64r=64, 30003000 steps 56.5556.55 37.5×37.5× DIV detect −- 18 whitebox E-E α=1α=1, λ=0.1λ=0.1 r=64r=64, 30003000 steps 7.407.40 4.90×4.90× 243243 detect −- 17′ whitebox E-E α=1α=1, λ=0λ=0 (seed 2) r=64r=64, 30003000 steps 9.719.71 6.44×6.44× 86,80986,809 detect −- 18′ whitebox E-E α=1α=1, λ=0.1λ=0.1 (seed 2) r=64r=64, 30003000 steps 57.3857.38 38.0×38.0× 35913591 detect −- Higher-rank LoRA frontier (P0-2 scale-up) 19a LoRA r==128 MSE, λ=0λ=0 r=128r=128, 30003000 steps 6.426.42 4.25×4.25× 50.3750.37 detect −- 19b LoRA r==128 MSE, λ=0.1λ=0.1 r=128r=128, 30003000 steps 47.0147.01 31.1×31.1× 35383538 detect −- Gemma-2-2B target, τGemma=1.09 _Gemma=1.09 19 mixed 8-attacker aggregate 44 lifts + 44 cross-family, legacy Mahalanobis only 1.9−5.8×τGemmamaha1.9-5.8×τ^maha_Gemma −- /8/8 detect 6/66/6 miss‡ 20 LoRA library-aware φ -only (Gemma) E14 base 205.56205.56 188.6×188.6× 18.4418.44 detect −- 21 LoRA E14 StageA λ=0λ=0 r=64r=64 105.89105.89 97.2×97.2× 8031.658031.65 detect miss 22 whitebox E-E Gemma α=1α=1, λ=0λ=0 r=64r=64 133.86133.86 122.8×122.8× 12,35612,356 detect −- 23 whitebox E-E Gemma α=1α=1, λ=0.1λ=0.1 r=64r=64 144.40144.40 132.5×132.5× 11321132 detect −- Gemma-2-9B target (P0-1 scale-up), L2020, width-131131k, τ9B=1.09 _9B=1.09, nhon=64n_hon=64 24 cross Qwen2.5-7B + public φ , scale-matched Pile 20002000, 80/2080/20 283.30283.30 260.0×260.0× −- detect −- 25 LoRA Gemma-2-2B-it + joint φ -only (baseline) no LoRA training 280.17280.17 257.0×257.0× 18.5018.50 detect −- 26 LoRA Gemma-2-2B-it + StageA λ=0λ=0 r=64r=64, 15001500 steps 43.7043.70 40.1×40.1× 560.16560.16 detect −- †Restart seeds s0,s1,s2\s0,s1,s2\: joint-z=6.89,7.80,4.99z=\6.89,7.80,4.99\; best-for-attacker 4.994.99 still >4.19>4.19 black-box. ∗E2 lifts reach TPR=1=1 at τpool _pool; legacy per-probe Mahalanobis: 56−148×τQwenmaha=1.5156-148×τ^maha_Qwen=1.51. ‡Of 88 aggregate attackers, 66 rerun under SVIP-routing (2/42/4 lifts + 4/44/4 cross-family); all 66 miss. Appendix E Second-Backbone Evidence: Gemma-2-2B with Gemma-Scope To test whether separability is a property of the (M,L,SAE)(M,L,SAE) tuple or of the protocol, we re-instantiate the pipeline on Gemma-2-2B + public gemma-scope-2b-pt-res canonical SAE [33] at L1212 (residual-post, dSAE=16,384d_SAE=16,384, top-3232). The 9696-probe library uses the same prompts via Gemma forwards; σi _i is recalibrated on 3232 backend configs (2 dtypes × 2 kernels × 4 positions × 2 seeds, disjoint from honest-pool seeds). Deployment threshold τGemma=1.09 _Gemma=1.09 (nhon=64n_hon=64, 0/640/64, CP one-sided upper 0.04570.0457). Eight Gemma attackers (four E2-equivalent lifts + four E3-equivalent cross-family substitutes: Gemma-2-2B-it, Pythia-1.4B, Qwen2.5-1.5B, Phi-3.5-mini-instruct) all clear τ at TPRjoint=1.00TPR_joint=1.00; per-probe Mahalanobis scores 109109–338338 (1.91.9–5.8×τGemmamaha=57.995.8×τ^maha_Gemma=57.99). Cross-family held-out R2R^2 is negative for all three non-Gemma families (Pythia-1.4B −1.29-1.29; Qwen2.5-1.5B −15.40-15.40; Phi-3.5-mini −0.39-0.39). Same-family Gemma-2-2B-it is the closest call (2.3×τGemmamaha2.3×τ^maha_Gemma) but clears on every replay. A library-aware rank-6464 StageA on Gemma-2-2B-it (Section˜5.4, Table˜4) reaches z=105.9z=105.9 (97.2×τGemma97.2× _Gemma) at Pile ppx 80328032 (436×436× clean) — detectable with catastrophic utility loss, but the utility-preserving Gemma frontier (λutil>0 _util>0 sweep) is unmeasured. Figure 6: Two-backbone separability on the per-probe Mahalanobis diagnostic scale (log y-axis, each panel uses its backbone’s own Mahalanobis threshold). (a) Qwen3-1.7B + transcoder L1414 (nhon=112n_hon=112 pooled; τQwenmaha=1.51τ^maha_Qwen=1.51). (b) Gemma-2-2B + Gemma-Scope canonical residual SAE L1212 (nhon=64n_hon=64, τGemmamaha=57.99τ^maha_Gemma=57.99). Four additional E3-equivalent cross-family attackers (Pythia-1.4B, Qwen2.5-1.5B, Phi-3.5-mini-instruct, Gemma-2-2B-it) also clear both backbones’ joint-z thresholds (not plotted). Appendix F Scale-Up to Gemma-2-9B with Gemma-Scope-9B This appendix substantiates the scale-up claims of Section˜5.5: that the commit-open primitive transfers without re-engineering to a 4.5×4.5× larger target, the honest threshold is numerically indistinguishable from the 2B value, and detection margins strictly grow with SAE width. Setup. Target: google/gemma-2-9b at L2020 (dmodel=3584d_model=3584, 2626 transformer blocks; L2020 matches the 46%46\% depth ratio of the 2B-L1212 reference); SAE: gemma-scope-9b-pt-res-canonical, layer_20/ width_131k/canonical, avg-L0=114L_0=114, dSAE=131,072d_SAE=131,072 [33]. The 9696 probe prompts and 88 circuit-class taxonomy from the 2B second-backbone replication (Appendix E) transfer verbatim; top-3232 feature IDs, μ, σ are recomputed on 99B residual activations. Honest pool: n=64n=64 (22 dtypes × 22 kernels × 44 positions × 44 seed groups), reusing the multi-kernel recipe of Appendix T. Compute: four Modal runs on A100100-4040GB and A1010G, total $2.43 2.43 of a $60 60 pre-committed scale-up budget. Honest threshold. τ9B=1.090 _9B=1.090 (pool p9999), versus τGemma=1.090 _Gemma=1.090 on 2B and τpool=1.509 _pool=1.509 on Qwen3. Gemma-2-9B and Gemma-2-2B thresholds agree to within 10−410^-4 on the same 9696-probe prompt set — strong evidence that the pooled-p9999 joint-z is a property of the probe library and scoring rule rather than of the model size. Cross-family (row 2424 of Table˜8). Qwen2.5-7B [27] with a public-corpus φ:ℝ3584→ℝ3584 ^3584\!→\!R^3584 fit by linear least-squares on 20002000 Pile samples (80/2080/20 train/hold-out; held-out R2=−0.091R^2=-0.091, consistent with the negative cross-family R2R^2 pattern seen on 2B). Joint-z=283.3z=283.3, per-probe minimum 253.1253.1. Detection is 260×τ9B260× _9B — a full two orders of magnitude tighter than the 2B cross-family rejection margin (median ∼3×τGemma \!3× _Gemma on the per-probe Mahalanobis scale). The mechanism: at width 131131k the per-probe reference pattern is higher-intrinsic-dim (Prop. 4.2 scales ∝log⁡dSAE \! d_SAE in the idim bound), and a φ that fits Pile-average activations does not bring 99B top-3232 features to Qwen2.5-7B’s manifold except in directions explicitly represented in the Pile marginal. Same-family φ -only baseline (row 2525). Gemma-2-2B-it through a joint φ to the 99B manifold, no LoRA: joint-z=280.2z=280.2 at Pile ppx 18.518.5 (baseline clean). This establishes that a 7×7× smaller same-family substitute, even with a trained linear lift, is rejected at 257×τ9B257× _9B before adaptation begins. Adaptive LoRA (row 2626). Same recipe as Qwen3 StageA: M′=M =Gemma-2-2B-it, r=64r=64, α=32α=32, seven projection modules, 15001500 steps AdamW lr 3×10−43×10^-4, λutil=0 _util=0 (pure-probe). Joint-z=43.70z=43.70, 40.1×τ9B40.1× _9B; Pile ppx 560.2560.2 (30×30× substitute-clean). In absolute z the margin is ∼2.4× \!2.4× tighter than the matching 2B point (z=105.89z=105.89), but the 99B SAE is 8×8× wider, so detection is strictly stronger per unit honest-pool noise. The attacker also pays a larger utility penalty in relative terms than on 2B (30×30× at 9B vs. 436×436× at 2B is superficially lower, but the 9B substitute starts from a 3×3× smaller clean ppx baseline and is pushed into a higher absolute ppx regime). A utility-preserving 99B frontier (λutil>0 _util>0 sweep) is not evaluated and, as on 2B, is an acknowledged caveat (Section˜6). Decision-gate verdict. The pre-committed gate of the scale-up plan required (a) τ9B∈[0.8,2.5] _9B∈[0.8,2.5], (b) adaptive LoRA >3×τ>\!3×τ, (c) cross-family >20×τ>\!20×τ. Observed: (a) τ9B=1.09 _9B=1.09; (b) 40×τ40×τ; (c) 260×τ260×τ. All three checks pass by >10×>\!10× the gate. Appendix G Within-Session Dependence and Robust τ The n=64n=64 honest pool partitions into 1616 (dtype,kernel,seed)(dtype,kernel,seed) tuples of 44 positions, letting us estimate cross-position correlation within a session. Mean off-diagonal correlation across the four within-tuple positions is ρ=0.883ρ=0.883 (Table˜9). A Gaussian-copula simulation at marginal α=0.01α=0.01 yields session-FPR 1.93%1.93\% at k=4k=4, vs. 3.94%3.94\% under independence and kα=4%kα=4\% union bound — positive cross-opening correlation deflates session-FPR below the naive bound. A follow-up multi-kernel spot check on Modal L4 draws 4848 additional honest configs at companion seeds 300−302300-302; MATH and EFFICIENT_ATTENTION medians agree to 4.6%4.6\% (0.9580.958 vs. 1.0021.002). The companion-seed family shifts joint-z median more than the kernel (0.6710.671 on seeds 100−107100-107 vs. 0.958−1.0020.958-1.002 on seeds 300−302300-302), so pooling n=112n=112 raises the empirical max to 1.5091.509 (adopted as τpool _pool). Table 9: Session-level FPR at nominal marginal α=0.01α=0.01 under (i) the naive kαkα union bound, (i) exact independence 1−(1−α)k1-(1-α)^k, and (i) a Gaussian-copula simulation at ρ=0.883ρ=0.883 (Qwen3 n=64n=64 pool, Nsim=105N_sim=10^5). k kαkα union independent copula (ρ=0.883ρ=0.883) 1 0.010 0.010 0.010 2 0.020 0.020 0.015 3 0.030 0.030 0.018 4 0.040 0.039 0.019 Appendix H Aggregator Sensitivity: Joint Consistency vs. Single Probe This is an aggregator sensitivity analysis, not an adversarial test: we interpolate attacker centers as αzattacker+(1−α)μhonestα\,z_attacker+(1-α)\, _honest with honest noise and sweep N∈1,…,96N∈\1,…,96\ at a fixed FPR budget. As α decreases, single-probe AUC collapses while joint consistency retains detection, peaking at ΔAUC=0.337 =0.337 for α=0.001α=0.001; all four attacker centers pass a ΔAUC≥0.03 ≥ 0.03 gate. The N-probe sweep (Figure˜7) shows AUC rising monotonically with N, largest benefit at α∈0.0005,0.001,0.002α∈\0.0005,0.001,0.002\; the curve flattens by N≈32N≈32, motivating N=96N=96 as a comfortable operating margin. The SPRT aggregator at α=β=0.01α=β=0.01 terminates at n∗=1n =1 under full-strength E3 attackers, so the joint aggregator only contributes in the weakened-mean regime; we do not claim robustness against probe-aware adaptive attackers. Figure 7: Aggregator N-probe sweep: mean AUC across four attacker centers at each α-weakened operating point. ΔAUC between N=1N=1 and N=96N=96 is largest for small α, and AUC plateaus by N≈32N≈32. Appendix I Partial Mechanistic Auditability We ablate the top-3232 features of each probe class and measure class-specific effect via ΔKL=KL⁡(pclean∥pabl)−KL⁡(pclean∥prec) =KL(p_clean\,\|\,p_abl)-KL(p_clean\,\|\,p_rec). Of the four circuit classes tested, three local-circuit classes admit class-specific attribution under our top-3232 sketch: IOI (4.35×4.35×), factual (2.14×2.14×), syntactic (1.73×1.73×); the fourth, induction, is known to be distributed across heads and layers [9] and does not localise (specificity −0.78×-0.78×). We therefore make only a partial auditability claim, restricted to local circuit classes for which the top-3232 sketch carries class-specific signal; this is not a general mechanistic-auditability guarantee. Appendix J Serving Overhead (E9) A single-GPU RTX 3090 sanity test (200200 prompts, output length 9696) gives strategy-(B) TPR =1.00=1.00 at τpool=1.509 _pool=1.509 while strategy-(C) parallel-commit deception costs 97.0%97.0\% of honest wall-clock (∼3% \!3\% gap on a single GPU; one operating point, not a dominance claim). At batched Modal L4 serving, amortised C/AC/A falls from 1.1851.185 (B=1B=1) to 1.0211.021 (B=32B=32, 2.1%2.1\% latency); payload is 224224 bytes/request (Figure˜8). Figure 8: E9 batched commit overhead. (a) Per-batch latency for forward-only (A) and forward + SAE + commit (C) with 20-trial standard deviation. (b) Commit overhead C/A−1C/A-1 collapses from 18.5%18.5\% at B=1B=1 to 2.1%2.1\% at B=32B=32. Appendix K Cross-Family φ Transfer Diagnostic Figure˜9 reports the fit and held-out R2R^2 of the public-corpus linear alignment map φ for each cross-family attacker, alongside the verifier-side joint z-score. Two attackers (Phi-3.5-mini, OLMo-2-7B) exhibit strongly negative held-out R2R^2, confirming that the public-corpus φ does not generalise across model families. Partial explanatory analysis for C3; detection is established independently by the joint z-score. Figure 9: E3 cross-family φ -transfer diagnostic. (a) Fit R2R^2 on the 2000-pair public-corpus training set vs. held-out R2R^2 on the 20%20\% split. (b) Joint z-score on the verifier side; all four attackers clear τ (dashed). Appendix L E4 Rank-Constrained LoRA Diagnostic Figure˜10 shows three diagnostic runs that confirm (a) LoRA trains (B-matrix Frobenius norm grows from 0 to 5.245.24 during stage A and stabilises through stage B); (b) gradients flow through the probe-layer hook (204204 of 392392 trainable parameters receive gradient); (c) with a random φ of the same Frobenius norm, eval- and train-mode outputs differ after training. The limiting factor is the min-norm structure of the learned φ , not a training bug: ‖Δh‖∞∼1\| h\|_∞ 1 projects to ‖Δproj‖∞≤0.01\| \|_∞≤ 0.01 through φ , below the SAE sparsity threshold. Figure 10: E4 rank-constrained LoRA diagnostic. (a) LoRA’s B-matrix Frobenius norm grows across stage-A and stage-B, yet stays below ‖φ‖F\| \|_F. (b) Per-category median joint z-score on held-out probes after full 2-stage training remains ≥12×≥ 12× the conservative deployment threshold τpool=1.509 _pool=1.509. Appendix M White-Box Joint-z Gradient Attacker (E-E) Table˜10 lists all eight evaluated operating points. The attacker objective is ℒ=MSE(φ(hM′),hM)+αjzℒjz+λutilNLLM′(Pile)L=MSE( (h_M ),h_M)+ _jz\,L_jz+ _util\,NLL_M (Pile) with ℒjzL_jz computed through the frozen SAE encoder over the top-3232 mask. Training uses AdamW lr 3×10−43×10^-4 at LoRA r=64r=64 for 30003000 steps. The strongest Qwen3 cell (αjz=0,λutil=0)( _jz=0, _util=0), pure MSE, yields joint-z=6.89z=6.89 (median over 3 restarts), weaker than the E4-v2 black-box optimum 4.194.19: ReLU-sparse SAE encoder yields near-zero gradients for inactive top-k features — the score-gradient term is a noisy small-support signal that fights the dense MSE. Adding λutil _util pulls the hidden state off the honest manifold (joint-z rises to 49.549.5); turning up αjz _jz without utility regularisation diverges (ppx>107ppx>10^7). Concrete white-box negative result within this budget: r=64×3000r=64× 3000 steps, gradient access to the scoring rule does not help the attacker. Compute: Qwen3 grid ≈40≈\!40 min L4 (≈$0.55≈ 0.55); Gemma ≈21≈\!21 min L4 (≈$0.29≈ 0.29). Table 10: White-box joint-z gradient attacker (E-E). LoRA r=64r=64, 30003000 steps, SAE weights frozen but gradients propagated through the ReLU encoder. The (αjz=0,λutil=0)( _jz=0, _util=0) row reports median over three restart seeds (6.89,7.80,4.99\6.89,7.80,4.99\); other rows are single runs. The best-for-attacker restart (lowest joint-z=4.99z=4.99) is still weaker than the E4-v2 black-box MSE optimum 4.194.19. “DIV”: ppx>106ppx>\!10^6. Backbone αjz _jz λutil _util joint-z Pile ppx Qwen3-1.7B, τpool=1.509 _pool=1.509 Qwen3 0 0 6.896.89 (med.) 31.931.9 3 seeds Qwen3 0 0.10.1 49.4649.46 36253625 Qwen3 0.10.1 0 57.3857.38 3.2×1073.2\!×\!10^7 DIV Qwen3 0.10.1 0.10.1 9.999.99 11881188 Qwen3 1.01.0 0 56.5556.55 1.0×10101.0\!×\!10^10 DIV Qwen3 1.01.0 0.10.1 7.407.40 243243 Gemma-2-2B, τGemma=1.09 _Gemma=1.09 Gemma 1.01.0 0 133.86133.86 12,35612,356 Gemma 1.01.0 0.10.1 144.40144.40 11321132 Appendix N Circuit-Ablation Effect Matrix Figure 11: E10 circuit-ablation effect matrix. Cell (i,j)(i,j) is KL⁡(clean∥ablated)−KL⁡(clean∥recon)KL(clean\|ablated)-KL(clean\|recon) when the top-32 features of probes in class j are zeroed, evaluated on probes of class i. Values are in units of 10−310^-3. Appendix O E6 Per-Category Attackability and Library Rotation Figure˜12 orders probe categories by the median joint z-score of the full E3 attacker suite. Robust classes (arithmetic, factual, commonsense) are retained for high-assurance regimes; attackable classes (induction, coreference, IOI) are candidates for periodic rotation. Figure 12: E6 per-category attackability under the E3 attack suite. Policy: rotate attackable classes; bias the library toward robust classes for high-assurance regimes. Appendix P Top-k Sweep We re-evaluate the Qwen3 honest pool and the strongest Qwen3 white-box attacker (E-E, αjz=0 _jz=0, λutil=0 _util=0, r=64r=64, 30003000 steps) at k∈4,8,16,32k∈\4,8,16,32\ with per-feature z-arrays captured post-hoc and re-aggregated. Detection margin is stable at 4.64−4.99×4.64-4.99× across the k range (Table˜11); both τ and the attacker score rise at smaller k because fewer features give individual strong activations more weight, and the two rise in lockstep. Table 11: k-sweep on Qwen3-1.7B with multikernel honest pool (n=48n=48, seeds 300−302300-302) and the strongest E-E attacker. Margin = attacker joint-z // pool-max τ at the same k. k honest median honest max (τ) attacker joint-z margin 44 1.2181.218 1.8501.850 9.2269.226 4.99×4.99× 88 1.2081.208 1.8621.862 9.1349.134 4.90×4.90× 1616 1.1661.166 1.8781.878 8.7918.791 4.68×4.68× 3232 0.9590.959 1.5091.509 7.0027.002 4.64×4.64× Caveats: the honest pool is the fresh seed-family spot-check pool (Appendix T), whose pool-max τ at k=32k=32 is 1.5091.509, not the headline τreal=1.13 _real=1.13 on the original n=64n=64 pool (which stored only scalar joint-z, foreclosing a post-hoc k-sweep). The attacker is the white-box Qwen3 pure-MSE point, which at joint-z=7.00z=7.00 is weaker than the E4-v2 black-box optimum (4.194.19); the k-sweep bounds but does not fully characterise the strongest-attacker frontier across k. Appendix Q Public/Secret Probe Split We partition the 9696 Qwen3 probes 64/3264/32 into public / secret hold-out (seed 4242), re-run the white-box attacker (E-E αjz=1.0 _jz=1.0, λutil=0.1 _util=0.1, r=64r=64, 30003000 steps) with ℒjzL_jz restricted to the public 6464 probes, and evaluate separately. Table 12: Public/secret probe hold-out for the strongest Qwen3 white-box operating point. 6%6\% gap between public and secret: no systematic overfit, and secret hold-out is slightly harder. Both remain ≳36× \!36× above τpool=1.509 _pool=1.509. Public (n=64n=64) Secret (n=32n=32) joint-z 54.5654.56 57.9257.92 per-probe max 137.8137.8 145.8145.8 margin vs. τpool=1.509 _pool=1.509 36.2×36.2× 38.4×38.4× Empirical public/secret gap is 6.2%6.2\%, comparable to seed variance, so the attacker neither generalises better nor worse to probes it was not trained against. A more aggressive evaluation would apply the same split to the strongest (black-box E4-v2, joint-z=4.19z=4.19) attacker; this requires retraining to capture probe-filtered gradients and falls outside the r=64r=64 white-box frontier evaluated here. Appendix R Mask-Flip Sensitivity We audit the stability of the top-3232 indices by randomising a fraction f per probe. At full E3-v2 attack strength (α=1α=1) AUC remains 1.001.00 for f∈0,0.05,0.10,0.20,0.40f∈\0,0.05,0.10,0.20,0.40\. At α=0.001α=0.001-weakened attackers (mixtures αM′+(1−α)Mα M +(1-α)M), worst-case AUC drops from 0.6130.613 to 0.5440.544 across the 40%40\% flip range (Table˜13). Monotone degradation is a sanity check; production deployment should combine secret-probe rotation and DP feature sketches (both compatible with Equation˜2 without protocol change). Table 13: Mask-flip sensitivity at α=0.001α=0.001 weakened attackers. AUC computed against nhon=2000n_hon=2000 honest draws and 2020 seeds per flip fraction. Attacker f=0f=0 f=0.05f=0.05 f=0.10f=0.10 f=0.20f=0.20 f=0.40f=0.40 Qwen2.5-1.5B 0.6130.613 0.6090.609 0.5980.598 0.5860.586 0.5440.544 Phi-3.5-mini-inst. 0.6240.624 0.6200.620 0.6110.611 0.5930.593 0.5510.551 OLMo-2-7B 0.9420.942 0.9370.937 0.9350.935 0.9190.919 0.8780.878 Qwen3-0.6B 0.6570.657 0.6530.653 0.6400.640 0.6230.623 0.5700.570 Mean 0.7090.709 0.7050.705 0.6960.696 0.6800.680 0.6360.636 Appendix S Gemma σ Recalibration The initial Gemma-2-2B pilot calibrated per-feature σ on 88 honest configurations (22 dtypes × 22 kernels × 22 positions); on a fresh n=64n=64 honest pool several per-feature σi,j _i,j sat at the numerical floor, so bf16-level drift produced 1000+×σ1000+×σ excursions on those features (joint-z≈58z≈58 at positions 0/11). A recalibration to 3232 configs (22 dtypes × 22 kernels × 44 positions × 22 seeds) shifted σ-tail mass (fraction below 10−310^-3: 0.42%→0.10%0.42\%→0.10\%) and yielded τGemmajoint=1.09τ^joint_Gemma=1.09 (nhon=64n_hon=64, 0/640/64 exceedances, CP 95%95\% upper 0.04570.0457). Protocol guidance: σ calibration for long-tailed SAE feature distributions (Gemma-Scope has p95σ=13.76p_95σ=13.76, max=624 =624) requires a calibration pool that spans every free axis (dtype, kernel, position, seed). Appendix T Multi-Kernel Honest-Pool Spot Check The original Qwen3 n=64n=64 pool ran on WSL2/CUDA, exposing only the MATH SDPA kernel. We re-run on Modal L4 to exercise the two SDPA backends dispatchable under our 1+31+3-companion opening format (MATH and EFFICIENT_ATTENTION); the spot check covers 2424 configs per kernel (22 dtypes × 44 positions × 33 seeds 300,301,302\300,301,302\). Table 14: Multi-kernel honest-pool joint-z on Qwen3-1.7B, Modal L4, seeds 300−302300-302. MATH and EFFICIENT_ATTENTION medians agree to 4.6%4.6\%; seed-family variance is the dominant source. Kernel n min median max p99 MATH 2424 0.4890.489 0.9580.958 1.4401.440 1.4401.440 EFFICIENT_ATTENTION 2424 0.4950.495 1.0021.002 1.5091.509 1.5091.509 Pooled fresh 4848 0.4890.489 0.9590.959 1.5091.509 1.5091.509 Original pool 6464 0.2430.243 0.6710.671 1.1201.120 1.1201.120 Pooled all 112112 0.2430.243 0.6770.677 1.5091.509 1.5091.509 The companion-seed family dominates: seeds 100−107100-107 give median 0.6710.671 whereas seeds 300−302300-302 give median 0.958−1.0020.958-1.002, a ∼43% \!43\% shift that swamps the 4.6%4.6\% kernel effect. 24/11224/112 pooled configs exceed τreal=1.13 _real=1.13, which is why headline claims use τpool=1.509 _pool=1.509 throughout. The strongest Qwen3 attacker (E4-v2 r=64r=64) lands at joint-z=4.19z=4.19; margin 2.78×2.78× under τpool _pool (vs. 3.71×3.71× under τreal _real). Wall-clock for the 4848-config run was 343343 s on one L4 (∼$0.08 \! 0.08). Appendix U Heuristic Explanation for the r=16r=16 Result Post-hoc empirical intuition, not a formal bound. In our Qwen3 setup, the public-corpus minimum-norm φ on npub=2000n_pub=2000 paired samples has ‖φ‖F=0.59\| \|_F=0.59. In the r=16r=16 LoRA runs, substitute-side perturbations of order ‖Δh‖∞∼1\| h\|_∞ 1 project through this φ to per-element changes of order ‖Δproj‖∞∼10−2\| \|_∞ 10^-2 on the honest-side residual stream — under the top-3232 SAE mask, insufficient to flip the mask on the tested probes. A formal capacity bound, mask-flip threshold, and scaling curve in npubn_pub remain future work. Appendix V Feature-Forgery Infeasibility (Proposition 4.2 in detail) This section provides the derivation of Proposition 4.2, the empirical intrinsic-dim numbers, and the F3 exact-optimum solver used in Section˜5.7. The claim is structural: an attacker who does not run M cannot beat τpool _pool regardless of optimisation budget. Setup. Fix the probe library (Si,μi,σi)i=1||\(S_i, _i, _i)\_i=1^|P| with ||=96|P|=96, |Si|=k=32|S_i|=k=32. Attacker picks T⊆0,…,dSAE−1T \0,…,d_SAE-1\ with |T|=k|T|=k and a value map v:T→ℝv T quantised to bf16. Per-slot penalty is pi,s(T,v)=|v(Si,s)−μi,s|/σi,sp_i,s(T,v)=|v(S_i,s)- _i,s|/ _i,s if Si,s∈TS_i,s∈ T, else |μi,s|/σi,s| _i,s|/ _i,s. Joint-z is z(t)=1||k∑i,spi,s(T,v)z(t)= 1|P|k _i,sp_i,s(T,v), which is separable in T. Per-feature gain. For a candidate feature f, let ℰ(f)=(i,s):Si,s=fE(f)=\(i,s):S_i,s=f\ be the set of probe-slot occurrences. The gain from including f at value v is G(f,v)=∑(i,s)∈ℰ(f)|μi,s|−|v−μi,s|σi,s,G(f,v)= _(i,s) (f) | _i,s|-|v- _i,s| _i,s, maximised at the weighted-median v∗(f)=wmed(μi,s,1/σi,s)(i,s)∈ℰ(f)v^*(f)=wmed\( _i,s,1/ _i,s)\_(i,s) (f) (convex ℓ1 _1 fit). Write G∗(f)=G(f,v∗(f))G^*(f)=G(f,v^*(f)) and γ(f)=G∗(f)/|ℰ(f)|γ(f)=G^*(f)/|E(f)| for the per-occurrence gain rate. Since the objective decomposes by feature, the attacker’s optimum is the top-k in G∗G^*: z∗=1||k(∑i,s|μi,s|σi,s−∑f∈T∗G∗(f)),T∗=arg⁡max|T|=k∑f∈TG∗(f).z^*= 1|P|k ( _i,s | _i,s| _i,s\;-\; _f∈ T^*G^*(f) ), T^*= _|T|=k _f∈ TG^*(f). Lower bound (Prop. 4.2). The total slot count is ||k|P|k; each of the attacker’s k features can cover at most |ℰ(f)||E(f)| slots. Let m¯=meanf|ℰ(f)|=||k/U m=mean_f|E(f)|=|P|k/U be the average multiplicity. A uniform upper bound on attacker coverage is km¯=k2||/Uk m=k^2|P|/U, so 1−k2||/(||k⋅U)=1−k/U1-k^2|P|/(|P|k· U)=1-k/U of the slots are forced to the zero-penalty floor c=mediani,s|μi,s|/σi,sc=median_i,s| _i,s|/ _i,s. In deployed numbers (U=1472U=1472, k=32k=32, ||=96|P|=96): attacker coverage rate ≈k/U⋅m¯=32⋅2.09/1472≈0.045≈ k/U· m=32· 2.09/1472≈ 0.045 of slots, so (1−0.045)c≈0.955⋅22.78=21.75(1-0.045)\,c≈0.955· 22.78=21.75 is the uniform-coverage lower bound. A tighter (data-dependent) bound evaluates the actual greedy optimum: it picks T∗T^* by G∗(f)G^*(f), achieving coverage 3.84/323.84/32 slots per probe on average and per-probe zi≥11.25z_i≥11.25, yielding the reported zgreedy∗=37.11=24.6τpoolz^*_greedy=37.11=24.6 _pool. Empirical library measurements. Computed offline from the deployed 9696 probes + σ-calibration (pilots/p1_probe_lib/20_intrinsic_dim_analysis.py): ||⋅k|P|· k total memberships 3,0723,072 |μ/σ||μ/σ| min/p5/med 0.00.0 / 0.00.0 / 22.7822.78 U=|⋃iSi|U=| _iS_i| 1,4721,472 |μ/σ||μ/σ| mean/max 57.3857.38 / 2,3502,350 membership mean / median / max 2.092.09 / 11 / 6666 greedy T∗T^* coverage / probe 3.843.84 / 3232 features in ≥2≥ 2 probes 379379 / 1,4721,472 greedy per-probe ziz_i min/med 11.2511.25 / 34.1234.12 pairwise overlap mean / pairs with 0 3.523.52 / 1,0641,064 of 4,5604,560 zgreedy∗z^*_greedy (Prop. 4.2) 37.1137.11 Exact-optimum solver vs. bound. The closed-form solver (e18_forgery_f3_local.py) picks T∗T^* by G∗(f)G^*(f) and assigns bf16-quantised v∗(f)v^*(f) to each, returning z∗=44.24z^*=44.24 (29.3τpool29.3 _pool) on Qwen3 and z∗=221.64z^*=221.64 (203τGemma203 _Gemma) on Gemma. The gap between the bound (37.1137.11) and the exact optimum (44.2444.24) reflects the fact that the bound used slot-averaged c without accounting for bf16 rounding, weighted-median fit, and that the greedy attacker’s coverage is not uniform. Critically, no gradient-descent, Gumbel-softmax, or longer-horizon discrete search can improve on z∗=44.24z^*=44.24 in our formulation, because the objective is separable and the inner optimisation admits the exact per-feature formula above. F4 (y-consistency) subsumption. A further tier F4 requires attacker-served y to be decodable text. Because the feature-library lives in SAE-feature-ID space and not in token space, the attacker’s optimal top-3232 is independent of y content. Any F4 constraint on y can only restrict the attacker’s strategy space and therefore zF4≥zF3∗=44.24z_F4≥ z^*_F3=44.24. F4 was therefore not run as a separate experiment. F2 (proxy-model) omission. F2 would run a smaller proxy (e.g. Qwen2.5-0.5B) through the M-matched SAE. But the M-SAE (dhidden=2048d_hidden=2048 for Qwen3-1.7B) cannot be applied to a proxy with mismatched hidden dimension, and a projection layer takes the attacker into F3/F4 territory. Since F3 is already the exact infimum of joint-z over discrete feature-fabrication strategies, F2 cannot produce a lower joint-z than F3. We report F2 as formally dominated and omit the experiment. Library-rotation spot check (Section˜5.7). Rather than building a larger probe pool (e.g. 256256 probes), we simulate rotation by cross-validating the F3 solver: attacker-train on 4848 probes, test on held-out 4848. Over 5050 random folds, train median joint-z is 40.2940.29 and test median is 52.6152.61 (+12.3+12.3 transfer gap; both 0/500/50 below τpool _pool). A session-time rotation draw from a larger pool inherits this gap structurally. The script is e19_library_split_generalization.py. (a) Feature-forgery ladder (log x-axis). Prop. 4.2 LB = 37.137.1 (dashed) and τpool _pool, τGemma _Gemma (dotted). (b) F3 train vs. held-out over 5050 random 48/4848/48 splits. Transfer gap Δmedian=+12.3 \,median=+12.3 joint-z. Figure 13: Feature-forgery attack ladder and library-rotation transfer gap for Qwen3 + Gemma. Even the exact-optimum discrete F3 attacker (top) clears τ by 29×29× on Qwen3; rotation (bottom) adds a further +12.3+12.3 transfer gap on held-out probes. Appendix W Weaker SVIP Attackers Section˜5.6 assumes the strongest SVIP attacker: full routing of each probe query to M. Weaker SVIP attackers share the same side-channel as long as the probe is separated in time from the commit. (i) Probe batching: accumulate probes and answer in a single batch on M. (i) Cached M responses: pre-compute M responses for a verifier-visible probe set. Both share the acceptance envelope of Figure˜5(a). Appendix X Caveat Index Table 15: Limitations index. Cluster → evidence → residual uncertainty → compatible mitigation. The mitigation column is not empirical: it marks which of the paragraphs below already sketches a concrete hardening path compatible with Equation˜2. “FW” == future work. Cluster Evidence Residual uncertainty Mitigation sketch FW? SAE/backend drift p95p_95 L2 drift 6.5%6.5\%, CV 1.9%1.9\% (Section˜3) recalibrate when outside env. per-probe σi _i + dtype×kernel grid partial MATH-kernel monoculture n=64n=64 MATH + n=48n=48 multi-kernel seeds 300–302 companion-seed coverage dominates pool to τpool=1.509 _pool=1.509 (CP 0.02640.0264) no Session-level FPR within-session ρ=0.88ρ=0.88; Gaussian-copula 1.9%1.9\% at k=4k=4 non-Gaussian tails unmeasured empirical MVN/copula sim (App. G) partial Gemma calibration nhon=64n_hon=64, v2 σ (3232 configs) cross-probe corr. unaudited τGemma=1.09 _Gemma=1.09 (CP 0.04570.0457) partial Black-box adaptive r≤64r≤64, ≤3000≤\!3000 steps, joint-φ (E4-v2) r>64r>64, longer, full-FT k−L−Nk-L-N margin stable (4.64−4.99×4.64-4.99×) yes White-box adaptive (E-E) SAE frozen, gradient through ReLU; 3 restarts at (0,0)(0,0) gradient zeroed by ReLU sparsity “lowest” 4.99>4.194.99>4.19 black-box (this paper) yes Gemma utility frontier one λutil=0 _util=0 catastrophic point λutil>0 _util>0 sweep r=64r=64 Gemma λ-sweep yes Public-probe overfit 64/3264/32 public/secret split, attacker z within 6%6\% secret-probe-aware attacker unmeasured rotation + DP noise (App. R) yes Mask-flip probe rotation up to 40%40\% flip, AUC 1.001.00 at strongest atk; worst-case 0.540.54 at α=0.001α=0.001 principled rotation schedule per-epoch secret subset (compatible w/ Equation˜2) yes Economic binding (not crypto) 0.97×0.97× parallel-commit + Prop. 4.2 structural floor heterogeneous HW / KV-share TEE attestation / append-only log yes Feature fabrication w/o M F3 exact-optimum 44.2444.24 (29.3τ29.3τ), Prop. 4.2 LB 37.1137.11 (App. V) library-independent attackers library rotation (Section˜5.7) no SVIP redesign variant not evaluated verifier-nonce-after-service untested +1+1 round-trip, closes parallel-serve yes Appendix Y Out-of-Scope Strategies and Rebuttal Map Table 16: Strategies considered. OOS = out of scope for this paper. Adversary class Status Why / what would be needed Substitute-serve, no adaptation in-scope Detected at τpool=1.509 _pool=1.509 on all 77 tested attackers. Parallel-commit (serve M′M , commit M) in-scope (one stack) Costs 0.97×0.97× honest wall-clock on RTX 3090; multi-GPU and KV-cache cost models OOS. Public-corpus φ + LoRA, r=16r=16, 30003000 steps in-scope Held-out per-category joint-z at ≥12×τpool≥ 12× _pool. Library-aware joint φ + LoRA, r=64r=64, 15001500 steps (Qwen3 + Gemma-2-2B + Gemma-2-9B) in-scope Four Qwen3 operating points + one Gemma-2B point (Table˜4) + one Gemma-2-9B point (row 2626, Section˜5.5). Higher-rank LoRA (Qwen3 r=128r=128, MSE/StageA) in-scope Joint-z=6.42z=6.42 (4.25×τpool4.25× _pool; rows 19a/19b19a/19b). White-box joint-z gradient LoRA (E-E), r=64r=64, 30003000 steps, 3 restart seeds in-scope Best Qwen3 restart joint-z=4.99z=4.99 — weaker than black-box (Table˜10). Higher-rank LoRA (r>128r>128) / full finetune beyond tested points OOS Frontier experiment; needs rank×steps×npub× n_pub sweep. Stronger white-box (secret-probe-aware, r>64r>64, >3000>\!3000 steps, full-finetune) OOS Evaluated E-E covers r=64r=64, 30003000 steps, public-probe joint-z. Full-model retraining matched to published library OOS Requires library rotation analysis + retraining-cost model. Non-linear φ (e.g. MLP) OOS Heuristic in App. U covers only the linear min-norm fit. Non-audited sessions OOS Per-session FPR/coverage depends on the audit-sampling policy. Three-backbone replication (Qwen3-1.7B + Gemma-2-2B + Gemma-2-9B) in-scope App. E, F; 1.71.7–99B range, scale-stable τ. Multi-tuple (≥4≥\!4) / flagship-class (≥70≥\!70B) generalisation OOS Sweep over backbones / layers / hook variants; 7070B SAEs not publicly released at the time of this work. Empirical SVIP side-by-side comparison in-scope Section˜5.6; weaker SVIP in App. W.

Instant research discovery