Paper deep dive

Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens

Matteo He, William F. Shen, Xinchi Qiu, Nicholas D. Lane

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 238

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/3/2026, 3:32:23 AM

Summary

The paper introduces Sparse Readout Prism (SRP), a method that decomposes the language model's unembedding matrix (readout) into sparse features using dictionary learning. This approach allows for explaining logit-lens scores in terms of readout features rather than tokens, addressing the ambiguity of token identities and the corpus-conditionality of fitted lenses. SRP demonstrates higher reconstruction fidelity for logit differences compared to geometric baselines and provides a corpus-independent control for analyzing intermediate model computations.

Entities (8)

Sparse Readout Prism · method · 98%Unembedding Matrix · model-component · 95%Corpus Conditionality · concept · 93%Logit Lens · method · 92%Gemma 4 · model · 90%Jacobian lens · method · 90%Qwen3.5 · model · 90%Sparse Autoencoder · method · 88%

Relation Signals (6)

Sparse Readout Prism → decomposes → Unembedding Matrix

confidence 95% · we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights

Sparse Readout Prism → uses → Sparse Autoencoder

confidence 95% · SRP fits a sparse autoencoder to the rows of W U

Sparse Readout Prism → outperforms → geometric baselines

confidence 92% · Reconstructs 8.9-17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows.

Sparse Readout Prism → evaluatedon → Qwen3.5

confidence 90% · The suite spans Qwen3.5... We fit one readout SAE per model... and evaluate it

Sparse Readout Prism → evaluatedon → Gemma 4

confidence 90% · The suite spans... Gemma-4... and evaluate it

Jacobian lens → exhibits → Corpus Conditionality

confidence 90% · Changing a Jacobian lens’s fitting corpus changes the language it reports for identical hidden states

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:A language model's prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading reflects both the hidden state and the readout (the unembedding matrix) used to decode it. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states. We call this dependence corpus conditionality. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses. Replacing the original readout with SRP's sparse approximation reconstructs 8.9-17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows. Ablating features shifts logit differences in proportion to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

234,746 characters extracted from source content.

Expand or collapse full text

Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens Matteo He, William F. Shen, Xinchi Qiu, Nicholas D. Lane Department of Computer Science and Technology, University of Cambridge A language model’s prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading reflects both the hidden state and the readout (the unembedding matrix) used to decode it. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states. We call this dependence corpus conditionality. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses. Replacing the original readout with SRP’s sparse approximation reconstructs 8.9–17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows. Ablating features shifts logit differences in proportion to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses. Correspondence: mh2274@cam.ac.uk Code: github.com/hematteo/sparse-readout-prism Dictionaries: huggingface.co/hematteo/sparse-readout-prism 1 Introduction In a transformer language model, each output logit is a dot product between the normalized final state presented to the language model head and one row of the unembedding matrixW U , also called the LM head. We call this matrix the readout, since the model reads out its prediction from the final state alone. Yet states at earlier layers already carry information about the eventual output. Methods in the lens style (nostalgebraist, 2020; Belrose et al., 2023; Pal et al., 2023; Ghandeharioun et al., 2024) exploit this, tracking how that information accu- mulates by decoding those states through the same readout. These readings are often taken as evidence about the computation behind them, as when pre- dominantly English tokens are interpreted as an English intermediate working language (Wendler et al., 2024; Schut et al., 2025). The logit lens ap- pliesW U unchanged, while fitted lenses such as the tuned lens (Belrose et al., 2023) and the Jaco- bian lens (Gurnee et al., 2026) first transport the state into the space the LM head reads, through a map estimated from a corpus. In either case, the lens shows which tokens receive high scores. Inter- preting those scores as evidence about intermediate computation treats token identity as a proxy for the readout structure through which the state is decoded. Figure 1 shows why this proxy is ambiguous. The logit lens scoresbugaboveinsecton a prompt about software, but every sense ofbug shares one row of the LM head, so the score alone cannot distinguish a software defect from an insect. Conversely, near-synonyms and translation equiva- lents occupy different rows that can share dominant directions, so distinct tokens can report the same underlying structure. Token identity is too coarse to distinguish senses of the same token and too fine to identify structure shared across tokens. A second ambiguity arises for a fitted lens, whose output can reflect the fitting corpus as well as the state. We find that lenses fitted on English and on Chinese report different languages for the Preprint. Under review.1 arXiv:2609.01936v1 [cs.CL] 1 Sep 2026 01020 logit bug insect 18.9 −1.5 A Logit lens: bug − insect = +20.37 012 signed contribution to bug − insect (logit units) defect/defects/缺陷 Crash/crash/crashes debug/Debug 漏洞/bug/Bug Issue/issue/Issues fix/Fix/fixing +1.6 f21804 дефек · flaw · défaut +1.0 f43419 crashed · Panic · smash +1.0 f5680 debugger · 调试 · verbose +0.8 f39674 vulnerability · loophole +0.6 f52267 이슈 · isu · vấn +0.6 f63433 fixes · 固定 · fixé B Sparse Readout Prism: the same margin, decomposed sparse feature sum +20.43 ⋅ residual −0.06 Figure 1: Where a lens reading comes from. (A) The logit lens scoresbugagainstinsectfor Qwen3.5-2B on the prompt “The programmer reproduced the crash and filed a”. (B) SRP decomposes that margin into signed readout feature contributions, in logit units. The leading features all concern software defects, with no feature for the insect sense among them. Bars give each feature’s id and the rows of the LM head that score highest on it, several of them non-English (§5). Small terms and token form terms (casing, spacing, and punctuation variants) are hidden. The sum and residual use the full decomposition. same fixed state (§5). The inference from reported tokens to a working language is therefore underde- termined. We introduce Sparse Readout Prism (SRP), which changes what a lens reports, from token identities to readout features, while leaving how it transports the state unchanged. The features reveal a new unit of analysis for lens readings, exposing readout structure that token identities can obscure. Because they are built from the readout shared by every lens, they also support comparisons across tokens, contexts, layers, and lenses. SRP applies dictionary learning directly to the rows of the readout and writes each row as sparse coefficients over a shared overcomplete basis plus a residual, and a readout feature is one direction of that basis. A single row can carry several features and a single feature can appear in many rows, so the features describe the organization of the read- out itself. A lens scores a token through one row of the readout and a margin between two tokens through the difference of their rows, so with each row written this way the score splits into one signed contribution per feature plus a residual (§3.2). SRP resolves thebugambiguity of Figure 1 within one token. In the software context the mar- gin favoringbugrests on defect features, and in an insect context the same token draws its support from features for the animal sense instead (Fig- ure 3). The cross-lens analysis takes up the con- verse case, where the same readout feature carries different reported tokens (§5). The paper makes two contributions, one method- ological and one empirical. 1.SRP (§3) provides one basis per model, fit once from the readout weights and shared by every lens and layer. It reconstructs 8.9 to 17.3 per- centage points more of the tested logit differ- ences than the strongest of six baselines built on row geometry, and ablating a feature shifts the score in proportion to its contribution (§4). 2. We show that a fitted lens’s report depends on the corpus it was fit on, a dependence we call corpus conditionality. Changing a Jacobian lens’s fitting corpus changes the language it re- ports for identical hidden states, while the domi- nant readout feature remains stable across fitting languages, across two lens constructions, and when the compared languages share a script. The basis therefore separates a change in the state from a change in the instrument (§5). Together the results caution against treating to- ken rankings as direct evidence about intermediate computation. They also supply the control such inferences need, a reference fixed in the weights before any reading is taken. 2 Related Work Methods in the lens style.The logit lens applies the unembedding unchanged (nostalgebraist, 2020), while the tuned lens, Future Lens, Patchscopes, and the Jacobian lens learn or transport the map first (Belrose et al., 2023; Pal et al., 2023; Ghande- harioun et al., 2024; Gurnee et al., 2026). All of them report in token identity, and SRP changes that unit without changing the lens, so differently fitted lenses can be compared on the same state (§5). Output geometry and sparse row dictionaries. Weight tying (Press and Wolf, 2017; Inan et al., 2017; Lopardo et al., 2026), softmax and target geometry (Yang et al., 2018a; Zhao et al., 2024), and tokenizer segmentation (Bostrom and Durrett, 2020) give the unembedding rows shared structure. Sparse coding of static word embeddings recov- ers interpretable codes from rows at the type level 2 (Murphy et al., 2012; Faruqui et al., 2015; Subra- manian et al., 2018; Arora et al., 2018). SRP carries this approach to the rows of a trained transformer’s LM head, judged by replacement and by recon- struction of selected scores. Row neighborhoods, clusterings, and principal directions summarize the same rows, but none of them is constrained to sum to the logit or margin under analysis. SRP satisfies that constraint by construction, and §4.2 measures how much of a selected score each method recov- ers. Activation SAEs and decompositions in param- eter space. To our knowledge, sparse autoen- coders (SAEs) have not previously been fit to a trained transformer’s static weights, only to its acti- vations. The two fits decompose opposite factors of the same score, since a lens score takes the form ̃ h ⊤ ℓ q, a decoded state against a readout directionq. Activation SAEs learn one overcomplete basis per layer for the hidden statesh ℓ behind the decoded state (Bricken et al., 2023; Huben et al., 2024; Gao et al., 2025a; Templeton et al., 2024), while SRP applies the same machinery toqitself, fitting one basis that serves every lens and layer. An activation SAE also depends on the corpus that produced its hidden states, so judging one lens reading against another through its basis would introduce a third instrument conditioned on a corpus. The cross-lens analysis of §5 instead needs a reference that does not move with the lens, the prompt, or the corpus. Decompositions in parameter space also factorize weights (Braun et al., 2025; Gao et al., 2025b), whereas SRP ties its factorization to the scores a lens reads. Appendix P extends this section with lens vari- ants, sparse feature infrastructure, readings of latent language, the geometry of the output space, and attribution to mechanisms. 3 Sparse Readout Prism Whether the score is a token’s logit or a margin be- tween two tokens, a lens reports it in token identity, and SRP resolves the same score into the readout features that push it up or down and by how much (Figure 2). The features come from a sparse dictio- nary learned from the rows of the LM head (§3.1). The contributions and an explicit residual sum to the score exactly (§3.2), one fixed dictionary makes decompositions comparable across contexts, layers, and lenses (§3.3), and fidelity diagnostics accom- pany every decomposition (§3.4). q α = P i β i (α)d i + r α s α = P i c i + ̃ h ⊤ ℓ r α q α β i (α)d i r α ̃ h ⊤ ℓ r α Figure 2: Sparse Readout Prism, schematically. Left. For coefficientsαover unembedding rows, SRP de- composes the selected readout directionq α into signed feature directionsβ i (α)d i and a residualr α . Right. For decoded state ̃ h ℓ , the scores α ( ̃ h ℓ )decomposes into one contribution per feature and ̃ h ⊤ ℓ r α , all in logit units. Blue bars supportA, orange bars supportB, and hatch- ing marks the residual (§3.2). Throughout,h ℓ ∈R d model denotes the hidden state at layerℓbefore a lens decodes it, andd model is the dimension of the model’s hidden states. Each lens applies a mapT ℓ that covers everything the lens does before the unembedding, namely the model’s fixed output normalization composed with whatever transport the lens adds, in the order the lens applies them. The logit lens adds no trans- port, so its map is that normalization alone, the model’s final LayerNorm or RMSNorm, and ev- ery lens reduces to the same map at the final layer ℓ = L. A lens is fitted whenT ℓ carries parameters estimated from data. A tuned lens adds the affine translator it fits, and the Jacobian lens compared in §5 adds an averaged Jacobian fitted on a corpus. We write ̃ h ℓ = T ℓ (h ℓ ) ∈R d model for the decoded state presented to the LM head. The unembedding matrixW U ∈R |V|×d model over the vocabularyV then produces the logitsW U ̃ h ℓ . Rowv, written as the column vectorw v := W U [v, :] ⊤ , gives tokenv the linear score logit v ( ̃ h ℓ ) = ̃ h ⊤ ℓ w v . Gemma applies a nonlinear logit softcap after this linear score, and Appendix F handles the softcap separately. SRP factorizesW U alone, and a lens changes only the decoded state the rows are scored against. 3.1 Factorizing the LM Head into Sparse Readout Features SRP fits a sparse autoencoder to the rows of W U (Bricken et al., 2023; Huben et al., 2024; Gao et al., 2025a). Each row then decomposes as w v = bw v + r v withbw v = μ + P D i=1 z v,i d i , where 3 bw v is the fitted row,r v is the residual,μis a shared offset,Dis the dictionary width,d i ∈R d model is a readout feature direction, andz v,i is the sparse coefficient of rowvon featurei. In SAE termsμ is the decoder bias andd i is a decoder direction. We call this dictionary the readout SAE. With the readout feature directionsd i fixed, the projection p i ( ̃ h ℓ ) = ̃ h ⊤ ℓ d i measures how strongly the decoded state aligns with feature i. We fit TopK sparse autoencoders (Makhzani and Frey, 2014; Gao et al., 2025a), whose encoder keeps theklargest pre-activations in each row, sok sets the sparsity budget directly. SRP requires only a sparse factorization of the rows, and any sparse autoencoder variant could supply it. Our reported operating points fixk = 256at 32×expansion, and individual score decompositions are sparser still (Appendix D). We center the rows and normalize their norms before fitting, the same preprocessing that TopK autoencoders apply to activations. Both steps are inverted afterward, so every term lies in the raw row space ofw v , in original logit units. Appendices C– F give width, sparsity, and preprocessing details. 3.2 Decomposing Selected Readout Scores For two tokensA,B ∈ V, the margin is the signed logit differencem A,B ( ̃ h ℓ ) = ̃ h ⊤ ℓ (w A −w B ) , and token logits and margins are the simplest scores we evaluate. We handle every score family through a coefficient vectorα ∈R |V| over unem- bedding rows, with the selected readout direction q α = P v∈V α v w v and the selected readout score s α ( ̃ h ℓ ) = ̃ h ⊤ ℓ q α , shortened below to the selected direction and the selected score. Settingα A = 1 and all other coefficients to zero recovers the token logitlogit A ( ̃ h ℓ ) = ̃ h ⊤ ℓ w A for a vocabulary token A∈V, and weightingbugby plus one andinsect by minus one gives the margin between those two tokens. A selected score whose coefficients sum to zero is a contrast, and any score component shared by the compared rows cancels, including components arising from output embedding geometry or tok- enization (Press and Wolf, 2017; Inan et al., 2017; Yang et al., 2018a; Bostrom and Durrett, 2020). Ev- ery margin is a contrast, and the remaining families replace either side of a margin with a uniform av- erage over a token set, or take the margin between the token the original LM head ranks first at the evaluated state and a competitor it ranks lower. Because each row decomposes into a fitted row plus a residual, every selected direction inherits that split. For a selected directionq α , define the row residual r α = P v∈V α v r v . Substituting the fitted reconstruction row by row and exchanging the sums gives s α ( ̃ h ℓ ) = P v α v ̃ h ⊤ ℓ μ + P D i=1 P v α v z v,i ̃ h ⊤ ℓ d i + ̃ h ⊤ ℓ r α For readout featurei, defineβ i (α) = P v∈V α v z v,i and the signed contributionc i ( ̃ h ℓ ,α)= β i (α)p i ( ̃ h ℓ ), so the middle sum above carries one signed contribution per feature. The resulting sum of shared offset, signed contri- butions, and residual is what we call a local score decomposition, shortened below to a decomposi- tion. The offset term( P v α v ) ̃ h ⊤ ℓ μ vanishes for every contrast, whose score then reduces to the signed contributions plus the residual. We writes recon for the reconstructed score, which sums the shared offset and signed contri- butions above. The exact score is s exact = s α ( ̃ h ℓ ), and the residual term isε = s exact − s recon = ̃ h ⊤ ℓ r α . Table 3 (Appendix A) instantiatesq α andβ i (α) for each score family we use. Before SRP is ap- plied, each evaluation fixes the decoded state, the coefficient vector, the competitor set, and the tok- enization filters that screen out brittle token cases (Appendix G.7). 3.3 Comparing Decompositions Across Contexts and Lenses Eachβ i (α)is fixed by the coefficient vectorαand the readout SAE, while the projectionsp i ( ̃ h ℓ )vary with the decoded state (Appendix L.6). For a fixed selected direction, two decompositions therefore differ only through the decoded state, and a change of context, layer, or lens supplies nothing else. De- compositions of different selected directions share the coordinate system and differ in their coeffi- cients. The dominant feature of a score is the fea- ture whose signed contribution is largest in absolute value, and two decompositions agree at the feature level when they share a dominant feature. 3.4 Reporting Fidelity Diagnostics The residual absorbs whatever the fitted rows miss, so the decomposition is exact by construction, and 4 we report three fidelity diagnostics with every de- composition to measure how well those rows stand in for the originals. Every display shows the resid- ual termεbeside the contributions (Appendix F.1). We apply the same diagnostics to simpler alterna- tives to test whether SRP reconstructs scores better than row geometry alone does (§4.2). Replacement fidelity works at the model level and measures whether held-out readout distribu- tions and rankings are preserved whenW U is re- placed by the reconstructed LM head d W U , the stacked fitted rows (§4.1). A dictionary can recon- struct rows accurately in aggregate yet reconstruct the selected logits poorly, so the remaining two diagnostics work at the level of individual scores. Relative reconstruction error weighs the residual against the size of the score it belongs to and is reported at two settings of the constant δ, where ρ δ ( ̃ h ℓ ,α) = |s exact − s recon | |s exact | + δ . Aggregate summaries useρ 0.5 ( ̃ h ℓ ,α), whose con- stant floors the denominator, while case study dis- plays and the baseline comparisons use the stricter unflooredρ 0 ( ̃ h ℓ ,α) =|ε|/|s exact |, the plain ratio of residual to score, and no table mixes the two settings (Appendix A). Sign agreement asks whethers exact ands recon carry the same sign, since either ratio bounds the er- ror’s magnitude but not its direction, and a flipped sign reverses which side of a contrast the recon- struction supports. Coverage combines the two, as the fraction of scores whose reconstruction matches the sign ofs exact and holdsρ 0 below one half, and we call those scores covered. 4The Decomposition Is Faithful, Predicts Intervention Effects, and Survives Retraining Setup. The suite spans Qwen3.5 (Qwen Team, 2026), Gemma-4 (Google DeepMind, 2026a,b), Ministral (Mistral AI, 2026), and Qwen/Llama readouts distilled for reasoning (DeepSeek-AI, 2025) at 0.8B–9B scale. We fit one readout SAE per model to the rows of its final LM head and evaluate it on the decoded states of 10,000 C4 con- tinuations (Raffel et al., 2020; Dodge et al., 2021), none of which enter the fit, and on the same banks of selected scores for every model,∼1,350 scores per model drawn from∼850 contrasts with their prompts fixed. The rewordings and context variants Table 1: Readout replacement and reconstruction of selected scores against baselines built on row ge- ometry. Q/Min/R1-Q/R1-L=Qwen3.5 / Ministral-3 / R1-Distill-Qwen / R1-Distill-Llama. top-1 is the frac- tion of held-out decoded states whose reconstructed and original readouts agree on the argmax. The SRP and “best alternative” columns give coverage, the fraction of the full bank of logit differences reconstructed with the correct sign and error below half the score’s magnitude, with 95% CIs from a cluster bootstrap. The best alterna- tive is the strongest of six such baselines, and lead is the difference in percentage points. Metric definitions and the full method grid are in Appendix G.2, full model names in Appendix B. Modeltop-1SRPBest alternativeLead Q-0.8B0.8910.754 [.733, .773]0.665 [.645, .686]+8.9 Q-2B0.8870.780 [.759, .798]0.625 [.606, .644]+15.5 Q-9B0.9000.812 [.792, .831]0.639 [.618, .658]+17.3 Min-8B0.9040.777 [.759, .795]0.648 [.629, .665]+12.9 R1-Q-7B0.7600.716 [.694, .738]0.579 [.560, .598]+13.7 R1-L-8B0.7540.751 [.728, .772]0.624 [.606, .642]+12.7 of one contrast stay together when the bootstrap re- samples. A positive margin says the readout favors AoverB, correctness is a separate question, and scores with small margins remain in the aggregates. All eight models enter the reconstruction and replacement analyses (Appendix E.4). The baseline and intervention comparisons cover the six softcap- free readouts, whose scores are a plain linear map of the decoded state (Table 1). The two Gemma-4 readouts sit behind a nonlinear logit softcap that SRP does not factorize, and the split was fixed on that ground before any contrast was scored. The retraining analysis of §4.4 trains several dic- tionaries per setting, so it is run on Qwen3.5-2B, the model the worked cases use throughout. In those cases the score exceeds its reconstruction error, the reconstruction keeps its sign, and the con- tribution mass is compact enough to list, the regime Appendix G.6 describes in full. The labels on the bars of the worked cases sum- marize each feature’s top unembedding rows and stay separate from the measured feature ids and signed contributions. Three independent blinded audit runs (Karvonen et al., 2025; Makelov et al., 2025) rate the large majority of the labels that sup- port a claim as coherent (Appendix L.8). Appen- dices B and G.7 describe the suite, bank families, provenance, and filters. 4.1 The Sparse Head Replaces the Readout On the six softcap-free readouts a readout SAE fitted to the rows of an LM head reproduces that head as a predictor over the vocabulary, row by 5 row, and on the selected scores. ReplacingW U with the reconstructed LM head preserves the held- out argmax on 0.89–0.90 of decoded states for the Qwen3.5 and Ministral readouts and on 0.75–0.76 for the two readouts distilled for reasoning (Ta- ble 1). The median KL between the original and reconstructed readout distributions splits the same way, at 0.09–0.14 bits for the Qwen3.5 and Minis- tral readouts and 0.43–0.49 bits for the two distilled for reasoning. Row reconstruction alone does not predict replacement, since the two Gemma-4 read- outs behind the softcap nearly match the six on explained variance over the centered rows while replacing far worse. Replacement metrics and row reconstruction for each readout are in Table 5 and Table 14. At the level of individual scores, sign agreement is at least0.91across all eight readouts, and the residual error and the few sign flips both concen- trate on the scores whose margin sits nearest a tie (Table 20, Appendix Figure 13, Appendix Ta- ble 18). Contrasts built from benchmark formats hold up at least as well, and the bank of 300 such contrasts, the only bank whoseAside comes from a gold label, reaches sign agreement 0.94 and cov- erage 0.84 on Qwen3.5-2B (Rajpurkar et al., 2018; Yang et al., 2018b; Koreeda and Manning, 2021; Guha et al., 2023; Siddiq and Santos, 2022) (Ap- pendix L.4, Table 36). 4.2SRP Outperforms Every Baseline Built on Row Geometry SRP is fitted to the rows ofW U , so the test that matters is whether its codes add anything to the geometry those rows already carry. We compare against six baselines built directly on that geometry, from methods that use the rows with no dictionary at all to dictionaries fitted by clustering. They are nearest row ridge and weighted kNN on the 128 closest rows, k-means centroid dictionaries at two widths, hard assignment of each row to a single centroid, and PCA at 256 components, each scored for coverage on the same banks under the same harness (§3.4, Appendix G.3). SRP outperforms all of them, reaching coverage 0.72–0.81 across the six softcap-free readouts, and the intervals do not overlap (Table 1). It also has the lowest mean, 95th-percentile, and maximum absolute error of the seven methods, so the ranking does not depend on how the error is summarized (Appendix G.4). To remove capacity as a confound, we build the k-means dictionary at SRP’s own width and spar- Table 2: Predicted versus measured local changes on the readout side. For each contrast, a feature’s predicted contributionc i = β i (α)p i ( ̃ h ℓ )is compared with the measured margin change when ̃ h ℓ is ablated along the unit decoder directiond i , with no new forward pass. Per model,∼260 bank contrasts, each with its top 10 features plus 10 random directions as controls. We reportr 2 and the slope of the regression of measured change on predicted contribution through the origin, over covered pairs with bootstrap 95% CIs, and random is the control r 2 . Modelr 2 [95% CI]SlopeRandomn Q-0.8B0.886 [.869, .903]1.6610.0101610 Q-2B0.834 [.802, .862]1.3380.0071820 Q-9B0.829 [.813, .845]1.6860.0581860 Min-8B0.926 [.913, .937]0.9570.0092050 R1-Q-7B0.918 [.899, .933]1.0820.0001500 R1-L-8B0.934 [.920, .947]1.1260.0341210 sity,D=65,536withk=256active centroids per row, and it trails SRP by 18 points of coverage on Qwen3.5-2B. Shuffled codes and random support leave coverage below 0.10 on the three Qwen3.5 and both Gemma-4 readouts (Table 19). 4.3Removing a Feature Moves the Margin as the Decomposition Predicts A decomposition is meant to say how much of the margin each feature accounts for, so removing that feature’s direction from the decoded state should shift the margin with the sign and roughly the size of its contribution. We test this by removing the direction, measuring the margin with the original readout, and comparing the measured change with the predicted one. The measured changes track the predicted ones atr 2 = 0.83–0.93on all six softcap- free readouts, against at most 0.06 when a random direction is removed instead, so the contributions say which features move the margin most and in which direction (Table 2, Appendix J). The size of the change matches as well on Ministral and the two readouts distilled for reasoning (slope 0.96– 1.13), while on the three Qwen3.5 readouts the margin moves further than predicted (slope 1.34– 1.69). 4.4 The Decomposition Survives Retraining We test whether a decomposition survives retrain- ing, since a dictionary trained from a new seed can fit the readout equally well with different de- coder directions. At each of two widths,32×and 16×, we train three dictionaries for Qwen3.5-2B that differ only in their seed. All six replace the readout equally well, at a held-out top-1 agree- ment of 0.805–0.848, so the comparisons below 6 are between dictionaries of equal fidelity. What we compare is the tokens a dictionary groups with each token of a contrast. For a contrast betweenA andB, pooling the top unembedding rows of the features whose coefficients most favor each side’s row gives the tokens the dictionary groups withA and withB. These lists depend on the two rows and the dictionary alone, and we compare them across dictionaries on all 63 distinct token pairs among the bank’s two-token margins. At the32×width, a feature that any of these lists draws on has a median cosine of∼0.33 between its direction and the nearest direction in another seed’s dictionary, so the directions themselves do not recur across seeds, as is documented for TopK dictionaries (Paulo and Belrose, 2026). Yet for the same token of the same contrast, the lists from two seeds share a mean Jaccard of 0.21–0.24 across the two widths, while the lists for two unrelated con- trasts share 0.014–0.018. The tokens a dictionary groups withAtherefore reproduce across seeds at 13–15×the overlap that unrelated contrasts show at the same width, and every one of the 63 contrasts lies above the 90th percentile of that unrelated over- lap. Across both widths and all seed pairs, 87–91% of the features these lists draw on find a coun- terpart in the other seed’s dictionary whose top rows overlap theirs beyond what the same search yields for random rows of matched frequency, so the row groups behind the features recur as well. Appendix I gives the full tables for both widths and shows that the variation grows when the prepro- cessing recipe changes along with the seed. 4.5 Context Decides Which Features Carry a Token So far we have shown that the decomposition stands in for the readout, predicts what removing a feature does to a margin, and survives retraining. We now ask what it adds to the score a lens already reports. Every sense ofbugshares one row of the LM head, so a lens score for the token cannot tell a software defect from an insect. SRP writes that row vas fixed coefficientsz v,i over the readout features, so from one context to the next only the projec- tionsp i ( ̃ h ℓ )change (§3.3). The context therefore decides which readout features carry the score, and those features tell the two senses apart. Worked cases. In Figure 3 we scorebugon Qwen3.5-2B in an insect context and in a software 1.00.50.00.51.0 f59571 M f13081 Error f58330 _error f18302 Beetle f4095 bee f36 mosquito -1.0 -0.7 -0.5 +0.6 +0.6 +1.1 A. Insect: bug - error 1.00.50.00.51.01.52.0 f59571 M f23681 M f34693 p f5680 debug f43419 Crash f21804 defect -0.7 -0.5 -0.4 +1.0 +1.0 +1.6 B. Software: bug - insect Figure 3: One token, carried by different features in different contexts. Signed readout feature terms in logit units forbugon Qwen3.5-2B, scored in (A) an insect context against the anchorerror, chosen for the software sense, and (B) a software context against insect. Green terms supportbugand orange terms oppose it. Bars carry each feature’s id and the rows of the LM head that score highest on it. context, each time against a token from the other sense so that the leading terms belong to the sense in play. In the insect context, againsterror, the margin is carried by features whose top rows are mosquito, bee, and beetle, with relative reconstruc- tion errorρ 0 = 0.185. In the software context of Figure 1, againstinsect, it is carried by defect, crash, and debug features (ρ 0 = 0.003), and no feature for the insect sense appears among the lead- ing terms. Appendix L.2 gives the displays and the terms for each case. Held-out senses.The worked cases show the two senses of one word carried by different features, and we test whether the same holds for 20 am- biguous words on CoarseWSD-20 (Loureiro et al., 2021), a benchmark with labelled senses and a fixed split of contexts into training and test. For each sense of a word we pick the eight features whose mean contribution on the training contexts differs most between that sense and the others. We then assign each test context to the sense whose eight features add up to the largest contribution. Both steps use feature ids and signed contributions only, so the labels on the bars play no part. Balanced accuracy is 0.90, 0.92, and 0.80 on Qwen3.5-2B, Qwen3.5-9B, and R1-Llama-8B, where shuffling the sense labels gives a null near 0.41. Every word beats its majority baseline, and the result holds whether we keep one, two, four, or eight features 7 per sense. A nearest centroid probe on the full decoded state reaches 0.96, 0.97, and 0.92, so the full state is the stronger predictor of the sense and eight features give a readable summary of it (Ap- pendix H). Attribution and edits. Because the readout fac- torizes independently of the hidden state, the same terms compose with direct logit attribution (Elhage et al., 2021; Wang et al., 2023; Nguyen, 2024), so that each layer’s contribution to a margin resolves into one term per feature (Appendix M). The terms also supply candidate directions for constrained edits on the readout side. At matched next-token KL those directions beat the mean and leading prin- cipal components of the rows they were discov- ered from on Qwen3.5-2B, while the mean wins on Qwen3.5-0.8B and Qwen3.5-9B (Appendices N and N.1). With the basis read on its own evidence, the next section applies it to a question token readings leave open. 5 A Fitted Lens Reports the Language of Its Fitting Corpus A fitted lens maps an intermediate hidden state into the space consumed by the LM head. The resulting tokens are often interpreted as evidence about the information represented at that layer, including the language in which the model is operating. For ex- ample, Gurnee et al. (2026) obtain English tokens from intermediate states for a Chinese prompt and interpret them as evidence that the intermediate computation occurs in English. Because this map is learned from a corpus, its token readings can reflect both the hidden state and the fitting corpus, and a token ranking alone cannot distinguish these sources. We call this dependence corpus conditionality. The fitting corpus affects the lens through the learned map ̃ h ℓ = T ℓ (h ℓ ) (§3), so our results apply only to fitted lenses. Studies of latent language that decode states directly through the unembedding use no fitting corpus (Wendler et al., 2024; Schut et al., 2025) and are therefore not subject to this dependence. Appendix P discusses a separate confound in token readings that affects those studies. 5.1 Design We test corpus conditionality by holding an inter- mediate hidden stateh ℓ fixed and varying only the corpus used to fit the lens. The two fitted maps can produce different decoded states from the same state, and hence different token readings, so any difference between the readings is attributable to the fitting corpus. To test whether different tokens reflect different readout structure, SRP decomposes both selected token scores in a single basis learned only from the model weights. Translation pairs, language labels, and fitting corpora do not enter this basis, which provides a reference independent of the fitting corpus. Using the reference implementation, we fit two Jacobian lenses (Gurnee et al., 2026) on Qwen3.5- 9B. The first uses 100 English C4 prompts, and the second uses 100 Chinese C4 prompts. We hold seeds, layers, and all other procedures fixed. We apply both lenses to the same frozen states from 80 evaluation prompts spanning seven task families. We also evaluate 12 controls—six digit prompts and six proper noun prompts—whose expected out- puts are language invariant. We evaluate layers 21, 24, 26, and 29, where the two lenses can produce different token readings. At each layer, we decompose the score assigned to the token reported by each lens and record the fea- ture with the largest contribution in absolute value (§3.3). We count a prompt as agreement when the same feature dominates under both lenses in at least two of the four layers. Requiring two of the four layers keeps a match at one layer alone from count- ing as agreement. We use two null comparisons to estimate chance agreement under this rule. One evaluates an unrelated token under the same lens, and the other shuffles prompts before pairing the two lenses. Appendix K provides the full protocol, token and feature agreement rates for each family, and results for each prompt. 5.2 Different Tokens Share a Dominant Feature Across layers 21, 24, 26, and 29, the two lenses re- port tokens in different scripts on 39 of 80 prompts, while their outputs converge at the final layer. On factual recall prompts, the lenses report different scripts on 9 of 10 (Figure 4). On the antonym prompt from Gurnee et al. (2026), the lens fitted on English reportslargeat layers 24 and 26 and bigat layer 29. The lens fitted on Chinese instead reports大的(“large” or “the large one”) at layers 24 and 26 and大(“large” or “big”) at layer 29. Interpreting either lens alone would therefore yield opposing conclusions about the model’s intermedi- ate language. 8 中国的首都是北京。英国的首都是 ... one frozen state ̃ h 26 English-fitted J-lens “London” 38.5 logits largest other feature +1.7 Chinese-fitted J-lens “伦敦” 39.2 logits largest other feature +1.6 ≠ f23180 dominant under both lenses bars give f23180’s share of each lens’s feature sum 75%63% Figure 4: Two lenses, two surface languages, one readout feature. The figure shows a Chinese factual recall prompt, which translates as “The capital of China is Beijing. The capital of the UK is . . . ”. For the same frozen Qwen3.5-9B state at layer 26, the Jacobian lens fitted on English reportsLondon, whereas the lens fitted on Chinese reports伦敦(“London”). Below each token is its logit under the corresponding lens. The bar shows the share of the lens’s feature sum attributable to the shared readout feature f23180, which dominates both scores. The feature sum is the signed sum of the feature contributions and excludes the offset and the residual. 0.25.50.751 dominant-feature agreement EN–ZH Jacobian main study EN–DE Jacobian same script EN–ZH ridge second lens family Jacobian vs ridge EN corpus, cross-family 77/80 84/90 67/80 76/80 floor: unrelated tokenfloor: shuffled pairing Figure 5: Feature agreement across language pairs and lens constructions. A prompt counts as agreement when the same feature dominates under both lenses in at least two of layers 21, 24, 26, and 29. Points show agreement rates, intervals show 95% binomial CIs, and counts appear at the right. Two baseline markers per setting show the matched null rates. All eight are at most 3/159 (1.9%). Appendix K provides tables for each family and construction details. The same SRP feature dominates under both lenses on 77 of 80 prompts (96%, 95% CI [0.90, 0.99]). In Figure 4, f23180 accounts for 75% of the feature sum for the lens fitted on English and 63% for the lens fitted on Chinese. Every other feature contributes at most+1.7logits. Neither null comparison produces a match, with 0/159 for unrelated tokens and 0/240 after shuffling prompts. The dominant feature agrees under both lenses on 5/6 digit controls and 5/6 proper noun controls. 5.3 The Pattern Persists Across Languages, Lens Constructions, and Corpus Sizes We extend the corpus conditionality analysis across a second lens construction, a language pair that shares a script, and larger fitting corpora. In each extension we test whether the token readings change while the same readout feature remains dominant. Figure 5 reports the agreement rates and matched null comparisons. Lens construction. We fit ridge translators re- lated to the tuned lens (Belrose et al., 2023) on the same English and Chinese corpora of 100 prompts. On 9 of 10 EN→ZH translation prompts, the trans- lator fitted on English reads Latin tokens and the translator fitted on Chinese reads CJK tokens. The same feature remains dominant under both transla- tors on 67/80 prompts (CI [0.74, 0.90]), compared with 3/159 and 1/240 matches under the two null comparisons. We then hold the English fitting cor- pus fixed to isolate the effect of lens construction. The Jacobian lens and ridge translator agree on the dominant feature for 76/80 prompts, and both null comparisons yield zero matches. Shared script.We next test English and German, which share the Latin script, and exclude cognates from the evaluation set. The Jacobian lenses fitted on English and German report different top-1 to- kens on 73/90 prompts. The same feature remains dominant on 84/90 prompts, compared with 1/204 and 0/270 matches under the two null comparisons. The dissociation therefore does not depend on the distinct scripts used in the English–Chinese com- parison. Corpus size.Finally, we triple each fitting corpus from 100 to 300 prompts to test whether the origi- nal corpus size accounts for the disagreement. Fea- ture agreement remains 77/80 for English–Chinese and increases from 84/90 to 85/90 for English– German. Token disagreement moves in opposite directions, falling from 39/80 to 33/80 for English– Chinese and rising from 73/90 to 76/90 for English– German. Estimation noise that decreases with ad- ditional fitting data would predict a decrease for both language pairs. Across the two fitting scales, eleven English–Chinese prompts change their re- ported language, and none changes its dominant feature. On Qwen3.5-9B, the token readings vary with the language and size of the fitting corpus, while the dominant readout feature remains stable across 9 these changes and across two lens constructions. 6 Future Work Safety audits can use SRP wherever the question admits a concrete readout score. A study can de- fine a group contrast for refusal, source verification, or a specified lexical family, compare its feature contributions across paraphrases, languages, ad- versarial prompts, and checkpoints, and treat the features that recur as nominations for causal tests or constrained interventions. The evaluation should report reconstruction error, transfer, distributional cost, and effects on other outputs, so the safety claim stays tied to the selected score. Appendix O.5 develops the methodological extensions, among them tracing feature contributions back to the com- ponents that supply them, joint activation and read- out factorizations, cross-model dictionary align- ment, and decompositions that cover more of the output than one selected score. 7 Conclusion Sparse Readout Prism offers an alternative unit for interpreting lens readings. Token identity is too coarse where one token carries several readout fea- tures and too fine where several tokens share one. SRP covers 8.9 to 17.3 percentage points more of the tested logit differences than the strongest of six baselines built on row geometry, and the tokens it groups with a contrast reproduce across seeds at 13 to 15 times the overlap of unrelated con- trasts. It shows what token rankings cannot, since lenses fitted on different corpora report different surface languages for the same hidden state while the same readout feature stays dominant under both. Lens studies therefore gain a reference fixed in the weights, one that tells a change in the state apart from a change in the instrument. These features describe the readout itself, com- plementing sparse dictionary methods that decom- pose hidden states. One row carries several senses and one feature spans surface forms and scripts, so the organization of a model’s output space becomes measurable from its weights alone. Projecting a decoded state onto the features then shows how the state aligns with that organization layer by layer, before any token is read off it. One dictionary per model serves every lens and layer, so a reading from any of them decomposes into the same terms with no further fitting, and those terms give au- diting work a control for what the fitting corpus contributed to a reading, along with terms for layer- by-layer attribution and candidate directions for constrained edits on the readout side. Limitations Fitting and basis choice. SRP adds a one-time sparse autoencoder fit for each model, taking roughly 6–12 GPU-hours on one NVIDIA A40 in our experiments. As in other sparse dictionary methods, individual decoder directions depend on dictionary size, sparsityk, training recipe, and seed (Elhage et al., 2022; Karvonen et al., 2025; Makelov et al., 2025; Paulo and Belrose, 2026). We therefore select operating points from fidelity sweeps and report stability across seeds and recipes (§4.4, Appendix I). Empirical scope.The cross-lens study evaluates Qwen3.5-9B across English–Chinese and English– German, two constructions of fitted lenses, and multiple fitting budgets. It establishes in these con- trolled settings that the surface language a lens reports can depend on the corpus it was fit on. The study of edits to the readout is run on Qwen3.5-2B, with cross-model losses reported in Appendix N. Interpretation boundary.SRP is designed to ex- plain selected readout scores and the organization of a model’s readout. Questions at the level of cir- cuits or of generation require interventions beyond the analyses on the readout side here. Feature la- bels summarize top unembedding rows and may in- herit tokenizer and vocabulary geometry (Press and Wolf, 2017; Inan et al., 2017; Yang et al., 2018a; Bostrom and Durrett, 2020). Quantitative conclu- sions therefore rest on feature ids, signed contribu- tions, and residual errors, with further discussion in Appendix O. Ethical Considerations This work studies the readouts of pretrained lan- guage models, collects no new data from human subjects, and evaluates no deployed systems. Ex- periments use public checkpoints and held-out de- coded states, including C4 continuations, which may contain offensive, private, copyrighted, or sen- sitive text (Raffel et al., 2020; Dodge et al., 2021). We use C4 only to measure readout reconstruction and make no claims about the social validity of indi- vidual tokens, labels, or generated outputs. Check- points and decoded states derived from C4 are used under the licenses or terms published with their re- 10 lease pages and model cards. We do not redistribute the checkpoints or raw C4 text. The released SRP dictionaries (hematteo/sparse-readout-prism) carry license and usage notes specific to each arti- fact. SRP makes lexical readout directions and row labels easier to inspect, and its stress test of edits to the readout moves specified lexical scores, which is a dual-use property of auditing tools. We report the edit with probes for distribution shift and for effects off target, and keep claims tied to measured quantities. Evaluation of safety filters, moderation, and deployment controls raises questions at the system level that lie beyond this work. References Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cun- ningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others. 2025. Circuit tracing: Revealing computational graphs in language models. Trans- former Circuits Thread. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483–495. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predic- tions from transformers with the tuned lens. Preprint, arXiv:2303.08112. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th Inter- national Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR. Kaj Bostrom and Greg Durrett. 2020. Byte pair encod- ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4617–4624. Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. 2025. Interpretability in parameter space: Minimizing mechanistic descrip- tion length with attribution-based parameter decom- position. Preprint, arXiv:2501.14926. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decompos- ing language models with dictionary learning. Trans- former Circuits Thread. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. BatchTopK sparse autoencoders. Preprint, arXiv:2412.06410. Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. 2025. Learning multi-level features with Matryoshka sparse autoencoders. In Proceed- ings of the 42nd International Conference on Ma- chine Learning, volume 267 of Proceedings of Ma- chine Learning Research, pages 6077–6101. PMLR. Woojin Chung and Jeonghoon Kim. 2025. Exploiting vocabulary frequency imbalance in language model pre-training. In Advances in Neural Information Pro- cessing Systems. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. 2023. Towards automated circuit discov- ery for mechanistic interpretability. In Advances in Neural Information Processing Systems. DeepSeek-AI. 2025. DeepSeek-R1 incentivizes reason- ing in LLMs through reinforcement learning. Nature, 645(8081):633–638. Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, Tianhao Li, Qian Cao, Xuancheng Ren, Xiaodong Deng, An Yang, Fei Huang, Dayi- heng Liu, and Jingren Zhou. 2026. Qwen-Scope: Turning sparse features into development tools for large language models. Preprint, arXiv:2605.11887. Jesse Dodge, Maarten Sap, Ana Marasovi ́ c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1286–1305. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2025. Separat- ing tongue from thought: Activation patching reveals language-agnostic concept representations in trans- formers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 31822–31841. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. 2024. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Process- ing Systems, volume 37. 11 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superpo- sition. Transformer Circuits Thread. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield- Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. 2015. Sparse overcom- plete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1491–1500, Beijing, China. Association for Computational Linguistics. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration problem in training natural language generation models. In In- ternational Conference on Learning Representations. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025a. Scaling and evaluating sparse autoencoders. In International Conference on Learning Representations. Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V. Govande, Bowen Baker, and Dan Mossing. 2025b. Weight-sparse transformers have interpretable cir- cuits. Preprint, arXiv:2511.13653. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Arya- man Arora, Zhengxuan Wu, Noah Goodman, Christo- pher Potts, and Thomas Icard. 2025. Causal abstrac- tion: A theoretical foundation for mechanistic inter- pretability. Journal of Machine Learning Research, 26(83):1–64. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lu- cas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden represen- tations of language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15466–15490. PMLR. Google DeepMind. 2026a.Gemma 4.https:// deepmind.google/models/gemma/gemma-4/. Of- ficial model-family page; accessed 2026-05-20. Google DeepMind. 2026b.Gemma 4 model col- lection.https://huggingface.co/collections/ google/gemma-4. Hugging Face model collection; accessed 2026-05-20. Neel Guha, Julian Nyarko, Daniel E. Ho, Christo- pher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Tal- isman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Ja- son Hegland, and 21 others. 2023. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, volume 36. Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Ma- teusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, T. Ben Thompson, David Abrahams, Subhash Kan- tamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. 2026. Verbalizable representations form a global workspace in language models. Trans- former Circuits Thread. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16410–16430. Arthur E. Hoerl and Robert W. Kennard. 1970. Ridge re- gression: Biased estimation for nonorthogonal prob- lems. Technometrics, 12(1):55–67. Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoen- coders find highly interpretable features in language models. In International Conference on Learning Representations. Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, and Nicholas D. Lane. 2025. DEPT: Decoupled embeddings for pre-training language models. In The Thirteenth International Conference on Learn- ing Representations. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In Interna- tional Conference on Learning Representations. Ian T. Jolliffe. 2002. Principal Component Analysis, 2 edition. Springer. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum Stuart Mcdougall, Kola Ay- onrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. 2025. SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 29223–29264. PMLR. 12 Yuta Koreeda and Christopher D. Manning. 2021. Con- tractNLI: A dataset for document-level natural lan- guage inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919. Benno Krojer, Shravan Nayak, Oscar Mañas, Vaib- hav Adlakha, Desmond Elliott, Siva Reddy, and Marius Mosbach. 2026.LatentLens: Revealing highly interpretable visual tokens in LLMs. Preprint, arXiv:2602.00462. Charles L. Lawson and Richard J. Hanson. 1974. Solv- ing Least Squares Problems. Prentice-Hall, Engle- wood Cliffs, NJ. Patrick Leask, Bart Bussmann, Michael T. Pearce, JosephIsaacBloom,CurtTigges,Noura Al Moubayed, Lee Sharkey, and Neel Nanda. 2025. Sparse autoencoders do not find canonical units of analysis. In International Conference on Learning Representations. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278–300. Association for Computational Lin- guistics. Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. 2024.Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread. Yiting Liu and Zhi-Hong Deng. 2026. Beyond acti- vation patterns: A weight-based out-of-context ex- planation of sparse autoencoder features. Preprint, arXiv:2601.22447. Llama Team. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. Antonio Lopardo, Avyukth Harish, Catherine Arnett, and Akshat Gupta. 2026. Weight tying biases token embeddings towards the output space. In Findings of the Association for Computational Linguistics: ACL 2026, pages 40795–40810. Association for Computa- tional Linguistics. Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021. Anal- ysis and evaluation of language models for word sense disambiguation. Computational Linguistics, 47(2):387–443. Anemily Machina and Robert Mercer. 2024. Anisotropy is not inherent to transformers. In Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pages 4892–4907. Aleksandar Makelov, Georg Lange, and Neel Nanda. 2025. Towards principled evaluations of sparse au- toencoders for interpretability and control. In Inter- national Conference on Learning Representations (ICLR). Alireza Makhzani and Brendan Frey. 2014. k-sparse au- toencoders. In International Conference on Learning Representations (ICLR). Julian Minder, Clément Dumas, Caden Juang, Bilal Chughtai, and Neel Nanda. 2025. Overcoming spar- sity artifacts in crosscoders to interpret chat-tuning. In Advances in Neural Information Processing Sys- tems. Mistral AI. 2026.Ministral 3.Preprint, arXiv:2601.08584. Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceed- ings of COLING 2012, pages 1933–1950, Mumbai, India. The COLING 2012 Organizing Committee. Thong T. Nguyen. 2024.Logit prisms: Decom- posing transformer outputs for mechanistic in- terpretability.https://neuralblog.github.io/ logit-prisms/. nostalgebraist.2020.Interpreting GPT:Thelogitlens.https://w. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. Charles O’Neill, Christine Ye, Kartheik Iyer, and John F. Wu. 2024. Disentangling dense embeddings with sparse autoencoders. Preprint, arXiv:2408.00657. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wal- lace, and David Bau. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. In Pro- ceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548– 560. Mihir Panchal, Deeksha Varshney, Mamta, and Asif Ekbal. 2026. Indic-TunedLens: Interpreting multi- lingual models in Indian languages. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 172–185, Rabat, Mo- rocco. Association for Computational Linguistics. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geome- try of large language models. In Proceedings of the 41st International Conference on Machine Learning. Yagyensh Chandra Pati, Ramin Rezaiifar, and P. S. Kr- ishnaprasad. 1993. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the Twenty-Seventh Asilomar Conference on Signals, Sys- tems and Computers, volume 1, pages 40–44. 13 Gonçalo Paulo and Nora Belrose. 2026. Sparse au- toencoders trained on the same data learn different features. In International Conference on Learning Representations (ICLR). Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaish- navi, Apoorv Saxena, and Koustava Goswami. 2025. Beyond logit lens: Contextual embeddings for ro- bust hallucination detection & grounding in VLMs. In Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 9661–9675, Albuquerque, New Mexico. Association for Compu- tational Linguistics. Ofir Press and Lior Wolf. 2017. Using the output embed- ding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Qwen Team. 2026.Qwen3.5 model collec- tion.https://huggingface.co/collections/ Qwen/qwen35. Hugging Face model collection; ac- cessed 2026-05-20. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. Preprint, arXiv:2407.14435. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784–789. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in English? Preprint, arXiv:2502.15603. William F. Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas D. Lane. 2025. LLM unlearning via neural activation redirection. In Advances in Neural Information Processing Systems. Mohammed Latif Siddiq and Joanna C. S. Santos. 2022. SecurityEval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applica- tions for Privacy and Security, pages 29–33. Felix Stollenwerk, Anna Lokrantz, and Niclas Hertzberg. 2026. Output embedding centering for stable LLM pretraining. Preprint, arXiv:2601.02031. Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. 2018. SPINE: SParse Interpretable Neural Embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 4921–4928, New Orleans, Louisiana, USA. AAAI Press. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Cal- lum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, and 7 others. 2024. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small. In International Con- ference on Learning Representations (ICLR). Zhenyu Wang. 2025. LogitLens4LLMs: Extending logit lens analysis to modern large language models. Preprint, arXiv:2503.11667. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? On the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-Math tech- nical report: Toward mathematical expert model via self-improvement. Preprint, arXiv:2409.12122. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018a. Breaking the softmax bottleneck: A high-rank RNN language model. In In- ternational Conference on Learning Representations. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b.HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 2369–2380. Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao, Qinli Yang, and Junming Shao. 2020. Revisiting rep- resentation degeneration problem in language mod- eling. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 518–527. 14 Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. 2024. Implicit geometry of next- token prediction: From language sparsity patterns to model representations. In First Conference on Language Modeling. 15 Appendix Roadmap Contents ASRP Score Decomposition Identities (p. 16). Readout SAE reconstruction identities, selected readout directions, reconstruction error, sign agreement, and fidelity diagnostics. BModel Suite and Selection Rationale (p. 17). Model-suite rationale, evaluated model families, tar- geted comparison rows, model roles, and interpretation details. CHow the Readout SAEs Were Selected (p. 18). Selection criteria, metric glossary, operating regimes, final dictionary settings, and sweep evidence map. DQwen Readout SAE Recipe and Capacity Sweeps (p. 19). Qwen3.5-9B recipe selection, capacity sweeps, and native operating-point claims. EReadout SAE Transfer and Model-Family Comparisons (p. 23). Smaller Qwen/Gemma transfer, cross-family rows, R1-distilled comparisons, and row-norm tail analysis. FReadout SAE Diagnostics and Reproducibility (p. 26). Support-selection diagnostics, rejected alternatives, denominator controls, and training/evaluation recipe details. GFidelity Checks and Controls (p. 28). Fidelity results with bootstrap intervals, C4 replace- ment, baselines and nulls, nearest row reading, token- level audits, and operating regimes. HSense-Labelled Evaluation (p. 33). Alignment against a permutation null that repeats the selection, sparsity references, and scope. IStability Across Independent Training Runs (p. 35). Cross-seed explanation stability, feature-group match- ing, held-out recovery of the stable core, and the disclo- sure across recipes. J Predicted Versus Realized Local Readout-Side Changes (p. 37). Intervention-validation protocol, falsifiers, six-model results, and slope diagnosis. K Cross-Lens Study Protocol, Controls, and Per-Family Results (p. 37). Corpus conditionality protocol, agreement per family, null floors, worked example, and scope. LAdditional Qwen Displays and Feature Audits (p. 40). Margin examples, examples of selected scores, fixed- token contexts, profiles, lens/prism comparisons, and token row audits. M Feature-Resolved Direct Logit Attribution (p. 47). Connection to direct logit attribution, DLA terms split by SAE feature, and additivity error. NConstrained Readout-Side Edit Test (p. 48). Constrained edit for SAE decoder directions on the readout side, held-out terms, cross-model repeats, and the matched-KL frontier with cross-model losses. ODiscussion and Future Work (p. 53). Interpretation details, model-family results, feature- resolved DLA terms, and extensions. PExtended Related Work (p. 54). Additional lens-method context and output-embedding related work. Table 3: Selected scores and their sparse-feature co- efficients. Rows instantiate the selected readout direc- tionq α and corresponding direction coefficientβ i (α). Barredwandzdenote uniform token-set averages, and for top competitor margins, ̄w π,R = P r π r w r and ̄z π,R,i = P r π r z r,i , whereπ r are the nonnegative com- petitor weights summing to one over the competitor set R. HereA,B ∈ Vare single tokens,A,B ⊂ Vare token groups, anduis the top-ranked token under the original LM head. Selected score family q α β i (α) Token logitw A z A,i Vocabulary-mean contrast w A − ̄w V z A,i − ̄z V,i Logit differencew A − w B z A,i − z B,i Group contrast ̄w A − ̄w B ̄z A,i − ̄z B,i Top competitor margin w u − ̄w π,R z u,i − ̄z π,R,i A SRP Score Decomposition Identities This appendix states the decomposition identities used by the figures and control tables and fixes the bookkeeping conventions for displaying a trained readout SAE. The identities are the same local score decomposition introduced in §3.2. All re- ported scores are evaluated in the model’s raw read- out space. If training used centering or row normal- ization, the SAE coefficients, reconstructed rows, and residual terms are first mapped back to that space. A.1 Raw-Space Row Decomposition Letw v ∈R d model be the column-vector form of unembedding rowv, following §3. Under the row preprocessing convention used by the reported anal- yses, x v = w v − μ a v , a v > 0, x v = X i z pre v,i d pre i + r pre v , witha v = 1for unnormalized rows. The raw-space decomposition then uses z v,i = a v z pre v,i , d i = d pre i , r v = a v r pre v . so that w v = μ + X i z v,i d i + r v . The displayed local score decomposition contains only this shared offset, SAE feature terms, and the explicit residual term, with no hidden dense base term added. 16 Why the offset stays outside the dictionary. Nothing forbids folding the shared offset into the dictionary, sinceμis a direction inR d model like any other and the row reconstruction could be writ- ten with one additional atom. We keep it explicit because that atom would carry coefficient one on every row by construction, so it would occupy one of thekactive slots for every token and spend sparse capacity on a quantity that is identical every- where. Its decoder direction would also be drawn toward the row mean, away from structure that distinguishes rows. An explicit centering term is the standard arrangement in sparse autoencoder practice (Bricken et al., 2023; Gao et al., 2025a), and centering output embeddings in particular is a documented stabilizer (Stollenwerk et al., 2026). Holdingμoutside the sparse code lets the features model deviations from the shared row mean, and it keeps the offset a separately visible term in the identity for a selected score below, where its can- cellation becomes a property of the selected direc- tion, independent of which atoms happened to be learned. A.2 Selected-Score Identity For a hidden stateh ℓ , let ̃ h ℓ = T ℓ (h ℓ )be the corre- sponding state presented to the LM head, following §3. A selected readout direction is a linear function of vocabulary rows, q α = X v∈V α v w v , s α ( ̃ h ℓ ) = ̃ h ⊤ ℓ q α . Define the direction coefficient, selected-direction residual, and SAE feature contribution as β i (α) = X v α v z v,i , r α = X v α v r v , c i ( ̃ h ℓ ,α) = β i (α)p i ( ̃ h ℓ ), wherep i ( ̃ h ℓ ) = ̃ h ⊤ ℓ d i is the readout feature pro- jection of §3.1. Substituting the raw-space row decomposition gives the exact grouped identity s α ( ̃ h ℓ ) = s μ ( ̃ h ℓ ,α) + s feat ( ̃ h ℓ ,α) + s resid ( ̃ h ℓ ,α), s μ ( ̃ h ℓ ,α) = X v α v ! ̃ h ⊤ ℓ μ, s feat ( ̃ h ℓ ,α) = X i c i ( ̃ h ℓ ,α), s resid ( ̃ h ℓ ,α) = ̃ h ⊤ ℓ r α . For zero-sum contrasts, including pairwise logit differences, group contrasts, vocabulary mean con- trasts, and top competitor margins, the offset term cancels. Figure 2 depicts the decomposition. A.3 Error, Sign, and Display Rules For a selected readout directionq α , the exact and reconstructed scores are s exact = ̃ h ⊤ ℓ q α , s recon = s μ ( ̃ h ℓ ,α) + s feat ( ̃ h ℓ ,α). The residual score is the reconstruction error, ε( ̃ h ℓ ,α) = s exact − s recon = s resid ( ̃ h ℓ ,α), and aggregate summaries use the parameterized relative error ρ δ ( ̃ h ℓ ,α) = |ε( ̃ h ℓ ,α)| |s exact | + δ , withδ = 0.5logits, denotedρ 0.5 , in aggregate summaries. We drop the arguments ofεandρ δ where the state and coefficient vector are clear from context. The floor prevents arbitrarily small margins from dominating median error summaries, and near the decision boundary sign agreement and margin-binned flip rates are the primary reli- ability checks. The unfloored variantρ 0 ( ̃ h ℓ ,α) = |ε( ̃ h ℓ ,α)|/|s exact |is used for identity checks, case- study displays, and all baseline comparisons. It upper-boundsρ 0.5 , and the two subscripts are never mixed within one table. For signed contrasts, side- specific feature support is evidence-bearing when sign(s exact ) = sign(s recon ). Each interpreted figure states the selected readout score, exact score, reconstructed score, reconstruc- tion error, relative reconstruction error, and sign status for contrasts. If a plot displays only the largest bars, the reported feature sum and residual are still computed from the full active feature sum. Token-level figures report tokenization or row au- dit information where it affects interpretation, and brittle single-token contrasts are handled as explicit group contrasts when needed. B Model Suite and Selection Rationale This appendix explains the model suite used to eval- uate Sparse Readout Prism, which spans the Qwen model suite (Qwen Team, 2025, 2026), the Gemma 17 model suite (Google DeepMind, 2026a,b), Minis- tral (Mistral AI, 2026), and targeted DeepSeek-R1 distilled comparison rows (DeepSeek-AI, 2025). Appendices C–F give the metric summary, report- ing the trained widths, nativeksettings (the TopK budget fixed at training), rowEV, top-1, KL, and usage statistics for each readout SAE setting. This appendix motivates the model choices. Sparse Readout Prism factorizes the output- token scoring matrix, so the model suite is orga- nized around readout fidelity, scale, and portability. The suite spans model size and readout geometry while preserving enough budget for tokenizer, row- norm, special-token, and matched-control analyses. B.1 Selection Rationale The Qwen3.5 rows provide a fixed qualitative dis- play setting and Qwen-family companion analy- ses. Qwen3.5-9B is the capacity anchor, large enough to be a meaningful test of readout fac- torization while still supporting recipe and width sweeps. Qwen3.5-0.8B and Qwen3.5-2B test the same recipe at smaller unembedding dimensions and cheaper rerun settings. The Gemma rows provide a second tied-readout family with a different tokenizer and final-logit softcap. These rows exercise the readout SAE recipe under a distinct readout geometry, and the softcap-correct readout evaluation is reported in Appendix E. Additional targeted rows broaden the coverage of readout geometries. Ministral-3-8B-Base tests transfer to an untied, softcap-free 8B-scale readout outside Qwen. R1-Distill-Qwen-7B and R1-Distill- Llama-8B evaluate the same recipe-selection mea- surements on reasoning-distilled comparison rows, where row reconstruction and LM head replace- ment fidelity are visible as separate result axes. B.2 Final Suite Table 4 lists the evaluated models and the role each one plays in the results. Qualitative feature analyses use a single low- error display setting so that examples share the same readout SAE and reporting convention. The choice is evidence-driven, since the additional rows show how the same metrics vary with readout ge- ometry and post-training regime. B.3 Interpretation Details We built the suite to evaluate readout factorization. We report token frequency, length, special-token behavior, row norm, tied embedding status, softcap handling, and matched-score identity bias so single- token logit decompositions can be compared on the same footing. The evaluation makes three commitments. First, model-specific readout layers are handled before reporting decision fidelity. Second, the SAE archi- tecture, training recipe, seed convention, dictionary settings, and evaluation settings are reported with the sweep records in Appendices C–F. Third, pri- mary reported claims are stated for selected readout scores, especially token logits and contrastive logit differences. C How the Readout SAEs Were Selected This appendix documents how the readout SAEs used in the main text are chosen. The selection criterion combines row reconstruction with readout- distribution preservation after substituting d W U for W U , namely rowEV, top-1 agreement, KL in bits, selected-logit reconstruction errors when available, and dead/rare feature usage. A feature is counted as rare when its evaluation firing rate is below10 −3 . All trained runs use seed0unless stated otherwise, and the Qwen3.5-0.8B 32×,k = 256dictionary setting has the three-seed stability window reported below. Metric glossary. rowEV is row-centered ex- plained variance for reconstructed unembedding rows. top-1 is agreement between the original and reconstructed vocabulary argmaxes over held-out decoded states. top5 is mean top-five set over- lap,|top5(W U ̃ h L ) ∩ top5( d W U ̃ h L )|/5 , and as a set-overlap metric it can be lower or higher than top-1. KL is the median over held-out decoded states ofD KL (p W U (· | ̃ h L )∥p d W U (· | ̃ h L ))in bits. Selected-logit reconstruction error is the absolute fractional error on the original top logit. Qwen3.5-9Bunembeddingobject. The Qwen3.5-9B experiments factorize the final unembedding matrixW U ∈R 248320×4096 . The prism loss is the auxiliary training term, weighted λ prism , that scores reconstruction in logit space on decoded states, on top of the row-space objective. The states used for it and for evaluation are the final RMSNorm outputs immediately before the language-model head.Anlm_headpre-hook confirms this, since the captured input reproduces the model readout with no additional post-readout transform. The remaining discrepancy between 18 Table 4: Evaluated model suite and role. Exact dictionary settings and metrics are in Appendices C–F. Model(s)Why includedRole in study Qwen3.5-0.8B Qwen3.5-2BSmall-model transfer and lower-cost repeats under the native k = 128 strict budget and k = 256 high fidelity regimes. Qwen transfer evidence. Qwen3.5-9BScale anchor for recipe selection, capacity sweeps, and high fidelity readout replacement. Qwen fidelity evidence. Gemma-4-E2B Gemma-4-E4B Cross-family evidence for whether the readout SAE recipe transfers beyond Qwen. Cross-family transfer evi- dence. R1-Distill- Qwen-7B Qwen-derived post-training comparison with an untied, softcap-free readout. rowEV vs. fidelity separa- tion evidence. R1-Distill- Llama-8B Reasoning-distilled comparison from a non-Qwen base family (both dictionary settings) for cross-architecture confirmation of the rowEV / fidelity gap. Cross-architecture post- training evidence. Ministral-3- 8B-BaseCross-family 8B-scale comparison with an untied, softcap- free readout (adds an 8B-scale non-Qwen readout geometry). Cross-family fidelity com- parison. reconstruction and readout is consistent with the model’s bf16 precision floor of the LM head as the relevant limit. Selection rule.The active budgetkis part of the trained SAE. We therefore report nativek = 128 strict budget rows and nativek = 256high fi- delity rows as separate operating regimes, and trun- catedk = 256dictionaries are over-budget compar- isons. Within each regime, rowEV is paired with top-1, KL, reconstruction error, achieved eval-L 0 where applicable, and usage statistics. Orthogonal matching pursuit (OMP) rows (Pati et al., 1993) are sparse-inference comparisons over fixed dictionar- ies, while deployed metrics use the learned TopK encoders. Per-model settings.The sweep supports two op- erating regimes. The strict budget rows preserve a nativek = 128interpretability budget, while the high fidelity rows use nativek = 256encoders when readout fidelity is the primary constraint. Ta- ble 5 summarizes these settings, and the model- by-model comparisons supporting them appear in Appendix E.1. C.1 Sweep Coverage and Evidence Map The selection evidence has three distinct roles, choosing a Qwen3.5-9B recipe, testing how that recipe scales with capacity, and evaluating whether the same recipe transfers to smaller Qwen and Gemma models. These roles are separated because the runs differ in model scale, active sparsity, con- vergence stage, and evaluation design. Table 6 maps each family to the decision it supports. The following readout SAE appendices follow this evidence map, taking first the Qwen3.5-9B recipe, then the capacity and operating-point com- parisons, then smaller-model transfer and addi- tional metric analyses. D Qwen Readout SAE Recipe and Capacity Sweeps This appendix gives the Qwen3.5-9B recipe- selection and capacity evidence summarized in Ap- pendix C. D.1 Qwen3.5-9B Fixed-k = 128 Recipe Selection The first Qwen3.5-9B sweep fixes the interpretabil- ity budget atk = 128and selects a no-PCA recipe. A 5k screening sweep identified the recipe family, a 20k continuation evaluated convergence of the selected recipe, and the older 50k matryoshka run (Bussmann et al., 2025) anchored the comparison against the previous long-trained baseline. The se- lected TopK recipe’s own convergence ladder pro- vides the step-count evidence, while the previous recipe is an architectural reference point. Table 7 maps each run to the evidence it supplies, and Ta- ble 8 gives the main effects the sweep isolates. The 9B fixed-k = 128recipe istopk,D = 32768(8×),k = 128, row-seeded initializa- tion,hybrid_50freq_50uniformrow sampling, warmup-cosine learning rate10 −3 → 10 −4 , de- layed prism ramp to10 −3 , 20k steps. It reaches rowEV0.6206, top-10.8457, KL0.2955, selected 19 Table 5: Representative dictionary settings from the Qwen/Gemma sweep. Nativek = 128rows support the strict budget regime and native k = 256 rows support the high fidelity regime. Gemma top-1/KL are softcap-correct. ModelSettingrowEVtop-1KLrare Role Strict-budget regime, native k = 128 Qwen3.5-0.8B16×, k = 1280.760 0.844 0.277 0.001 strict budget result Qwen3.5-2B16×, k = 1280.712 0.858 0.261 0.004 strict budget runner-up Gemma-4-E2B16×, k = 1280.714 0.6231.94 0.031 strict budget, softcap-correct Gemma-4-E4B16×, k = 1280.693 0.6691.82 0.040 strict budget, softcap-correct High-fidelity regime, native k = 256 Qwen3.5-0.8B32×, k = 2560.877 0.891 0.135 0.001 high fidelity result Qwen3.5-2B32×, k = 2560.847 0.887 0.136 0.010 high fidelity runner-up Qwen3.5-9B16×, k = 2560.761 0.874 0.167 0.061 compact 9B comparison Qwen3.5-9B32×, k = 2560.857 0.900 0.105 0.292 reported 9B point Gemma-4-E2B32×, k = 2560.834 0.3336.37 0.038 fidelity row, softcap-correct Gemma-4-E4B32×, k = 2560.827 0.7361.22 0.078 fidelity row, softcap-correct Table 6: Sweep coverage for Appendix C. Run families and the decision each supports, without listing every cell. Sweep familyCoverageAxes variedSelection use Original Qwen3.5-9B baseline unembedding measurements, short- run comparisons, and a 50k run matryoshka width,k, λ anchors the old 50k baseline at rowEV 0.511, top-1 0.807 Qwen fixed-k = 128 recipe search unembedding measurements, short- run recipe comparison, and 20k run init, width, sampling, TopK/matryoshka objective, delayed ramp selects the topk 8×, k = 128 recipe Qwen BatchTopK control (Bussmann et al., 2024) target-L 0 measurements and evaluation-L 0 controls eval-L 0 targeting, random vs. row- seeded init rejects BatchTopK for this setting Qwen capacity sweephigher-kscreening runs, 16×and 32× 20k runs, and OMP comparison width and active budget k compares the 32×,k = 256high- capacity point with the 16×,k = 256 compact point Qwen JumpReLU control (Rajamanoha- ran et al., 2024) four JumpReLU cells and matched TopK controls LR, prism ramp, controller target-L 0 rejects JumpReLU as a reported readout SAE Small-model transfer complete four-model width/kscreen- ing grid plus 20k runs model family, width, k selects representative nativek = 128 andk = 256transfer rows under the shared metric suite Figure 6: Qwen3.5-9B fixed-k = 128recipe-selection frontier. Points are 5k recipe variants and the black curve is the 5k non-dominated frontier. The final 20k TopK recipe improves both reconstruction and KL at the fixed interpretability budget. logit reconstruction error0.027, dead0.000, rare 0.021, and Gini0.256. The 15k→20k rowEV gain was+0.0025, consistent with the stop rule (Fig- ure 7). Recipe-selection conclusion. The selected no- PCA TopK recipe improves over the previous long- trained matryoshka baseline on reconstruction, top- 1 agreement, KL, and feature usage, and its 5k– 20k ladder supports the 20k stop rule. The main drivers are row-seeded initialization, delayed prism ramping, plain TopK, and theD = 32768width choice. D.2 Qwen3.5-9B Capacity Sweep After recipe selection, we vary capacity along dic- tionary width and active sparsity budget. This tar- geted sweep separates two questions that rowEV 20 Table 7: Evidence map for the Qwen3.5-9B fixed-k = 128recipe-selection experiment. Per-run metrics for the 5k screening sweep are provided with Figure 6. Evidence roleRun or run familyWhat it establishes Previous baselineMatryoshka TopK,D = 16384,k = 128, 50k steps Long-trained previous recipe used as an architectural reference, at rowEV0.511, top-10.807, KL0.416, dead0.119, rare 0.236. 5k selected recipe TopK,D = 32768,k = 128, row-seeded, hybrid row sampling, delayed prism ramp Best recipe-region representative in the screening sweep, at rowEV0.567, top- 10.775, KL0.572, reconstruction error 0.038, rare 0.060. 5k screening sweep18 recipe variants varying initialization, architecture, row sampling, prism schedule, and width Supports the selected recipe by matched comparisons, with row-seeded initialization, TopK, delayed ramp, andD = 32768each selected from the screening grid. 20k selected recipeTopK,D = 32768,k = 128, row-seeded, hybrid row sampling, delayed prism ramp Final reported Qwen3.5-9B fixed-k = 128 recipe, at rowEV0.621, top-10.846, KL 0.296, reconstruction error0.027, dead 0.000, rare 0.021. Table 8: Main effects in the fixed-k = 128Qwen3.5-9B recipe sweep. FactorEvidenceConsequence Initialization row_seededwas2.0× randomat 5k (rowEV 0.563vs.0.278atD = 32768). Use row-seeded initialization. Prism-loss schedule Delayed ramp kept rowEV unchanged at 0.567while reducing se- lected logit reconstruction error from0.064to0.038 in the matched TopK hy- brid cell. Rampλ prism to 10 −3 after70% of training. ArchitectureAtD = 32768/5k, plain TopK exceeded matryoshka variants (Bussmann et al., 2025) in rowEV (0.567vs. 0.559/0.554) at about 3.6× lower cost. Use plain TopK for Qwen/Gemma. WidthD = 16384trailed D = 32768by about 0.06rowEV in the fixed- k = 128 recipe sweep. UseD = 32768 for the 9Bk = 128 recipe. alone can blur, namely which nativek = 128read- out SAE should be used for fixed-budget compar- isons, and which nativek = 256readout SAE should be the high fidelity operating point. A 5k screen selected which larger-capacity cells received 20k training. At 16×, thek = 256cell had better rowEV and rare feature rate thank = 192(0.695/0.100vs.0.675/0.164), despite lower 5k top-1, so it was continued. The 32×,k = 256 cell had enough rowEV headroom (0.799) to justify a width continuation. The native 20k rows below Figure 7: Convergence ladder for the selected Qwen3.5- 9B fixed-k = 128recipe. The final 15k→20k rowEV gain is small while decision-fidelity and usage metrics remain stable, which is why we stop at 20k. Figure 8: Qwen3.5-9B capacity sweep. Increasing width and active sparsity improves rowEV and top-1 agreement, while rare feature usage is an additional us- age statistic. provide the evidence for the operating-point claims. Table 9 and Figure 8 show the tradeoff. The D = 131072,k = 256run has the best rowEV, top- 1, and KL in this Qwen3.5-9B capacity sweep. Its rare feature ladder0.300→ 0.294→ 0.292shows a persistent rare-feature tail, which is reported as a usage property alongside fidelity metrics. 21 Table 9: Converged 20k native Qwen3.5-9B capacity rows under the selected recipe family. Nativek = 128and native k = 256 rows are separate operating regimes. RunWidth Eval k rowEV top-1KLrare Interpretation D = 32768, train k = 1288×1280.621 0.846 0.296 0.021 strict budget recipe D = 65536, train k = 25616×2560.761 0.874 0.167 0.061 compact k = 256 comparison D = 131072, train k = 25632×2560.857 0.900 0.105 0.292 best rowEV/top-1/KL, high rare feature rate Table 10: Sparsity budget versus dictionary usage. Ranges are over independently seeded runs at otherwise identical settings (20,000 steps each), and single-seed rows give the one measured value. Dead counts features never active on the evaluation rows, and rare an evaluation firing rate below10 −3 (Appendix C). Width multipliers are relative to d model ,2048for Qwen3.5-2B and1024for Qwen3.5-0.8B. The dead and rare rates are the quantities at issue in these rows, and rowEV, which is not what disqualifies a budget here, is entered as — where the run artifact does not carry it. ModelSettingSeedsDeadRarerowEVtop-1KL Q-2B16× (D = 32768), k = 3210.6940.776—0.7660.476 Q-2B16× (D = 32768), k = 6420.491–0.512 0.632–0.645 0.683–0.684 0.783–0.816 0.367–0.370 Q-2B16× (D = 32768), k = 25620.036–0.037 0.213–0.2180.8010.807–0.848 0.226–0.243 Q-2B32× (D = 65536), k = 12830.336–0.352 0.628–0.637 0.769–0.771 0.816–0.826 0.241–0.253 Q-0.8B16× (D = 16384), k = 3210.6410.687—0.7710.434 Q-0.8B16× (D = 16384), k = 6410.3170.412—0.8010.296 The 16×,k = 256comparison provides the compact high fidelity point. It also improves during convergence, since from 5k to 20k steps its rowEV ladder is0.719 → 0.749 → 0.758 → 0.761, and its rare feature rate falls from 0.100 to 0.061. D.3 Lower Sparsity Budgets Activation SAEs are often trained at smaller ab- solute budgets than ours, which raises the ques- tion of whether a readout SAE would also work at k = 64or below. We retrained Qwen3.5-2B and Qwen3.5-0.8B under an otherwise identical recipe (20,000 steps, batch4,096, same data, schedule, and initialization family, with only width,k, and seed differing) and measured how much of each dictionary is ever used (Table 10). At matched width, lowering the budget strands capacity. Going fromk = 256tok = 64at 16×raises the dead-feature rate from under4%to about half the dictionary (0.491–0.512), while re- construction and replacement both degrade (rowEV 0.80→ 0.68, KL0.23→ 0.37). Halving the bud- get again strands two thirds of the dictionary (0.694 atk = 32), and Qwen3.5-0.8B repeats the progres- sion at its own 16×width (0.317atk = 64,0.641 atk = 32), so the pattern is not specific to one model. The comparison is between budgets at a fixed width, since thek = 128rows sit at twice the width, where the same budget leaves about a third of the dictionary dead. What governs usage is the budget relative to width, which is why we re- port sparsity ask/Dand as the number of features a score actually recruits (Appendix D.4). Seed spread at each setting is small compared with the differences between settings, so the ordering does not depend on the seed. D.4 Operating-Point Claims The final presentation uses two native operating regimes. The strict budget readout uses native k = 128readout SAEs for fixed-budget compa- rability, and after recipe transfer the selected strict budget reconstruction point is Qwen3.5-0.8B 16× (Appendix E.1). The high fidelity readout uses na- tivek = 256readout SAEs when readout accuracy is the limiting concern. On Qwen3.5-9B the re- ported point is the 32×,k = 256run, with rowEV 0.857, top-10.900, KL0.105, and rare0.292, and every 9B number in the main text comes from it. The 16×,k = 256run is retained as a compact comparison, with rowEV0.761, top-10.874, KL 0.167, and rare 0.061. Sparsity of the operating points. Activation SAEs are often reported at smaller absolutek, but kis comparable across dictionaries only relative to width. AtD = 65,536,k = 128andk = 256ac- tivate0.20%and0.39%of the dictionary per row. What a reader inspects is sparser still, because a selected score concentrates its mass on a few fea- tures. Across the six softcap-free readouts, a me- 22 Figure 9: Truncatingk = 256-trained dictionaries back tok = 128can raise rowEV relative to the nativek = 128 recipe while sharply reducing top-1 agreement. This rowEV vs. fidelity inversion motivates treating the fixed interpretability budget as part of the trained object. dian of65–107features carries80%of a score’s ab- solute contribution mass (76on the display model), against dictionaries of tens of thousands. The two operating points therefore differ only in budget, withk = 128the stricter interpretability budget andk = 256prioritizing reconstruction fidelity. Budgets below this range are not simply sparser, since they leave much of the dictionary unused (Ap- pendix D.3). Truncating ak = 256dictionary back tok = 128after training does not substitute for the native recipe, since it can raise rowEV while sharply reducing top-1 agreement (Figure 9). E Readout SAE Transfer and Model-Family Comparisons This appendix reports how the selected read- out SAE recipe transfers across the smaller Qwen/Gemma suite and targeted cross-family or post-training rows. E.1 Transfer to Smaller Models The fixed Qwen3.5-9B recipe transfers to four smaller models. The transfer tests both whether a fixed active budget reconstructs lower-dimensional unembedding rows more effectively, and whether the higherk = 256fidelity regime remains use- ful away from 9B. As above, nativek = 128and nativek = 256rows are reported as separate op- erating regimes, and truncation comparisons are handled as capacity controls. Coverage. The 5k screen covered all 24 model/width/kcells (Figure 10). In every model, the largest screening rowEV occurred at 32×, k = 256, giving Qwen3.5-0.8B0.812, Qwen3.5- 2B0.765, Gemma-4-E2B0.711, and Gemma-4- E4B0.742. These screening values select contin- Figure 10: Small-model screening grid. Each model evaluated at 8×, 16×, and 32×under nativek = 128 andk = 256, while final claims use the native 20k rows in Table 11. Figure 11: Qwen scaling summary. Panel A gives the Qwen rows used for the scaling claim, and panel B the rare feature rate as a usage statistic. Additional model- family rows appear in the surrounding tables. uations, and the 20k rows in Table 11 supply the reported dictionary settings. The nativek = 256evaluation in Table 12 is the reported dictionary setting, and thek = 384row is an over-budget comparison. KL is softcap-correct, while rowEV and top-1 are cap-independent / argmax-invariant. Across the native Qwen dictionary settings in Figure 11, additional capacity generally improves both rowEV and top-1 agreement. The selected strict budget transfer point is Qwen3.5-0.8B 16×, k = 128, and the selected high fidelity small- model point is Qwen3.5-0.8B 32×,k = 256, with Qwen3.5-2B giving the same qualitative pattern. The scaling figure focuses on Qwen so that model size is varied within a single family, and the sur- rounding tables evaluate the same recipe under ad- ditional readout geometries. Rare feature rate is a usage statistic, and it stays low for the smaller Qwen 32×,k = 256rows and rises to0.292for the 9B 32×,k = 256capacity point. Qwen3.5-2B has higher strict budget top-1, so the 0.8B selection is a joint reconstruction-and-fidelity dictionary set- ting with top-1 as one component. Table 13 repeats that 0.8B setting across three seeds. The seed window shows that the headline re- placement metrics are stable under this recipe, 23 Table 11: Converged 20k small-model results. The 16×,k = 128rows instantiate the strict budget regime and the 32×, k = 256 rows the high fidelity regime. Gemma top-1/KL are softcap-correct. ModelDictionary setting rowEVtop-1KLdead/rare Role Qwen3.5-0.8B16×, k = 1280.760 0.844 0.277 0.000/0.001 strict budget transfer result Qwen3.5-0.8B32×, k = 256 0.877 0.891 0.135 0.000/0.001 high fidelity transfer result Qwen3.5-2B16×, k = 1280.712 0.858 0.261 0.000/0.004 strict budget transfer Qwen3.5-2B32×, k = 2560.847 0.887 0.136 0.000/0.010 high fidelity transfer Gemma-4-E2B16×, k = 1280.714 0.6231.94 0.001/0.031 softcap-correct fidelity Gemma-4-E2B32×, k = 2560.834 0.3336.37 0.000/0.038 high rowEV, lower replacement fidelity Gemma-4-E4B16×, k = 1280.693 0.6691.82 0.012/0.040 softcap-correct fidelity Gemma-4-E4B32×, k = 2560.827 0.7361.22 0.002/0.078 fidelity row, softcap-correct Table 12: Per-kevaluation curve for the completed Gemma-4-E4B 32×, k = 256 convergence. k eval rowEVtop-1KL (bits) 640.1130.00838.5 1280.5680.03731.6 1920.7740.32110.8 256 native0.8270.7361.22 3840.6840.7201.37 since rowEV, top-1, and KL have standard devi- ation at most0.02. Rare feature rate varies more across seeds, so we report it as a usage statistic and use the seed-window mean when making seed- robust statements about that tail. E.2 Cross-Family 8B Comparison (Ministral) Ministral-3-8B-Base is included as a cross-family 8B-scale comparison.It has the same hid- den widthd model = 4096as Qwen3.5-9B but an untiedlm_headand no post-readout softcap, and the factorized readout object is language_model.lm_head.weightwith shape 131,072× 4096(readout-extraction reconstruction relative error≈ 3.2×10 −6 , an order of magnitude cleaner than the bf16 Qwen floor). The transferred TopK recipe applies at the matched widths. Both Ministral dictionary settings have healthy usage and decision-fidelity metrics (dead< 0.03, KL< 0.14bits). Decision fidelity (top-1, KL) falls in the high fidelity range of the matched 32×, k = 256comparison rows, which supports the claim that the row-factorization recipe transfers beyond the Qwen family when the readout is ge- ometrically clean (untied, softcap-free). The rare feature rate is high at both points, and we report it as a usage statistic, following the same convention as the Qwen3.5-9B 32×, k = 256 row. E.3 R1-Distilled Comparison Rows R1-Distill-Qwen-7B is included as a post-training comparison row for rowEV and decision fidelity. It is a Qwen-derived 7B readout with an untied lm_headand no post-readout softcap, which gives a clean comparison under the same transferred TopK recipe family at the strict budget and high fidelity dictionary settings in Table 14. The 32×,k = 256R1 row preserves high row reconstruction, within0.044rowEV of the matched Ministral point, while top-1 and KL separate from the base-model comparison. This gives a third pat- tern on the rowEV vs. fidelity axis, since row recon- struction and decision-level readout replacement are distinct evaluation quantities. R1-Distill-Llama-8B adds a second reasoning- distilled checkpoint from a non-Qwen base family at both dictionary settings. At the strict budget, its rowEV is close to the Ministral strict budget row (0.796vs.0.806), with different top-1/KL behav- ior (0.725/0.536vs.0.885/0.130). At the fidelity setting row reconstruction matches the Ministral fidelity row to three decimals (0.888vs.0.888), while top-1/KL remain separated (0.754/0.434vs. 0.904/0.087). The pattern is consistent across both base families (Qwen2.5-Math-derived (Yang et al., 2024) and Llama-3.1-derived (Llama Team, 2024)), in that reasoning-distilled readouts can pre- serve row reconstruction while requiring separate decision-fidelity measurement. Appendix E.4 re- ports the row-norm tail analysis that helps explain this separation. Attribution of the post-training effect.The R1- distilled checkpoints differ from the base-model rows in architecture, tokenizer, and pretraining lin- eage as well as post-training. The repeated pattern 24 Table 13: Qwen3.5-0.8B 32×,k = 256seed-variation window. Seed 0 is the archived dictionary setting, and seeds 1 and 2 repeat the same recipe. SeedrowEVtop-1KLrareWall 0 (archived)0.8770.8910.1350.001– 10.8860.8750.1040.13439 min 20.8870.8790.1050.14251 min mean± std 0.883± 0.006 0.882± 0.009 0.115± 0.018 0.092± 0.080– Table 14: Cross-family and post-training metrics under the transferred TopK recipe. “Strict” denotes nativek = 128 and “Fidelity” native k = 256. ModelDictionary settingWidth / krowEVtop-1KLdeadrareRole Ministral 3-8BStrict16×, k = 1280.806 0.885 0.130 0.026 0.456strict budget comparison Ministral 3-8BFidelity32×, k = 2560.888 0.904 0.087 0.018 0.529fidelity comparison R1-Distill Qwen-7BStrict16×, k = 1280.709 0.695 0.777 0.016 0.298strict budget comparison R1-Distill Qwen-7BFidelity32×, k = 2560.844 0.760 0.489 0.016 0.378fidelity gap R1-Distill Llama-8BStrict16×, k = 1280.796 0.725 0.536 0.009 0.356strict budget comparison R1-Distill Llama-8BFidelity32×, k = 2560.888 0.754 0.434 0.011 0.392 fidelity gap (rowEV- matched, top-1/KL apart) across two distillation lineages makes post-training a plausible contributor, and a matched base-against- distill ablation within one architecture would fur- ther isolate the effect. R1 contrastive score reconstruction. We also ran the held-out contrastive score-reconstruction check on the two R1-distilled checkpoints. Ta- ble 15 reports these results using the same bank version as the five-model run. Sign agreement re- mains close to the five-model baseline, and the fraction with relative error below one half is com- parable in the fidelity setting and lower under the stricter setting. E.4 Readout Row-Norm Tail Analysis The TopK recipe operates on row-centered, row- normalizedW U rows, which treats all rows uni- formly inL 2 . When the underlying row-norm distribution is heavy-tailed, however, a small per- row reconstruction error is amplified disproportion- ately on the high-norm rows, and those are the rows that disproportionately affect top-1 / KL after the softmax. We therefore report row-norm sum- mary statistics for the base-model and reasoning- distilled checkpoints as a usage analysis for the separation between rowEV and fidelity observed in Appendix E.3. Table 16 reports those statistics. Two effects appear. First, the upper tail is heav- ier on the distilled checkpoints, with max/median at1.5–1.6for the base-model rows and1.7–2.0for Table 15: Held-out contrastive score-reconstruction met- rics for the R1-distilled checkpoints under the same contrast-bank version as the five-model run. “sign” is contrastive sign agreement, “median resid.” the me- dian relative residual|∆margin|/|exact margin|, and “resid.< 0.5” the fraction of held-out contrasts with relative residual below 0.5. Model Setting (cfg) sign median resid. resid. < 0.5 Ministral 3-8B-Base fidelity (32×, k = 256) 0.949 0.079 0.831 Ministral 3-8B-Base strict (16×, k = 128) 0.947 0.102 0.812 R1-Distill Llama-8B fidelity (32×, k = 256) 0.926 0.146 0.782 R1-Distill Llama-8B strict (16×, k = 128) 0.907 0.193 0.752 R1-Distill Qwen-7B fidelity (32×, k = 256) 0.921 0.163 0.738 R1-Distill Qwen-7B strict (16×, k = 128) 0.890 0.233 0.700 5-model baseline0.934–0.773 Table 16:W U row-norm statistics for the base-model and reasoning-distilled checkpoints, computed on the readout extraction artifacts used to train each SAE. “CoV” is the coefficient of variationσ/μ. “max/median” summarizes the heavy-tail end. Modelminp50maxmeanCoV max/med Qwen3.5-9B (base)0.379 0.980 1.545 0.983 0.1201.58 Ministral-3-8B (base)0.250 0.447 0.684 0.449 0.1161.53 R1-Distill-Qwen-7B (sft) 0.573 1.054 1.756 1.029 0.1591.67 R1-Distill-Llama-8B (sft) 0.424 0.897 1.787 0.904 0.1391.99 25 the distilled rows, R1-Distill-Llama-8B the most extreme. Second, the lower tail is compressed, since minimum row norms rise from0.25–0.38on the base-model rows to0.42–0.57on the distilled rows. In these comparison rows, the row-norm dis- tribution is wider for the reasoning-distilled check- points than for the base-model rows. The coeffi- cient of variation tracks this (0.12on base models vs. 0.14–0.16 on distilled models). The row-centered-and-normalized SAE recipe operates on a heavier-tailed norm distribution for the distilled rows, which is consistent with the ob- served pattern, where row reconstruction (which is uniform inL 2 ) is preserved, while decision fidelity (which is exponential in the logit error and domi- nated by the heaviest rows) separates from rowEV. This characterizes a second-order property of these reasoning-distilled comparison rows and motivates norm-aware SAE recipes as a natural extension for those settings. F Readout SAE Diagnostics and Reproducibility This appendix collects secondary diagnostics, com- parisons of alternative recipes, denominator con- trols, and reproducibility details for the selected readout SAEs. Artifact availability. All code, with per-model training and evaluation recipes, is released atgithub.com/hematteo/sparse-readout-prism (MIT). The selected readout SAE checkpoints for all eight models (both thek=128and k=256recipes per model) are released at hematteo/sparse-readout-prism.The con- trast banks, both the diagnostic bank and the bank derived from benchmarks, and the blinded label-audit protocol accompany the code release. F.1 Additional Metric Analyses These analyses ask whether the remaining error comes from coefficient fitting, support selection, low-rank structure, or a mismatch between recon- struction and readout metrics. Support selection. On the Qwen3.5-9BD = 32768,k = 128recipe, the trained encoder reaches rowEV0.621. Nonnegative least-squares (Law- son and Hanson, 1974) on the encoder’s own sup- port reaches0.625, nonnegative OMP (Pati et al., 1993)0.647, and signed-OMP0.655. The coeffi- cient gap is therefore small (about0.004), while support-selection headroom within the same coeffi- cient class is about0.026. Dense rank-128 recon- struction reaches only0.101, far below the sparse dictionary, ruling out low-rank PCA structure (Jol- liffe, 2002). Figure 12 shows both the strict budget recipe and the high fidelity 32×,k = 256run. The right-hand table labels the latter OMP numbers as sparse-reconstruction ceilings, while the deployed encoder’s metrics are the trained-encoder rows. For the 32×,k = 256capacity run, coefficient refitting contributes+0.006rowEV, while support selection contributes+0.038. The encoder reaches 95.1%of the signed-OMP proxy, similar to the D = 32768,k = 128ratio. Dense rank-256 re- construction is0.155, so thek = 256sparse code is about5.9×better by unexplained-variance re- duction than the same-rank dense reference, since (1− 0.155)/(1− 0.858)≈ 5.9. Small-model support-selection analysis.We re- peat the analysis on the 16×,k = 128small-model transfer 5k checkpoints to isolate the source of the remaining reconstruction gap with the learned dictionary fixed, and the converged transfer met- rics are reported in Appendix E.1. Across all four models, refitting coefficients on the encoder’s own support gives a much smaller gain than changing the support, so support selection is the dominant re- maining bottleneck on the recipe side. The signed- OMP gaps are+0.158for Qwen3.5-0.8B,+0.099 for Qwen3.5-2B,+0.182for Gemma-4-E2B, and +0.125 for Gemma-4-E4B. Metric mismatch.The capacity sweep motivates selecting readout SAEs with metrics at both the row level and the readout level. TheD = 65536,k = 256model truncated tok = 128has higher rowEV than the fixed-k = 128recipe (0.695 > 0.621), while top-1/KL favor the fixed-k = 128recipe (0.721/0.469vs.0.846/0.296). TheD = 131072 truncation shows the same metric separation, with rowEV/top-1/KL at0.816/0.738/0.425. These truncated comparisons motivate the selection rule, which uses rowEV, top-1, KL, reconstruction-error metrics, and usage statistics jointly. F.2 Alternatives and Negative Controls Alternative recipes and denominator controls. Matched comparisons rule out the non-selected recipes.AtD = 32768/5k, plain TopK ex- ceeded the matryoshka variants in rowEV (Buss- mann et al., 2025) (0.567vs.0.559/0.554) at lower cost, and theD = 16384fixed-k = 128cell 26 32×, k = 256 ceiling comparison QuantityrowEV95% CI Trained encoder0.858 [0.856, 0.859] Signed LS support0.864 [0.863, 0.866] Signed-OMP proxy 0.902 [0.900, 0.903] Figure 12: Support-selection analyses for the Qwen3.5-9B readout SAE. The left panel gives the strict budget k = 128recipe and the high fidelity 32×,k = 256capacity run, and the right panel the 32×,k = 256EV- ceiling numbers with 95% bootstrap CIs. OMP (orthogonal matching pursuit) is an achieved sparse-reconstruction comparison. trailedD = 32768by about0.06rowEV. Cali- brated BatchTopK (Bussmann et al., 2024) reached rowEV0.531on the matched frontier cell. Qwen JumpReLU (Rajamanoharan et al., 2024) missed the intended targetL 0 = 256(achieved about38– 68), and step- or sparsity-matched TopK controls had higher rowEV (0.42–0.44vs.0.26–0.31), with mixed logit metrics. Finally, tokenizer and non- token row audits support the denominator choice, since structural non-token rows were5.4%of rows and excluding non-token readout rows changed rowEV by +0.0016. F.3 Reproducibility and Selection Details The main experiments report exact and recon- structed scores, sign agreement, relative error, and null/reference baselines for interpreted scores. Ap- pendices B–G specify models, dictionary settings, seeds, row sampling, selection order, tokenization filters, score construction, and denominator con- trols. The remaining risks are computational cost, checkpoint and tokenizer versioning, and tokenizer- specific filtering for single-token scores. Local de- compositions include explicit reconstruction errors, and we restrict qualitative interpretations to suffi- ciently reconstructed scores. Artifact contents. The artifact bundle has four components. First, the code artifact includes the readout-SAE training scripts, hidden-state capture and evaluation scripts, baseline implementations, and figure/table reproduction scripts. Second, the dictionary artifact includes the reported readout- SAE checkpoints when permitted by upstream model licenses, covering decoder directions, sparse row codes, shared offsets, preprocessing metadata, configuration files, seeds, and model checkpoint revision identifiers. For any checkpoint that can- not be redistributed directly, we provide the ex- act recipe needed to regenerate it from the public model weights. Third, the prompt artifact includes the prompt templates and filled prompts used for reconstruction of selected scores, qualitative dis- plays, feature-resolved DLA examples, and the con- strained lexical-edit probes. Fourth, the score-set artifact includes token ids or token groups, row- combination coefficientsα, split/base-case identi- fiers, tokenization filters, bootstrap clusters, and the exact score-family metadata needed to recompute the reported sign and reconstruction metrics. We do not redistribute original model checkpoints or raw C4 text, and C4-based evaluations are repro- duced from public dataset/model identifiers and the artifact sampling, filtering, and scoring code. Dictionary selection used row reconstruction, replacement-logit metrics, and feature usage statis- tics, and the logit difference contrast set en- ters only the reconstruction checks.Most re- ported dictionary settings use seed0, with the Qwen3.5-0.8B 32×,k = 256three-seed win- dow providing the seed-variation snapshot above. For Qwen3.5-9B, tokenizer-free denominator con- trols replace tokenizer-aligned text-token-only EV and frequency-stratified fidelity, since the tok- enizer row count did not cleanly align with the W U rows and no vocab-aligned unigram table was available.The tokenizer-free non-token row audit is therefore the denominator-control result for that model. Gemma rowEV is mea- sured on the linear pre-softcap unembedding rows and is cap-independent, while Gemma top-1/KL are evaluated under the model’s30 tanh(·/30) final_logit_softcapping.Since the cap is strictly monotone, top-1 is cap-independent and the cap affects only KL. The selected operating points 27 and reported claims come from the Qwen/Gemma sweeps and the targeted Ministral/R1 comparison rows above, and earlier Pythia-160M architecture and continuity sweeps (Biderman et al., 2023) are design-history records. Exact replication depends on using the same model checkpoint revisions, tok- enizers, final-normalization and softcap code paths, score filters, and single-token eligibility rules. For this reason, every interpreted local score decompo- sition reports the exact score, reconstructed score, reconstruction error, and, for signed contrasts, sign agreement. Training and evaluation recipe. All reported read- out SAEs are TopK sparse autoencoders trained on the model’s final readout rows. The reported Qwen/Gemma recipe uses row-seeded initialization,hybrid_50freq_50uniformrow sampling, a warmup-cosine learning-rate schedule from10 −3 to10 −4 , and seed0unless stated otherwise. The Qwen3.5-9B recipe-selection run usesD = 32768, nativek = 128, a delayed prism-loss ramp to10 −3 after70%of training, and 20k optimization steps, while the capacity and transfer runs keep the same recipe family while varyingDand nativek as reported in Tables 5, 9, and 11. Hidden-state evaluation uses the final readout after the relevant final normalization and, for Gemma, the model’s monotone final-logit softcap for top-1/KL evaluation. The reported metrics are rowEV, top-1 agreement, KL in bits, feature usage, and the reconstruction error and sign agreement for selected scores defined above. Compute and software. Readout-SAE train- ing and the headline evaluations ran on a single NVIDIA A40 per job on an academic Slurm clus- ter. A 24 GB-class GPU suffices for inference and evaluation, while training the larger readouts (e.g. Qwen3.5-9B) needs roughly 40 GB. One readout SAE costs about 6–12 A40 GPU-hours depend- ing on vocabulary size and dictionary width (the Ministral-3-8B131072× 4096head takes 12.2 h at32×,k = 256, and 6.3 h at16×,k = 128, for 20k steps), and the three-model constrained- edit evaluation totals about 2.5 GPU-hours. Fi- delity evaluations of selected scores for the three late-added models ran on a single rented cloud GPU, and some figure and analysis passes ran on Apple Silicon (PyTorch MPS, bfloat16). Ex- periments use Python 3.11–3.12 with pinned de- pendencies, includingtorch2.7.1 (CUDA 11.8) on the cluster andtorch2.12.0 on Apple Sili- con,transformers5.9.0,numpy2.4.6,datasets 4.8.5, andsafetensors0.7.0. The released code includes the full lockfile. G Fidelity Checks and Controls This appendix collects the checks and controls that support interpreting and reproducing the local score decompositions, covering tokenization and filtering rules, row-norm and frequency checks, split con- struction, dictionary settings, and aggregate met- rics. The decomposition definitions themselves are stated in Appendix A. G.1 Controls Table 17 groups the measurements used to read a local score decomposition. Aggregate outcomes for these checks appear in the quantitative summaries below. G.2 Metric Definitions and Fidelity Results Metric definitions for Table 1.Dictionary width is the cfg multiplier times the hidden dimension (width selection in Appendices C–D). rowEV (row- centered explained variance) is the fraction of vari- ance in row-centered unembedding rows captured by the sparse reconstruction. top-1 is the fraction of held-out decoded states where SRP and the dense LM head agree on the argmax token. KL is the median KL divergence between dense and SRP- reconstructed next-token distributions. Sign is the fraction of contrasts where the SRP-reconstructed margin shares sign with the exact margin. Replace- ment uses held-out decoded states, while score met- rics use the main held-out logit difference set at each model’sk = 256operating point. Med.ρ 0.5 andρ 0.5 < 0.5use the flooredρ 0.5 of §3.4. The main text reports top-1 and the baseline compar- ison, while rowEV and KL are in Table 5, and the per-score columns in Table 20. The main-text table lists the six softcap-free readouts, and the two Gemma-4 readouts are reported at the same operating point in Table 5, with their per-score re- construction in Table 20 and their null controls in Table 19. Gemma-4-E4B reaches top-1 0.736, while its median KL of 1.22 bits lies well above the six softcap-free readouts, whose largest value is 0.489 bits. Fidelity on selected scores is the central quantita- tive check. Across the eightk = 256readout SAEs (32×width), the sparse reconstruction preserves contrast sign for0.914–0.958of selected readout scores, with medianρ 0.5 = 0.068–0.255(Table 20, with cluster-bootstrap 95% intervals). The broader contrast set over all rows supplies a supporting di- agnostic. Aggregating both held-out halves, the 28 Table 17: Checks and controls supporting local score decompositions. The checks verify that a displayed local score decomposition tracks the intended readout score and control for prompt, tokenization, and SAE artifacts. The evaluation setup specifies the token choices, coefficients of the readout directions, split assignment, and tokenization rules used for the aggregate metrics. Check or controlWhat it testsHow it is specified Reconstruction error and sign agreement Feature bars are reported with relative recon- struction error and, for contrasts, the signs of both the exact and reconstructed scores. Reported local score decompositions include exact score, reconstructed score, reconstruction error, and ρ 0.5 . Swapped and matched contrasts A/B swaps flip signs consistently, and matched negative-control contrasts underperform the intended contrast. Evaluation sets specify paired A/B contrasts and matched negative controls before aggregate evaluation. Template and context variants Prompt rewordings and context variants are clustered before aggregation, so correlated variants contribute through their base case. Rows are clustered by score or base-case id before aggregate evaluation. Token-level auditsSingle-token claims can be brittle under token length, frequency, whitespace, capitalization, row norm, special-token status, or tokenization collisions. Tokenization filters and row-norm/frequency controls are applied before aggregate evalua- tion, and ambiguous cases are handled through feature-audit flags or explicit group contrasts. Null and reference base- lines Learned token-to-feature support is compared against shuffled support, random support, dense PCA, and lexical nearest row references. Baselines use the same score set and the same reconstruction error and sign metrics. Qwen3.5-0.8B/2B/9B settings give sign agreement 0.940–0.951, medianρ 0.5 = 0.091–0.107, and ρ 0.5 < 0.5fractions0.790–0.818. These per-score figures cover the high fidelity subset of this evalua- tion, which is why the Qwen sign and reconstruc- tion error ranges are tighter than the full set’s. The Qwen3.5-2B reference panel below evaluates ev- ery method on that model’s 1,348-row full logit difference set under the baseline harness, and the redesigned six-model comparison behind the main- text baseline table is in Appendix G.3. Many C4 positions are local near-ties, which lowers headline numbers, yet Qwen still reaches0.822–0.838top-1 agreement and0.634–0.667preservation of the top- 1–top-2 ordering at relative error0.5on 10,000 neu- tral C4 continuations per model (Raffel et al., 2020; Dodge et al., 2021). This appendix writesm exact andm recon for the exact and reconstructed margins, the quantitiess exact ands recon of §3.2 evaluated on the logit difference bank. For confident Qwen margins,|m exact |≥ 2, C4 top-1 agreement rises to 0.97–1.00and top-1–top-2 ordering preservation to0.84–0.96. Additional model-family rows use the same metric suite, separating row reconstruc- tion, replacement fidelity, and reconstruction of selected scores. Null and reference baseline com- parisons preserve the same ordering, since shuffled and random support approach chance sign agree- ment and dense PCA underperforms Sparse Read- out Prism at comparable compactness. Cluster- bootstrap 95% intervals on the Qwen3.5-2B panel under the baseline harness (848 base cases, 400 resamples, unflooredρ 0 identically for all meth- ods) give SRP coverage0.774(0.755–0.792) and medianρ 0 = 0.081(0.071–0.092), against near- est row ridge at0.622(0.603–0.643) and0.260 (0.242–0.295). PCA-1024 reaches0.539(0.514– 0.563) using a median of 337 components per score, while PCA-256, matched to SRP’s real- ized compactness (a median of 78 components car- rying80%of absolute contribution mass against SRP’s 76 features), drops to0.272(0.247–0.296). The redesigned harness of Appendix G.3 gives 0.780 for SRP on this model, the value in Table 1. Table 19 repeats the two null controls on every Qwen/Gemma readout at the operating point that prioritizes fidelity. Each model–null cell lowers coverage by0.58–0.71and sign agreement to0.38– 0.52, so the learned row–code assignment carries the reconstruction on every evaluated model, in- cluding the lower-fidelity Gemma rows. Figure 16 reports the same score set and metrics as the Sparse Readout Prism row. Tokenization filters, row-norm controls, frequency controls, KL, sign agreement, and relative reconstruction error use the same def- initions across the score-group and baseline com- parisons. Fidelity also varies by family of selected scores. Table 18 breaks the maink = 256comparison point down by family, ordered by how close the competitor sits to the selected token, and the main text reads the digest for each family from this table. 29 Table 18: Score and contrast groups at the maink = 256setting. Rows group selected readout score families at thek = 256fidelity setting. Row counts aggregate evaluated model–case cells across the five Qwen3.5 and Gemma-4 readout SAEs (excluding Ministral and the R1-distilled readouts), after tokenization filters. Medianρ 0.5 is the floored variant of §3.4. Source banks and per-family definitions of the score objects are in Appendix G.7. GroupSelected scorecasesrowssign agreementmedian ρ 0.5 Vocabulary-mean con- trast top-1 vs vocabulary mean30115051.0000.020 Reference-row contrasttop-1 vs row-norm distractor30115051.0000.048 Logit differencesource vs prior823920.9640.172 Logit differenceaction/tool alternatives964800.9060.283 Group contrastabstention family844200.8880.198 Logit differencecurrent vs stale924600.8610.369 Top competitortop-1 vs sampled competitor30115050.9930.113 Top competitortop-1 vs top5 competitor30115050.9400.215 Top competitortop-1 vs top2 competitor30115050.8240.342 Table 19: Cross-model null collapse. Coverage, mean- ing sign agreement together withρ 0 < 0.5, with sign agreement in parentheses, for SRP and the two null controls on each model’s full logit difference set at the operating point that prioritizes fidelity (k = 256,32× width), under the baseline harness’s unflooredρ 0 . The Qwen3.5-2B SRP row is computed on the same banks as the main-text baseline table, whose redesigned har- ness gives 0.780 for SRP on this model (Appendix G.3). Anchor sign agreement on this full set sits below the per-model values of Table 20 (the main-text0.91sign floor refers to that evaluation). ModelSRPShuffled codes Random support Qwen3.5-0.8B 0.752 (0.926) 0.088 (0.519)0.085 (0.484) Qwen3.5-2B0.774 (0.941) 0.081 (0.502)0.069 (0.407) Qwen3.5-9B0.784 (0.927) 0.094 (0.467)0.079 (0.424) Gemma-4-E2B 0.665 (0.890) 0.085 (0.484)0.087 (0.377) Gemma-4-E4B 0.701 (0.900) 0.095 (0.476)0.072 (0.438) <0.50.5-11-22-5>=5 |m exact | bin 10 −1 10 0 10 1 error / margin A <0.50.5-11-22-5>=5 |m exact | bin 0% 10% 20% 30% 40% 50% decision flip rate B Qwen 0.8B Qwen 2B Qwen 9B Gemma E2B Gemma E4B Figure 13: Held-out logit difference fidelity. Per- bin reconstruction across five readout SAEs vs. exact- margin magnitude|m exact |. A. Median relative error |m exact − m recon |/|m exact |(log). B. Decision-flip rate. Solid lines are Qwen3.5-0.8B/2B/9B, dashed Gemma- 4-E2B/E4B. Both concentrate at small margins, with calibration in Figure 14. Figure 14: Exact versus reconstructed logit differ- ences. Binned exact versus reconstructed margins on the five-model held-out contrast set, where the dashed line marks perfect reconstruction. This is the global calibration companion to the reconstruction error and decision-flip summaries of Figure 13, and the decision boundary is better captured by margin-binned views. Figure 15: Margin size controls the reliability of signed contrasts. Aggregate reconstruction error is tight on the held-out contrast set, while sign agreement is lowest near zero exact margins. Raising the mini- mum|m exact |increases sign agreement while reducing retained coverage. Feature displays therefore report re- construction error and sign agreement alongside row reconstruction. 30 Table 20: Reconstruction of selected scores with cluster-bootstrap intervals. Per-model sign agreement, median ρ 0.5 andρ 0.5 < 0.5on the held-out logit difference set, with 95% intervals (flooredρ 0.5 , base-case clusters resampled with replacement, 400 resamples). Rows and base cases per model are 1,853/649 (Qwen3.5 and R1-distilled), 1,859/655 (Gemma-4), 1,856/652 (Ministral). ModelSignMed. ρ 0.5 ρ 0.5 < 0.5 Qwen3.5-0.8B0.950 (0.942, 0.960)0.075 (0.069, 0.084)0.891 (0.877, 0.905) Qwen3.5-2B0.958 (0.947, 0.967)0.071 (0.064, 0.080)0.890 (0.875, 0.904) Qwen3.5-9B0.949 (0.938, 0.959)0.088 (0.079, 0.096)0.878 (0.863, 0.893) Gemma-4-E2B0.914 (0.900, 0.928)0.255 (0.233, 0.278)0.710 (0.687, 0.730) Gemma-4-E4B0.940 (0.927, 0.951)0.155 (0.141, 0.165)0.807 (0.784, 0.828) Ministral-3-8B0.949 (0.939, 0.958)0.068 (0.061, 0.078)0.888 (0.873, 0.902) R1-Distill-Qwen-7B0.921 (0.908, 0.933)0.147 (0.134, 0.160)0.787 (0.771, 0.806) R1-Distill-Llama-8B0.926 (0.913, 0.938)0.134 (0.125, 0.143)0.830 (0.814, 0.846) Figure 16: Baseline and reference comparisons for Qwen3.5-2B. Sparse Readout Prism is compared with nearest row ridge, dense PCA references, shuffled sparse codes, and random support under the same reconstruction error and sign summaries. The readout SAE preserves sign and keeps a larger low-error tail than the null controls, while PCA references lose that low-error tail as the compactness constraint tightens. G.3 Direct-Geometry Alternatives Across the Primary Readouts The main-text baseline table (Table 1) summarizes a six-model comparison against methods built di- rectly fromW U geometry, and Table 21 gives the full grid. Each method reconstructs the same se- lected scores on the same banks and is scored iden- tically (sign agreement and unflooredρ 0 , cluster- bootstrap CIs by base case). The methods are near- est row ridge (ridge regression of each selected di- rection on its top-128 nearest rows), weighted kNN on the rows (zero-fit weighted average of the top- 128 nearest rows), k-means centroid dictionaries at D = 65,536andD = 16,384withk = 256active centroids per row (width- and sparsity-matched to SRP), hard cluster assignment (each row explained by its singleD = 65,536cluster centroid), and PCA-256. An earlier version of the clustering and kNN baselines over-reconstructed through a full- rank projection artifact and was redesigned. The k-means centroid dictionary at matchedDandk is the sharpest test, since it grants the alternative SRP’s exact capacity, and it still trails SRP’s cover- age on every model. G.4 Error Tails and Margin Stratification A median can hide large failures in the tail, so we re- port the full error distribution for every method on the same banks and harness as Appendix G.3 (Ta- ble 22). SRP has the lowest mean, 95th-percentile, and maximum absolute error of the seven methods, so its coverage lead in Table 1 does not depend on the choice of summary statistic. Per model, SRP’s mean absolute error is0.43–1.22logits, its p951.53–3.83, and its maximum4.4–10.4, against 1.16–2.95,2.98–8.00, and8.5–17.4for nearest row ridge, the strongest alternative. Where the remaining tail sits decides whether it threatens interpretation. Table 23 stratifies the same contrasts by exact margin magnitude. SRP’s 31 Table 21: Coverage for SRP and six alternatives built from direct row geometry. Coverage is sign agreement together with unflooredρ 0 < 0.5, on each softcap-free readout’s full logit difference bank (∼1,350 rows over∼850 base-case clusters), with cluster bootstrap 95% CIs in brackets. Bold marks the best method per model. MethodQ-0.8BQ-2BQ-9BMin-8BR1-Q-7BR1-L-8B SRP0.754 [.733, .773] 0.780 [.759, .798] 0.812 [.792, .831] 0.777 [.759, .795] 0.716 [.694, .738] 0.751 [.728, .772] Nearest-row ridge (top-128)0.665 [.645, .686] 0.625 [.606, .644] 0.639 [.618, .658] 0.648 [.629, .665] 0.579 [.560, .598] 0.600 [.580, .620] k-means dict, D=65,536, k=2560.585 [.566, .603] 0.596 [.575, .614] 0.619 [.598, .639] 0.628 [.610, .648] 0.479 [.457, .503] 0.624 [.606, .642] k-means dict, D=16,384, k=2560.570 [.550, .591] 0.561 [.540, .581] 0.602 [.580, .623] 0.604 [.585, .622] 0.312 [.284, .339] 0.537 [.516, .558] Hard cluster, D=65,5360.430 [.405, .451] 0.397 [.370, .422] 0.411 [.386, .435] 0.499 [.475, .521] 0.353 [.325, .377] 0.492 [.470, .512] Weighted row-kNN (top-128)0.420 [.397, .442] 0.331 [.308, .353] 0.403 [.377, .426] 0.480 [.456, .504] 0.163 [.143, .184] 0.257 [.234, .279] PCA-2560.462 [.439, .486] 0.269 [.244, .291] 0.244 [.224, .268] 0.338 [.317, .367] 0.084 [.070, .099] 0.105 [.090, .120] Table 22: Full error distribution for SRP and the six alternatives built from direct row geometry. Absolute reconstruction error|ε|in logits and unfloored relative errorρ 0 , pooled over the six softcap-free readouts (8,009 contrasts per method, under the same banks, dictionaries, and harness as Table 21). Absolute errors are pooled across models of different logit scales, and per-model ranges are given in the text. Coverage (sign agreement and ρ 0 < 0.5) repeats the pooled figure for reference. Lower is better except sign agreement and coverage. MethodMeanp95MaxMedian ρ 0 SignCoverage SRP0.692.3410.40.1010.9260.765 Nearest-row ridge (top-128)1.975.8817.40.2890.8580.626 k-means dict, D=65,536, k=2562.627.3414.60.3640.8390.588 k-means dict, D=16,384, k=2563.589.8922.50.4600.8320.531 Hard cluster, D=65,5364.4711.5721.40.5710.7940.430 Weighted row-kNN (top-128)5.0913.1226.10.6680.7590.343 PCA-2565.7415.6930.70.7540.7780.252 Table 23: SRP error by exact-margin magnitude. The same 8,009 pooled contrasts as Table 22, binned by|m exact |. Share is the fraction of contrasts in the bin, Mean and p95 are absolute errors in logits, and the last column is the coverage. |m exact | Share Meanp95 Med. ρ 0 Sign ρ 0 < 0.5 < 0.50.140.64 2.151.864 0.6870.227 0.5–10.120.67 2.180.556 0.8430.459 1–20.140.66 2.170.256 0.9310.679 ≥ 20.610.71 2.490.034 0.9960.966 absolute error is nearly flat across bins (mean0.64– 0.71logits), so the relative-error tail is a denomina- tor effect. It concentrates in the14%of contrasts with|m exact | < 0.5, where the two outputs are close to tied, and falls monotonically as the margin widens. On the61%of contrasts with|m exact |≥ 2, sign agreement is0.996and0.966of contrasts are covered. Nearest-row ridge shows the same shape with roughly double the absolute error in every bin (1.30–2.39logits) and0.057coverage in the near-tie bin against SRP’s0.227. The diagnostics of §3.4 make the near-tie regime visible, so this tail is measured and reported. Figure 15 traces the resulting trade-off, since raising the minimum |m exact |buys sign agreement at the cost of retained coverage. G.5 Nearest-Row Reading of Selected Directions The zero-fitting reading of a selected readout direc- tionqranks vocabulary rows by cosine alignment withqand inspects the top tokens. Table 24 ap- plies this reading to two case-study contrasts on Qwen3.5-2B, ranking row-centered unembedding rows with the two contrast tokens themselves ex- cluded, and raw-cosine rankings behave the same. Forw bug − w insect the listing is dominated by casing and morphological variants of the contrast words, the first sense-bearing rows (a malfunction row,debug,issue,glitch,flaw) appear at ranks 13–17 at roughly half the leading alignment, and the orthographic neighborblogenters by rank 30. Forw bug −w error the top 30 consists entirely of cas- ing, sub-word, and cross-lingual variants of the two contrast words. The nearest row reading recovers the contrast’s token family, while the sense struc- ture that the SRP decomposition of the same scores separates into signed feature terms (Figure 18) sits below its morphological band. G.6 Operating-Regime Cases A faithful decomposition is a local reading of the readout side, tied to this dictionary and its budget of active features, since a different dictio- nary or budget could split the same contribution across other features. Operating-regime cases de- 32 Table 24: Nearest centered unembedding rows for two selected directions. Qwen3.5-2B, top 12 rows by |cosine|with each direction, contrast tokens excluded. □marks a leading space, and bracketed entries gloss non-Latin tokens. For signed cosines, positive supports the first token of the contrast. w bug − w insect w bug − w error Rank TokencosTokencos 1□Bug+0.49□Error−0.53 2 bug+0.46 error−0.50 3 Bug+0.44□errors−0.49 4□bugs+0.42 _error−0.47 5□BUG+0.34 Bug+0.45 6 _bug+0.31 -error−0.44 7 BUG+0.31 [tab]error −0.44 8□insects −0.30 Error−0.44 9 bugs+0.29□Bug+0.42 10□buggy+0.29 [zh: numerical error] −0.42 11□Bugs+0.28 bug+0.42 12 [zh: insect] −0.24□bugs+0.40 fine when a compact local score decomposition is evidence-bearing. Qualitative interpretation uses cases where the selected readout score is larger than the reconstruction error, the reconstructed con- trast preserves sign, the exact model readout sup- ports the intended side of the contrast, contribution mass is compact enough to summarize, and token- boundary, special-token, row-norm, frequency, or tokenization-collision effects do not dominate the contrast. The remaining cases are retained as quan- titative records under the same reporting rule. Each failing condition routes the reading to the part that still holds. When the residual dominates, only the selected score itself is read. When the reconstructed contrast flips sign, the exact con- trast is used for direction alone. When the exact readout does not support the intended target, the score is reselected. When the contribution mass is too diffuse for a compact reading, the aggre- gated mass is reported in place of individual terms. And when a single-token contrast turns on token- boundary, special-token, row-norm, frequency, or tokenization-collision effects, the reading falls back to the safer token-family contrast. Each promoted display names the constraint it met and the reading that constraint leaves in place. Where the resid- ual dominates or terms repeatedly change sign, the score is reported without a feature-level read- ing and retained in the record. The Qwen3.5-2B readout used for the main-paper displays sits well within the operating regime on every one of these measurements. G.7 Selected-Score Bank Provenance The banks of selected scores define the analyzed object behind the headline fidelity numbers. A row record contains the prompt or hidden-state source, token ids or token-family strings before tokeniza- tion, the score-family label, tokenizer-audit fields, and the exact and reconstructed scores used to com- puteρ 0.5 . Table 25 lists the source banks. The controlled A/B banks are deterministic synthetic probes that build benchmark-like margin types, while the separate C4 bank is sampled from C4 val- idation and supports the model-native replacement and frontier diagnostics only. Table 26 maps each main-text score family (Table 18) to its source bank and gives a one-line definition of the score object, and Table 27 shows representative A/B records. A separate contrast bank derived from benchmarks builds answer-vs-distractor, abstention-vs-entity, label, and safe-vs-action readout contrasts from SQuAD2, HotpotQA, LegalBench (ContractNLI), and SecurityEval formats. It tests reconstruction of fixed readout contrasts, with task accuracy out of scope, and its per-format results are in Table 36. H Sense-Labelled Evaluation §4.5 reads individual scores for a fixed token under two contexts. This appendix tests the same feature groups against labelled sense data and measures what the sparse account gives up against the full decoded state. Data and coverage. CoarseWSD-20 (Loureiro et al., 2021) supplies 20 ambiguous words with coarse sense labels and a fixed train and test split, 33,566 contexts in total. We keep the shipped splits and score the final position of a cloze prompt that asks for the ambiguous word. All 20 words resolve to a single token under the Qwen3.5 vocabularies. Seven do not under the R1-Llama-8B vocabulary, so that model is evaluated on 13 words and 16,785 contexts and its column is not comparable item for item with the other two. Row-gate coverage is 1.00 on the Qwen models and 0.976 on R1-Llama-8B. Sense distributions are skewed, so we report bal- anced accuracy throughout. Unbalanced accuracy gives the same ordering and is released with the run artifacts. Protocol. For each word we select one group of eight features per sense on the training split by largest mean contribution difference between that sense and the others, then assign each held-out con- 33 Table 25: Source banks for the checks on selected scores. The main-text score-family table uses the model- native, curated A/B, and case-candidate banks at the operating point that prioritizes fidelity, while the C4 bank supports the separate model-facing replacement diagnostic. BankRecordsScore construc- tion Provenance Model- native prompts v2 301 prompt records Run-time top-1 contrasts against the vocabulary mean, a row- norm matched distractor, a sam- pled competitor, the rank-5 token, and the rank-2 token. Hand-written prompt set cov- ering factual QA (50), instruction (30), paragraph (40), code (30), dialogue (30), retrieval-style updates (96), and safety/tool prompts (25). Curated A/B v2 320 prompt- target records Single-token A− Bmargins, plus abstention- family versus forced-answer- family margins. Deterministic template expan- sion over 80 current-vs-stale, 80 source-vs- prior, 80 ac- tion/tool, and 80 abstention cases. Case candidates v2 40 prompt- target records The same con- trast types as the curated A/B bank, used for figure-candidate and coverage rows when tok- enization checks pass. 16 action/tool, 8 source-vs- prior, 12 current- vs-stale, and 4 abstention records. C4 model- native v3 10,000 prompt prefixes Model-native replacement, top-token, KL, and top-1–top-2 frontier checks. Sampled from allenai/c4En- glish validation at pinned revision 1588ec45..., seed 0, 110- word prefixes after length and control-character filters. text to the sense whose group carries the larger summed contribution. Feature scales are standard- ized by training statistics alone, and no test label enters the selection. The null permutes sense la- bels within the training split and repeats the whole selection, so it inherits the selection procedure. Sense alignment. Table 28 gives the result. Se- lected groups recover held-out senses at 0.90, 0.92, and 0.80 balanced accuracy against a permutation null near 0.41. Every word on Qwen3.5-2B, 19 of 20 on Qwen3.5-9B, and 12 of 13 on R1-Llama- 8B exceed that null’s 95th percentile, and every word on all three models exceeds its majority base- Table 26: How source banks map to the main-text score-family rows. Cases count distinct base cases after tokenization filters at the operating point that pri- oritizes fidelity, and rows count evaluated model–case cells across the five-model Qwen/Gemma suite. Score familySource bankCases Rows Score object top-1 vs vo- cab mean Model-native prompts v2 301 1505Exact top token against the vocabulary mean row. top-1 vs row- norm distrac- tor Model-native prompts v2 301 1505Exact top token against a sampled row-norm matched distractor. source vs prior Curated A/B + case candidates 82392Source- supported an- swer against a plausible prior answer. action/tool alternatives Curated A/B + case candidates 96480 Task- appropriate action against a plausible wrong action. abstention family Curated A/B + case candidates 84420Mean abstention- token fam- ily against forced-answer tokens. current vs stale Curated A/B + case candidates 92460Current or corrected value against an earlier stale value. top-1 vs sam- pled competi- tor Model-native prompts v2 301 1505Exact top token against a sampled lower-ranked competitor. top-1 vs top5 competitor Model-native prompts v2 301 1505Exact top token against the rank-5 token. top-1 vs top2 competitor Model-native prompts v2 301 1505Exact top token against the rank-2 token. line. The finding does not depend on the account size. Across sizes one, two, four, and eight the majority baseline is exceeded for every word on every model, and the count above the null’s 95th percentile moves by at most one word. The last two rows of the same table bound the result from above. A nearest centroid probe on the full decoded state reaches 0.96, 0.97, and 0.92, and the full 256-dimensional account without any selection reaches 0.95, 0.97, and 0.89. The three rows form a sparsity ladder in which factorizing the row costs little. Compressing to eight features 34 Table 27: Representative A/B prompt records. Repre- sentative rows from the synthetic controlled banks, used as controlled probes of readout margins. The full bank is machine-readable. FamilyPrompt endingSpecified con- trast current vs stale Initial note: the password was blue. Update: the password is now red. The current password is red minus blue source vs prior The document says: “The meeting city is Paris.” Ignore prior assumptions. The meeting city is Paris minus Lyon action/tool The request is ambiguous and might damage files. The appropriate next action is to ask minus run abstention family The passage says Alice owns a key. It does not say her favorite color. Alice’s favorite color is mean(unknown, Unknown, unclear,not) minus forced- color tokens costs a further 5 to 9 points against the full account, and 5 to 12 points against the full state. That is what the account costs, and it buys a set of features a reader can name. This is an alignment result for the selected groups. The comparison against directW U ge- ometry is made on reconstruction and coverage in Table 1 and on grouping recovery in §4.4, where the alternatives are matched by construction. A classifier framing.A different framing trains a per-word classifier on the ten features that carry the largest absolute contribution, skipping the group selection. Under that framing the unweighted pro- jectionsp i ( ̃ h ℓ )beat the signed contributions on all three models, at 0.87 against 0.84, 0.90 against 0.76, and 0.73 against 0.67. Row-coefficient shuf- fles and randomly drawn features sit close to the signed contributions. That ordering follows from the factorization. Holding a vocabulary rowvfixed freezes its co- efficientsz v,i and leaves onlyp i ( ̃ h ℓ )free (§3.2), so for a fixed word every weighted variant is a constant per-coordinate rescaling of the same pro- jections. A per-word supervised classifier gains nothing from that rescaling, and a shuffle of the coefficients preserves the projections it is meant to remove. The signed contributions name which readout directions support a selected score. They Table 28: Sense alignment on CoarseWSD-20. Bal- anced accuracy on held-out contexts for groups of eight features selected on the training split, against a label- permutation null that repeats the selection, a majority baseline, and two references that use no selection. Qwen3.5-2BQwen3.5-9BR1-Llama-8B Words202013 Selected groups0.8990.9230.798 Permutation null0.4080.4070.422 Majority0.4100.4100.429 Words above null p9520/2019/2012/13 Words above majority20/2020/2013/13 Full account, no selection0.9470.9700.892 Hidden state0.9580.9710.920 are not a sense representation, and this second fram- ing measures the second quantity. Sense structure in embedding geometry has been studied directly (Arora et al., 2018). Scope. The evaluation covers one dataset, one sparsity setting per model, and one prompt tem- plate. No behavioral filter is applied, and the model ranks the ambiguous word at median rank 174, 10, and 1131 on the three models, so a share of the scored contexts sits outside the operating regime of Appendix G.6. Filtering on target rank would change the reported values in an untested direction. I Stability Across Independent Training Runs This appendix documents the reproducibility evi- dence summarized in §4.4, together with a held- out comparison of the grouping structure against row geometry that the main text does not report. All experiments use Qwen3.5-2B with three dic- tionaries per width configuration (32×/k=256and 16×/k=128), trained from scratch with indepen- dent initialization, data-order, and sampling seeds. Held-out reconstruction is equal across seeds (val- idation top-1 agreement 0.805–0.848), so differ- ences below are not fidelity artifacts. All state- ments in this appendix hold within a recipe, mean- ing seeds vary while width, sparsity, and prepro- cessing are fixed. Variation across recipes is larger and is disclosed at the end. I.1 Explanation-Level Stability The unit under test is the decomposition of a se- lected score. For each of 63 curated contrasts (126 signed side-sets), we compare the token sets that each seed’s decomposition places on each side of the contrast. Individual decoder directions do not recur across 35 Table 29: Cross-seed stability of contrast explana- tions. Qwen3.5-2B, three seeds per width, with means over seed pairs. Same-side Jaccard compares side token- sets for the same contrast across seeds, and the cross- contrast null pairs decompositions of unrelated contrasts through the identical pipeline. Metric32×/k=256 16×/k=128 Same-side Jaccard (mean)0.2140.240 Cross-side leakage0.0030.005 Cross-contrast null0.0140.018 Contrasts above null p90100%100% Matched-projection correlation0.53–0.540.63–0.66 Matched decoder cosine (used features) 0.31–0.340.52–0.55 Table 30:Cross-seed feature-group matching. Qwen3.5-2B, ranges over the three seed pairs per width. The null passes frequency-matched pseudo- groups through the identical search (best-single Jaccard median 0.059, p99≈ 0.11, recall@3 0.25). Metric (range over pairs)32×16× Best-single Jaccard, median0.32–0.420.37–0.43 Groups above null p9989–90%87–91% Strong equivalence (J ≥ 0.5)36–41%40–47% Recall@3, median0.83–0.900.86–0.88 Recall@3 above null p9991–93%92% Matched decoder cosine, med / p90 0.20–0.26 / 0.80–0.89 0.40–0.42 / 0.90–0.94 seeds (matched decoder cosines average∼0.33 at32×), consistent with the documented seed- dependence of TopK dictionaries at scale (Paulo and Belrose, 2026). Yet every tested contrast re- produces its explanation above the null’s 90th per- centile, at∼15×the null level and with negligible cross-side leakage (Table 29). The decomposition of a selected score is therefore the trustworthy unit of interpretation. I.2 Feature-Group Matching A complementary question is whether the feature- level token groups themselves have an equivalent in another seed’s dictionary, with indices re-shuffled. We sample 100 features per seed pair from the bank- used set. Each feature’s group is its top-12 centered- row token set, and the counterpart is the best- Jaccard match over the full candidate dictionary. The null passes 500 frequency-matched pseudo- groups through the identical best-of-Dsearch, pric- ing in the inflation of searching 65,536 candidates, and recall@3 admits a greedy union of at most three candidate features, testing whether instability is feature splitting. About 90% of used feature groups have an above- chance counterpart in an independently trained dic- Table 31: Held-out recovery of the stable core. Qwen3.5-2B, showing recovery of the leave-one-out core by the held-out SRP dictionary from the same recipe, weighted kNN on the rows, and hard cluster- ing, on identical cores. The held-out SRP value (≈0.75) is the operative anchor, and recall should be read against it. Method32×16× Held-out SRP (same recipe) 0.749 [.73, .77] 0.720 [.70, .74] Weighted row-kNN0.493 [.44, .54] 0.432 [.38, .48] Hard clustering0.249 [.21, .29] 0.206 [.17, .24] SRP/kNN ratio1.521.67 Cross-contrast null0.0140.018 tionary, roughly 40% a near-exact single match, and the median group is 83–90% covered by a union of at most three features (Table 30), so token- group structure survives retraining even though fea- ture indices are reassigned, and the residual instabil- ity takes the form of feature splitting. The decoder- cosine distribution is bimodal, with a stable core recurring almost exactly while the rest recombine, and the narrower16×directions are more stable, consistent with Paulo and Belrose (2026). For the below-null tail (32/300 and 33/300 groups at the two widths), a direction-level check finds decoder counterparts at cosine≥ 0.5for 5/32 and 10/33. A qualitative pass suggests most of the remainder resemble the audit’s mixed/ambiguous class, but we have not rubric-scored that tail and do not quote it as an audited rate. The strong-equivalence and recall@3 thresholds were fixed for this analysis, with no sweep, so we report them descriptively alongside the threshold-free medians. I.3 Held-Out Recovery of the Stable Grouping Core The grouping comparison quoted in the main text asks which methods recover reproducible grouping structure, on identical targets. For each held-out seed, the core is the set of grouped tokens stable in both remaining seeds (leave-one-out construc- tion, all methods evaluated on identical cores, 95% contrast-clustered bootstrap CIs). Held-out SRP recovers 72–75% of the stable core, while kNN on the rows recovers 43–49% and clustering 21–25%, with SRP/kNN ratio CIs excluding parity (Table 31). Direct row geometry overlaps the grouping structure well above the null but misses most of the reproducible core that an independently trained dictionary from the same recipe recovers. 36 I.4 Cross-Recipe Disclosure The stability results above hold the training recipe fixed, while across recipes the agreement is weaker. The released paper dictionary, trained under an ear- lier preprocessing recipe than the seed-variation family, shares only 36% of the stable core and recovers 0.444 of the leave-one-out cores, below kNN on the rows in that comparison. Variation across recipes therefore exceeds seed variation within a recipe, and every stability claim in this paper is scoped to a fixed recipe. Comparisons in this appendix use the seed-variation family end-to- end, and no table mixes dictionary families. J Predicted Versus Realized Local Readout-Side Changes This appendix documents the intervention- validation protocol behind Table 2. Claim under test. For a selected contrast direc- tionq α and decoded state ̃ h ℓ , SRP asserts that featureicontributesc i = β i (α)p i ( ̃ h ℓ )to the realized score. If this attribution is correct, re- moving thed i -component of the decoded state should change the exact score by−c i . We there- fore ablate ̃ h ′ ℓ = ̃ h ℓ − ( ̃ h ⊤ ℓ d i )d i for unit-norm de- coder directiond i and measure the realized change s α ( ̃ h ′ ℓ )− s α ( ̃ h ℓ ) with the dense LM head, so the prediction uses the factorization while the measure- ment does not. The claim is falsifiable in three independent ways, since sparse-code misattribu- tion (a contribution assigned to the wrong feature), decoder non-orthogonality (overlapping features absorbing the ablated mass), and the row residual (score mass outside the fitted basis) each break the predicted–realized agreement. Random unit direc- tions supply the control. Protocol.Each model contributes approximately 260 contrasts from the main bank, and for each we take the top-10 features by absolute contribution plus 10 random-direction controls, giving 1,210– 2,050 prediction–realization pairs per model. We reportr 2 and the slope of realized change on pre- dicted contribution, fit through the origin, over cov- ered pairs (§3.4), with bootstrap 95% CIs, and re- sults over all pairs differ by less than 0.02 inr 2 throughout. A harness self-test on a residual-free synthetic factorization recoversr 2 = 1.000(slope 1.04) with zero-scoring controls, so the pipeline itself does not manufacture agreement. Results and reading. All six softcap-free read- outs giver 2 = 0.83–0.93against random-control r 2 ≤ 0.06(Table 2). Slopes are close to 1 on the 7–8B readouts (0.96–1.13) and rise to 1.34–1.69 on the Qwen3.5-0.8B/2B/9B readouts, where real- ized changes exceed predicted magnitudes while preserving sign and rank. A candidate explanation is feature interference, with overlapping decoder directions sharing ablated mass, but we have not isolated it and report slopes as measured. The vali- dated claim is local, since ablations act on the read- out side of a realized forward pass (linear in ̃ h ℓ , no re-forward), so the result licenses the claim that feature contributions predict local changes on the readout side. Circuit-level necessity, sufficiency, and generation-level effects call for separate inter- ventions (Conmy et al., 2023; Geiger et al., 2025). K Cross-Lens Study Protocol, Controls, and Per-Family Results This appendix documents the corpus conditionality study of §5. Lenses. We use the reference implementation of the Jacobian lens (Gurnee et al., 2026), which reads an intermediate hidden state by transport- ing it through an averaged Jacobian into the space consumed by the unembedding and decod- ing through the LM head. On Qwen3.5-9B we fit two lenses that differ only in fitting corpus, 100 seeded English C4 prompts against 100 seeded Chinese C4 prompts, with identical seeds, posi- tions, and procedure. Transport is evaluated at layers2, 5, 7, 10, 12, 14, 17, 19, 21, 24, 26, 29of 31 at the final prompt position. The cross-lens comparison uses layers 21, 24, 26, and 29, where the two lenses’ token outputs can diverge, while both lenses converge at the final layer. Compo- sition is validated numerically, since the decoded states ̃ h ℓ = T ℓ (h ℓ ) reproduce the lens’s own output scores to within 0.2%. SRP basis and comparison operation.Because the implemented lens feeds the unembedding di- rectly, the score each lens reports is a selected read- out score on ̃ h ℓ , and SRP decomposes it in the shared basis fitted from the weights alone (§3.3). The dictionary is an independently trainedk=128 (seed 0) readout SAE for Qwen3.5-9B, a different operating point from the32×/k=256configuration of the main tables, disclosed here per the rule of Appendix I for comparisons across recipes. No 37 Table 32: Cross-lens agreement on the dominant fea- ture by prompt family. Agreement means the same SRP feature is dominant under the English and the Chi- nese lens in at least two of layers 21, 24, 26, and 29, with 95% binomial CIs. “Lens-only” is the surface base- line, counting prompts where the English lens reports top-1 tokens in Latin script and the Chinese lens in CJK script (modal script across the same four layers), i.e. where token outputs alone suggest disagreement. Control families (below the rule) are excluded from the 77/80 headline. FamilyAgreement95% CILens-only split Antonym (ZH)14/14[0.78, 1.00]4/14 Cloze (EN)10/10[0.72, 1.00]7/10 Cloze (ZH)16/18[0.67, 0.97]6/18 Exemplar (ZH)8/8[0.68, 1.00]7/8 Factual recall (ZH)9/10[0.60, 0.98]9/10 Translation EN→ZH10/10[0.72, 1.00]6/10 Translation ZH→EN10/10[0.72, 1.00]0/10 All cross-lens77/80[0.90, 0.99]— Digit controls5/6[0.44, 0.97]0/6 Proper noun controls5/6[0.44, 0.97]0/6 translation pairs, language labels, or fitting corpora enter the basis, which is fitted to the rows of the LM head only. For each prompt, each lens’s reported score is decomposed at layers 21, 24, 26, and 29. The dominant feature at a layer is the one whose signed contribution is largest in absolute value. We count a prompt as agreement when the same fea- ture id dominates under both lenses in at least two of the four layers. Prompts.Eighty evaluation prompts span seven families, namely Chinese antonym (14), English cloze (10), Chinese cloze (18), Chinese exemplar (8), Chinese factual recall (10), and translation in both directions (10 each). A further 12 control prompts (6 digit, 6 proper noun) have an expected surface form that is language-invariant. Two null comparisons price in loose matching. Under the unrelated token null, the dominant feature of each reading is compared against the decomposition of an unrelated token under the same lens on the same state (159 comparisons). Under the shuffled pairing null, the two lenses’ readings are re-paired across different prompts (240 comparisons). Results. The same SRP feature is dominant un- der both lenses on 77 of 80 cross-lens prompts (96%, CI [0.90, 0.99]), uniformly across families (Table 32). Within a single lens, one feature car- ries both surface forms (e.g. the English and Chi- nese realizations of the same concept) on 61/80 (English lens) and 62/80 (Chinese lens) prompts. Both null floors are empty, at 0/159 matches un- Table 33: The antonym case, layer by layer. Top-1 token and its softmax share under each lens on iden- tical Qwen3.5-9B hidden statesh ℓ for the prompt of Gurnee et al. (2026) (“小”的反义词是, and the model answers大). The two Jacobian lenses differ only in fitting corpus. LayerEN-fittedZH-fitted 24 large 40%大的36% 26 large 26%大的73% 29 big 58%大93% final大98%大98% der the unrelated token null and 0/240 under the shuffled pairing null, so the headline agreement is not an artifact of a permissive matching criterion. The “Lens-only” column shows the surface picture the decomposition corrects, since on factual recall prompts the two lenses report different scripts on 9/10 prompts while the dominant feature agrees on 9/10. The worked example is the prompt Gurnee et al. (2026) use for their multilingual illustration, “the opposite of小”. The lens fitted on English re- portslarge/bigat layers 24, 26, and 29, while the lens fitted on Chinese reports大的/大, and both readings are carried by the same readout feature, whose top unembedding rows are the大family. Table 33 gives that case layer by layer. The shared feature f112 is the top contributor to大under both fitted lenses at each comparison layer (+12.5and +13.4at layer 24,+14.5and+17.6at layer 29) and is also the top contributor tobigunder both, so the two surface forms are carried by one coordinate within each lens as well as across the pair. A second language pair. The English–German bank removes the script confound, since both lan- guages use the Latin script, so agreement can- not come from the script alone. Ninety cross- lens prompts in seven families mirror the English– Chinese bank (German antonym, cloze in both languages, German exemplar and factual recall, translation in both directions), plus the same 12 controls. Every decomposed surface is validated single-token under the Qwen tokenizer, cognate pairs are excluded at normalized edit distance≤ 2 after diacritic folding (30 of 132 candidates ex- cluded, with the exclusion report released alongside the bank), and calls on the surface language use a lexical criterion, since the two languages share a script. Jacobian lenses fitted on English and on German (identical recipe, seeds, and 100-prompt fitting budget) diverge in their top-1 token readings on 73/90 prompts while the same SRP feature stays 38 dominant on 84/90 (0.93, CI [0.86, 0.97]), against floors of 1/204 (unrelated token) and 0/270 (shuf- fled pairing). Per family the counts are German antonym 7/9, German cloze 14/14, English cloze 15/15, exemplar 7/10, factual recall 8/8, translation DE→EN 17/17 and EN→DE 16/17, with controls at 5/6 (digits) and 6/6 (proper nouns). Within a single lens, one feature carries both surface forms on 62/90 (English lens) and 53/90 (German lens) prompts, weaker than in the English–Chinese study and reported as measured. One prompt read by three fitted lenses. Be- cause the two banks share a model and a basis, one hidden stateh ℓ can be read by three transports at once, which the pairwise design does not display. We takeantonym_de_01, the German analogue of the published antonym case (“Das Gegenteil von hoch ist tief. Das Gegenteil von klein ist”, model answergroß), and read its frozen states with the Jacobian lenses fitted on English, Chinese, and Ger- man (Table 34). At layer 21 all three return format- ting tokens, and at layer 24 each fills its ranking with its own fitting language. The English lens readslarge,large,-large,Large, the German lens readsgroß,große,large,großes, and the Chinese lens readslargefollowed by大了,巨大, 大的. The same shift is visible on a single token, since the transported logit forgroßis 15.16 under the English lens and 15.60 under the Chinese lens against 22.19 under the German lens. By layer 29 the lenses fitted on Chinese and on German both readgroßand the lens fitted on English alone still reads large. The decomposition separates the two language pairs here.großis carried by f6764 andlargeby f12474, whilebigand大share f112. The English and Chinese realizations of this concept share a readout feature and the English and German ones do not, which is consistent with the weaker carrying rate within a single lens reported for German above. That contrast comes from the row codes of the LM head alone, and this single prompt is reported as a worked example. A second lens family. To test whether corpus conditionality is specific to the Jacobian construc- tion, we fit a second family of ridge translators (Hoerl and Kennard, 1970), one per layer and with- out a bias term, into the final-layer residual basis, in the tuned-lens style (Belrose et al., 2023) but solved in closed form in state space, on exactly the same 100-prompt English and Chinese corpora Table 34: A German antonym prompt read by three fitted lenses. Top-1 token and its transported logit on identical Qwen3.5-9B hidden statesh ℓ for antonym_de_01, whose answer isgroß. The three lenses differ only in fitting corpus. LayerEN-fittedZH-fittedDE-fitted 24 large 22.4 large 21.5 groß 22.2 26 large 28.8 large 29.9 large 29.1 29 large 19.0 groß 19.5 groß 21.2 and layer set as the Jacobian lenses. Controlling the fitting corpus is what the test requires, so a re- leased tuned lens is unusable here. It arrives fitted on a corpus we did not choose, and matching our corpora would mean retraining it in any case. A closed-form translator keeps that retraining cheap, and because it shares no fitting procedure with the Jacobian lens, agreement between the two families cannot be an artifact of a common construction. We tune the ridge strength per layer, and at every layer the tuning picks the largest value we allow. Even that value contributes negligibly to the fit, so the translators are effectively ordinary least squares. HoldoutR 2 on a 10-prompt tail rises monotonically with depth (0.46–0.92 English, 0.46–0.86 Chinese), and the resulting decoded states match the model’s top-1 token on 61–75% of positions at the two deepest fitted layers. Under this family the corpus conditionality pattern reproduces (e.g. EN→ZH translation prompts read Latin under the translator fitted on English and CJK under the translator fitted on Chinese on 9/10), while the same SRP feature stays dominant on 67/80 prompts (floors 3/159 and 1/240). Finally, holding the fitting corpus fixed (English) and varying only the construction, the Jacobian and ridge lenses read the same dominant feature on 76/80 prompts with both floors empty (0/159, 0/240), so the decomposition is invariant to the choice of transport across the two families. A larger fitting corpus. The 100-prompt fitting budget is small, so a divergence between two lenses could in principle reflect estimation noise. We refit the English, Chinese, and German Jaco- bian lenses on 300 prompts each, drawn from the same seeded pools under an identical recipe, and re-read the same frozen states. The decomposi- tion does not move, since English–Chinese agree- ment is 77/80 again and the agreement decision is identical prompt for prompt, with the same 77 prompts passing, the same three failing, and the counts unchanged in every family and in both con- 39 Table 35: Cross-lens extension matrix. Agreement on the dominant feature (at least two of layers 21, 24, 26, and 29) with 95% binomial CIs and the two null floors (unrelated token / shuffled pairing). Row 1 is the main study, rows 2–3 vary one axis each, row 4 holds the corpus fixed and varies only the lens construction, and rows 5–6 hold corpus language and construction fixed while tripling the fitting budget. ComparisonAgreement95% CIFloors EN–ZH, Jacobian77/80[0.90, 0.99]0/159· 0/240 EN–DE, Jacobian84/90[0.86, 0.97]1/204· 0/270 EN–ZH, ridge translator67/80[0.74, 0.90]3/159· 1/240 Jacobian vs ridge, EN76/80[0.88, 0.98]0/159· 0/240 EN–ZH, Jacobian, n=30077/80[0.90, 0.99]0/159· 0/240 EN–DE, Jacobian, n=30085/90[0.88, 0.98]1/204· 0/270 trol groups. English–German rises from 84/90 to 85/90. The null floors are unchanged in both pairs (0/159 and 0/240 for English–Chinese, 1/204 and 0/270 for English–German). The surface reading does move, since script divergence on the English– Chinese bank falls from 39/80 to 33/80 and the factual recall figure quoted in §5 falls from 9/10 to 8/10, while token divergence on the English– German bank rises from 73/90 to 76/90. Eleven English–Chinese prompts change their reported surface language across the two fitting scales, and none changes its dominant feature. If the diver- gence were estimation noise that shrinks as the fitting corpus grows, it would shrink for both pairs, and it does not. What the fitting budget changes is how often the instrument’s surface report flips, while the readout feature carrying the reading stays fixed. Table 35 collects the agreement rates and both null floors for the main study and all three extensions. Caveats and scope. The Chinese and German C4 fitting corpora contain incidental English text, which biases the lens pairs toward each other and makes the token-level divergence conservative. The study covers one model, now across two language pairs, two constructions of fitted lenses, and two fitting scales, and the ridge translator is our min- imal instantiation in the tuned-lens style, simpler than the exact construction of Belrose et al. (2023). The released records for each case include every prompt, layer, dominant feature id, and top-1 token reading. L Additional Qwen Displays and Feature Audits This appendix collects Qwen-family companion analyses that support the main text. It proceeds from additional margin examples to variants of se- lected scores, fixed-token context shifts, projection profiles, logit-lens comparisons, and token row au- dits. All figures use the same Qwen3.5-2B32×, native-k = 256SAE trained on unembedding rows and the reconstruction error reporting conventions from Appendix A.3. L.1 All-Layer Logit-Lens Dense Table Figure 17 gives an all-layer dense logit-lens dis- play for the literature promptIt was the best of times, it was the worst of times. The dis- played columns cover the first eight prompt tokens. The top panel is the ordinary logit-lens top-token trajectory after applying the model’s final normal- ization at each layer. The bottom panel lists the two largest positive decomposition terms support- ing each cell’s vocabulary mean top-token contrast. The feature labels are decoded from top Qwen un- embedding rows as token-label summaries for the feature ids. L.2 Additional Logit-Difference Case Studies Figure 18 collects the four curated pairwise logit differences read in §4.5, namely the twobugcon- trasts, and thebarkandbassdifferences. In the tree context,barkdraws on bark/wood SAE fea- tures against dog/cat features, and in the music contextbassdraws on music/percussion-related features against trout/salmon features (relative re- construction errors 0.126 and 0.097). L.3 Additional Selected Readout Score Examples Figure 19 gives the single-state example summa- rized in the main text, where raw-token, vocabulary mean contrast, and pairwise logit difference scores yield different local score decompositions for the same decoded state. Figure 20 adds a group con- trast and a top competitor margin. L.4 Benchmark-Format Readout Contrast Examples Table 36 reports reconstruction for answer, absten- tion, label, and safe/action contrasts derived from benchmarks and instantiated as fixed combinations of rows of the LM head (Rajpurkar et al., 2018; Yang et al., 2018b; Koreeda and Manning, 2021; Guha et al., 2023; Siddiq and Santos, 2022). These rows are used as reconstruction checks for read- out contrasts. Lower-error QA examples appear in Figure 21. 40 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 output Layer Itwasthebestoftimes,it ItwaslekbestTEN-timesspace-it ItwasumobestspaceTimesspace-it j ambledadamndcircuspaceavic zea rij cemdamnkedspacespaceTA 9amer ayer spacethemspacespaceTA zeahto.......damnicablespacespaceirem 9 rij ...,damnamblelichsten..., chy zeaspace...,damnspacespace..., chy Magdalena lah...,damnspacespace _, alic Magdalena ____ ......spaceicablespace Subject uration Magdalena ________ sgi :space y uration Magdalena anth ___ sgi "].(AMPLapparspace Magdalena ___ ...' sgi ']*AMPL y chy <empty> /is _____ space:pszappid**** Matent ____ spaceminepsz**- chy gra ntac -selling "..AMPLtisspace chy spacedarkest -selling possible ighted ..."space chy _____________ -casepokeperk...,space chy ____ same -selling timesperk..."was chy ntsame -selling timesperktimeswas chy decidedhottest-oftimesheetstimeswas chyraining first-oftimes...**itwas isafirst-oftimes,itwas isafirstoftimes,itwas Qwen3.5-2B Logit Lens Prediction Itwasthebestoftimes,it Input token 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 output Layer f55643 It/It f5969 it/ f323 was/was f5636 was/ was f596 the/ f55418 the/ f52709 Best/Best f20718 Best/ Best f751 of/ c a f12688 Of/Of f39502 _TIMES/_times f54036 time/ Time f786 ',/,' f2996 ("/ (( f5969 it/ f55643 It/It f55643 It/It f5969 it/ f323 was/was f5636 was/ was f1607 lek/ Lek f52820 leader/Leader f52709 Best/Best f20718 Best/ Best f58422 Ten/ Ten f18530 ten/ Ten f344 times/ f39502 _TIMES/_times f4328 / f5634 / f5969 it/ f55643 It/It f55643 It/It f23681 M/P f323 was/was f5636 was/ was f18420 UMO/umo f10125 UMA/uma f52709 Best/Best f20718 Best/ Best f25315 live/live f357 / f39502 _TIMES/_times f54036 time/ Time f31692 / f12890 / per f5969 it/ f55643 It/It f58618 j/ f1664 0/ f53323 / f25536 alliances/ all... f39062 da/ da f23361 d/dT f27722 damn/ damn f9918 darn/ damn f52341 Fucking/ fucki... f4687 TED/TED f2895 / f49022 circ/circ f41725 / f4361 or/ f32987 av/AV f974 bic/ bic f36302 za/ZA f40389 / f58367 nij/nije f24292 mittel/mitteln f42390 cem/cami f46542 cem/ Cem f9918 darn/ damn f27722 damn/ damn f54402 kers/ker f21275 ked/ Ked f17125 _href/Href f39035 ONTAL/ENTAL f10135 / f56584 / f22393 -Ta/Ta f48523 T/ T f7766 9/8 f167 6/ f25483 ame/AME f56338 amera/AMERA f31481 ayers/ayer f13376 / f41670 / f873 and/ v f19228 them/them f23334 they/ f56844 / f11989 filmu/ projektu f50840 / f10144 epid/ Epid f22393 -Ta/Ta f6403 TD/ TD f36302 za/ZA f40389 / f29870 ht/ht f6714 parto/ childbir... f2528 ......./.......... f7580 / f27722 damn/ damn f56113 freaking/ fuck... f50653 icable/hetic f53323 / f52537 / f832 / f31221 / f34915 / f5319 betracht/ betra... f43806 irea/ire f7766 9/8 f167 6/ f28404 zij/ zij f2865 Prij/ prij f40331 ...,/ , f7580 / f27722 damn/ damn f9918 darn/ damn f60479 / f1362 amblea/amble f7146 lichsten/lichem f57455 fts/lie lich f40331 ...,/ , f1481 ,(/ ," f56592 chy/chy f35307 / f36302 za/ZA f34982 cea/cee f18260 / f63046 colder/ cooler f40331 ...,/ , f7580 / f27722 damn/ damn f9918 darn/ damn f46707 / f21707 / f63640 / f16320 / f40331 ...,/ , f1481 ,(/ ," f56592 chy/chy f35307 / f62612 Marta/ Magdale... f26031 Manuela/ Gabri... f54621 lah/lah f8906 la/las f40331 ...,/ , f7580 / f27722 damn/ damn f9918 darn/ damn f59656 / f24505 / f27500 Platform/Platf... f12640 / f17491 _,/_, f35898 _./_, f60747 / f65446 ali/ALI f26031 Manuela/ Gabri... f62612 Marta/ Magdale... f33953 ______/ ______... f59548 __/ __ f2528 ......./.......... f7580 / f37166 / f6152 maintenant/ Mai... f50653 icable/hetic f10725 able/ABLE f14484 / f26111 / Retro f11553 subject/ subje... f22944 _subject/_SUBJ... f8106 URATION/uration f63793 ization/isation f26031 Manuela/ Gabri... f62612 Marta/ Magdale... f33953 ______/ ______... f59548 __/ __ f33953 ______/ ______... f59548 __/ __ f55113 cgi/cgi f47310 agner/ Wagner f50944 ,/ . f52412 / ) f14484 / f26111 / Retro f32451 m/ n f28246 / f8106 URATION/uration f63793 ization/isation f62612 Marta/ Magdale... f26031 Manuela/ Gabri... f22468 anth/ Anth f13896 anthropology/ ... f5977 ___/_____ f59548 __/ __ f55113 cgi/cgi f47310 agner/ Wagner f26965 "].(/ faptul f34315 ><?=/><?=$ f51300 Szk/ szk f28153 mpl/ mpl f15751 appar/appar f47180 / f61900 / f2548 / f62612 Marta/ Magdale... f26031 Manuela/ Gabri... f5977 ___/_____ f59548 __/ __ f7580 / f12013 ...*/...** f55113 cgi/cgi f47310 agner/ Wagner f10965 ']*/]* f12952 ']//"]/ f41182 AM/AM f51300 Szk/ szk f32451 m/ n f28246 / f56592 chy/chy f35307 / f24754 <empty>/ulator f167 6/ f3403 -is//is f48134 IS/IS f33953 ______/ ______... f1243 ___/ ____ f41670 / f25465 / f50944 ,/ . f28781 ':/": f16730 psz/psz f64406 /qry f45172 appid/appId f53486 app/ f18999 ****/ **** f44769 *****/****** f64969 Mate/ Mate f62967 Mat/Mat f30711 nte/nten f38018 .nt/ nt f33953 ______/ ______... f59548 __/ __ f41670 / f25465 / f6888 mine/ Mine f48868 min/max f16730 psz/psz f64406 /qry f62604 **-/** f53323 / f31403 latino/ Latin f28082 / f9987 ger/gers f36302 za/ZA f30711 nte/nten f38018 .nt/ nt f41964 ac/ ac f37853 / f38633 selling/ selli... f45206 -quality/quali... f21963 / f38372 )../".. f28153 mpl/ mpl f20387 Sampling/ Samp... f52672 tis/ tis f9221 t/ t f1975 is/ f25600 / f56592 chy/chy f54646 (timer/ f43903 / f19279 / Need f2504 brightest/ brig... f59481 largest/ Large... f38633 selling/ selli... f39867 Sellers/ selle... f56357 Possible/Possi... f39797 posible/ possi... f29366 ighter/ighting f13357 ighth/igth f14442 ...,/ ...) f5433 ...../ .... f58688 / f45898 / f56592 chy/chy f42325 / f33953 ______/ ______... f59548 __/ __ f4229 _________/_____... f33953 ______/ ______... f2189 case/ Case f65353 case/cases f18239 poke/ poke f21963 / f14459 _IS/_is f14233 / f14442 ...,/ ...) f1481 ,(/ ," f41670 / f30982 / f56592 chy/chy f35307 / f33953 ______/ ______... f59548 __/ __ f52602 same/same f435 <space>/ f38633 selling/ selli... f6711 sell/ sells f54036 time/ Time f39502 _TIMES/_times f14459 _IS/_is f1615 period/ periods f14442 ...,/ ...) f1243 ___/ ____ f323 was/was f1975 is/ f35307 / f54646 (timer/ f30711 nte/nten f41887 nd/ nd f52602 same/same f435 <space>/ f38633 selling/ selli... f6711 sell/ sells f54036 time/ Time f39502 _TIMES/_times f14459 _IS/_is f13649 perte/pert f54036 time/ Time f344 times/ f323 was/was f5636 was/ was f56592 chy/chy f54646 (timer/ f38976 decides/ decide f40289 / f59481 largest/ Large... f2504 brightest/ brig... f751 of/ c a f31862 of/ f54036 time/ Time f39502 _TIMES/_times f17163 sheet/ Sheet f34881 _sheet/Sheet f54036 time/ Time f344 times/ f323 was/was f1975 is/ f56592 chy/chy f26979 chi/Chi f38319 raining/ rains f53323 / f12548 first/first f7400 /first f751 of/ c a f31862 of/ f39502 _TIMES/_times f54036 time/ Time f12013 ...*/...** f7580 / f5969 it/ f55643 It/It f323 was/was f1975 is/ f1975 is/ f323 was/was f7055 a/ f596 the/ f12548 first/first f435 <space>/ f751 of/ c a f31862 of/ f39502 _TIMES/_times f344 times/ f786 ',/,' f1481 ,(/ ," f5969 it/ f55643 It/It f323 was/was f1975 is/ f1975 is/ f323 was/was f7055 a/ f596 the/ f12548 first/first f40106 that/ the f751 of/ c a f40106 that/ the f54036 time/ Time f39502 _TIMES/_times f786 ',/,' f40106 that/ the f5969 it/ f55643 It/It f323 was/was f1975 is/ Current-SAE Sparse Readout Features for the Logit-Lens Top Token Figure 17: All-layer Qwen3.5-2B logit-lens dense table. The prompt isIt was the best of times, it was the worst of times, and columns are the first eight tokens. The top panel gives the ordinary logit-lens top-token prediction at each layer, and the bottom panel the top positive decomposition terms for each cell’s vocabulary mean top-token contrast under the Qwen3.5-2B 32×, native-k = 256 readout SAE used throughout this appendix. 41 −1 0 1 1 2 3 4 5 A Insect: bug-error ex +7.7; sp +6.2; r +1.4 1 f59571M, J, G, K 2 f13081 Error, error, _... 3 f18302Beetle, beetle,... 4 f4095bee, bees, Bee,... 5 f36mosquito, mosqu... 0 1 1 2 3 4 5 B Software: bug-insect ex +20.3; sp +20.4; r -0.1 1 f2996M, G, a, F 2 f39674 bug, Bug, BUG, ... 3 f5680 debug, Debug, .... 4 f43419Crash, crash, c... 5 f21804defect, defects... 0 2 1 2 3 4 5 C Tree: bark-dog ex +10.9; sp +9.6; r +1.4 1 f13856 dog, dogs, Dog,... 2 f40106that, the, and,... 3 f30192wood, Wood, boi... 4 f61230bark, äsident, ... 5 f55275Bark, bark, Arm... −2 0 1 2 3 4 5 D Music: bass-trout ex +13.0; sp +11.7; r +1.3 1 f44118trout, Trout, s... 2 f40106that, the, and,... 3 f62935Music, music, M... 4 f42908percussion, Per... 5 f37669 bassi, bajos, a... Figure 18: Sparse Readout Prism case studies. Four selected Qwen3.5-2B (D = 65536,k = 256) pairwise logit differences. Positive terms support the first token in each contrast and negative terms support the competitor. The examples arebug-vs-errorin an insect context,bug-vs-insectin a software context,bark-vs-dogin a tree context, andbass-vs-troutin a music context. Small bars are suppressed for readability. All four panels report the unflooredρ 0 < 0.5and preserve the sign of the logit difference (display rules in Appendix A.3). Table 39 lists the bug-panel feature rows. −10010 feature contribution f29839 culprit / / offen... f19085 gu / Gu / gu f39358 happy / happy / H... f16310 Gu / Gu / gu f13473 accused / acusado... f9249 responsible / Res... f37898 convicted / sente... f18716 guilty / guilt / ... +0.2 +0.2 +0.3 +0.4 +0.4 +0.6 +0.7 +11.2 A Selected token: Selected token score exact +18.4; sparse +20.2; rho 0.01 −10010 feature contribution f29839 culprit / / offen... f19085 gu / Gu / gu f39358 happy / happy / H... f16310 Gu / Gu / gu f13473 accused / acusado... f9249 responsible / Res... f37898 convicted / sente... f18716 guilty / guilt / ... +0.2 +0.2 +0.3 +0.4 +0.4 +0.6 +0.7 +11.2 B Availability: Token vs vocabulary mean exact +20.4; sparse +20.2; rho 0.01 −10010 feature contribution f47071 not / not / Not f2073 Not / Not / _not f254 in / / f2996 (" / (( / M f23681 M / P / S f46953 the / an / a f18884 are / has / is f18716 guilty / guilt / ... -7.9 -1.9 -1.6 -1.4 +1.5 +1.9 +2.5 +11.2 C Named arbitration: Guilty over not exact +2.7; sparse +2.4; rho 0.10 Figure 19: Same decoded state, different selected scores. All panels use the same final state presented to the LM head for the prompt “After reviewing the evidence, the jury found the defendant”, selecting the rawguiltyscore, theguilty-versus-vocabulary mean contrast, and theguilty-minus-notlogit difference. Positive bars increase the selected readout score and negative bars reduce it. −202 feature contribution f41099 / / f37327 / / Blue f31242 green / green / f46269 BLUE / BLUE / Blue f19687 black / black / f46953 the / an / a f47071 not / not / Not f2359 unknown / unknown... -2.4 -1.6 -1.6 -1.1 -1.0 +0.9 +1.3 +1.8 A Token-family arbitration: Abstention family exact -5.9; sparse -6.4; rho 0.09 −10010 feature contribution f786 ', /, ' /, \ f2996 (" / (( / M f25997 effects / Effects... f254 in / / f46953 the / an / a f23681 M / P / S f59571 M / J / G f18716 guilty / guilt / ... -4.6 -3.4 -2.0 -1.9 +2.3 +6.1 +6.6 +11.2 B Selection pressure: Winner vs rank-5 competitor exact +3.6; sparse +3.5; rho 0.04 Figure 20: Additional Qwen examples of group contrasts and top competitor margins. On the left is the abstention-group versus forced-color-group contrast, and on the right is how far the top-rankedguiltyrow leads its rank-5 competitor. Bars increase or reduce the selected readout score by sign, and labels give feature ids and token summaries. 42 202468 signed feature contribution f56784 Norman f18515 German f27564 French f2513 Sweden f22163 Fr f22161 France -1.7 -0.3 +0.3 +0.4 +0.5 +8.2 A. SQuAD2: France - distractors 202468 signed feature contribution f15365 Santa f24306 Tenerife f53323 webElementX f42651 can f64201 Spanish f25361 Canary -1.7 -1.5 +0.4 +0.4 +0.4 +7.3 B. HotpotQA: Canary - distractors Figure 21: Local score decompositions for contrasts derived from benchmarks. Positive bars support the gold/intended side and negative bars support the distractor side. Both Qwen3.5-2B panels use the selected32×, k = 256 readout SAE and have correct sign with low relative error. Table 36: Reconstruction of readout contrasts de- rived from benchmarks. Rows evaluate answer-vs- distractor, abstention-vs-entity, label, or safe-vs-unsafe readout contrasts. Relative errors are the unflooredρ 0 , and the last column gives sign agreement withρ 0 < 0.5. All aggregate rows use 300 contrasts, and family rows break down Qwen3.5-2B. ModelContrast sourcen Med. ρ 0 Sign Sign & ρ 0 < .5 Q-0.8B all contrasts from benchmarks 3000.180 0.9830.933 Q-2Ball contrasts from benchmarks 3000.129 0.9430.840 Q-2BSQuAD2 answerable500.150 1.0000.900 Q-2BHotpotQA distractor750.125 0.9330.907 Q-2BLegalBench ContractNLI500.053 1.0001.000 Q-2BSQuAD2 unanswerable750.157 0.9200.787 Q-2BSecurityEval500.393 0.8800.600 L.5 Fixed-Token Context Comparisons The fixed-token analyses of §4.5 subtract two prompt contexts for the same selected token. Fig- ure 3 gives the signed feature terms for the two bugcontrasts discussed there. Figure 22 shows thebugandbridgecases, and Figure 23 gives the additional ring example. Figure 24 gives the companion prompt-support view for the fixed-token cases, retaining shared positive support that the signed difference plot sub- tracts away. Figure 25 adds one technical-domain context comparison. The most clearly separated panels are graph,memory, andvirus, whilepipeillustrates a mixed Unix/plumbing case. Across the four panels, the maximum rank of the selected token is17and the maximum relative reconstruction error is 0.05. L.6 Readout-Feature Profiles The examples above decompose selected token rows or contrasts. As a complementary view, we 1.00.50.00.51.0 f36 mosquito f4095 bee f32574 sect f52267 Issue f21804 defect f43419 Crash +0.8 +0.6 +0.3 -0.5 -0.9 -1.0 A. bug: insect - software 0.40.20.00.20.40.60.8 f54636 jembatan f55298 river f62208 boat f1070 hybrid f9991 engineer f54655 address +0.8 +0.4 +0.2 -0.2 -0.2 -0.3 B. bridge: structure - network Figure 22: Same selected token, different prompt contexts. Qwen3.5-2B (D = 65536,k = 256). Each panel fixes the output token and plots the first-minus- second change in feature contribution to ̃ h ⊤ L (w v − μ), the token logit with the shared offset term removed, with teal more in the first context and orange more in the second. A.bug: insect−software. B.bridge: structure− network. ring companion: Figure 23. 1.21.00.80.60.40.20.00.20.4 f53287 band f435 <space> f1594 -S f65502 ringrazi f3987 rang f32300 ringing +0.3 +0.2 +0.2 -0.2 -0.2 -1.0 ring: jewelry - phone Figure 23: Additional fixed-token context compari- son forring. Selected token fixed, with the jewelry context compared against the phone context, using the same signed-difference convention as Figure 22. 43 0.00.51.01.5 feature contribution f2996 (", ((, M, (! f43419 Crash, crash, crash... f36 mosquito, mosquitoe... f40106 that, the, and, ... f21804 defect, defects, 缺陷... A bug: insect / software rank 6/1; centered +16.7/+19.7 insect software 0246 feature contribution f25681 环, 環, ring, Ring f32300 ringing, ring, Ring... f40106 that, the, and, ... f3118 rings, ring, -ring,... B ring: jewelry / phone rank 8/2; centered +15.4/+18.7 jewelry phone 0246 feature contribution f40106 that, the, and, ... f9237 bridge, bridge, bri... f38597 Bridge, Bridge, bri... C bridge: structure / network rank 1/2; centered +19.3/+17.0 structure network Figure 24: Prompt-support companion for the main fixed-token examples. Each panel holds the selected token fixed and compares positive SAE feature contributions in two prompt contexts. Bars show the union of the four largest positive contributions per context, with teal the first context in the panel title and orange the second. Missing bars are features outside that context’s top-four set. This view highlights shared positive support that the signed main-text difference plot subtracts away. Figure 25: Technical-domain fixed-token context comparisons. Positive bars are larger in the left context and negative bars larger in the right.graphseparates chart/diagram support from edge/vertex-like graph-structure support,memoryseparates human memory from computer-memory/RAM support, andvirusseparates biological- virus from malware/antivirus features. pipe shows a mixed technical/plumbing local score decomposition. rank readout features directly from the decoded state before choosing a token or logit difference. For the final decoded state ̃ h L , we score each SAE decoder direction by ̃ h ⊤ L d i . Vocabulary rows then label the selected directions by their highest- associated rows. The resulting figures therefore show projections in readout space before score se- lection, while score-specific figures provide the local score contributions. Figures 26 and 27 give these profiles for theshell,Java,port, andcell prompts. L.7 What SRP Adds Beyond the Logit Lens Figure 1 gives the main-text lens-versus-prism com- parison. Table 37 lists four additional Qwen3.5-2B examples where the ordinary logit lens gives a local A-vs-B logit difference and the SRP local score de- composition separates the competing feature terms behind that contrast. In the prompt-injection case SRP decomposes an undesirable local readout rank- 44 201001020 f45045: shell / Shell / Shell / shell / shells / shell / shell / / (shell / .shell / / / / / / bash / kernel f12747: coil / Coil / coils / coil / / cun / strand / winding / tube / wound / / bulb / piant beam / strand / END / loop / clamp f46747: pattern / Pattern / pattern / pattern / Pattern patterns / pattern / Patterns / patterns / Patterns (pattern / .pattern / / attern / patterns Muster / padrões / .Pattern / PATTERN / mø f15782: shoe / shoes / Shoe / / Shoes / sepatu / / chaussures / zapatos / giày / / / / footwear / glove / Schuh / scarpe f751: of / ca / of / Of / Of / .of / of / / of / OF / OF / / Ca thereof / / / / ca +21.4 +19.3 +17.6 +17.5 +17.4 shell: beach Walking along the beach, she picked up a spiral logits: shell:+17.38; of:+17.12; notebook:+15.88; -shaped:+15.81;... 201001020 f45045: shell / Shell / Shell / shell / shells / shell / shell / / (shell / .shell / / / / / / bash / kernel f6459: Support / Support / support / support / Control support / support / Compliance / Expect / Access Use / .Support / Supply / Supports / Performance Output / Service / Consider / Interpret / Release f9075: command / command / Command / commands / Command / (command / / command / Commands command / commanded / commands / comando / COMMAND cmd / .command / command / comand / commanding f35005: parser / parser / parser / .parser / parser / Parser / router / parsers / decoder parser / tokenizer / parsing / Parser / (parser serializer / scheduler / lexer / encoder / compiler f58324: editor / Editor / editors / Editor / editor / editor / / editor / (editor editor / / EDITOR / Editors / editorial .editor / .Editor / / EDITOR / ditor +23.0 +20.7 +20.5 +20.0 +18.5 shell: terminal The command was typed into the Unix logits: shell:+17.25; command:+16.88; terminal:+16.62; system:+15... 201001020 f21282: tea / Tea / tea / teas / / / / coffee / / trà / aj / / / Te / / / f11678: Coffee / Coffee / coffee / coffee / / / caffè / / café / kopi / Café / / / Espresso / / / f33592: specialty / Specialty / specialties / specialist / Spezial specialize / / spécial / specialists / specialized / niche / specializes / obligated / Spécial Special / hobby / specializing / / specialization f48858: / the / was / is / has can / have / had / we / are will / he / would / t / must a / pe / could / la / l f23681: M / P / S / K / G A / D / B / N / J I / Y / H / O / C F / T / X / In / L +22.3 +20.5 +19.2 -19.1 -19.1 Java: coffee The cafe served a strong cup of logits: coffee:+17.88; tea:+15.38; hot:+14.69; black:+14.06; espr... 201001020 f11066: Focus / Focus / focus / focus / .Focus focus / focusing / focuses / Fokus / focus / / Fit / / Shift ocusing / Match / .focus / Trigger / FOCUS f2309: JavaScript / JavaScript / javascript / Javascript / javas... Python / TypeScript / Java / Javascript / Ruby JS / javascript / PHP / Python / js .Java / typescript / PowerShell / Java / .js f47549: Angular / Angular / angular / angular / angular Vue / angular / React / Vue / Flutter Artificial / Cold / Chromium / / Bootstrap TypeScript / Kotlin / Conservation / vue / Rails f42050: Entity / Entity / entity / entity / Entities entity / entities / .Entity / / entity entity / ENTITY / (Entity / .entity / ENTITY entidade / Entities / entidad / / Entity f44239: Spring / Spring / spring / springs / spring (Spring / .Spring / pring / / Springs / .spring / / Autumn / Springer @Spring / Winter / / Summer / .springframework +24.5 +22.5 +22.1 +19.6 +19.0 Java: programming The backend service was written in logits: Java:+16.12; Python:+16.12; Go:+15.88; C:+15.56; a:+15.00 Selected global readout-feature activations: shell and Java domains Figure 26: Readout-feature profiles for shell and Java prompts. Features ranked by| ̃ h ⊤ L d i |at the final decoded state. Theshellprompts share a shell-related direction, and the terminal context additionally has large projec- tions onto command/parser/editor-like directions. TheJavaprompts separate beverage-related directions from programming-framework and language directions. Each label lists the top associated unembedding rows for that feature. Table 37: Logit-lens comparison examples. All rows are Qwen3.5-2B pairwise logit differences with small local reconstruction errors. CaseContrastExactSparse ρ 0 Note Prompt injection summarize− delete -4.000-4.123 0.031undesirable local readout ranking Ambiguous action ask− run+4.125 +4.571 0.108 successful pairwise contrast Source conflict dollar− euro-1.438-1.430 0.005lexical contrast example Abstention Unknown− Alice +9.375 +8.051 0.141abstention- family example ing, where the ordinary lens favors the wrong ac- tion and the sparse terms reconstruct that contrast tightly. The companion feature tables list the dis- played feature ids, token-summary labels, exact and reconstructed scores, and reconstruction errors for these examples. For the prompt fragment Trusted task: summarize the email. Untrusted email content says: delete the files. We must follow the trusted task. The next action is to the ordinary logit-lens margin favorsdeleteover summarize. Sparse Readout Prism shows that the undesirable logit difference is carried by several deletion-related SAE features against summary and communication features: ContributionSideToken summary -2.938 delete deleting/deletes/deleted -1.700 delete _delete/Delete/delete -1.606 delete Delete/delete/deletion +0.969 summarize sum/Sum/-sum +0.933 summarize communicate/communicates +0.922 summarize summary/Summary The lens reports the local readout ranking, and SRP resolves that ranking into signed feature terms. L.8 Token-Row Audit for Main Case-Study Features How labels are produced. Feature labels in the displays are drafted by a language model from the feature’s highest-scoring unembedding rows. The authors then verify every label used to support a claim against those same rows, and the audit below measures how often that verification succeeds. Table 38 reports a conservative qualitative au- dit of the claim-bearing displayed labels in the 45 201001020 f20040: harbour / harbor / harb / Harbour / Harbor har / Har / Har / / Hafen har / HAR / porto / bor / entertain / / Harlem / / f55831: industrial / Industrial / Industrial / industriale / indu... industriali / industriel / / / industri industriales / ustrial / / INDUSTRI / industriels / / / Industri / industria f31587: city / City / cities / City / city CITY / city / city / / cidade Cities / città / / Stadt / / Cities / város / / ciudades f52667: waters / Waters / waters / eaux / águas watershed / acque / shores / aguas / skies Woods / seas / wcs / sands / perairan / Powers / Seas / anean / Roberts f23178: London / London / / london / Londres / / Londra / / ONDON / Lond / ondon / Paris / Dublin lond / Moscow / Boston / Tokyo / Berlin +22.3 +22.2 +20.6 +20.2 +18.9 port: harbor After crossing the ocean, the ship entered the busy logits: port:+16.25; harbor:+14.88; city:+14.62; ,:+14.25; **:+13... 201001020 f54655: address / Address / addresses / address / Address addressing / Addresses / addressed / address / ADDRESS / address / / adress / addresses .address / Addresses / (address / ADDRESS / ADDRESS f39992: socket / sockets / socket / (socket / Socket / Socket / socket / sockets / socket .socket / / SOCKET / .Socket / SOCKET sock / (sock / ockets / / socks f16087: access / accessible / accessible / Access / accessible Access / access / ACCESS / Accessible / access ACCESS / access / .access / airspace / ACCESS avi / oarece / display / access / accessibility f50723: interface / interface / Interface / / interfaces Interface / interface / interface / (interface / / / / .interface / interface INTERFACE / interfaces / Interfaces / / f31182: resolver / Resolver / resolver / Resolver / resolver resolv / resolving / resolves / responder / resolve registrar / risolvere / resolved / mover / resol mun / vender / resolution / resolutions / solucionar +24.5 +24.2 +23.1 +22.7 +21.7 port: network The web server was configured to listen on a network logits: interface:+20.50; port:+19.62; address:+19.50; IP:+18.25;... 201001020 f5751: wooden / Wooden / / / / madera / / madeira / hout Holz / g / rustic / Wood / sandy legno / / timber / / sturdy f53825: warehouse / Warehouse / Warehouse / warehouses / warehouse arehouse / / / wholesale / bookstore truck / ware / magazzino / / supermarket Ware / iller / bunker / mall / lih f27709: iron / Iron / Iron / iron / / hierro / / / / st / / / steel / / / manganese / besi f64070: castle / castle / Castle / fortress / Castle / palace / cathedral / castell / castello chapel / château / astle / monastery / casino mansion / / dungeon / splendida / princess f62208: boat / Boat / boat / boats / bateau barco / boats / canoe / kayak / barca yacht / thuyn / / / ferry / / / / +24.5 +21.9 +20.9 +20.8 +20.6 cell: prison After the trial, the prisoner was locked in a small logits: cell:+17.00; room:+17.00; ,:+16.75; wooden:+15.06; dark:+... 201001020 f23681: M / P / S / K / G A / D / B / N / J I / Y / H / O / C F / T / X / In / L f50498: / / / / organismo organismo / organism / organisme / organisms / / organisme / szervezet / / organizmu / / / organismos / f17948: lizard / rept / izard / monkey / shark squirrel / dinosaur / salam / ape / Bird / monkey / frog / sharks / Martian Monkey / ília / Shark / izards / Monkey f5112: employee / Employee / Employee / employee / employees / .employee / employee / student / (employee worker / Employees / / / inmate / / customer / employer / shareholder f23628: coral / Coral / / salmon / reef reefs / Fred / Reef / ivory / Dino turquoise / tuna / peach / / yster moss / / dolphin / cJSON / Ivory -21.9 +19.0 +19.0 +18.9 +18.6 cell: biology Under the microscope, the biologist examined a single living logits: cell:+19.12; organism:+17.38; animal:+15.81; bacter:+15.5... Selected global readout-feature activations: port and cell domains Figure 27: Readout-feature profiles for port and cell prompts. Theportprompts separate harbor/city/water directions from address/socket/interface directions. Thecellprompts separate prison-like physical-enclosure directions from biology and organism-related directions, while also exposing a generic uppercase-letter direction as a non-lexical high-magnitude component. Qwen feature figures. The audit classifies label readability, a separate question from feature valid- ity. Coherent labels name a clean semantic lexi- cal family read off the top-20 token rows, while token form labels are surface artifacts (letter cas- ing, leading-space, and sub-word surface forms, punctuation-like rows) that carry no semantic fam- ily. A stricter on-target test then asks whether a coherent family also matches the sense named in its panel and points to the side indicated by the signed feature term. The source table for each row, refined feature descriptions, and full top-20 token row audit accompany the figure artifacts, and the shipped artifact records the earlier three-way co- herent/ambiguous/token form classification, which the binary scheme above refines by folding ambigu- ous labels into the coherent count and adding the on-target test. Blinded rater runs. The author audit is com- plemented by the blinded audit behind the labels of §4.5, namely three independent rater runs of gpt-5.6-solat high reasoning effort, each in a fresh agent context with a frozen prompt, rating only the top-20 unembedding rows of each claim- bearing feature (no access to panel claims, signed terms, or the author classification). The three runs agree unanimously on 93.4% of items (Fleiss’ κ = 0.87), and their majority-rule consensus over the 99 display occurrences is 76 coherent, 4 mixed or ambiguous, and 19 token form, agreeing with the author audit on 81.8% of items, with disagreements skewing toward rating labels more coherent than the author did. We report this as a same-model reproducibility audit under a fixed protocol and retain the frozen prompt, per-run outputs, and ag- gregation rule in the released artifacts. Item order was randomized under a fixed seed, and generation temperature was not exposed by the agent runtime. Across the displayed panels, 85 of 99 claim- bearing labels name a coherent lexical family at top-20 token depth, and the remaining 14 are token form surface artifacts. By the stricter on-target test, 66 of the 85 coherent families are on-target, all side-consistent, with no case where an on-target label opposed the side claimed in the prose. The remaining 19 coherent families are off-target, being clean families that name an off-panel sense or that are dominated by the panel’s own target token. In 46 Table 38: Qualitative feature-label audit for claim-bearing Qwen displays. Rows count the displayed, claim- bearing feature labels. Coherent labels are clean semantic lexical families read off the top-20 token rows, while token form labels are surface artifacts (letter casing, leading-space and sub-word surface forms, punctuation-like rows) that carry no semantic family. The on-target column is a stricter subset of the coherent count, holding coherent families whose sense matches the named family used in that panel and whose signed contribution points to the claimed side. Feature setLabels Coherent Token-form On-target Scope Main bug panels11838 Figure 18, detailed rows in Table 39. bark/bass margins12937 Figure 18. Fixed-token context panels121209 Figure 22. Technical context stress3231124 Figure 25. verify/assume DLA8807 Feature heatmap Figure 30, also Figures 28 and 32. General readout examples2417711 Figures 19 and 20. Total99851466 Conservative count over displayed, claim-bearing labels. Table 39: Top associated unembedding rows for the main case-study features. Rows cover all unique features displayed in the twobugpanels of Figure 18. Leading-space token variants are shown without the leading space, and non-Latin synonyms are summarized. Feature Top associated rows, shortenedInterpretationConfidence f36 mosquito,mosquitoes,Mos,mos, mosquito tokens in Chinese and Vietnamese,insect,flea, malaria, bug, moth mosquito / insect / disease-vector bug sense high f4095 bee, bees, Bee, bee tokens in Chinese, hive, butterfly, insect, ants, honeybees / insectshigh f18302 Beetle, beetle, beet, Beet, Bug, bug, _bug, BUG, buggy beetle / bug token family, mixed insect and software string forms medium-high f58330 _error, _ERROR, error, -error, Error, .error, _errors, /error, _errprogramming/error-token featurehigh f13081 Error, error, _error, -error, ERROR, errors, .Error, _Error, Japanese error tokenerror / exception surface formshigh f59571 M, J, G, K, N, P, I, L, A, B, ..., newline-like token single leading-space capitals / low- level formatting-like feature low f21804 defect ,defects, defect tokens in Chinese and Russian,flaw,defe,defective,flaws,bug, fault defect / flaw / software-bug sensehigh f43419 Crash, crash, crashes, crashed, crashing, Bug, bug, bugs, paniccrash / bug / panic software failurehigh f5680 debug, Debug, .debug, _debug, DEBUG, debugging, debugger, dbg, Chinese debug tokendebug / debugging code tokenshigh f34693 p, j, d, b, t, g, m, l, h, s, c, nsingle lowercase leading-space letterslow f23681 M, P, S, K, G, A, D, B, N, J, I, Y, H, O, C, Fsingle uppercase letters / alphabetic token family low the general-readout set, the six off-target coherent families are the positive-affect family, twogu/Gu fragment families, two blue-surface families, and an effects family. Panel-level claims in the main text draw on the on-target subset. Table 39 summarizes every feature shown in the two mainbugpanels. For each sparse feature, we score centered and normalized unembedding rows with that feature’s encoder row and list the highest- scoring tokens. The result supports the main-text interpretation, since most large contributors at the row level are coherent lexical families, while three displayed features are marked as low-level token form directions. M Feature-Resolved Direct Logit Attribution Sparse Readout Prism terms sit inside ordinary di- rect logit attribution as an additive identity (Elhage et al., 2021; Wang et al., 2023; Nguyen, 2024), the composition §4 points to. This appendix states that identity and gives the worked case. Direct logit attribution splits a margin across layers and SRP splits it across features, so composing the two re- solves each layer’s contribution into one term per feature, and the basis that says which structure a margin recruits also says which layer supplies it. For a selected readout directionq α = P v α v w v , let r α = P v α v r v . Sparse Readout Prism gives q α = X v α v μ + X i β i (α)d i + r α , whered i is an SAE decoder direction andr α is the readout reconstruction residual for the selected readout score. The displayed scores below are zero- sum vocabulary mean or token-token contrasts, so P v α v = 0and the offset term cancels, where raw token logits would retain it. At the final decoded state ̃ h L , ̃ h ⊤ L q α = X v α v ̃ h ⊤ L μ + X i β i (α) ̃ h ⊤ L d i + ̃ h ⊤ L r α . Under the fixed final-normalization convention used for DLA, decompose ̃ h L ≈ P j ̃ h L,j , index- ing residual-stream components byj. The compo- 47 nent offset, SAE feature, and residual terms are O j (α) = P v α v ̃ h ⊤ L,j μ, M j,i (α) = β i (α) ̃ h ⊤ L,j d i , R j (α) = ̃ h ⊤ L,j r α . For the zero-sum contrasts used in the figures, O j (α) = 0. We therefore report two separate gaps, the DLA additivity error between ̃ h ⊤ L q α and P j ̃ h ⊤ L,j q α , and the readout SAE residual ̃ h ⊤ L r α (or its componentized version P j R j (α)under the same DLA approximation). Because these iden- tity checks sit on well-separated margins, every relative residual quoted below is the unfloored ratioρ 0 (§3.4), computed on full-precision mar- gins before display rounding. The difference be- tween ̃ h ⊤ L q α and a componentized feature-only sum P j,i M j,i (α)contains both sources unless the residual and DLA additivity terms are included separately. The main display uses the pairwise logit differ- enceq α = w verify − w assume , for whichβ i (α) = z verify,i − z assume,i . The same identity also covers the vocabulary mean contrastq α = w verify − ̄w V , which decomposes one resolved token’s readout above the vocabulary mean row across residual- stream components. The component view is an ad- ditive decomposition over a realized forward pass. Figure 28 shows theverify-minus-assumemar- gin discussed in the main text, grouped by seman- tics. Figure 29 gives the per-component view of the same margin, with each component’s readout fea- ture terms stacked against ordinary direct logit at- tribution for that component. Figure 30 resolves theverify-minus-assume margin to individual readout features across residual-stream stages, where Figure 28 groups them by semantics. The margin is carried by sev- eral distinct same-family directions, where verifi- cation and checking features supply the positive mass, concentrated in the later layers, against a smaller assumption-family group. The exact mar- gin is+4.31, the component attribution sum is +4.36, and the sparse feature sum is+4.28, with relative residual ρ 0 = 0.009. Figure 31 shows the vocabulary mean contrast version. The prompt is the same source-verification prompt, and the selected score is ̃ h ⊤ L (w verify − ̄w V ) , whereas the main figure uses the token-token logit difference. verify/ check assumeother sparse W U resid. direct DLA embed L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 L14 L15 L16 L17 L18 L19 L20 L21 L22 L23 residual component +0.6+0.5 +1.3+1.5 +0.5 +0.6 +0.7+1.0 -0.6 +3.2-0.9-1.7+0.7 1.500.750.000.751.50 supports assumesupports verify Figure 28: Feature-resolved component attribution. Qwen3.5-2Bverify-assumemargin on the source- verification prompt (“The source has not been checked yet. The appropriate next action is to”). Rows are embedding and per-layer post-norm contributions, and columns group verify/check, assume, other sparse, LM head residual, and ordinary DLA terms. Red supports verify and blue supports assume. Figure 32 is the pairwise companion to the vo- cabulary mean view above, adding theassumeside of the contrast so that verification/checking and assumption-family terms appear on the same com- ponent axis. N Constrained Readout-Side Edit Test This appendix reports a constrained readout-edit test for readout SAE decoder directions, ask- ing whether directions associated with a fixed profanity-related candidate lexicon can move spec- ified logit differences when edited at the read- out, and how that effect behaves under held-out terms, dose sweeps, comparison rows, and off- target probes. The test asks a different question from the ablation of §4.3. The ablation removes a feature from a state the model produced and checks that the margin moves by the feature’s contribution. The edit test selects features from a few discovery tokens, edits the readout along their decoder direc- tions, and asks how far held-out terms of the same lexicon are suppressed. A direction that reaches terms which played no part in selecting it carries the lexical family beyond the rows it was read from. Table 40 lists the checks the test reports and what each one establishes. Setup. The primary run uses Qwen3.5-2B with the 32×,k = 256readout SAE reported in Ap- pendix C. The evaluation vocabulary contains 22 48 embed L0L1L2L3L4L5L6L7L8L9 L10L11L12L13L14L15L16L17L18L19L20L21L22L23 Residual component −0.5 0.0 0.5 1.0 1.5 logit contribution Qwen3.5-2B 32x Prism-DLA: verify - assume direct DLAfeaturesW_U residual Figure 29: Component attribution resolved by readout feature. Theverify-minus-assumemargin on the source-verification prompt for Qwen3.5-2B, decomposed across residual-stream components (embedding and layers 0–23). Each bar stacks that component’s readout feature terms with its LM head reconstruction residual, and black points mark ordinary direct logit attribution for the same component, which the stacked terms track. Verification mass concentrates in the later layers. f25948 verify f47092 check f64402 assume f62519 Assuming f27888 assumptions f16851 Verify f40205 verify f4592 assume Readout feature embed L0-5 L6-11 L12-17 L18-23 Residual stage Qwen3.5-2B 32x Prism-DLA: verify - assume −2 −1 0 1 2 signed logit contribution Figure 30: Feature-resolved component accounting. Theverify-minus-assumemargin decomposed by residual- stream stage (rows) against individual sparse readout features (columns) for Qwen3.5-2B. Red cells supportverify and blue cells supportassume. The contrast is distributed over several same-family feature directions, with the leading verification feature (f25948) dominant in the later layers (L18–23). Exact margin+4.31, component attribution sum +4.36, sparse feature sum +4.28, relative residual ρ 0 = 0.009. non-hateful profanity terms and 20 affect/control terms, and all single-token claims are filtered through the model tokenizer. Features are selected from five discovery terms, fuck,shit,hell, sucks,bastard, and then evaluated on held- out terms fucking,fucked,bullshit,shitty, damned,asshole,bitch,crap,piss,pissed. The intervention is a one-step logit edit, where for selected decoder directionsd i ,i ∈ F, we add eW ⊤ U to the logits withe = −γ P i∈F d i /∥d i ∥ 2 . With this sign convention, positiveγsuppresses the selected profanity-associated directions. The reported intervention scale isγ = 16. The pri- mary candidate-constrained grid has 16 prompts, 9 profane-to-reference candidate pairs, 10 interven- tion scales, and 5 methods (none, readout SAE direction edit, discovery token bias, oracle token bias, and norm-matched random feature edit), giv- ing 7200 evaluated rows. Feature specificity. The selected directions are lexically specific. In the Qwen3.5-2B run, 12 of the top 13 profanity-associated SAE features have specificity1.0against the 20 control terms, and the remaining feature has specificity0.988. The token summaries partition recognizable lexical families, 49 02468 component term for row-centered verify readout (logit units) embed L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 L14 L15 L16 L17 L18 L19 L20 L21 L22 L23 Residual component Feature-resolved DLA for row-centered verify verification/checking featuresother sparse features W U residual direct DLA Figure 31: Vocabulary-mean contrast component attribution. Vocabulary-meanverifycontrast ̃ h ⊤ L (w verify − ̄w V ) for the source-verification prompt, decomposed by residual-stream component. Each row stacks readout terms grouped into verification/checking features, remaining sparse features, and the unembedding reconstruction error term, and black points mark ordinary direct logit attribution for that residual-stream component. The vocabulary mean contrast is+17.34, the component DLA sum is+17.34, and the sparse SAE feature sum is+17.43, with relative reconstruction error 0.005. 3210123 signed logit contribution to verify - assume embed L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 L14 L15 L16 L17 L18 L19 L20 L21 L22 L23 Residual component Feature-resolved DLA by residual layer verification/checking featuresassumption featuresother sparse features W U residual direct DLA Figure 32: Pairwise logit difference DLA by SAE feature.verify-minus-assumelogit difference for the source- verification prompt, decomposed by residual-stream component. Each row stacks sparse readout terms grouped into verification/checking features, assumption features, remaining sparse features, and the unembedding reconstruction error term, and black points mark ordinary direct logit attribution for that residual-stream component. Positive terms supportverifyoverassume, and negative terms supportassume. The exact logit difference is+4.31, the component DLA sum is+4.36, and the sparse SAE feature sum is+4.28, with relative reconstruction error0.009. 50 Table 40: Readout SAE decoder-direction edit summary. The constrained edit on the readout side supports an intervention result at the readout level, in that editing SAE decoder directions selected from discovery tokens can move constrained lexical logit differences. The table records the evidence used to pair intended margin shifts with off-target lexical probes. MeasurementWhat it measuresSetting Lexical specificityDegree to which the selected features and token summaries are specific to the profanity-related lexical family. Supports feature selection for this edit test. Score-level interven- tion effect Whether editing along the selected SAE decoder directions changes the specified profanity-vs- reference logit differences in the intended direc- tion. Identifies a score-level effect at the edited read- out, and circuit tracing (Ameisen et al., 2025) can localize upstream sources. Held-out templates Whether the same intervention transfers to held- out lexical prompts and prompt templates after feature selection. Measures held-out template transfer after feature selection. Dose response Whether larger intervention strengths produce ordered margin shifts before quality or off-target behavior dominates. Measures dose ordering and separates threshold artifacts from graded response. Cross-model replica- tion Whether the same intervention pattern appears in more than one evaluated model. Replication is evidence for a recurring pattern at the readout level, while each model is reported alongside its own readout fidelity metrics. Off-target lexical probes Whether the intervention leaves unrelated lexical contrasts and neutral prompts stable. These probes measure obvious off-target regres- sions alongside the intended margin movement. Open-generation probes How the edit at the readout level propagates when the model is allowed to generate freely. Complements the candidate-constrained lexical- score measurements. Table 41: Primary Qwen3.5-2B readout-edit comparison. All methods use intervention scale 16. Discovery token bias penalizes only the five discovery token ids, while oracle token bias penalizes the full evaluation lexicon. Random features are norm-matched to the selected readout SAE direction edit.∆p bad is reported as a positive reduction in probability assigned to the profane-token side. Method Discovery flip Held-out flip Held-out ∆p bad Median KL Interpretation Readout SAE directions (10)0.4500.3590.3980.679Transfers from discovery terms to held-out variants. Discovery token bias0.4500.0000.0000.001Token-id edit restricted to discovery terms, so held-out ids receive no direct bias. Oracle token bias0.4500.3590.4040.004Upper-bound token edit with full lexicon knowledge. Random features0.0130.1560.0781.363Norm-matched control with a different effect/cost profile. No intervention0.0000.0000.0000.000Reference condition. including f-word, sh-word, hell/damn, and insult- related directions. Table 41 reports the key intervention compari- son. The readout SAE direction edit matches the oracle token bias held-out flip rate while using only directions selected from discovery tokens, whereas the equally informed discovery token bias has zero held-out effect. The match is exact because scale 16 saturates the evaluation. A candidate can flip only when the unedited model prefers the profane side, which holds for 23 of the 64 held-out and 36 of the 80 discovery candidates. At scale 16 the SAE direction edit and the oracle flip exactly the same 23 held-out candidates, so both rates equal the ceiling23/64 = 0.359(cluster-bootstrap 95% CI0.250–0.469for both methods). The methods separate below saturation, where at scale 4 the or- acle flips 20 of the 23 against the SAE edit’s 11 (intersection 10), and the oracle saturates by scale 6 while the SAE edit reaches the ceiling at scale 12. KL summaries cluster at the prompt level, since the 16 prompts are shared across both splits. The KL column records the distributional cost of each edit, and the gap between the oracle’s0.004and the di- rection edit’s0.679reflects what each method is al- lowed to know. The oracle bias penalizes the logits of the evaluation lexicon itself, so it moves exactly the tokens being scored and leaves the rest of the distribution untouched. A readout direction is se- lected from five discovery tokens with the held-out 51 Table 42: Cross-model readout SAE direction edit measurements. All rows use ten readout SAE decoder directions selected from discovery tokens at intervention scale 16. Discovery directions are selected from five profanity tokens, and held-out terms are withheld from both the feature selector and the discovery-only token bias baseline. The Ministral tokenizer leaves the held-out split below the single-token threshold, so that row reports the comparison on the discovery side. ModelSAE used Feature discovery flip Feature held-out flip Discovery-token bias held-out Off-target top-1 regression Setting Qwen3.5-2B32×, k = 2560.4500.3590.0000.444 Primary readout-edit run. R1-Distill Qwen-7B32×, k = 2560.3880.4690.0000.222Same tokenizer family, reasoning-distilled check- point. Ministral 3-8B-Base16×, k = 1280.562–0.278Tokenizer-filtered held-out split, evaluated with an earlier strict SAE checkpoint and reported as a discovery-side comparison. terms withheld from the selector, so it acts through the whole vocabulary and pays for whatever else it moves. The oracle is therefore a label-informed up- per bound, and the comparisons that carry the claim are the label-free mean-row and PCA directions of Appendix N.1, which are matched on distributional cost. Across the intervention-scale sweep, the dis- covery flip rate under the readout SAE direction edit increases monotonically from0.038to0.450, and the held-out flip rate tracks it within about0.04 across the mid-range scales (6 to 10), widening to roughly0.07at scale 4 and0.09at the head- line scale 16. The result establishes a constrained logit difference intervention pattern, while open- generation behavior remains a separate evaluation layer beyond candidate scoring. Table 42 records the quantitative readout-edit outcome. The Qwen3.5-2B and R1-Distill-Qwen- 7B rows show the intended transfer pattern, since the readout SAE direction edit transfers to held- out lexical variants, while the discovery-only token bias baseline has zero held-out effect because those token ids were not in its bias set. The Ministral row provides a cross-family comparison on the discovery side because its tokenizer leaves too few held-out single-token terms for the held-out split. SAE decoder directions move constrained lex- ical logit differences when edited at the read- out. The intended scores move under interven- tion, held-out templates and scale-sweep measure- ments provide supporting evidence, and the two Qwen-family rows show the same held-out lexi- cal transfer pattern. The Ministral row is a cross- family comparison on the discovery side because too few held-out terms pass the tokenizer filter. Off- target lexical probes map regressions beyond the intended candidate scores. The Qwen3.5-2B off- target top-1 regression of0.444(Table 42) is the candidate-scoring counterpart of the median-KL distributional cost, and it shows that the edit is constrained by its construction, while its measured effect reaches beyond the intended scores. N.1 Matched-KL Frontier and Cross-Model Outcome The comparison above matches intervention scale, while a stricter comparison matches distributional cost, since a direction can buy more suppression by moving the next-token distribution further. The comparison against the mean row and leading prin- cipal components of the same discovery rows tests whether the dictionary supplies more of the lexical family than the row geometry alone. The upgraded protocol enlarges the held-out lexicon from 64 to 200 candidates (10 held-out pairs×20 prompts), extends scales to 64 so that baseline frontiers reach SRP’s KL range, and adds three label-free control directions, namely the mean row of the five dis- covery tokens, PCA rank-1, and PCA rank-4 (the full rank of the discovery rows, hence capacity- fair), while the oracle, discovery-bias, and random brackets are retained. The frontier plots held-out suppression (∆p bad primary, flip rate secondary) against median next-token KL, and paired cluster bootstraps (same candidates across methods, clus- tered conservatively by the 10 held-out terms and, separately, by the 20 prompts, with 10,000 resam- ples) test differences at matched KL. On the display model SRP outperforms every label-free baseline at every matched KL (Fig- ure 33), with all 12 paired comparisons signifi- cant (term-clustered CIs exclude zero). Against the strongest baseline, the mean-row direction, the paired∆p bad advantage is+0.032[+0.014, 52 10 3 10 2 10 1 10 0 distribution disturbance median next-token KL (bits, log) 0.0 0.1 0.2 0.3 0.4 held-out suppression ( profane probability) SRP readout features mean-row direction oracle (uses eval labels) same disturbance read vertically gap at KL=0.05: +0.067 Qwen3.5-2B (display model replacement top-1 0.887) 10 3 10 2 10 1 10 0 distribution disturbance median next-token KL (bits, log) mean-row direction SRP readout features R1-Distill-Qwen-7B (mid-fidelity readout top-1 0.760) Matched-KL frontier held-out lexical suppression per unit of distribution damage Bands: 95% cluster-bootstrap CI (decisive pair only). n=200 held-out cases per model; oracle edits the evaluation tokens directly and is a labeled upper bound, not a label-free competitor. SRP readout featuresmean-row directionPCA rank-1PCA rank-4oracle (uses eval labels)random features Figure 33: Held-out suppression versus distributional cost at matched KL (Qwen3.5-2B). Held-out∆p bad (left) and flip rate (right) against median next-token KL for the SRP direction edit, mean-row, PCA rank-1/rank-4, and the oracle and random brackets. SRP leads every label-free baseline at every matched KL on this model and reaches the oracle ceiling at scale 12, while trivial directions reach it only at 1.3–5× the KL. +0.054 ] at KL 0.02,+0.087[+0.065,+0.115] at 0.05,+0.085[+0.062,+0.109] at 0.10, and +0.074[+0.048,+0.103] at 0.20. Against PCA rank-4 the advantages span+0.036to+0.097, all significant. The advantage is model-specific, and we report the losses. At matched KL, Qwen3.5-0.8B under- performs the mean-row direction (∆ −0.030to −0.094, significant) and Qwen3.5-9B underper- forms it more clearly (−0.086to−0.135, signifi- cant), while R1-Distill-Qwen-7B is mixed (under- performs mean-row at two of four matched-KL points, outperforms PCA rank-4). Ministral is ex- cluded because the Tekken tokenizer leaves too few single-token terms for feature discovery (the dictionary itself is validated in Appendix J), and R1-Distill-Llama-8B is excluded with one tokeniz- able pair and no held-out terms. Two candidate explanations for the losses were tested and rejected, namely replacement fidelity (the 9B readout has high fidelity and still loses) and dictionary width (a registered prediction that the narrower 0.8B dictio- nary should do better, which it did not). The two comparisons separate. Against capacity-matched PCA rank-4, the dictionary directions match or lead on nearly every model and matched-KL point. Against the mean discovery row, the simplest label- free direction available, they lead decisively on the display model and lose on two others. The sup- ported claim is therefore scoped. On the display model, dictionary-direction edits outperform trivial category directions at matched distributional cost, the advantage is not general across models, and the mean row is the strongest label-free baseline. O Discussion and Future Work This appendix expands the discussion and future- work map. It covers the scope of selected scores, relation to activation SAEs, cross-family interpre- tation, DLA composition, and possible extensions. O.1What Sparse Readout Prism Decomposes Selected readout score.Each local score decom- position selects one scalars α ( ̃ h ℓ ) = ̃ h ⊤ ℓ q α . This may be a selected vocabulary logit, a specified pair- wise logit difference, a group contrast, a vocab- ulary mean contrast, or a top competitor margin. The readout feature projectionsp i ( ̃ h ℓ ) = ̃ h ⊤ ℓ d i are available before selecting such a score, and support for a particular token or contrast becomes defined only after they are combined with the direction co- efficientsβ i (α). Distribution-wide decompositions therefore require measuring reconstruction over the relevant vocabulary slice or token group. This dis- tinction matters because a token can be supported above a reference while still losing to a named al- ternative or a group contrast, and a top-1 token can have only a small margin over top-ranked com- petitors. Reference contrasts, pairwise logit differ- ences, group contrasts, and top competitor margins therefore define distinct selected readout scores. Local score decomposition.SRP provides local additive accounting for a selected readout score. For that score, the displayed terms express the exact value as SAE feature terms plus preprocessing and reconstruction error terms under the conventions in Appendix A. Contribution terms are reported with those error terms, so the local score decomposition 53 is tied to its reconstruction quality. O.2 Relation to Activation SAEs Fitted object.Sparse Readout Prism uses sparse autoencoder machinery, but its fitted object dif- fers from activation-SAE work. Activation SAEs learn sparse codes for a distribution of hidden states (Bricken et al., 2023; Huben et al., 2024; Gao et al., 2025a). Sparse Readout Prism learns sparse codes for rows of the unembedding matrixW U . The learned dictionary features are therefore readout directions used by token rows. Context dependence. The decoded state enters after the readout SAE has been learned. A fea- ture contributes to a selected readout score when two conditions hold together. The selected read- out direction must use that SAE feature through its sparse row coefficients, and the current decoded state must have a large dot product with the corre- sponding decoder direction. Hence the same token row can have different dominant feature contribu- tions across contexts, since the row coefficients are fixed while the projection terms ̃ h ⊤ ℓ d i change with the state being decoded. O.3 Model-Family Results Qwen evidence.Across Qwen sizes, the selected dictionary settings preserve held-out signed read- out scores and readout-distribution behavior on decoded states with reconstruction errors small enough to support the reported local score decom- positions. These rows provide the fixed display setting and Qwen-family companion analyses used for most qualitative figures. Additional families. The additional model- family rows report the same metric suite under different tokenizer, readout, and post-training con- ditions. They separate row reconstruction, LM head replacement, and reconstruction of selected scores, and the paper reports all three beside its feature contributions. O.4 Feature-Resolved DLA Readout versus components. Analyses on the readout side and the component side answer dif- ferent questions. The SRP decomposition asks which SAE terms on the unembedding side score a selected logit or contrast, while component-level attribution asks which residual-stream components project onto the selected readout direction in the realized forward pass. Residual terms.Combining the two views yields additive component-by-feature terms. The analy- sis reports both the DLA additivity error for the selected readout score and the readout SAE recon- struction error left outside the displayed feature terms. O.5 Future Work Extensions. Natural extensions include con- trolled interventions on SAE decoder directions, cross-model feature-alignment analyses, distribu- tional decompositions of local top-kfrontiers, and broader model-family sweeps. The basis also extends to the effective linear readouts used by tuned lenses, logit-prism variants, and future-token lenses, so lens methods can be compared by sparse signed feature terms as well as decoded tokens, and terms nominated by SRP give targets for causal patching, ablation, and generation-level evaluation. A further extension factorizes both sides of a se- lected margin at once, pairing an activation dictio- nary forh ℓ with the readout SAE forq, so that each term records the alignment between a feature on the activation side and one on the readout side. All depend on the fidelity diagnostics used in the main text, with feature terms read against the reported residual of the selected readout score. Residual-stream census.SRP also exposes what the readout can express before a decoded state is reduced to a single score. The readout feature pro- jectionsp i ( ̃ h ℓ )can be spread widely, while the se- lected coefficientsαselect the termsβ i (α)p i ( ̃ h ℓ ). Future work could quantify projection mass used by the selected readout score, assigned to com- petitors, or active but unused. The unused cate- gory is local to the selected readout score, and it may support alternative tokens, later positions, or other computations. Comparing the split across lay- ers, ambiguous prompts, and incorrect generations could test whether models carry broad task state beyond the immediate readout decision. P Extended Related Work This appendix records broader context relevant to positioning but not required for the main argument. Modern lens variants. Beyond the lens work cited in the main text, LogitLens4LLMs updates logit-lens tooling for modern Hugging Face model families (Wang, 2025), ContextualLens and La- tentLens use richer contextual representations for 54 vision-language models (Phukan et al., 2025; Kro- jer et al., 2026), and Indic-TunedLens adapts de- coding in the tuned-lens style to Indian languages (Panchal et al., 2026). These variants are com- plementary front ends to hidden-state inspection, orthogonal to SRP’s factorization of readout rows. Recent sparse-feature infrastructure. Model- family-specific activation-SAE releases include Gemma Scope and Qwen-Scope (Lieberum et al., 2024; Deng et al., 2026), and recent weight-based SAE interpretation studies how activation-SAE fea- tures interact with model weights and output-token effects (Liu and Deng, 2026). A separate line trains SAEs on the outputs of dense text-embedding mod- els (O’Neill et al., 2024), and those dictionaries decompose embedding vectors computed for in- puts, whereas SRP decomposes the weight rows of the LM head itself. Crosscoders extend dictio- nary learning across layers, models, or training time within a shared feature space (Lindsey et al., 2024), with crosscoder-specific sparsity artifacts documented in chat-model diffing (Minder et al., 2025). SRP shares the model-family-specific em- phasis but fits its sparse dictionary directly to un- embedding rows, and does not evaluate or replace activation-feature infrastructure. Activation-SAE evaluation has standardized diagnostics (Karvo- nen et al., 2025; Makelov et al., 2025) and known caveats about unit canonicity (Leask et al., 2025). Latent-language readings. The inference re- tested in §5 also appears in the latent-language literature (Wendler et al., 2024; Schut et al., 2025), which reads through the unembedding directly. Those lenses have no fitting corpus and are not ex- posed to corpus conditionality. A weaker and sepa- rate confound applies to them instead, since which of several realizations surfaces in a ranking is fixed by the row codes ofW U alone, and one readout feature carries both realizations on 61/80 and 62/80 of our own within-lens comparisons. Dumas et al. (2025) establish language-agnostic concept repre- sentations causally, by activation patching, without relying on a ranking at all. Output-space geometry and confounds. The linear-representation view treats token distinctions as directions or contrasts in vocabulary space (Park et al., 2024), and vocabulary-embedding directions can steer generation (Han et al., 2024), and SRP’s edits on the readout side act in the same space at the granularity of individual features. The same view also motivates edits on the activation side, where redirecting the representations of a target set into a region the model already uses to decline an answer suppresses knowledge through a single down-projection matrix (Shen et al., 2025). Those edits are stated in hidden states, whereas SRP’s are stated in readout rows. Which rows are available to either kind of edit is itself a training outcome, since embeddings can be decoupled from the trans- former body and trained with a separate vocabulary per data source (Iacob et al., 2025), one reason the paper reports its metric suite per model family (Ap- pendix O.3). Degeneration and frequency analyses show that mean directions, row norms, and token frequencies can move token scores independently of the property under test (Gao et al., 2019; Zhang et al., 2020; Chung and Kim, 2025). Anisotropy varies with training and data (Machina and Mer- cer, 2024), and output-embedding centering targets the shared mean direction during pretraining (Stol- lenwerk et al., 2026). These effects motivate the row-norm and frequency controls of Appendix G. Additive attribution and mechanisms. Direct logit attribution splits a token logit or logit differ- ence into per-component terms (Elhage et al., 2021; Wang et al., 2023). Despite the similar names, logit prisms (Nguyen, 2024) split who wrote the hidden state across residual-stream components, while SRP splits what reads that state across the row structure of the LM head, and the two decom- positions compose (Appendix M). Interventions (Conmy et al., 2023), causal abstraction (Geiger et al., 2025), and transcoder or attribution-graph work (Dunefsky et al., 2024; Ameisen et al., 2025) address complementary mechanism-level questions about which components and paths produce a be- havior. 55

Instant research discovery