Paper deep dive
Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence
Katarzyna Filus, Sebastian Pokuciński
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 7/21/2026, 5:38:41 AM
Summary
This paper introduces the Tversky Monosemanticity Score (TMS), a label-free metric for measuring monosemanticity in Sparse Autoencoders (SAEs) by evaluating the activation-set coherence of binarized latents. The authors demonstrate that TMS is less sensitive to encoder anisotropy compared to embedding-based metrics like the MonoSemanticity Score (MS), particularly when using highly anisotropic encoders such as CLIP. TMS is evaluated on SAEs trained on features from DINOv3, CLIP, and BLIP2 models, showing strong alignment with established indicators and revealing distinct training dynamics.
Entities (12)
Relation Signals (10)
Tversky Monosemanticity Score → measures → Monosemanticity
confidence 95% · TMS operationalizes monosemanticity as activation-set coherence of binarized SAE latents
Sparse Autoencoder → usedin → Mechanistic Interpretability
confidence 92% · mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features
Tversky Monosemanticity Score → islesssensitiveto → Encoder Anisotropy
confidence 90% · TMS is less affected by encoder anisotropy than its embedding-based alternative
MonoSemanticity score → issensitiveto → Encoder Anisotropy
confidence 88% · MS exhibits a strong dependence on the encoder... highly anisotropic encoders inflate embedding-based scores
Tversky Monosemanticity Score → correlateswith → MonoSemanticity score
confidence 85% · TMS remains aligned with established monosemanticity indicators... strong positive relationship is observed... with MS
Tversky Monosemanticity Score → evaluatedon → CLIP
confidence 85% · We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (...CLIP...)
Tversky Monosemanticity Score → evaluatedon → BLIP2
confidence 85% · We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (...BLIP2)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Within Explainable Artificial Intelligence, mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features from neural representations. However, assessing their monosemanticity, and thus explanation quality, remains challenging. Existing metrics require external concept labels or depend on pretrained embedding models, making them sensitive to encoder's geometry. We introduce the Tversky Monosemanticity Score (TMS), a label-free metric that operationalizes monosemanticity as activation-set coherence of binarized SAE latents, and does not require external embedding encoders. We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (DINOv3, CLIP, BLIP2), two common SAE regimes (TopK, BatchTopK), multiple sparsity levels, and expansion factors. Our results show that TMS is less affected by encoder anisotropy than its embedding-based alternative, while remaining aligned with established monosemanticity indicators. TMS also reveals distinct SAE training dynamics across base models. Moreover, under encoder anisotropy, TMS provides a stronger indication of probe-based concept deletion effectiveness, while being competitive otherwise.
Tags
Links
- Source: https://arxiv.org/abs/2607.17770v1
- Canonical: https://arxiv.org/abs/2607.17770v1
Trouble viewing inline? Open PDF directly →
Full Text
48,687 characters extracted from source content.
Expand or collapse full text
Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence Katarzyna Filus 1 () and Sebastian Pokuciński 2 1 Institute of Theoretical and Applied Informatics, Polish Academy of Sciences, Bałtycka 5, 44-100 Gliwice, Poland 2 Department of Applied Informatics, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland Abstract. Within Explainable Artificial Intelligence, mechanistic in- terpretability uses Sparse Autoencoders (SAEs) to extract more inter- pretable features from neural representations. However, assessing their monosemanticity, and thus explanation quality, remains challenging. Ex- isting metrics require external concept labels or depend on pretrained embedding models, making them sensitive to encoder’s geometry. We introduce the Tversky Monosemanticity Score (TMS), a label-free met- ric that operationalizes monosemanticity as activation-set coherence of binarized SAE latents, and does not require external embedding en- coders. We evaluate TMS on SAEs trained on features from pretrained vision and vision-language models (DINOv3, CLIP, BLIP2), two com- mon SAE regimes (TopK, BatchTopK), multiple sparsity levels, and ex- pansion factors. Our results show that TMS is less affected by encoder anisotropy than its embedding-based alternative, while remaining aligned with established monosemanticity indicators. TMS also reveals distinct SAE training dynamics across base models. Moreover, under encoder anisotropy, TMS provides a stronger indication of probe-based concept deletion effectiveness, while being competitive otherwise. Keywords: Mechanistic interpretability· Sparse Autoencoders· Monose- manticity· Explainable Artificial Intelligence Evaluation. 1 Introduction Within Explainable Artificial Intelligence (XAI), mechanistic interpretability aims to improve the transparency of deep networks by analyzing their inter- nal representations. A crucial challenge is feature entanglement, where individ- ual neurons encode multiple unrelated concepts, making representations diffi- cult to interpret [2]. Sparse autoencoders (SAEs) emerged as a key technique in this area, encouraging more disentangled features by projecting activations into higher-dimensional sparse latent spaces [8,19]. In this context, monoseman- ticity plays a key role [2], describing whether a neuron corresponds to a single coherent concept and reflects reduced feature entanglement. Beyond serving as interpretability indicator, monosemanticity holds functional importance and aids in targeted manipulations of model behavior [9]. arXiv:2607.17770v1 [cs.CV] 20 Jul 2026 2K. Filus and S. Pokuciński Besides its acknowledged importance, currently there is no consensus on how monosemanticity of SAE neurons should be defined and measured. Existing ap- proaches are grounded in supervised concept-predictive evaluation [9,5] or simi- larity among stimuli that activate a given neuron [13]. While these methods pro- vide useful proxies, they typically rely on external resources such as annotated concept labels [9,5], auxiliary classifiers [9], or pretrained embedding models [13]. While methods based on embedding models eliminate the need for concept la- bels [13], their estimates still largely depend on the representational geometry of external encoders, and factors such as anisotropy. As a consequence, monose- manticity estimates may vary substantially across encoders, complicating direct comparisons between models. These observations highlight the need to develop a monosemanticity measure independent of external labels and embedding models. Motivated by the limitations of existing monosemanticity metrics, we intro- duce a novel Tversky Monosemantic Score (TMS), a metric that operationalizes monosemanticity as activation-set coherence within sparse autoencoder latent features. TMS is independent of both external concept labels and test-time reference encoders. The proposed formulation uses a set-based, non-geometric similarity measure inspired by symmetric Tversky’s similarity model from cog- nitive psychology [17]. By operating on binary activation patterns rather than continuous embeddings, TMS is less sensitive to factors induced by embedding geometry, including anisotropy, that distort similarity-based measures. This in- dependence from external resources simplifies the evaluation and makes it more self-contained. We evaluate TMS on SAEs trained on representations from pre- trained vision and vision-language models across different sparsity regimes and expansion factors, using Mini-ImageNet as a natural image benchmark. Our results show that, despite TMS’ simplicity, it remains aligned with established monosemanticity indicators, while showing stronger agreement with the MonoSe- manticity Score (MS) [13] in low-anisotropy settings (DINOv3). TMS trajecto- ries show that monosemanticity is not static but evolves during training, with clear differences across SAE regimes and base models, pointing to a direction for further investigation in SAE research. TMS as a model-level measure exhibits stronger association with target drop in linear probes trained on SAE activations, compared to the reference measure for anisotropic embedders such as CLIP. We provide our code in an anonymous Zenodo repository for reproducibility [6]. 2 Related work Mechanistic interpretability aims to uncover the internal structures responsible for neural networks behavior [12]. A key challenge is polysemanticity, meaning that individual neurons in a network respond to multiple unrelated concepts, preventing straightforward interpretation. It is often connected to the super- position hypothesis stating that networks encode more features than neurons through overlapping activation directions [10]. Sparse autoencoders (SAEs) are a promising approach to decompose activations into sparse feature dictionaries that are more disentangled and interpretable, capturing more coherent, single- Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence3 concept representations – a property commonly referred to as monosemanticity. SAE-based analyses have demonstrated monosemantic feature discovery not only in language models, which were the main focus of the initial studies [3,8,15], but also in vision–language or vision settings, where learned features also align with human concepts and can be used for targeted model steering [13,4]. Available monosemanticity measures include Feature Monosemanticity Score (FMS) [9], Jensen-Shannon divergence-based score (JSDS) [5], and MonoSeman- ticity score (MS) [13]. FMS uses performance difference for removed or added features in concept classification. JSDS measures how activations separate across concepts with Jensen–Shannon divergence. The primary limitation of JSD and FMS is that they need external concept labels. A recent study introduced the Monosemanticity Score (MS) [13] to avoid this limitation. MS treats a latent neu- ron as monosemantic if the input features that activate it are mutually similar, by embedding each image with an external encoder and computing a weighted mean cosine similarity between embeddings. While this is an important step toward label-free measures, it remains dependent on the external encoder geometry, and on the encoder choice, which adds variability and complicates cross-model com- parisons. Contrastingly, our Tversky Monosemanticity Score is independent of both external concept labels and embedding models. Moreover, while prior stud- ies typically focus on SAEs trained over a single-type modality (e.g. image-text [13], text [9]), we provide an analysis across different modality types – purely vi- sual (DINOv3) and multimodal (CLIP, BLIP2), under consistent SAE training. Finally, beyond the predominantly static monosemanticity evaluation [13,9,5], we complement our analysis with a dynamic tracking of monosemanticity train- ing evolution, offering a broader view on SAE interpretability. 3 Tversky Monosemanticity Score We define our novel metric – Tversky Monosemantic Score (TMS) – follow- ing the feature-based similarity model of Tversky [17] from cognitive psychology to operationalize monosemanticity.A =a,b,c,... denotes the domain of stim- uli (images), which are fed to the deep learning model under interpretation. In Tversky’s theory, each object is represented as a set of binary features, and sim- ilarity between two objects depends on their shared and distinctive features. For objects a,b∈A with feature sets A and B, similarity can be defined as a func- tion S(a,b) = F(A∩ B, A\ B, B \ A), increasing with the number of shared features and decreasing with the number of distinctive features. In our case, we interpret a trained Sparse Autoencoder (SAE) as a concept extractor. Each latent neuron of the trained SAE is treated as a learned concept proxy (in the sense of Tversky’s feature) and as a detector of recurring activation patterns, rather than a ground-truth semantic concept. Each stimulus (features generated with a model under interpretation for a given image) is described by the set of active latent concepts. Let B n ⊆ 1,...,K denote the binary set of active latent neurons for stimulus n, obtained via a predefined activation rule – following recent work showing that binarization of SAE features can improve 4K. Filus and S. Pokuciński interpretability and provide an effective alternative representation [7,1]. For a latent neuron k and its active set of samples A k = n : k ∈ B n , for which k activated, we measure similarity between pairs n i ,n j ∈A k using Tversky index: TI α,β (B n i ,B n j ) = |B n i ∩ B n j | |B n i ∩ B n j | + α|B n i \ B n j | + β|B n j \ B n i | .(1) For the purpose of this paper, we set α and β to 1, which reduces the score to Jaccard index, as a parameter-free default. The Tversky framing, however, provides a principled cognitive-science foundation and naturally accommodates future asymmetric extensions [17]. A latent neuron is considered more monose- mantic if, on average, stimuli in its active set share more common than dis- tinctive features, which is reflected in high Tversky’s within-set similarity: s k = E n i ,n j ∈A k TI α,β (B n i ,B n j ) If the latent neuron’s active set is considered empty, i.e., A k = ∅, its monosemanticity score becomes s k = 0. In practice, we com- pute TI α,β on binarized latent activation vectors. Given activations Z ∈ R N×K , we obtain a binary matrix B ∈ 0, 1 N×K by thresholding each latent k from the test set at its mean training activation μ k (i.e., B n,k = I[Z n,k > μ k ]) for a parameter-free, neuron-wise and data-derived threshold, following prior work of [1] using mean-based thresholds computed on SAE training data for its indi- vidual neurons. For each latent k, and its active set A k = n : B n,k = 1, we estimate the within-set similarity by sampling at max M distinct pairs (n i ,n j ) with n i ̸= n j uniformly at random from A k , and averaging similarities. The score reflects internal conceptual coherence of stimuli activating a given latent neuron, rather than geometric properties of the embedding models. The intuition for grounding our operationalization of monosemanticity in latent co- herence is that a latent encoding a relatively selective concept may tend to acti- vate on stimuli sharing not only that concept, but also its contextual structure. For example, a ‘bird wings’ latent may often co-occur with ‘beaks’ and ‘feath- ers’, giving consistent activation patterns, whereas polysemantic latents would produce more heterogeneous ones. 4 Experimental Setup We consider pretrained models from the HuggingFace repository 3 : a purely visual DINO-v3 with ConvNeXt backbone (facebook/dinov3-convnext-tiny-pretrain- lvd1689m)[16], and vision-language CLIP (openai/clip-vit-base-patch32)[14] and BLIP2 (Salesforce/blip2-opt-2.7b)[11]. We refer to them as base models (when used to extract features for SAE training) and encoders (when used to em- bed images for MS [13] computation). The experiments are conducted on Mini- ImageNet [18], whose natural categories facilitate qualitative inspection of SAE features. We use DINOv3’s global average pooled vector, and CLIP’s/BLIP’s CLS tokens as embeddings. We train sparse autoencoders (SAEs) adopting two 3 https://huggingface.co/ Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence5 common strategies: TopK and BatchTopK [8], using overcomplete 4 . We vary the active features number K ∈ 10, 32, 64 and the expansion factor exp∈ 2, 4, 8, yielding diverse SAE configurations. We train all SAEs for 25 epochs using the Mean Squared Error loss and AdamW optimizer with 3× 10 −4 learning rate and 512 batch size, while applying the overcomplete’s reanimation loss with weight 10 −2 . For TMS computation, we set the max number of sampled pairs at M = 1000. An auxiliary stability analysis on an example SAE (BatchTopK, DINOv3, K = 32, expansion factor×4) showed that TMS remains stable across 10 runs with M ∈50, 100, 250, 500, 1000, 2500, 5000, with values consistent to three decimal places and standard deviation on the order of 10 −5 at M = 1000. The number of active neurons that contribute to TMS after binarization re- mained substantial in all configurations, with average TMS always computed over at least several hundred neurons. Detailed statistics for all M and results on active neurons are provided in Appendices C and D. We use MonoSeman- ticity score (MS) [13] as embedding-based monosemanticity reference. The test environment was a PC with an Intel i9 12900 CPU, RTX 3090 24GB GPU, and 196 GB of DDR5 RAM. Our evaluation has several stages: 1. Embedding anisotropy analysis. We quantify the anisotropy of the base models when used as external encoders for embedding-dependent MS score [13]. This motivates the need for external embedding-free metrics. 2. Monosemanticity estimation. For all SAEs, we compute the proposed TMS alongside the reference – MS [13]. To complement quantitative results, we present example top activating images for neurons for all base model. 3. Metric correlation. We compute the correlation between monosemanticity estimators by measuring per-neuron Spearman correlations with p-values between TMS and alternative statistics (MS, within-active-set variance with the activation set defined via TMS) for neurons with non-zero TMS. 4. Training dynamics. Beyond static evaluation, we examine how monose- manticity evolves during SAE training by tracking TMS and MS values across epochs and visualizing their trajectories using scatter plots. 5. Concept deletion experiment. We evaluate the functional relevance of SAE features via a concept deletion experiment. For each SAE, we train a linear probe classifier on Mini-ImageNet and identify the most relevant SAE features for a given class using probe weights (max values), ensuring inde- pendence from monosemanticity metrics. We then analyze target accuracy degradation as the top-N features are removed and compute correlations be- tween MS/TMS and the accuracy drops across SAE configurations.To train the probes, we use a scikit-learn’s linear probe implemented as Standard Scaler and logistic regression (saga solver, C = 1.0, max_iter=25). In Appendices A and B, we report SAE reconstruction and health metrics (R 2 , in-out cosine similarity, latent L 0 , dead neuron ratio) and pre-deletion probe accuracies (on SAE latents and base model features) to show that the SAEs and probes are sufficiently functional, making our monosemanticity and concept- deletion experiments under a shared setup meaningful. 4 https://github.com/KempnerInstitute/overcomplete 6K. Filus and S. Pokuciński Modelσ cosine EVR DINOv3 0.07 0.02 CLIP0.52 0.07 BLIP20.33 0.05 (a) Anisotropy statistics of embedding models: mean cosine similarity (σ cosine ) and the first principal component’s explained variance ratio (EVR). Bold: the highest anisotropy. (b) Aggregated MS for non-zero neurons (TOPK SAE, K=10). Fig. 1. Anisotropy-induced bias in embedding-based MS. (a) External embedding mod- els exhibit different degrees of anisotropy. (b) MonoSemanticity Score (MS) [13] aggre- gated over non-zero-MS neurons depends strongly on the embedding source—more anisotropic encoders produce inflated MS results across setups. 5 Experimental results The impact of anisotropy level on monosemanticty level estimation Fig. 1 shows how external encoder geometry can affect monosemanticity estimation based on pairwise sample similarity. We first quantify anisotropy of the feature-source models (Fig. 1a) using mean pairwise cosine similarity between L2-normalized embeddings and the explained variance ratio (EVR) of the first principal compo- nent (PC1). Both statistics indicate that CLIP is significantly more anisotropic (σ cosine = 0.52, EVR = 0.07) than BLIP2 (0.33, 0.05), while DINOv3 has the most isotropic space (0.07, 0.02). Higher σ cosine and EVR suggest a stronger concentration of embeddings along some direction, which can create a bias and inflate similarity estimates. This effect is reflected in embedding-based monose- manticity measures such as MS [13]. Fig. 1 shows MS computed over non-zero values for all pretrained models and configurations for TOPK SAE with K = 10. Pretrained models are used both as base models and external encoders for MS (Fig. 1b). MS exhibits a strong dependence on the encoder: CLIP-based MS is consistently the highest, BLIP2 – intermediate, and DINOv3 – the lowest. This ordering follows the anisotropy ranking from Fig. 1, showing that highly anisotropic encoders inflate embedding-based scores, without necessarily reflect- ing a corresponding increase in monosemanticity. These observations motivate creating a monosemanticity score that is not dependent of external encoders. Measuring the monosemanticity level In Fig. 2, we present the barplots of TMS values obtained for all the examined base models and SAEs. Mean is computed for the set of non-zero TMS per-neuron values. We present the reference barplot of MS values in Fig. 3 (for the same set of neurons for a fair comparison). In this setup, we use each base model as the encoder for its SAE’s MS compu- tation. A clear difference between the two measures emerges. For MS, CLIP dominates with the highest score for all cases. TMS produces a more balanced ordering of models with two examined multimodal models sharing the highest Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence7 scores – CLIP (in 13 cases) and BLIP2 (in 5 cases). The overall inter-model gap is notably reduced. When the barplots of TMS and MS are compared in configu- ration pairs, the agreement between TMS and MS is strongest for DINOv3-based SAEs – likely due to the smallest score inflation due to anisotropy. It suggests that embedding-based MS is less affected by geometric similarity inflation in the more isotropic regime. In contrast, the divergence between MS and TMS for multimodal models indicates that embedding geometry with higher anisotropy can amplify apparent monosemanticity independently of the feature coherence. Overall, the results suggest that TMS captures a notion of within-feature co- herence that is more robust to embedding-space anisotropy, leading to a score less dominated by embedding geometry and more reflective of intrinsic feature coherence. Consequently, differences in monosemanticity across SAEs trained on different feature sources appear less pronounced under TMS than under purely geometry-based measures. (a) TOPK (b) BatchTOPK Fig. 2. Tversky Monosemantic Score (TMS) results. Fig. 4 provides qualitative examples of top-activating images for neurons with decreasing TMS values (from left to right) across SAEs trained on DI- NOv3, CLIP, and BLIP2 features for SAE TopK, K = 64, expansion factor 2. A consistent pattern can be observed, in which neurons with high TMS values ex- hibit visually coherent top activating image sets, sharing a clear semantic/visual motif, whereas lower TMS corresponds to more heterogeneous sets. We high- light neurons for which the TMS and MS orderings differ. While the overall ranking of neurons is largely consistent across measures, the highlighted cases reveal apparent divergences. For example, in the DINOv3 row, neurons with TMS values of 0.12 and 0.05 (showing mixtures of islands/musical instruments, and of animals/textures/boats/food, respectively) receive relatively higher MS scores (approximately 0.3) and an inverse ordering. The latter neuron exhibits a 8K. Filus and S. Pokuciński (a) TOPK (b) BatchTOPK Fig. 3. MonoSemantic (MS) [13] score results. clearly polysemantic activation pattern, and the substantially lower TMS value reflects smaller overlap in its activation sets. For CLIP-based SAEs, neurons with relatively high MS values (e.g. compared to DINOv3) display visually and semantically diverse image sets despite their geometric embedding space prox- imity. In contrast, TMS assigns lower scores to such neurons, reflecting reduced consistency in their activation behavior. These qualitative observations support the interpretation of TMS as capturing within-feature coherence as proxies for monosemanticity rather than embedding-driven similarity, and provide an intu- itive complement to the quantitative results in Figs. 2 and 3. Correlation with existing monosemanticity indicators To further validate TMS as an indicator of monosemanticity through feature coherence, we analyze its re- lationship with two neuron-level properties: (i) our main reference – the MonoSe- manticity Score (MS), and (i) the embedding variance within the activation sets. The second reference serves as an independent structural proxy, as monosemantic neurons are expected to activate on more homogeneous sets of stimuli (expressed with features of base models), which should result in lower variability in their activation sets (based on TMS binarization). Tab. 1 presents the per-neuron correlations between TMS and MS across all SAE variants. MS is computed with a base model used as encoder. Overall, a strong positive relationship is observed, particularly for DINOv3-based SAEs (the least anisotropic encoder), where correlations exceed 0.75 in most cases (16/18). This indicates that TMS captures a notion of feature monosemanticity largely overlapping with embedding-based measures, despite being computed without reliance on external representations. For CLIP-based SAEs, correlations exhibit greater variability, reflecting the influence of embedding geometry and anisotropy. Nevertheless, moderate to strong correlations are still present in all configurations, showing partial alignment between measures. Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence9 0.28 (MS = 0.77)0.15 (MS = 0.46)0.12 (MS = 0.29)0.09 (MS = 0.14)0.05 (MS = 0.32) 0.23 (MS = 0.81)0.15 (MS = 0.66)0.10 (MS = 0.56)0.08 (MS = 0.51)0.07 (MS = 0.41) 0.24 (MS = 0.56)0.17 (MS = 0.67)0.11 (MS = 0.47)0.08 (MS = 0.37)0.05 (MS = 0.32) Fig. 4. Example top 9 activating images for neurons with different TMS. Values de- crease from left to right. Top row includes examples for SAE-DINOv3, middle row — CLIP, and bottom row – BLIP2. All SAEs are TopK, expansion factor 2 and K = 64. We use bold for cases where ordering is different for TMS and MS. In Tab. 2 we present correlations between TMS and within-activation-set variance of feature embeddings via the base model. Across all models and SAE setups, the correlations are strongly negative, frequently exceeding −0.9. This result supports the interpretation that higher TMS values correspond to more co- herent activation patterns, aligned with the expected behavior of monosemantic features. The effect is particularly pronounced for BLIP2-based SAEs, where cor- relations approach −0.9 across nearly all configurations. These findings support that TMS reflects both the alignment with existing monosemanticity indicators and coherence of neuron activations. Evolution of monosemanticity during SAE training To see how monoseman- ticty changes over the course of optimization, we analyze the dynamics of MS and TMS throughout SAE training. Figs. 5 and 6 report how both measures evolve across epochs for all the base models and SAE configurations. It is visi- ble that both measures have similar trajectory trends for the majority of plots when compared in pairs – again mostly visible for DINOv3-based SAEs (in- crease at the beginning, followed by drop and stabilization). Trajectories show that monosemanticity in SAEs trained with different base model features evolves differently (e.g. the trend described for DINOv3 vs the TMS decreasing trend for BLIP2 (without the initial "bump"). TMS shows that monosemanticity of TopK SAEs with smaller K regime (K = 10) results in significantly higher scores over training than for higher K values. Contrastingly, for the BatchTopK vari- 10K. Filus and S. Pokuciński Table 1. MS correlation grouped by expansion factor and K for TOPK. All reported correlations are statistically significant (p-values << 0.05). SAEModelx2x4x8 K=10 K=32 K=64 K=10 K=32 K=64 K=10 K=32 K=64 TopKDINOv3 0.88 0.92 0.87 0.87 0.86 0.86 0.86 0.78 0.77 CLIP 0.54 0.77 0.86 0.53 0.75 0.83 0.47 0.70 0.78 BLIP2 0.68 0.75 0.79 0.62 0.75 0.77 0.64 0.75 0.75 BatchTopK DINOv3 0.87 0.93 0.84 0.65 0.89 0.89 0.50 0.78 0.85 CLIP 0.35 0.60 0.90 0.22 0.58 0.90 0.27 0.61 0.87 BLIP2 0.81 0.79 0.82 0.80 0.68 0.78 0.79 0.69 0.76 Table 2. Variance correlation itself grouped by K and expansion factor for TOPK. All reported correlations are statistically significant (p-values << 0.05). SAEModelx2x4x8 K=10 K=32 K=64 K=10 K=32 K=64 K=10 K=32 K=64 TopKDINOv3 -0.85 -0.85 -0.76 -0.87 -0.83 -0.83 -0.84 -0.80 -0.82 CLIP -0.81 -0.90 -0.88 -0.79 -0.90 -0.91 -0.74 -0.90 -0.91 BLIP2 -0.90 -0.93 -0.92 -0.90 -0.94 -0.94 -0.91 -0.93 -0.93 BatchTopK DINOv3 -0.81 -0.82 -0.70 -0.64 -0.86 -0.85 -0.49 -0.81 -0.88 CLIP -0.45 -0.71 -0.92 -0.47 -0.65 -0.93 -0.60 -0.68 -0.90 BLIP2 -0.90 -0.86 -0.88 -0.91 -0.78 -0.90 -0.90 -0.84 -0.89 ant, all SAE configurations converge more closely and obtain lower values. MS exhibits more irregular epoch-to-epoch fluctuations than TMS (e.g. MS for the CLIP-based TopK SAE, in which MS increases quickly in early epochs and then oscillates with relatively high-frequency fluctuations). Despite these differences, both measures tend to display synchronized local fluctuations, indicating that certain training phases induce genuine variability in feature emergence (e.g. for DINOv3-based TopK SAE with k=32/expansion factor 8). These observations suggest that monosemanticity is not static during SAE training but undergoes distinct phases of feature development, while its dynamics are different for SAEs trained with different base model features and sparsity. TMS and MS can capture these phenomena, highlighting the need for such dynamic examinations. Concept deletion effectiveness To assess whether model-level TMS monoseman- ticity translates into effective feature-level interventions for a given SAE, we perform a probe-based concept deletion experiment. For each SAE, we train a linear probe on SAE representations and mini-ImageNet classes. We zero the top-N latent features most relevant to a given class, ranked by the magnitude of the probe weights, treating classes as higher-level semantic targets. The resulting drop in target-class accuracy (target drop) serves as a measure of the functional relevance of SAE features. Fig. 7 shows target drop as a function of the number Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence11 of removed features N. Distinct behaviors emerge across base models. For DI- NOv3 SAEs, smaller K and lower expansion factors yield the strongest deletion effects. For CLIP and BLIP2 SAEs, smaller K also tends to produce stronger target drops at small N (even N = 1). This is consistent with our earlier TMS results, where these configurations exhibited higher monosemanticity, suggesting that tighter sparsity yields more concentrated features. For CLIP, the K = 10 settings show comparable or smaller drops at larger N, likely due to their lower probe accuracies (∼ 60–70%). Similar behavior for BLIP2 BatchTopK SAEs despite high probe accuracies (> 90%) may indicate greater feature overlap. To quantify the link between monosemanticity and deletion effectiveness, we compute correlations between SAE-level mean TMS/MS and target drop across all trained SAEs for each N (Tab. 3). For small N, both metrics show strong positive correlations with target drop for all models, indicating that more monosemantic SAEs enable more effective interventions. For CLIP-based SAEs, correlations become non-significant for N = 16 and N = 32, likely due in part to lower downstream probe performance. Comparing the two measures, TMS gives stronger correlations for CLIP-based SAEs (where MS is impacted by higher anisotropy), while comparable, but slightly weaker results in less anisotropic settings (DINOv3 and BLIP2 at smaller N). Overall, both metrics capture func- tional feature relevance, with TMS offering better predictive value when embed- ding geometry may distort similarity-based assessments. Table 3. Correlation analysis of TMS/MS-target drop across different models and N values. All correlations computed with corresponding p-values. Best results for each model marked in bold. Non-significant results (p-value > 0.05) in italics. ModelN Target drop - TMSTarget drop - MS Correlationp-valueCorrelationp-value DINOv3 10.8370.0000150.8780.000002 20.8780.0000020.9150.000000 40.8760.0000020.8930.000001 80.8430.0000110.9050.000000 160.8250.0000260.9170.000000 320.8680.0000030.9170.000000 CLIP 10.7830.0001210.7210.000728 20.8270.0000230.7070.001037 40.8640.0000040.6530.003286 80.6780.0019850.5340.022588 160.4060.0949530.2570.303309 320.1580.531483-0.0010.996757 BLIP2 10.7190.0007670.8180.000033 20.6740.0021660.7850.000113 40.6220.0058190.7500.000335 80.6700.0023600.7710.000181 160.6970.0013180.6620.002791 320.7320.0005580.5770.012195 12K. Filus and S. Pokuciński 0510152025 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Epoch MS DINO: TopKSAE 0510152025 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 Epoch DINO: BatchTopKSAE 0510152025 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 Epoch MS CLIP: TopKSAE 0510152025 0.55 0.60 0.65 0.70 Epoch CLIP: BatchTopKSAE 0510152025 0.40 0.45 0.50 0.55 Epoch MS BLIP: TopKSAE 0510152025 0.35 0.40 0.45 0.50 0.55 0.60 0.65 Epoch BLIP: BatchTopKSAE Exp=2, K=10Exp=4, K=10Exp=8, K=10 Exp=2, K=32Exp=4, K=32Exp=8, K=32 Exp=2, K=64Exp=4, K=64Exp=8, K=64 Fig. 5. Epoch evolution of MS for TopKSAE (left column) and BatchTopKSAE (right column) across DINO, CLIP, and BLIP models. Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence13 0510152025 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Epoch Tversky similarity DINO: TopKSAE 0510152025 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Epoch DINO: BatchTopKSAE 0510152025 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Epoch Tversky similarity CLIP: TopKSAE 0510152025 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Epoch CLIP: BatchTopKSAE 0510152025 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 Epoch TMS BLIP: TopKSAE 0510152025 0.05 0.10 0.15 0.20 0.25 Epoch BLIP: BatchTopKSAE Exp=2, K=10Exp=4, K=10Exp=8, K=10 Exp=2, K=32Exp=4, K=32Exp=8, K=32 Exp=2, K=64Exp=4, K=64Exp=8, K=64 Fig. 6. Epoch evolution of TMS for TopKSAE (left column) and BatchTopKSAE (right column) across DINO, CLIP, and BLIP models. 14K. Filus and S. Pokuciński 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of deleted features (N) Target drop DINO: TopKSAE 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of deleted features (N) DINO: BatchTopKSAE 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of deleted features (N) Target drop CLIP: TopKSAE 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of deleted features (N) CLIP: BatchTopKSAE 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of deleted features (N) Target drop BLIP: TopKSAE 2 0 2 1 2 2 2 3 2 4 2 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Number of deleted features (N) BLIP: BatchTopKSAE Exp=2, K=10Exp=4, K=10Exp=8, K=10 Exp=2, K=32Exp=4, K=32 Exp=8, K=32 Exp=2, K=64Exp=4, K=64Exp=8, K=64 Fig. 7. Deletion curves showing target drop as a function of number of deleted features (N). Left column: TopKSAE, right column: BatchTopKSAE. Rows: DINO, CLIP and BLIP models, respectively. Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence15 6 Conclusions In this paper, we introduced the Tversky Monosemanticity Score (TMS), a met- ric that operationalizes monosemanticity as activation-set coherence of bina- rized SAE latents. Across multiple SAEs trained on DINOv3, CLIP, and BLIP2 features, we showed that TMS is less affected by encoder anisotropy than the reference embedding-based metric, while remaining correlated with established monosemanticity indicators. Moreover, TMS more clearly revealed differences in training dynamics across base models and SAE regimes during optimization. Fi- nally, our probe-based concept deletion study showed that TMS as a model-level metric was more strongly associated with deletion effectiveness under higher en- coder anisotropy than the reference while remaining competitive otherwise. By operating on SAE latents, we eliminate the cost connected to external embedding generation of methods relying on external encoders, varying depending on hard- ware used, which in our experiments was up to app. 30 mins on the BLIP2 test set. Although our method operates on higher-dimensional SAE representations, its similarity computation is based on sparse vectors, thresholding and binary operations, which are typically computationally cheaper than the floating-point arithmetic on dense vectors, and allow implementation-level optimization. We adopted Tversky’s feature-based similarity model because it provides the conceptual basis for representing latent activations and their comparisons through shared and distinctive components, as a coherence-based measure. TMS’s correlation with other monosemanticity indicators supports its use as a use- ful estimation aid, especially for natural-image data, where concepts often co- occur with reasonable contextual structure. Practically, against degenerate co- activations, the presented within-active-set variance can be used as a comple- mentary sanity check. Our future work will focus on extending the proposed parameter-free baseline with the asymmetric Tversky variant, more advanced binarization [7,1], and a broader evaluation across tasks. Nevertheless, our cur- rent study with multiple base models, SAE regimes, sparsity levels, expansion factors, a natural-image task and from different perspectives, shows the potential of TMS for SAE verification without external concept labels or encoders. Preprint notice This is a preprint version. A shorter version of this paper has been accepted for presentation and publication in the post-workshop proceedings of the 8th International Workshop on eXplainable Knowledge Discovery in Data Mining (XKDD 2026), co-located with ECML PKDD 2026. The appendix is included only in this preprint and is not part of the peer-reviewed proceedings paper. Acknowledgments. This research was supported by the START Scholarship of the Foundation for Polish Science (FNP) for outstanding young scholars, agreement No. START 017.2025, the Polish Minister of Science and Higher Education scholarship for outstanding young researchers, agreement No. SMN/20/1546/2024, and the Research Funds for Young Scientists at the Silesian University of Technology (agreements No. 02/100/BKM25/0045 and 02/100/BKM26/0055). The authors utilized LLMs for lan- guage polishing under human supervision and take fully responsible for the manuscript. 16K. Filus and S. Pokuciński Disclosure of Interests. The authors have no competing interests. References 1. Aswal, D., Hudelot, C.: Llmsymguard: A symbolic safety guardrail framework leveraging interpretable jailbreak concepts. arXiv e-prints p. arXiv–2508 (2025) 2. Bereska, L., Gavves, E.: Mechanistic interpretability for ai safety–a review. arXiv:2404.14082 (2024) 3. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J.E., Hume, T., Carter, S., Henighan, T., Olah, C.: Towards monosemantic- ity: Decomposing language models with dictionary learning. Transformer Circuits Thread (2023) 4. Cywiński, B., Deja, K.: Saeuron: Interpretable concept unlearning in diffusion mod- els with sparse autoencoders. arXiv:2501.18052 (2025) 5. Fereidouni, M., Haider, M.U., Ju, P., Siddique, A.: Evaluating sparse autoencoders for monosemantic representation. arXiv:2508.15094 (2025) 6. Filus, K., Pokuciński, S.: Measuring monosemanticity in sparse autoencoders via latent activation coherence - source code (2026), Zenodo repository (private review link, will be made fully public on acceptance) 7. Gallifant, J., Chen, S., Sasse, K., Aerts, H., Hartvigsen, T., Bitterman, D.: Sparse autoencoder features for classifications and transferability. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. p. 29927–29951 (2025) 8. Gao, L., la Tour, T.D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., Wu, J.: Scaling and evaluating sparse autoencoders. arXiv:2406.04093 (2024) 9. Härle, R., Friedrich, F., Brack, M., Wäldchen, S., Deiseroth, B., Schramowski, P., Kersting, K.: Measuring and guiding monosemanticity. arXiv:2506.19382 (2025) 10. Huben, R., Cunningham, H., Smith, L.R., Ewart, A., Sharkey, L.: Sparse autoen- coders find highly interpretable features in language models. In: The Twelfth In- ternational Conference on Learning Representations (2023) 11. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. p. 19730–19742 (2023) 12. Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., Mordv- intsev, A.: The building blocks of interpretability. Distill 3(3), e10 (2018) 13. Pach, M., Karthik, S., Bouniot, Q., Belongie, S., Akata, Z.: Sparse autoencoders learn monosemantic features in vision-language models. arXiv:2504.02821 (2025) 14. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763 (2021) 15. Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., Nanda, N.: Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv:2407.14435 (2024) 16. Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M., Massa, F., Haziza, D., Wehrstedt, Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence17 L., Wang, J., Darcet, T., Moutakanni, T., Sentana, L., Roberts, C., Vedaldi, A., Tolan, J., Brandt, J., Couprie, C., Mairal, J., Jégou, H., Labatut, P., Bojanowski, P.: Dinov3 (2025) 17. Tversky, A.: Features of similarity. Psychological review 84(4), 327 (1977) 18. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. Advances in Neural Information Processing Systems 29 (2016) 19. Zaigrajew, V., Baniecki, H., Biecek, P.: Interpreting clip with hierarchical sparse autoencoders. arXiv:2502.20578 (2025) 18K. Filus and S. Pokuciński Appendices: Measuring Monosemanticity in Sparse Autoencoders via Latent Activation Coherence A Accuracy of SAEs In Tab. A.1, we provide the standard training metrics obtained for all the SAE setups used in our study after the final epoch 25: R 2 , L 0 , input-output Cosine Similarity and dead neurons ratio. R 2 and dead neurons ratio are obtained di- rectly via overcomplete pipeline, while we compute L 0 and input-output Cosine Similarity on the 10% split of our test dataset. Overall, the results indicate that the large majority of SAEs trained with our unified procedure are at least relatively functionally strong. In particular, the minimum input-output Cosine Similarity is 0.6636, 20 out of 27 BatchTopK models exceed 0.8, all TopK models exceed 0.8, and the best-performing setups achieve values above 0.9. Similarly, most models obtain R 2 > 0.7, while dead neurons are effectively eliminated due to the reanimation loss recommended in the overcomplete library. Therefore, although a configuration-specific strategy to tune the parameters would likely provide us with even stronger reconstruction metrics, the obtained models are sufficiently functional to support a fair comparative study of monosemanticity under a shared setup. B Accuracy of linear probes In Tab. B.1, we report the probe accuracy obtained for the concept deletion experiment before the deletion. Specifically, we include the accuracy of probes trained on raw features extracted from the base models (DINOv3, CLIP, and BLIP), as well as on the latent activations of all SAEs considered in our experi- ments. The results show that, across all data sources, it is possible to train rela- tively accurate probes in the 100-class classification setting via mini-ImageNet. The best overall performance is obtained for BLIP, for which all accuracies ex- ceed 0.9, and for DINO, where 18/19 cases achieve accuracy above 0.9. Weaker results are observed for CLIP embeddings and the latents of SAEs trained on CLIP representations. However, even in this case, the majority of accuracies re- main above 0.85. Overall, although both the SAEs and the probes were trained under a unified procedure rather than being individually tuned for a particular configuration, the obtained results are sufficiently strong to support the experi- ments regarding the concept deletion experiment. Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence19 Table A.1. Training metrics of SAE models across different feature extractors and configurations. R 2 denotes explained variance, L 0 is the fraction of zero activations in latent codes, CS denotes mean cosine similarity between input and reconstruction, and Dead features ratio is the fraction of latent dimensions never activated during training. Best values per feature extractor marked in bold. Model Exp. Factor K R 2 L 0 CS Dead Features Batch TopK TopK Batch TopK TopK Batch TopK TopK Batch TopK TopK DINO ×2 100.67 0.680.99 0.990.80 0.810.001 0.000 320.71 0.710.97 0.980.84 0.830.000 0.000 640.72 0.720.95 0.960.85 0.840.000 0.000 ×4 100.69 0.700.99 1.000.81 0.830.003 0.000 320.73 0.730.99 0.990.85 0.840.000 0.000 640.75 0.740.98 0.980.86 0.850.000 0.000 ×8 100.70 0.721.00 1.000.82 0.830.036 0.003 320.76 0.750.99 0.990.86 0.850.000 0.000 640.77 0.760.99 0.990.86 0.860.000 0.000 CLIP ×2 100.67 0.710.96 0.990.76 0.840.000 0.000 320.85 0.850.92 0.970.91 0.920.000 0.000 640.89 0.890.92 0.940.94 0.940.000 0.000 ×4 100.67 0.710.96 1.000.74 0.840.000 0.000 320.85 0.850.95 0.980.89 0.920.000 0.000 640.89 0.900.95 0.970.94 0.940.000 0.000 ×8 100.69 0.700.98 1.000.75 0.840.004 0.000 320.86 0.860.97 0.990.88 0.920.000 0.000 640.90 0.900.97 0.980.94 0.940.000 0.000 BLIP ×2 100.62 0.650.97 1.000.68 0.810.004 0.001 320.80 0.790.97 0.990.88 0.890.001 0.002 640.83 0.810.97 0.980.91 0.900.004 0.030 ×4 100.62 0.670.98 1.000.68 0.820.012 0.006 320.80 0.800.98 0.990.87 0.890.006 0.004 640.83 0.830.98 0.990.91 0.910.011 0.058 ×8 100.64 0.670.99 1.000.66 0.820.044 0.020 320.81 0.810.99 1.000.87 0.890.018 0.013 640.83 0.790.99 0.990.91 0.900.030 0.121 20K. Filus and S. Pokuciński Table B.1. Reconstruction accuracy of SAE models across different feature extractors and configurations. Baseline Acc denotes linear probe accuracy on raw features. SAE Acc denotes linear probe accuracy on SAE latent representations. Best SAE result per feature extractor marked in bold. ModelExp. FactorKBaseline Acc SAE Acc BatchTopKTopK DINO ×2 10 93.08% 90.90%90.31% 3291.85%91.55% 6492.36%92.09% ×4 1090.49%90.26% 3291.65%91.45% 6492.06%91.99% ×8 1089.80%90.05% 3291.50%91.54% 6491.82%91.77% CLIP ×2 10 89.83% 70.93%68.36% 3286.56%85.34% 6487.47%87.25% ×4 1059.67%68.96% 3286.93%84.87% 6487.87%86.89% ×8 1056.45%72.06% 3286.67%84.55% 6488.24%86.84% BLIP ×2 10 96.49% 90.14%89.42% 3295.35%94.85% 6495.47%95.09% ×4 1091.07%90.33% 3295.14%94.71% 6495.34%94.89% ×8 1089.46%90.75% 3294.85%94.55% 6495.09%94.82% Preprint: Monosemanticity via Sparse Autoencoders Latent Coherence21 C TMS stability under sampling To examine the stability of the sampling-based TMS computation, we conducted an additional experiment on an example SAE configuration: BatchTopK trained on DINOv3 features with sparsity K = 32 and expansion factor ×4, using the same training setup as for all other SAEs considered in the paper. Table C.1 reports the mean and standard deviation of the estimated mean TMS for differ- ent values of samples used as an upper limit on the number of pairs M over 10 runs. The results show that the estimate is rather stable across M values, with the mean remaining unchanged at approximately 0.084 to three decimal places across all tested values. However, as could be predicted, the variability slightly decreases as the number of sampled pairs increases: for K = 1000, which is the value used throughout the paper, the standard deviation is already on the order of 10 −5 , indicating sufficient stability. Even smaller budgets appear reasonably stable for fast comparisons in our case, while larger values provide even higher consistency across repetitions. Table C.1. Stability of mean TMS for different numbers of upper limit on the number of sampled pairs (M). Mmeanstd 500.0841.88× 10 −4 1000.0841.99× 10 −4 2500.0849.30× 10 −5 5000.0846.78× 10 −5 10000.0846.14× 10 −5 25000.0842.94× 10 −5 50000.0842.46× 10 −5 D Active latent neurons after binarization To better contextualize the Tversky Monosemanticity Score (TMS), we report the number of latent neurons with non-empty active sets A k after applying our mean-based binarization rule in Tab. D.1. For reference, the original feature di- mensionalities of the analyzed base models are 768 for DINO, 512 for CLIP, and 1408 for BLIP, which correspond to latent dimensionalities scaled by the consid- ered expansion factors×2,×4, and×8. The reported counts indicate how many latent neurons are considered active after binarization, therefore, how many neu- rons effectively participate in TMS computation. These statistics reveal a differ- ence between TopK and BatchTopK SAEs regimes. For BatchTopK, the number of active neurons is typically closer to the full latent dimensionality, indicating the expected broad utilization of the SAE dictionary, which is especially visible for CLIP-based SAEs. In this case, nearly all neurons become active across con- figurations. DINOv3-based SAEs also exhibit high coverage, while BLIP2-based 22K. Filus and S. Pokuciński SAEs, although still broadly utilized under BatchTopK, remain visibly below full saturation, especially for larger expansion factors. In contrast, TopK SAEs often use only a subset of their latent space after binarization, which depends on both K and expansion factor. For all base models, increasing K leads to a visi- ble increase in the number of active neurons, whereas increasing the expansion factor reduces the fraction of neurons that are active. This means that although larger expansion increases the overall latent dimensionality, the effective bina- rized dictionary size does not grow proportionally under TopK, which is mostly observed for BLIP-based SAEs. Overall, the raw latent dimensionality can dif- fer from the number of neurons effectively contributing to TMS. Active-neuron statistics therefore can help contextualize the reported TMS values by clarifying and confirming sufficiently large neuron sets for model-level aggregation. Table D.1. Active neurons grouped by K and expansion factor for TOPK. SAEModelx2x4x8 K=10 K=32 K=64 K=10 K=32 K=64 K=10 K=32 K=64 TopKDINO 1131 1533 1535 1460 2908 3043 1811 4376 5732 CLIP 569 954 1021 659 1606 2015 677 2466 3766 BLIP 670 1639 2300 633 2192 3627 630 2475 4200 BatchTopK DINO 1535 1535 1536 2714 3058 3070 4492 5928 6095 CLIP 1024 1024 1024 2048 2047 2046 4080 4084 4088 BLIP 2644 2735 2685 5071 5282 5096 9104 9613 9243