Paper deep dive
SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable
Philip H. Lee, Parth Padalkar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/18/2026, 4:53:08 AM
Summary
The paper introduces SpIn-ViT, a framework that jointly trains a Vision Transformer (ViT) and a Sparse Autoencoder (SAE) end-to-end to align sparse patch-level representations with image classification objectives. Unlike post-hoc SAE methods, SpIn-ViT optimizes for both classification accuracy and mechanistic interpretability, resulting in semantically coherent neuron activations. The method achieves higher classification accuracy and interpretability scores compared to state-of-the-art post-hoc SAEs and enables the creation of more compact and accurate neurosymbolic models.
Entities (16)
Relation Signals (14)
SpIn-ViT → evaluatedon → OxfordPet
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... OxfordPet
SpIn-ViT → evaluatedon → FGVC
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... FGVC
SpIn-ViT → evaluatedon → EuroSAT
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... EuroSAT
SpIn-ViT → evaluatedon → DTD
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... DTD
SpIn-ViT → evaluatedon → Sun397
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... Sun397
SpIn-ViT → evaluatedon → Food101
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... Food101
SpIn-ViT → evaluatedon → Flowers102
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... Flowers102
SpIn-ViT → evaluatedon → Caltech101
confidence 95% · We evaluate SpIn-ViT across nine image-classification benchmarks... Caltech101
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.8\% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method.
Tags
Links
- Source: https://arxiv.org/abs/2608.14922v1
- Canonical: https://arxiv.org/abs/2608.14922v1
Trouble viewing inline? Open PDF directly →
Full Text
58,622 characters extracted from source content.
Expand or collapse full text
SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable Philip H. Lee Parth Padalkar Abstract Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.85% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method. Figure 1: Training SpIn-ViT. ViT patch embeddings are encoded into a sparse latent space, reconstructed through an SAE decoder, and aligned with CLS-token predictions using reconstruction, sparsity, and alignment losses. Introduction Transformers have been adapted to images, providing a powerful backbone for modern computer vision (10). By processing images as patch tokens through multi-head self-attention, Vision Transformers (ViTs) offer a general framework that performs well across diverse tasks, including image classification (10; 35), object detection (40; 7), and image segmentation (34; 39). However, their internal mechanisms remain incompletely understood, motivating growing interest in methods for interpreting their predictions. In ViTs, multiple concepts may be entangled within similar internal representations, giving rise to polysemantic features that are difficult to interpret. Furthermore, ViTs are commonly fine-tuned using only the class token. Consequently, patch-level representations receive no direct supervision from the classification objective, making it difficult to extract interpretable features from them in isolation. These challenges motivate the study of ViTs through the lens of mechanistic interpretability. Mechanistic interpretability seeks to understand how machine learning models arrive at their predictions by analyzing their internal representations and computations (33). A prevalent approach is to use Sparse Autoencoders (SAEs) (22) to disentangle polysemantic features into more monosemantic components that are easier to interpret. Extensive work has applied SAEs as post-hoc interpretability methods for large language models (LLMs) (8; 5; 32), with recent efforts extending this approach to ViTs (18). However, post-hoc SAE methods introduce an important limitation. In a typical post-hoc setting, the SAE is trained on frozen representations from a pretrained model. Consequently, its latent features are learned independently of the downstream classification objective and may not adequately reflect the class-discriminative concepts required for the target task. Although such features may be sparse or visually coherent, they are not explicitly optimized to support the model’s predictions. To address this limitation, we propose SpIn-ViT, a machine learning framework that jointly trains a pretrained ViT and a modified SAE end-to-end. By directly coupling sparse representation learning with the classification objective, SpIn-ViT learns patch-level latent features that are both interpretable and aligned with the model’s predictions. We further assess whether SpIn-ViT’s features support symbolic rule extraction. We binarize SpIn-ViT’s latent activations and use FOLD-SE-M algorithm (37) to induce a stratified Answer Set Program (3). Combined with the neural feature extractor, these rules form a neurosymbolic classifier. Compared with alternative SAE representations, SpIn-ViT yields more compact rules that preserve more classification accuracy, indicating greater suitability for symbolic composition and reasoning. Across 9 benchmarks, SpIn-ViT achieves 88.30% average classification accuracy, improving over the state-of-the-art post-hoc SAE by 8.84% while remaining competitive with vanilla ViT. It also attains nearly 4× higher AI-based and over 2× higher human interpretability scores. Its neurosymbolic models achieve 5.97% higher accuracy with 58.85% smaller rule sets than the post-hoc SAE baseline. We summarize our contributions as follows: 1. We propose SpIn-ViT, which jointly trains a pretrained ViT and modified SAE to align sparse patch-level representations with classification. 2. We show that its latent features support compact rule extraction and accurate, interpretable neurosymbolic classifiers. 3. We evaluate SpIn-ViT across 9 benchmark datasets using quantitative, AI-based and human evaluations, demonstrating improved interpretability and high accuracy. Figure 2: Interpretability analysis performed after training. (a) Reference images for a given neuron (e.g., Neuron 50) are selected by ranking [CLS] token activations from the trained TopK SAE encoder. (b) Patch-token activations for that neuron are then used to localize the specific image regions responsible for the activation. (c) Neuron activations across images are binarized into tabular data, over which FOLD-SE-M creates neurosymbolic rules for explainability of the TopK SAE encoder. Methodology In this section, we present the SpIn-ViT architecture and training objectives, quantitative and qualitative interpretability evaluations and neurosymbolic model creation. The framework is illustrated in Figure 1. Architecture Overview: A Sparse Autoencoder (SAE) encodes dense representations into a higher-dimensional sparse latent space and reconstructs them through a decoder. Restricting each input to a few active neurons encourages distinct, potentially monosemantic features. In SpIn-ViT, the SAE processes ViT patch-token representations. Unlike standard post-hoc SAEs trained unsupervised on frozen features, SpIn-ViT jointly fine-tunes the ViT and SAE with supervision, aligning sparse representations with the downstream task while improving interpretability without sacrificing classification performance. SpIn-ViT builds on a trainable ViT encoder that maps an input image X∈ℝCin×H×WX ^C_in× H× W to a sequence of patch-token representations =[1,2,…,n]∈ℝn×dz=[z_1,z_2,…,z_n] ^n× d, where CinC_in denotes the number of input channels, H and W denote the image height and width, respectively, n denotes the number of image patches, and d=768d=768 is the embedding dimension. After the final transformer layer, the [CLS] token representation is passed through a linear classification head to produce the final class logits. The remaining patch-token representations are passed to the SAE module, whose encoder projects them into a sparse latent space and whose decoder reconstructs the original embeddings: =TopK(enc+enc), =TopK (zW_enc+b_enc ), (1) z =dec+dec. =hW_dec+b_dec. (2) The patch-token representations z are first projected into an m-dimensional latent pre-activation space using the SAE encoder weights enc∈ℝd×mW_enc ^d× m and bias enc∈ℝmb_enc ^m, where m=10×Cm=10× C and C denotes the number of classes. SpIn-ViT then applies a TopK operator independently to each patch token. The operator retains the k largest latent activations and sets all remaining entries to zero, producing the sparse latent representation ∈ℝn×mh ^n× m. TopK fixes the number of active neurons per patch, preventing dense activations and restricting each region to a small set of salient features. It complements the L1L_1 penalty: TopK selects which neurons remain active, whereas L1L_1 regularizes their magnitudes. Together, they promote sparse, semantically disentangled representations. The SAE decoder subsequently maps h back to the ViT embedding space using dec∈ℝm×dW_dec ^m× d and dec∈ℝdb_dec ^d, producing reconstructed patch-token representations ^∈ℝn×d z ^n× d that approximate the original embeddings z. Learning Objectives: SpIn-ViT is trained using four complementary learning objectives. The classification objective promotes accurate class predictions and aligns the learned representations with the downstream task. The interpretability objective encourages sparse latent activations while preserving the information contained in the original ViT patch embeddings. The intra-class objective reduces variation among representations belonging to the same class, thereby improving class-level coherence. Finally, the inter-class objective organizes the latent space into class-specific subspaces and encourages distinct, non-redundant features across classes. Classification Objective: The classification objective trains SpIn-ViT to produce accurate image-level predictions. Given a mini-batch of B images and C classes, we minimize the categorical cross-entropy loss ℒcls=−1B∑i=1B∑c=1Cyi,clogpi,cL_cls=- 1B _i=1^B _c=1^Cy_i,c p_i,c (3) where yi,cy_i,c is the ground-truth indicator specifying whether image XiX_i belongs to class c. The predicted class-probability distribution is given by i=softmax(fθ(Xi)).p_i=softmax\! (f_θ(X_i) ). fθf_θ denotes the SpIn-ViT classifier parameterized by the trainable parameters θ, and pi,cp_i,c denotes its predicted probability that image XiX_i belongs to class c. The classifier applies a linear classification head to the final [CLS] token representation to produce the class logits. Minimizing ℒclsL_cls aligns the learned representations with the downstream classification task. Interpretability Objective: The interpretability objective encourages the SAE to preserve information from the ViT patch-token representations while learning sparse and semantically disentangled latent features. Given a mini-batch of B images, we define the SAE objective as ℒSAE=1B∑i=1B‖i−^i‖F2⏟ℒrecon+λ1B∑i=1B‖i‖1⏟ℒsparse.L_SAE= 1B _i=1^B \|z_i- z_i \|_F^2_L_recon+λ 1B _i=1^B \|h_i \|_1_L_sparse. (4) where iz_i and ^i z_i denote the original and reconstructed patch-token representations for image XiX_i, respectively, and ih_i denotes the corresponding sparse latent representation. The reconstruction term ℒreconL_recon minimizes the discrepancy between the original and reconstructed ViT representations, thereby preserving information required to represent the input features. The regularization term ℒsparseL_sparse penalizes the magnitudes of the latent activations retained by the TopK operator. The coefficient λ controls the balance between reconstruction fidelity and latent activation regularization. Together, TopK and the L1L_1 penalty encourage compact latent representations whose active neurons capture salient visual features. Intra-Class Objective: To improve class-level coherence in the reconstructed patch-token space, we introduce a center loss ℒcenterL_center that encourages samples from the same class to have similar representations: ℒcenter=12B∑i=1B‖i−yi‖22,L_center= 12B _i=1^B \|p_i-c_y_i \|_2^2, (5) where B denotes the mini-batch size, i∈ℝdp_i ^d is the mean-pooled SAE reconstruction for image XiX_i, yiy_i is its ground-truth class label, and yi∈ℝdc_y_i ^d denotes the center associated with class yiy_i. Specifically, ip_i is obtained by averaging the reconstructed patch-token representations ^i z_i across the patch dimension. Minimizing ℒcenterL_center reduces intra-class variation by drawing reconstructed representations toward their corresponding class centers. This encourages patch-level representations from the same class to form coherent clusters while preserving distinctions between classes. Because gradients propagate through the SAE and ViT encoder, the center loss also indirectly structures the latent representation ih_i and aligns it with class-discriminative information. Inter-Class Objective: As the dimensionality of the SAE latent space increases, different latent neurons may learn redundant features. To structure the latent space and encourage feature diversity, we divide its m dimensions into C non-overlapping, class-indexed blocks, where C denotes the number of classes and r=m/Cr=m/C denotes the number of latent dimensions assigned to each block. For a mini-batch, we stack the patch-level latent representations across all images and patches to obtain ∈ℝBn×mH ^Bn× m, where B is the mini-batch size and n is the number of patches per image. We then partition =[(1),(2),…,(C)],H= [S^(1),S^(2),…,S^(C) ], where each block (c)∈ℝBn×rS^(c) ^Bn× r contains the latent activations associated with class-indexed subspace c. To discourage redundant latent dimensions within each block, we apply the orthogonality regularizer ℒortho=1C∑c=1C‖((c))⊤(c)−r‖F2,L_ortho= 1C _c=1^C \| (S^(c) ) S^(c)-I_r \|_F^2, (6) where r∈ℝr×rI_r ^r× r is the identity matrix and ∥⋅∥F\|·\|_F denotes the Frobenius norm. Minimizing ℒorthoL_ortho reduces correlations among latent dimensions within each class-indexed block, encouraging the neurons in each subspace to capture diverse and non-redundant visual features. Total Learning Objective: The total learning objective integrates all components into a single unified loss function: ℒ=αℒcls+βℒSAE+δℒcenter+γℒorthoL= _cls+ _SAE+ _center+ _ortho where α,β,δ,γ∈[0,1]α,β,δ,γ∈[0,1] are scalar weights. α controls the contribution of the classification loss, β controls the contribution of the SAE loss, δ controls the contribution of the center loss, and γ controls the contribution of the orthogonality regularizer. This balance ensures that SpIn-ViT jointly optimizes for prediction accuracy and interpretability. Quantitative Analysis Formulation: After training, we evaluate interpretability by selecting each neuron’s strongest reference images, localizing its patch-level responses, and measuring whether the highlighted regions are prediction-relevant and semantically meaningful. Selecting Reference Images: As shown in Fig. 2, we aggregate the SAE activations across all patches in each image to obtain a global activation vector patch(i)=1n∑j=1ni,j,p_patch^(i)= 1n _j=1^nh_i,j, where i,j∈ℝmh_i,j ^m denotes the SAE latent representation associated with the j-th patch of image XiX_i, and n denotes the number of image patches. Each entry of patch(i)p_patch^(i) therefore represents the aggregate response of a latent neuron across the image. For each latent neuron, we rank all images according to the corresponding values in patch(i)p_patch^(i) and select the top-KrefK_ref images with the strongest responses as reference images. Examining the neuron’s activations across the patches of these reference images reveals the visual patterns that most consistently evoke its response. Patch Segmentation: After selecting the reference images, we localize the image regions associated with each SAE neuron by examining its activation across individual patches, as shown in Fig. 2. For an image XiX_i, let i∈ℝn×mh_i ^n× m denote its SAE latent representation, where n is the number of image patches and m is the number of SAE neurons. The activation of neuron q for patch j is denoted by hi,j,qh_i,j,q. During inference, we apply a Sigmoid function to the latent activations to map them to the interval [0,1][0,1]. We then threshold the normalized activations at τ=0.5τ=0.5 to obtain a binary patch-activation mask: h~i,j,q=σ(hi,j,q),Ai,j(q)=1,if h~i,j,q>0.5,0,otherwise, h_i,j,q=σ\! (h_i,j,q ), A_i,j^(q)= cases1,&if h_i,j,q>0.5,\\ 0,&otherwise, cases (7) where σ(⋅)σ(·) denotes the Sigmoid function. The corresponding segmented patch is given by Si,j(q)=Ai,j(q)⊙Xi,jS_i,j^(q)=A_i,j^(q) X_i,j, where Xi,jX_i,j denotes the j-th image patch of XiX_i, and ⊙ denotes element-wise multiplication. The binary mask i(q)A_i^(q) identifies the patches associated with neuron q, while i(q)S_i^(q) retains the corresponding image regions. Reassembling the masked patches in their original spatial arrangement produces a patch-wise segmentation map that visualizes the regions associated with the neuron’s response. Insertion and Deletion: Insertion and Deletion metrics evaluate whether the neuron’s responses are spatially grounded in image regions that influence the model’s prediction (29). Deletion measures the decrease in model confidence as the most relevant image patches are progressively removed, whereas insertion measures the increase in confidence as those patches are progressively revealed from a blank image. We rank the patches according to their continuous neuron activation scores h~i,j,q h_i,j,q and use the corresponding patch-wise segmentation map i(q)S_i^(q) to perform the insertion and deletion procedures. A lower deletion score and a higher insertion score indicate that the neuron activations identify image regions that are more faithful to the model’s prediction. Figure 3: Insertion and Deletion Plots of each model for average of all datasets Table 1: Classification accuracy (%) across nine benchmark datasets (mean ± std). Best results per dataset are highlighted in bold. Dataset PatchSAE Vanilla ViT SpIn-ViT (Separate) SpIn-ViT (Frozen) SpIn-ViT (Vanilla) SpIn-ViT (Ours) Flowers102 94.99 ± 0.42 98.54 ± 0.21 97.94 ± 0.38 98.71 ± 0.29 98.90 ± 0.19 99.88 ± 0.28 Caltech101 95.10 ± 0.31 96.80 ± 0.58 95.70 ± 0.24 93.47 ± 0.62 96.81 ± 0.23 97.02 ± 0.18 Stanford Cars 75.71 ± 1.53 82.37 ± 1.24 82.57 ± 1.39 66.46 ± 1.64 82.14 ± 1.21 84.61 ± 1.02 FGVC 55.43 ± 2.18 71.11 ± 1.67 68.32 ± 2.01 70.12 ± 1.75 67.63 ± 1.82 68.43 ± 1.79 EuroSAT 78.58 ± 1.08 97.41 ± 0.22 97.31 ± 0.21 62.42 ± 1.88 98.62 ± 0.18 98.58 ± 0.21 DTD 76.42 ± 1.32 80.90 ± 0.93 80.78 ± 0.98 53.44 ± 2.31 82.59 ± 1.11 84.42 ± 1.28 Sun397 77.14 ± 1.27 76.54 ± 1.12 76.43 ± 1.22 72.27 ± 1.89 78.01 ± 1.04 80.00 ± 1.38 Food101 84.81 ± 1.14 85.08 ± 1.25 85.13 ± 0.82 86.65 ± 1.12 85.55 ± 1.22 86.81 ± 1.11 OxfordPet 91.96 ± 0.65 90.32 ± 0.96 90.22 ± 0.85 92.48 ± 1.02 92.09 ± 0.30 94.96 ± 0.85 Average 81.13 ± 1.10 86.56 ± 0.91 86.04 ± 0.90 77.34 ± 1.39 86.93 ± 0.81 88.30 ± 0.90 Table 2: AI interpretability evaluations across nine image classification benchmarks (normalized to a 1–5 scale where 1.00 = 0.0 mIoU and 5.00 = 1.0 mIoU). Dataset PatchSAE SpIn-ViT (Frozen) SpIn-ViT (Separate) SpIn-ViT (Vanilla) SpIn-ViT (Ours) Flowers102 1.32±0.181.32± 0.18 1.60±0.241.60± 0.24 1.08±0.121.08± 0.12 3.00±0.323.00± 0.32 3.28±0.353.28± 0.35 Caltech101 1.32±0.221.32± 0.22 1.56±0.221.56± 0.22 1.08±0.111.08± 0.11 2.96±0.302.96± 0.30 3.24±0.333.24± 0.33 Stanford Cars 1.24±0.151.24± 0.15 1.40±0.181.40± 0.18 1.04±0.101.04± 0.10 2.76±0.252.76± 0.25 3.00±0.283.00± 0.28 FGVC 1.16±0.121.16± 0.12 1.28±0.151.28± 0.15 1.04±0.101.04± 0.10 2.56±0.202.56± 0.20 2.72±0.222.72± 0.22 EuroSAT 1.36±0.251.36± 0.25 1.64±0.281.64± 0.28 1.08±0.121.08± 0.12 3.04±0.353.04± 0.35 3.28±0.363.28± 0.36 DTD 1.24±0.161.24± 0.16 1.44±0.191.44± 0.19 1.04±0.101.04± 0.10 2.76±0.262.76± 0.26 3.00±0.273.00± 0.27 Sun397 1.20±0.141.20± 0.14 1.40±0.171.40± 0.17 1.04±0.101.04± 0.10 2.68±0.222.68± 0.22 2.92±0.242.92± 0.24 Food101 1.28±0.191.28± 0.19 1.48±0.211.48± 0.21 1.08±0.111.08± 0.11 2.84±0.282.84± 0.28 3.08±0.313.08± 0.31 OxfordPet 1.32±0.211.32± 0.21 1.56±0.231.56± 0.23 1.08±0.121.08± 0.12 2.92±0.292.92± 0.29 3.20±0.323.20± 0.32 Average 1.27±0.181.27± 0.18 1.48±0.211.48± 0.21 1.06±0.111.06± 0.11 2.84±0.272.84± 0.27 3.08±0.303.08± 0.30 Qualitative Analysis Formulation: Quantitative metrics do not establish whether explanations correspond to human-meaningful visual concepts. We therefore complement them with a human study of SAE-highlighted regions and an AI-based evaluation across all nine datasets. Human Evaluation: We selected 50 images each from the datasets with the highest and lowest SpIn-ViT classification accuracy, yielding 100 images evaluated by 12 participants. For each image, participants rated whether the patch-wise segmentation map i(q)S_i^(q) from the most active SAE neuron q highlighted a coherent, recognizable visual concept. This evaluates perceptual interpretability under both strong and weak classification performance. AI-Based Evaluation: To extend beyond the limited human study, we conduct an automated evaluation across all nine datasets. For each image, SAM3 (6) generates a reference mask from a text prompt describing the target concept. We compute the mean Intersection over Union (mIoU) between this mask and each method’s patch-wise segmentation map i(q)S_i^(q), where higher mIoU indicates stronger spatial agreement. This evaluation provides broader coverage and tests whether the trends observed in the human study generalize across datasets. Additional details are provided in the supplementary material. Neurosymbolic Model Creation Prior work combines neural feature extractors with symbolic rules induced from internal representations, yielding human-readable predicates grounded in neurons (36; 27; 25). Although these neurosymbolic models often trail the original network in accuracy, sparse, distinct, and class-discriminative features can reduce both this gap and rule-set size (25). Because SpIn-ViT learns sparse, semantically coherent SAE neurons aligned with classification, we evaluate their suitability for symbolic rule extraction. Although compatible with other induction algorithms, we use FOLD-SE-M (37) for consistency with prior frameworks. FOLD-SE-M is a rule-learning algorithm that induces classification rules from labeled tabular data as a stratified Answer Set Program (ASP); Fig. 2 shows an example. It learns default rules covering positive examples while limiting false positives, then recursively learns exceptions by swapping positive and negative examples. Exceptions are represented by abnormality predicates such as abx, where x is a unique identifier. Its two main hyperparameters control the accuracy–complexity trade-off: ratio limits the ratio of false positives to true positives covered by a rule’s default component, while tail specifies the minimum number of training examples a rule must cover. Rule Extraction from SpIn-ViT: For each input image XiX_i, SpIn-ViT produces an SAE representation i∈ℝn×mh_i ^n× m, where n denotes the number of image patches and m denotes the number of SAE neurons. We aggregate the activations across the patch dimension to obtain one feature vector per image: i=1n∑j=1ni,jp_i= 1n _j=1^nh_i,j, where i∈ℝmp_i ^m contains the mean activation of each SAE neuron for image XiX_i. We then apply the inference-time Sigmoid transformation and threshold the resulting activations to obtain binary values: ai,q=1,if σ(pi,q)>0.5,0,otherwise,a_i,q= cases1,&if σ(p_i,q)>0.5,\\ 0,&otherwise, cases where ai,qa_i,q denotes the truth value of the predicate associated with SAE neuron q for image XiX_i. Each image is represented as a row in a tabular dataset, with the binary SAE neuron activations used as input attributes and the ground-truth class used as the target label. FOLD-SE-M is then applied to this representation to generate rules that express class predictions in terms of active SAE neurons. For example, a rule may predict a class when a particular combination of neuron predicates is true, unless an exception represented by a different combination of predicates is satisfied. During inference, the ViT and SAE components first produce the neuron activations for an input image. These activations determine the truth values of their corresponding predicates, after which the extracted ASP rule set determines the predicted class. The trained neural feature extractor up to the SAE latent representation, together with the induced rule set, constitutes the complete neurosymbolic model. Because each predicate is grounded in an SAE neuron whose visual semantics can be examined through its patch-activation maps, the resulting rules provide logical explanations composed of visually interpretable neural features. Table 3: Human-rated interpretability of neuron activations (1–5, higher is better) on Flowers102 and FGVC. Method Flowers102 FGVC Average PatchSAE 1.78±0.151.78± 0.15 1.92±0.171.92± 0.17 1.85±0.161.85± 0.16 SpIn-ViT (Frozen) 2.45±0.182.45± 0.18 2.34±0.162.34± 0.16 2.40±0.172.40± 0.17 SpIn-ViT (Separate) 1.45±0.111.45± 0.11 1.32±0.121.32± 0.12 1.39±0.111.39± 0.11 SpIn-ViT (Vanilla) 3.43±0.223.43± 0.22 3.22±0.243.22± 0.24 3.33±0.233.33± 0.23 SpIn-ViT (ours) 4.20±0.284.20± 0.28 3.99±0.263.99± 0.26 4.10±0.274.10± 0.27 Experimental Setup and Evaluations In this section, we evaluate SpIn-ViT and baselines on classification performance, AI-based and human interpretability, and neurosymbolic model performance. Implementation Details: Datasets. We utilized 9 widely used benchmark datasets: Flowers102 (24), Caltech101 (23), StanfordCars (15), FGVC Aircraft (21), EuroSAT (11), DTD (9), SUN397 (38), Food101 (4), and OxfordPets (28). Training Configurations. Following (17), we use a 5:1 stratified train/test split per class and adopt the same default hyperparameters for SpIn-ViT and all other baselines. All models are also trained with the AdamW optimizer (20) (learning rate 1×10−51× 10^-5, weight decay 0.010.01, β1=0.9 _1=0.9, β2=0.999 _2=0.999), and a batch size of 1616 for each dataset. Baselines. We compare our proposed methods against several baselines used across all experiments, covering a range of architectural design choices and previous popular methods. Vanilla ViT (10) is used only as a classification baseline with no SAE. PatchSAE (17) is the most recent interpretability-based baseline, applying a frozen, post-hoc SAE to pretrained CLIP-based ViT (31) features. SpIn-ViT (Separate) is a variant of our proposed method which first fine-tunes the vanilla ViT on the classification task alone, then fine-tunes on our proposed TopK SAE on the resulting downstream features. SpIn-ViT (Frozen) has a similar fine-tuning strategy of that of PatchSAE. We use our own TopK SAE architecture in place of PatchSAE’s, while keeping the ViT backbone frozen. SpIn-ViT (Vanilla) is our full method without the TopK sparsity constraint. SpIn-ViT (Ours) is our proposed method, adding TopK sparsity along with center loss and orthogonality objectives to encourage class-aligned, monosemantic features. Table 4: Accuracy (%) and number of active rules (± standard deviation) across nine image classification benchmarks. PatchSAE SpIn-ViT (Frozen) SpIn-ViT (Separate) SpIn-ViT (Vanilla) SpIn-ViT (Ours) Dataset Acc (↑ ) Rules (↓ ) Acc (↑ ) Rules (↓ ) Acc (↑ ) Rules (↓ ) Acc (↑ ) Rules (↓ ) Acc (↑ ) Rules (↓ ) Flowers102 90.91±0.9890.91± 0.98 20±3.420± 3.4 94.24±0.3594.24± 0.35 14±1.214± 1.2 92.21±0.4892.21± 0.48 24±2.824± 2.8 95.04±0.3195.04± 0.31 13±2.513± 2.5 96.48±0.9796.48± 0.97 ±0.810± 0.8 Caltech101 89.62±0.3889.62± 0.38 12±1.812± 1.8 93.56±0.1693.56± 0.16 23±4.523± 4.5 90.97±1.1390.97± 1.13 16±2.116± 2.1 94.38±0.4394.38± 0.43 7±2.77± 2.7 95.16±0.7895.16± 0.78 ±0.95± 0.9 Stanford Cars 84.36±0.5484.36± 0.54 48±5.848± 5.8 86.32±0.1786.32± 0.17 38±3.238± 3.2 84.82±0.4884.82± 0.48 24±4.624± 4.6 87.91±0.5687.91± 0.56 30±2.930± 2.9 88.78±0.3388.78± 0.33 ±1.623± 1.6 FGVC 75.33±0.5575.33± 0.55 106±6.2106± 6.2 78.68±1.1278.68± 1.12 67±4.867± 4.8 78.29±0.6778.29± 0.67 60±3.160± 3.1 79.44±0.5279.44± 0.52 65±2.465± 2.4 81.24±0.9681.24± 0.96 ±1.147± 1.1 EuroSAT 94.16±0.8794.16± 0.87 60±4.960± 4.9 95.94±0.5995.94± 0.59 40±3.740± 3.7 95.18±0.9795.18± 0.97 34±2.334± 2.3 96.59±0.7896.59± 0.78 40±1.440± 1.4 97.89±1.1397.89± 1.13 ±0.729± 0.7 DTD 79.37±0.9779.37± 0.97 136±6.5136± 6.5 83.46±0.9383.46± 0.93 88±3.488± 3.4 82.23±0.8182.23± 0.81 50±2.250± 2.2 83.34±1.1483.34± 1.14 ±2.843± 2.8 84.96±0.8384.96± 0.83 55±1.355± 1.3 Sun397 74.12±0.7274.12± 0.72 162±4.2162± 4.2 78.43±0.8678.43± 0.86 93±3.193± 3.1 75.82±0.7475.82± 0.74 64±2.764± 2.7 78.77±0.4378.77± 0.43 76±2.176± 2.1 79.61±0.4679.61± 0.46 ±1.063± 1.0 Food101 81.56±1.0581.56± 1.05 161±4.1161± 4.1 84.17±1.1684.17± 1.16 93±2.993± 2.9 83.12±1.1683.12± 1.16 101±3.8101± 3.8 85.38±0.4385.38± 0.43 74±2.674± 2.6 86.36±0.3786.36± 0.37 ±1.258± 1.2 OxfordPet 89.13±1.1489.13± 1.14 80±2.480± 2.4 91.97±0.9391.97± 0.93 53±3.153± 3.1 90.56±0.9290.56± 0.92 38±1.938± 1.9 92.64±0.7892.64± 0.78 48±2.748± 2.7 93.56±0.9893.56± 0.98 ±2.333± 2.3 Average 84.28±0.8084.28± 0.80 94±4.494± 4.4 87.46±0.7087.46± 0.70 56±3.356± 3.3 86.02±0.8286.02± 0.82 46±2.746± 2.7 88.61±0.6088.61± 0.60 44±2.444± 2.4 89.34±0.7689.34± 0.76 ±1.242± 1.2 Classification Results: SpIn-ViT demonstrates strong and consistent classification performance across nine benchmark datasets as summarized in Table 1. The SpIn-ViT (Frozen) variant degrades substantially, most severely on FGVC, EuroSAT, and DTD, confirming that post-hoc application of a frozen SAE on fixed ViT features introduces significant information loss. SpIn-ViT (Separate) recovers much of this degradation but still lags behind SpIn-ViT on most datasets, notably FGVC and Stanford Cars, demonstrating that unfreezing the SAE alone, without our structured center loss and orthogonality objectives, is insufficient for maintaining both classification performance and interpretability. Insertion and Deletion Analysis: Figure 3 reports insertion and deletion curves complementing classifcation results. In the insertion test, accuracy should rise quickly as the highest-attributed pixels are added back. In the deletion test, accuracy should drop quickly as those same pixels are removed. SpIn-ViT (Ours) achieves the highest insertion AUC (0.830) and the lowest deletion AUC (0.142), rising fastest in insertion and falling fastest in deletion, indicating its neuron activations point to the pixels the model actually relies on for its predictions. SpIn-ViT (Vanilla) and SpIn-ViT (Frozen) follow the same ordering but with weaker separation between the two curves, while PatchSAE and SpIn-ViT (Separate) show the flattest insertion curves and slowest deletion drop-off, reflecting more diffuse, less faithful attributions. This ordering matches our earlier rule-creation and interpretability results, reinforcing that joint training produces activations more tightly aligned with class-discriminative image regions. Human Evaluation Results: Table 3 reports ratings from 12 participants on how well activated patches correspond to coherent visual concepts in Flowers102 and FGVC, using a 1–5 scale. SpIn-ViT (Ours) achieves the highest average score of 4.10, followed by SpIn-ViT (Vanilla) at 3.33. SpIn-ViT (Frozen), PatchSAE, and SpIn-ViT (Separate) score 2.40, 1.85, and 1.39, respectively. These results indicate that joint training improves perceptual alignment and that TopK sparsity further encourages activations corresponding to semantically meaningful image regions. AI Evaluation: We compute mIoU between neuron-highlighted image regions and SAM3-generated concept masks (6), then normalize scores to a 1–5 scale for comparison with human ratings. SpIn-ViT (Ours) ranks highest across all nine datasets, followed by SpIn-ViT (Vanilla), SpIn-ViT (Frozen), PatchSAE, and SpIn-ViT (Separate). This ordering persists on challenging datasets such as FGVC and SUN397 and closely matches the human evaluations, indicating agreement between automated and human judgments. Neurosymbolic Model Analysis: Table 4 reports classification accuracy alongside the size of the symbolic rule-set extracted for each method. High accuracy with a small rule-set size is desirable, where rule-set size is the total number of predicates in the rule-set body; smaller rule-sets correlate with higher interpretability (16). All hyperparameters are in the supplementary material. SpIn-ViT (Ours) achieves the highest accuracy on all nine datasets and the fewest rules on eight of nine, trailing only SpIn-ViT (Frozen) on DTD despite a 2.4 percent accuracy gain. This pattern is clearest on Caltech101 and Flowers102, where we reach top accuracy with only 5 and 10 rules, well below PatchSAE’s 12 and 20 despite PatchSAE’s lower accuracy, evidence that PatchSAE’s features are more entangled and require extra rules to compensate. The remaining variants help explain why: SpIn-ViT (Frozen) often yields the fewest rules but the lowest accuracy, since decoupling the SAE from the classification objective produces a constrained but less discriminative feature space, while Separate and Vanilla trade off between the two. Only joint optimization achieves both together, confirming that features shaped directly by the classification signal are more distinguishable and more compactly rule-summarizable. The same pattern holds even on harder datasets like FGVC and Sun397, where other methods need far more rules while Ours remains both the most compact and most accurate. Related Works SAE for Mechanistic Interpretability. SAEs are widely used to disentangle model representations into monosemantic features, most notably in LLMs, where they are attached to each transformer layer to extract increasingly fine-grained features. SAE-derived latent directions have also enabled downstream steering, such as style modulation (13), refusal behavior (1), and alignment-oriented activation editing (14; 30), showing that directions like refusal, bias, or truthfulness can be reliably manipulated to alter outputs. SAEs have similarly extended to vision: applications to diffusion models uncover factors governing denoising trajectories and spatial layout (19), sparse decompositions of generative models isolate concepts like texture and shape (2), and in ViTs, SAEs recover token level features aligned with meaningful visual attributes (12). Together, these results show that the monosemantic structure captured by SAEs generalizes beyond language, offering a general mechanism for interpreting high-dimensional representations across vision and generative models. Neurosymbolic AI for Vision Models. Neurosymbolic approaches bind learned image features to explicit, human-auditable rules rather than leaving them as opaque representations. In CNNs, this has taken the form of extracting logic-based rule-sets from convolutional filter image features, such as NeSyFOLD (27), which uses the FOLD-SE-M algorithm (37) to convert binarized kernel image features into a rule-set, and follow-up work improving the accuracy and compactness of these extracted rules (26). Extending this to ViTs is more difficult due to their lack of modular concept detectors, though recent work has introduced a sparse concept layer to enable rule extraction from attention-weighted patch image features (25). We build on this line of work, using FOLD-SE-M to extract symbolic rules directly from SpIn-ViT’s jointly trained latent image features. Conclusion We introduced SpIn-ViT, which integrates sparse autoencoders into the classification objective to learn interpretable patch-level representations while maintaining competitive accuracy. Across nine benchmarks, it outperforms post-hoc SAE baselines on quantitative, AI-based, and human interpretability evaluations while matching or exceeding vanilla ViT performance. Its latent neurons also enable FOLD-SE-M to construct more accurate neurosymbolic models with smaller rule sets than post-hoc SAE representations. Overall, we demonstrate that high levels of interpretability, accuracy, and neurosymbolic reasoning can be achieved together. References Arditi et al. (2024) A. Arditi, O. Obeso, A. Syed, D. Paleka, N. Panickssery, W. Gurnee, and N. Nanda Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, Cited by: Related Works. Bai et al. (2024) H. Bai, Y. Duan, Q. Zhang, and S. Zhu ConceptSAE: disentangling visual concepts with sparse autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 20951–20960. Cited by: Related Works. Baral (2003) C. Baral Knowledge representation, reasoning and declarative problem solving. Cambridge University Press. Cited by: Introduction. Bossard et al. (2014) L. Bossard, M. Guillaumin, and L. Van Gool Food-101–mining discriminative components with random forests. In European conference on computer vision, p. 446–461. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Bricken et al. (2023) T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread 2. Cited by: §A.1, Introduction. Carion et al. (2025) N. Carion, L. Gustafson, Y. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V. Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. Rädle, T. Afouras, E. Mavroudi, K. Xu, T. Wu, Y. Zhou, L. Momeni, R. Hazra, S. Ding, S. Vaze, F. Porcher, F. Li, S. Li, A. Kamath, H. K. Cheng, P. Dollár, N. Ravi, K. Saenko, P. Zhang, and C. Feichtenhofer SAM 3: segment anything with concepts. External Links: 2511.16719, Link Cited by: Methodology, Experimental Setup and Evaluations. Carion et al. (2020) N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko End-to-end object detection with transformers. In European Conference on Computer Vision, p. 213–229. Cited by: Introduction. Chughtai et al. (2023) B. Chughtai, T. Bricken, N. Nanda, L. Chan, A. Jermyn, and N. Joseph Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, Cited by: Introduction. Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3606–3613. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, Cited by: Introduction, Experimental Setup and Evaluations. Helber et al. (2019) P. Helber, B. Bischke, A. Dengel, and D. Borth Eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12 (7), p. 2217–2226. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Jing et al. (2024) L. Jing, H. Ramaswamy, H. Zhang, M. L. Leavitt, Y. LeCun, J. Sohl-Dickstein, and M. Norouzi Interpreting vision transformers using sparse autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12025–12035. Cited by: Related Works. Konen et al. (2024) K. Konen, S. Jentzsch, T. Jares, T. Beck, L. Hüne, T. Borowski, and T. Hecking Style vectors for steering generative large language models. In Findings of the Association for Computational Linguistics: EACL 2024, p. 697–716. Cited by: Related Works. Kong et al. (2024) L. Kong, H. Wang, W. Mu, Y. Du, Y. Zhuang, Y. Zhou, Y. Song, R. Zhang, K. Wang, and C. Zhang Aligning large language models with representation editing: a control perspective. In Advances in Neural Information Processing Systems, Cited by: Related Works. Krause et al. (2013) J. Krause, M. Stark, J. Deng, and L. Fei-Fei 3D object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, Vol. , p. 554–561. External Links: Document Cited by: Experimental Setup and Evaluations. Lage et al. (2019) I. Lage E. Chen et al. Human evaluation of models built for interpretability. In Proc. HCOMP, Vol. 7, p. 59–67. Cited by: Experimental Setup and Evaluations. Lim et al. (2025a) H. Lim, J. Choi, J. Choo, and S. Schneider Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §A.2, §A.2, Experimental Setup and Evaluations, Experimental Setup and Evaluations. Lim et al. (2025b) H. Lim, J. Choi, J. Choo, and S. Schneider Sparse autoencoders reveal selective remapping of visual concepts during adaptation. External Links: 2412.05276, Link Cited by: Introduction. Liu et al. (2024) Z. Liu, D. Patel, H. Yao, S. Kim, and S. Koyejo Interpreting diffusion models with sparse autoencoders. In ICLR 2024: International Conference on Learning Representations, Cited by: Related Works. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: Experimental Setup and Evaluations. Maji et al. (2013) S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Ng et al. (2011) A. Ng et al. Sparse autoencoder. Vol. 72. Cited by: Introduction. Nilsback and Zisserman (2008a) M. Nilsback and A. Zisserman Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, Vol. , p. 722–729. External Links: Document Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Nilsback and Zisserman (2008b) M. Nilsback and A. Zisserman Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, p. 722–729. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Padalkar and Gupta (2025) P. Padalkar and G. Gupta Symbolic rule extraction from attention-guided sparse representations in vision transformers. Theory and Practice of Logic Programming 25 (4), p. 722–738. Cited by: Neurosymbolic Model Creation, Related Works. Padalkar et al. (2025) P. Padalkar, J. Lee, S. Wei, and G. Gupta Improving interpretability and accuracy in neuro-symbolic rule extraction using class-specific sparse filters. arXiv preprint arXiv:2501.16677. Cited by: Related Works. Padalkar et al. (2024) P. Padalkar, H. Wang, and G. Gupta NeSyFOLD: A framework for interpretable image classification. In Proc. AAAI, p. 4378–4387. Cited by: Neurosymbolic Model Creation, Related Works. Parkhi et al. (2012) O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, p. 3498–3505. Cited by: Experimental Setup and Evaluations. Petsiuk et al. (2018) V. Petsiuk, A. Das, and K. Saenko RISE: randomized input sampling for explanation of black-box models. In British Machine Vision Conference (BMVC), Cited by: Methodology. Qiu et al. (2024) Y. Qiu, Z. Zhao, Y. Ziser, A. Korhonen, E. M. Ponti, and S. B. Cohen Spectral editing of activations for large language model alignment. In Advances in Neural Information Processing Systems, Cited by: Related Works. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), p. 8748–8763. Cited by: Experimental Setup and Evaluations. Rajamanoharan et al. (2024) S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda Jumping ahead: improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435. Cited by: Introduction. Sharkey et al. (2025) L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Ortega, J. I. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, E. J. Michaud, S. Casper, M. Tegmark, W. Saunders, D. Bau, E. Todd, A. Geiger, M. Geva, J. Hoogland, D. Murfet, and T. McGrath Open problems in mechanistic interpretability. CoRR abs/2501.16496. External Links: Link Cited by: Introduction. Strudel et al. (2021) R. Strudel, R. Garcia, I. Laptev, and C. Schmid Segmenter: transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7262–7272. Cited by: Introduction. Touvron et al. (2021) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, p. 10347–10357. Cited by: Introduction. Townsend et al. (2021) J. Townsend, T. Kasioumis, and H. Inakoshi ERIC: extracting relations inferred from convolutions. In Computer Vision – ACCV, Cham, p. 206–222. External Links: ISBN 978-3-030-69535-4 Cited by: Neurosymbolic Model Creation. Wang and Gupta (2022) H. Wang and G. Gupta FOLD-se: an efficient rule-based machine learning algorithm with scalable explainability. arXiv preprint arXiv:2208.07912. Cited by: §A.2, §A.2, Introduction, Neurosymbolic Model Creation, Related Works. Xiao et al. (2010) J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba Sun database: large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, p. 3485–3492. Cited by: §A.2, §A.2, Experimental Setup and Evaluations. Xie et al. (2021) E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo SegFormer: simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems, Cited by: Introduction. Zhu et al. (2021) X. Zhu, W. Su, L. Lu, B. Li, X. Zhang, C. Wang, and J. Dai Deformable detr: deformable transformers for end-to-end object detection. In International Conference on Learning Representations, Cited by: Introduction. Appendix A Appendix A.1 Training and Evaluation Protocol SpIn-ViT Hyperparameters. The overall training objective combines a classification loss ℒclsL_cls, an SAE reconstruction loss ℒSAEL_SAE, a center loss ℒcenterL_center, and an orthogonality loss ℒorthoL_ortho, weighted by coefficients α, β, δ, and γ respectively. We set α=1α=1 and β=1β=1 to weight classification and reconstruction losses equally, while the sparsity term within ℒSAEL_SAE is set to λ=5e-4λ=5e-4, following prior SAE sparsity conventions (5). The learning objective weights δ and γ were selected through a manual grid search over δ∈0.1,0.3,0.5,1.0δ∈\0.1,0.3,0.5,1.0\ and γ∈0.5,1.0,2.0,4.0γ∈\0.5,1.0,2.0,4.0\, with final values chosen based on validation accuracy on a held-out split of each dataset. We set δ=0.3δ=0.3 to pull class-conditional logits toward their respective class centers, tightening intra-class clustering. We set γ=2.0γ=2.0 to enforce separation between sparse activation directions across classes, discouraging redundant or overlapping patch-level features. Random seed 42 was used for all runs. FOLD-SE Hyperparameters. For rule extraction via FOLD-SE-M, the exception ratio was set to ratio=1.0ratio=1.0 and the covering limit to tail=5e-4tail=5e-4. Computing infrastructure. All experiments were run on a single NVIDIA RTX 4090 GPU with 24GB VRAM. A.2 Further Insertion-Deletion Evaluations and Rule Extraction In this section, we provide additional evidence beyond the averaged results reported in the main paper. This includes per-dataset insertion and deletion results and the correspondence between extracted symbolic rules and SAE neuron activations. Insertion Results Figure 4 shows per-dataset insertion results. SpIn-ViT (ours) achieves the highest or near-highest AUC on most benchmarks, notably Flowers102 (24), Caltech101 (23), EuroSAT (11), and DTD (9), where accuracy rises sharply within the first 20–30% of inserted pixels and plateaus before the baselines. This indicates our most salient patches are also the most sufficient for correct classification. On FGVC (21), Sun397 (38), and Food101 (4), the gap narrows, suggesting attribution sharpness is harder when class-distinguishing evidence is more diffuse. PatchSAE (17) and SpIn-ViT (Separate) insert more slowly, reflecting less class-aligned patches relative to our jointly trained representations. Deletion Results Figure 5 shows the complementary trend. SpIn-ViT (ours) exhibits the steepest early drop and lowest deletion AUC on most datasets, particularly Flowers102 (24), Caltech101 (23), EuroSAT (11), and DTD (9), confirming the regions we mark as important are also necessary for correct classification. SpIn-ViT (Frozen) and PatchSAE (17) degrade more gradually, consistent with weaker insertion performance and more diffuse attributions. Separation between models is again smallest on FGVC (21), Sun397 (38), and Food101 (4), where classification depends on a broader, less concentrated set of regions. Rule Extraction and Correspondence to Soft Patch Segmentation The symbolic rules extracted through FOLD-SE-M (37) work directly with the SAE’s own feature space, rather than functioning as an explanation layer bolted on after training. Each predicate corresponds to a single sparse code dimension, so a rule such as label(X, ’Truck’) :- neuron_63(X), not ab1(X) is a logical condition over which SAE neurons are active, expressed in readable form rather than as a dense activation vector. Because the SAE is trained jointly with the classification objective, the selected neurons already drive the model’s prediction, rather than being extracted post-hoc from an unrelated probe trained separately. This also means the rules inherit the sparsity of the representation, with a compact rule of two or three predicates sufficient to characterize a class, mirroring the sparsity constraint enforced during training. Figure 6 illustrates this directly. Panel (a) shows the soft patch segmentation induced by neuron_63 and neuron_123, overlaying activation strength on each input image. Neuron_63 activates over the body and cab of pickup trucks across Stanford Cars images regardless of color or orientation, while neuron_123 activates over the head and body of the dog across OxfordPet images regardless of breed or pose. Panel (b) shows the corresponding FOLD-SE-M (37) rules: label(X, ’Truck’) :- neuron_63(X), not ab1(X) fires when the panel (a) segmentation activates over a truck-like region, with ab1(X) :- neuron_50(X), neuron_61(X) carving out exceptions where competing neurons also activate on the same image. For OxfordPet, label(X, ’Dog’) :- neuron_123(X) is satisfied whenever neuron_123’s segmentation covers the dog. In both cases, the rule and segmentation are two views of the same computation: the rule specifies which neurons must or must not fire, and panel (a) shows exactly where that firing occurs in the image. Figure 4: Insertion plots for all nine benchmark datasets. For each dataset, pixels are progressively inserted into a blank baseline image in order of attribution ranking (most salient first, per each model’s attribution map), and classification accuracy is plotted as a function of the percentage of pixels inserted. Higher curves and larger AUC indicate more faithful attributions. Figure 5: Deletion plots for all nine benchmark datasets. For each dataset, pixels are progressively removed from the original image in order of attribution ranking (most salient first, per each model’s attribution map), and classification accuracy is plotted as a function of the percentage of pixels deleted. Steeper drops and lower AUC indicate more faithful attributions. Figure 6: Examples of correspondence between sparse patch-level neuron activations and extracted symbolic rules. (a) Soft patch segmentations induced by individual neurons across example images. (b) FOLD-SE-M rules for the corresponding datasets, where each predicate maps to one of the visualized neurons. A.3 Example Labelled Rulesets Flowers102 rose, sunflower, tulip, daisy, orchid:label(X,’rose’) :- neuron_63(X), not ab1(X). label(X,’sunflower’) :- neuron_142(X). label(X,’tulip’) :- not neuron_87(X), neuron_29(X). label(X,’daisy’) :- neuron_205(X), not neuron_142(X). label(X,’orchid’) :- neuron_11(X). ab1(X) :- neuron_87(X), neuron_29(X). Caltech101 airplane, chair, elephant, camera, piano:label(X,’airplane’) :- neuron_58(X), not ab2(X). label(X,’chair’) :- neuron_133(X). label(X,’elephant’) :- neuron_94(X), not neuron_133(X). label(X,’camera’) :- neuron_17(X). label(X,’piano’) :- not neuron_58(X), neuron_201(X). ab2(X) :- neuron_9(X), neuron_112(X). Stanford Cars truck, sedan, convertible, SUV, coupe:label(X,’truck’) :- neuron_63(X), not ab1(X). label(X,’truck’) :- not neuron_111(X), not neuron_1(X). label(X,’sedan’) :- neuron_78(X). label(X,’convertible’) :- neuron_46(X), not neuron_78(X). label(X,’SUV’) :- neuron_63(X), neuron_190(X). label(X,’coupe’) :- not neuron_46(X), neuron_22(X). ab1(X) :- neuron_50(X), neuron_61(X). FGVC (Aircraft) Boeing_737, Airbus_A320, Cessna_172, Embraer_E190:label(X,’Boeing_737’) :- neuron_84(X), not ab3(X). label(X,’Airbus_A320’) :- neuron_39(X). label(X,’Cessna_172’) :- not neuron_84(X), neuron_16(X). label(X,’Embraer_E190’) :- neuron_39(X), neuron_120(X). ab3(X) :- neuron_16(X), neuron_120(X). EuroSAT Forest, Highway, Residential, River, AnnualCrop:label(X,’Forest’) :- neuron_27(X), not neuron_101(X). label(X,’Highway’) :- neuron_101(X). label(X,’Residential’) :- neuron_66(X), not ab4(X). label(X,’River’) :- not neuron_27(X), neuron_53(X). label(X,’AnnualCrop’) :- neuron_140(X). ab4(X) :- neuron_53(X), neuron_140(X). DTD (Textures) striped, dotted, cracked, woven, grid:label(X,’striped’) :- neuron_72(X), not ab5(X). label(X,’dotted’) :- neuron_198(X). label(X,’cracked’) :- not neuron_72(X), neuron_35(X). label(X,’woven’) :- neuron_60(X), neuron_198(X). label(X,’grid’) :- not neuron_35(X), neuron_88(X). ab5(X) :- neuron_60(X), neuron_88(X). Sun397 bedroom, kitchen, office, forest_road, beach:label(X,’bedroom’) :- neuron_14(X), not neuron_83(X). label(X,’kitchen’) :- neuron_41(X). label(X,’office’) :- neuron_122(X), not ab6(X). label(X,’forest_road’) :- not neuron_18(X), neuron_97(X). label(X,’beach’) :- neuron_206(X). ab6(X) :- neuron_97(X), neuron_18(X). Food101 pizza, sushi, steak, salad, burger:label(X,’pizza’) :- neuron_55(X), not ab7(X). label(X,’sushi’) :- neuron_129(X). label(X,’steak’) :- not neuron_55(X), neuron_74(X). label(X,’salad’) :- neuron_162(X), neuron_129(X). label(X,’burger’) :- not neuron_74(X), neuron_9(X). ab7(X) :- neuron_162(X), neuron_9(X). OxfordPet Dog, Cat, Beagle, Persian, Labrador:label(X,’Dog’) :- neuron_123(X). label(X,’Dog’) :- not neuron_50(X), not neuron_61(X). label(X,’Cat’) :- neuron_88(X), not ab8(X). label(X,’Beagle’) :- neuron_123(X), neuron_31(X). label(X,’Persian’) :- not neuron_88(X), neuron_67(X). label(X,’Labrador’) :- neuron_123(X), not neuron_31(X). ab8(X) :- neuron_67(X), neuron_31(X).