Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition

Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition Brianna Chrisman * 1 , Lucius Bushnaq 2 , and Lee Sharkey 2 1 Independent 2 Apollo Research April 2, 2025 Abstract Much of mechanistic interpretability has focused on understanding the activation spaces of large neural networks.However, activation space- based approaches reveal little about the under- lying circuitry used to compute features. To better understand the circuits employed by mod- els, we introduce a new decomposition method calledLocal Loss Landscape Decomposition (L3D). L3D identifies a set of low-rank subnet- works—directions in parameter space—of which a subset can reconstruct the gradient of the loss be- tween any sample’s output and a reference output vector. We design a series of progressively more challenging toy models with well-defined subnet- works and show that L3D can nearly perfectly recover the associated subnetworks. Additionally, we investigate the extent to which perturbing the model in the direction of a given subnetwork af- fects only the relevant subset of samples. Finally, we apply L3D to a real-world transformer model and a convolutional neural network, demonstrat- ing its potential to identify interpretable and rele- vant circuits in parameter space. 1. Background Mechanistic interpretability aims to uncover the internal mechanisms responsible for the behavior of large models, enabling developers to better understand, intervene in, and align models (Bereska & Gavves, 2024). One goal of the field is to decompose model behavior into subcomponents that are less complex and more human-interpretable while still fully explaining a model’s behavior. The most pop- ular method in this space is Sparse Dictionary Learning (SDL) (Cunningham et al., 2023; Bricken et al., 2023; Gao et al., 2024), which identifies latent features by decompos- * Email:brianna.chrisman@gmail.com ing a model’s activation space into an overcomplete basis of sparsely activating components. These learned basis vectors represent distinct features that can then be used to reconstruct the original activations. 1.1. From Activation to Parameter-Based Interpretability However, decomposing the activation space of a model has various limitations. Current SDL algorithms struggle with reconstructing features of certain geometries, such as nonlinear features, feature manifolds, and certain types of superposition (Engels et al., 2025a;b; Merullo et al., 2025; Lindsey et al., 2024). Such issues could become more pro- nounced in models with a less clearly defined read/write stream, such as diffusion models and recurrent networks. (Pascanu et al., 2013; Ho et al., 2020). Additionally, acti- vation space captures thefeaturesextracted by a model’s underlying circuits, but it says little about what mechanisms derived them. Alternatively, to understand a model’s underlyingmecha- nisms, we might interpret models through the lens ofparam- eter space. Parameters are the fundamental objects updated during training, and can capture information about a model’s internal mechanisms, the training process, and the mecha- nistic relationship between outputs. We hypothesize that parameter space can hold interpretable units of computa- tion (Sharkey et al., 2025): models can be decomposed into simplersubnetworks, where each subnetwork is involved in the predictions of a subset of training data. To understand how we might go about identifying such sparsely active subnetworks, we first must understand some key insights about loss landscape geometry. 1.2. Loss Landscape Geometry Singular learning theory (SLT) describes how the structure of parameter space influences model behavior (Watanabe, 2000; 2005) and has been used to characterize model topolo- 1 arXiv:2504.00194v1 [cs.LG] 31 Mar 2025 gies (Bushnaq et al., 2024; Lau et al., 2024) as well as different phases of the training process (Wang et al., 2024; Hoogland et al., 2025; Davies et al., 2023). A key insight from SLT is that large models are highly degenerate in parameter space: they can have many different parameter configurations that achieve minimal loss on the training set (Wei et al., 2022; Watanabe, 2007). In fact, gradient descent tends to converge on configurations with many of these de- generate directions. Our work extends this hypothesis one step further:If models are highly degenerate with respect to the full training distribution, then with respect to a subset of the training data, they likely exhibit additional subset-specific degeneracies. Another key phenomenon our method relies on is that, at least in the current set of foundation models, local attribution methods appear to be good approximations of global rela- tionships between pairs of samples. For example, attribution patching can successfully modify a model’s output by tar- geting specific activations determined by the first-order gra- dients of paired outputs (Nanda, 2023; Kram ́ ar et al., 2024; Syed et al., 2023). Similarly, steering vectors—derived from differences in activations between paired samples—can ef- fectively guide models toward specific behaviors, even when applied beyond the original magnitude of those activation differences (Turner et al., 2024; Subramani et al., 2022). 1.3. Loss Landscape Decomposition Our goal in this paper is to identify directions in parameter space that correspond to subnetworks, as defined in Section 1.1. Two existing methods in particular address a similar prob- lem of decomposing models into subnetworks. An earlier work (Matena & Raffel, 2023) decomposes parameter space by computing principal directions of a per-sample Fisher In- formation matrix to identify meaningful features. A more re- cent method, Attribution Parameter Decomposition (Braun et al., 2025), decomposes model weights by identifying subnetworks where (1) the sum of subnetwork weights ap- proximates the original model parameters, (2) for any given input, the outputs of the sum of topk-attributed networks has low behavioral loss when compared to those of the original model, and (3) subnetworks are individually simpler than the whole network. Rather than the parameter values themselves (Braun et al., 2025) or an approximate second order gradient of the param- eters (Matena & Raffel, 2023),the object we decompose is the gradient of the loss between a sample’s output and a reference output.We aim to identify directions in parame- ter space that strongly affect this loss for some samples, and have little effect on the loss for other samples. In practice, we choose the reference output as another output w 1 w 2 (f (x 1 , W), y r ) w 1 w 2 (f (x 2 , W), y r ) w 1 w 2 (f (x 3 , W), y r ) Figure 1: Decomposing a loss landscape into a set of pa- rameter directions, or subnetworks, where a smaller subset of directions can approximately reconstruct the gradient of divergence/loss between any sample’s output and a refer- ence output. Here,Dis a loss, ordivergencemeasure,fis our model,Wis the set of parameters in the model,x i is a sample input, andy r is a reference output sampled from the training distribution, and we discuss our reasoning in Section 2.2. Our goal is to identify low-rank directions in parameter space—henceforth referred to as subnetworks—such that for any pair of samples, a small number of these directions can be used to reconstruct this gradient (Figure 1). We call our decomposition method Local Loss Landscape Decomposition (L3D). In this work, we first describe the mathematical foundation of our approach. We then develop progressively more complex toy models to evaluate the ef- ficacy of L3D and characterize its limitations. Finally, we present preliminary results on real-world models to demon- strate L3D’s potential for scaling beyond toy settings. 2. Methodology In the next sections, we will formally set up our decom- position problem (Section 2.1), define the criteria that we will use for our subnetwork/parameter directions (Section 2.2), describe how to efficiently decompose parameters into these directions (Section 2.3), walk through our training algorithm (Section 2.4), and then explain how to use these decompositions to intervene on a model’s behavior (Section 2.5). From now on, we will use the word “subnetworks” to refer to the directions in parameter space we wish to learn. 2 2.1. Set up Consider a modelfthat takes a batch of inputsX(with number of samplesn s and input dimensionn i ) and param- eter values ofW, and computes a batch of outputs (with output dimensionn o ). f(x,W) :R n s ×n i →R n s ×n o (1) Our approach assumes that for a given input, there are many components of a model’s parameters that are not involved in inference. Changing parameters in the direction of these components will not change the model’s output. Conversely, changing parameters in the direction of components that areinvolvedwouldchange the model’s output. Moreover, we are interested in finding parameter directions that, when perturbed,meaningfullychange a model’s output. The next section will explain what constitutes “meaningful.” 2.2. Divergences of Paired Outputs Intervening in a relevant parameter direction should move a sample’s output either closer to or further from areference output. This reference output should serve as a neutral and representative baseline that captures the typical behavior of the model’s output distribution. We considered three candidates for this reference: 1.A uniform output:This reference consists of a vec- tor with uniform values. However, it fails to account for the training distribution, leading to a bias toward learning subnetworks that influence outputs that skew toward particularly high or low values. 2. Mean of outputs:This reference is computed by av- eraging each output index across the training distribu- tion or a batch. While it is grounded in the data, it risks averaging away meaningful correlations between outputs, producing a reference that may still be out-of- distribution relative to the training data. 3. Another sample as the reference:For each sample, we use the output of a randomly selected sample as the reference. This approach preserves the nuances of the output distribution but may lead to slow convergence due to high variance in reference selection. We thought (3) was the most principled, and least biased of the three. Although not tested rigorously, in early prototypes all three choices seemed to produce reasonable results on toy models and we did not find any issues with convergence using (3). For this work we use (3) as our reference output, but we believe other choices are possible and may have different strengths and weaknesses. Therefore, we decompose gradients of the loss between pairs of outputs with the aim of finding directions that move a model’s output towards or away from the reference.Be- cause we use the term “loss” later on when we describe our training process, we will refer to this metric instead as “divergence.” The gradient of the divergence of a sample’s output and a reference can be written as as: ∇ W D(f(x i ,W),y r )| W=W 0 (2) wherex i ∈X,y r ∈f(X) Here,Dis a divergence measure,fis our model,x i is our input of interest,y r is a reference output (chosen as another output sampled from the training distribution),W is a set of parameters andW 0 is the model’s original pa- rameters. Our toy models are regression-type models, so we use normalized MSE as divergence. For the real-world transformer and CNN models, which output probabilities, we used KL-divergence. We abbreviate the expression in Eq. 2 as∇ W D. 2.3. Sparse Principal Directions We want to decompose our per-sample gradients with re- spect to parameters into low-rank components. Each sam- ple’s gradient should be able to be expressed as a lin- ear combination of a small set of these components. We will do this by learning transformsV in ∈R n v ×n w and V out ∈R n w ×n v wheren w is the number of parameters in the model, andn v is the number of components (sub- networks) we wish to use to represent the parameter space. (For those familiar with the sparse dictionary learning set up, this is similar to learning a transform from activation space into feature space, and vice versa). V in effectively transforms a gradient from the parameter space to the subnetwork space, so that: ∇ V D=V in ∇ W D(3) We want to findV in andV out such that for any given pair of samples, a small subset of subnetworks can approximately reconstruct the gradient of divergence. ∇ W D≈V out ΛV in ∇ W D(4) whereΛ i,j =    1ifi=jandi∈argTopK i (|∇ v i D|) 0otherwise argTopkrelies on a hyperparameterkthat controls the num- ber of components we wish to use to reconstruct each sam- ple. In practice, we use abatchTopK(Bussmann et al., 2024) and a fraction for thekhyperparameter rather than an absolute number.k= 0.1means that we select the top 10% of∇ V Dmagnitudes overvandxto reconstruct our batch of gradients. 3 2.3.1. LOWRANK PARAMETER DIRECTIONS Learning a set of full rank parameter directions would be extremely expensive. We also expect that modular, sparsely active circuits would be lower rank than their full-model counterparts because they are processing smaller numbers of features. Therefore, we use low-rank representations of ourV in andV out , and correspondingly learn low-rank circuits (Appendix B.1). Specifically, we use a Tucker de- composition described in Section B.1. 2.4. Training We wish to learn the decomposition-related transformsV in andV out that minimize thebatchTopKreconstruction loss of our divergence gradient described above. We use a (nor- malized) L2 norm loss. L= ||∇ W D−V out ΛV in ∇ W D|| 2 ||∇ W D|| 2 (5) For each batch of samples, we randomly select a reference samplex r to be paired with each samplex i in the batch. We then compute the gradient of divergence betweenf(x i ) andf(x r )at the target model’s parametersW 0 . We trans- form that gradient into the subnetwork space usingV in , and compute thetopKcomponents. We transform those com- ponents back into the original parameter space usingV out , and compute the loss between the reconstructed gradient and the original gradient. We apply a learning update to V in andV out with the goal of minimizing this loss. We also normalizeV out to be a unit vector after each update in order to keep the magnitudes ofV in andV out similar. Algorithm 1L3D algorithm for learningV in andV out trans- forms of parameter space. 1:foreach epochdo 2:foreach minibatchXdo 3:foreachx i ∈Xdo 4:Randomly selectx r ∈X 5:∇ w D i =∇ w D(f(x i ,W),f(x r ))| W=W 0 6:end for 7:∇ v D=V in ∇ w D 8:τ=topK(abs(∇ v D)) 9: ˆ ∇ w D=V out (∇ v D⊙(abs(∇ v D)> τ)) 10:L= ||∇ w D− ˆ ∇ w D|| 2 ||∇ w D|| 2 11:L.backward() 12:UpdateV in andV out 13:NormalizeV out to be unit vectors. 14:end for 15:end for 2.5. Measuring and Intervening Our learned subnetworks will just be the columns ofV out , restructured into the same tensor structure asW. After identifying subnetworks, we may want to intervene on a specific circuit. If we wish to “intervene” on a model using a single sub- network, we can update the model’s parameters by moving them in their unit direction, multiplied by a scalar factor (δ). To tune our model in the direction of subnetworkv i and compute predictions onx, we evaluate: f(x,W+δv i )(6) We also may want to quantify the impact of a subnetwork in on a certain sample. First, we can compute the impact of a subnetwork on a specific output’s (f(x i )) divergence with a single reference outputy j . The impactIof subnetwork v k on the gradient of divergence betweenf(x i )can be measured by: I(x i ,y j ,v k ) = V in k,: ∇ w D(f(x i ,W),y j ) (7) Because we are randomly sampling outputs from our train- ing distribution as the reference output, we then average the impacts of a subnetworkv k and an inputx i over many different reference samples to better quantify the impact of the subnetwork on a single sample’s predictions overall. Al- though more computationally expensive, this gives a more robust measurement for the impact of a subnetwork on a specific sample. I(x i ,v k ) = 1 n j n j X j=1 I(x i ,y j ,v k )(8) 3. Results To evaluate L3D’s ability to decompose models, we focused on developing toy models that involve well-defined subnet- works. We designed several toy models to test the efficacy of L3D. Our toy models all consist of several well-characterized computations being performed by the same model, with an sparse input space designed in a way that only a small number of computations are being performed for each input sample. Our toy models progressively test more complex types of circuits. Table 1 describes our 4 toy models and the differ- ent attributes of circuitry the are designed to capture. The specific hyperparameters used to train our toy models are described in Appendix B.2, as well as the hyperparameters used for each decomposition in Appendix B.3 4 Toy model of super- position Circuit Superposition (TMCS) Higher Rank Circuit Superposition Complex Loss Land- scape x 1 x 2 x 3 x 1 x 2 x 3 x 1 x 2 x 3 A 1 X A 2 X A 3 X x 1 x 2 x 3 A 1 X A 2 X A 3 X x 1 x 2 x 3 x 1 2 x 2 2 x 3 2 X7→X7→AXX7→AXX7→X 2 FeatureSuperposi- tion ✓ Circuit Superposition×✓ Circuits>rank 1×✓probably✓ Complex Loss Land- scape ×✓ Table 1: Our toy models and their various properties. Toy models are designed to test progressively more complicated phenomenon present in model circuitry, 3.1. Toy Model of Superposition 3.1.1. SETUP We started off by validating our algorithm on a well-studied toy problem, the toy model of superposition (TMS). TMS is simple linear autoencoder with a low-dimensional hid- den layer followed by a ReLU activation function at the output (Elhage et al., 2022). The model is trained on a dataset of samples where few features are active at a time, and “superimposes” these features in the hidden layer such that features’ embeddings in the hidden layer have minimal interference with each other. We trained a toy model of superposition with 5 features and 2 hidden dimensions (with sparsity=.05) to test L3D’s ability to resolve models with superimposed features. 3.1.2. DECOMPOSITION We decomposed the TMS model into 5 subnetworks, using rank-1 parameter tensors. L3D successfully decomposed the model into subnetworks corresponding to the encoding and decoding of each feature (aX i : ˆ X i circuit). Figure 2 shows the decomposition. Moreover, the encoder directions of the learned subnetworks are nearly perfectly parallel to the original embedding of each input index (Figure 3). One thing to note is that parameter vectors do not have a preferred direction. L3D is equally likely to identify a parameter vector in the direction ofθas it is in the direction of−θ. This is why, for example, the weights in subnetwork 1 are in the opposite direction as the original network (Table 1). This decomposition resulted in a reconstruction error of 19%. The reconstruction error is related to the interfer- ence between features when multiple features are active in the same sample. We expect decompositions of higher dimensional networks to exhibit less reconstruction error, as the amount of nearly orthogonal parameter vectors (non- interfering) that can be compressed into parameter space scales exponentially with dimension. We see this effect in the next higher-dimensional toy model where the reconstruc- tion loss is in fact lower. 3.1.3. INTERVENTION Parameter vectors learned by L3D can be used to intervene on model behavior. In principle, we could finetune a model using only selected subnetworks (See 4.2). While we did not go the extent of finetuning a model, we explored the effect of perturbing a model’s parameter space in the direction of a subnetwork (by an increment ofδ), as described in Section 2.5. If subnetworks do in fact represent sparse computations, we hope that intervening on a subnetwork has a strong effect on the predictions of relevant samples, and little effect on others. As shown in Figures 4 and 5, moving the TMS model in the direction of a single subnetwork did in fact achieve this. Perturbing in the direction of subnetwork 1 primarily affected samples where feature 1 was active, with a small effect on the inputs that had interference with feature 1’s embeddings. In fact, for TMS, we could successfully fully “turn off” a computation by moving far enough in the direction of the subnetwork. (Although for models with more complex loss landscapes, “turning off” a computation is not as straightforward, as we will later discuss). 5 Figure 2: L3D subnetwork decomposition of TMS. Each subnetwork corresponds to the encoder/decoding of a single input feature. 1.51.00.50.00.51.01.5 W i, 0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 W i, 1 0 1 2 3 4 Figure 3: The encoder/decoder directions of the original model (solid lines) and each subnetwork (dashed lines). The directions learned by each subnetwork are nearly perfectly parallel to the encoding for each input feature. The colors of the lines refer to the input index each embedding represents. 1 0 1 2 3 4 v 0 v 1 v 2 v 3 =1 v 4 0.00.51.0 1 0 1 2 3 4 0.00.51.00.00.51.00.00.51.00.00.51.0 = 1 X 0 X 1 X 2 X 3 X 4 X i X i Figure 4: The effect of intervening on the TMS model in the direction of each subnetwork. We generated 1000 inputs from the TMS input distribution (x-axis), intervened on each subnetworkv i with magnitudeδand measured the change in outputs (y-axis) for each sample. The outputs corresponding to the index relevant to each subnetwork experienced the most change. 202 0.0 0.5 1.0 E[f(X, W 0 + v i )f(X)] subnetwork = 0 202 subnetwork = 1 202 subnetwork = 2 202 subnetwork = 3 202 subnetwork = 4 X i 0 1 2 3 4 Figure 5: The effect of intervening at various values ofδin the direction of each subnetwork. The y-axis represents the average amount an output changed (data points colored by output index), when perturbed an amountδin the direction of a subnetwork. 3.2. Toy Model of Circuit Superposition 3.2.1. SETUP TMS exhibitsfeature superposition- the input features’ low dimensional embeddings are non-orthogonal. However, the sparsecircuitsin the original TMS we decomposed are no- tablynotin superposition - a given weight or parameter is only relevant for a single circuit and circuits. It seems highly unlikely that real world model circuits would decompose this way, since learning circuits composed of perfectly or- thogonal parameter vectors limits the amount of circuits that can be contained in a given set of parameters. We therefore developed a toy model ofcircuit superposition(TMCS) in order to analyze L3D’s ability to resolve such circuits. We definecircuit superposition as a phenomenon by which subnetworks share parameter elements, and even more generally have non-orthogonal parameter vectors. Our toy model of circuit superposition (Toy model 2 in Table 1) uses the same architecture and input data distribution as TMS, but is trained to predictlinear combinations of the input features(X7→AX). We set the entries ofAas uniform random values between 0 and 3 (chosen arbitrarily) and generate input-output pairs to train the toy model with. We used an model with 10 inputs, 5 hidden layers, and 10 output features (although such a model does not need to have the same number of input and outputs). If subnetworks are only relevant to a small set of inputs, then we would expect each subnetwork to compute an input feature’s contribution to the output vector. If this is the case, then individual parameters would be involved in multiple subnetworks:W dec i,1 (the set of parameters connecting the hidden nodes to the first output node) will contain in- formation about bothA 1,1 ,A 2,1 ,A 3,1 .... Put another way, 6 the subnetworks will interfere with each other - parameter directions associated with each subnetwork will be non- orthogonal. 3.2.2. DECOMPOSITION We decomposed TMCS into 10 subnetworks of rank-1 pa- rameter tensors (Figure 6) with a reconstruction loss of 6.4%. The subnetworks each strongly corresponded to a single input feature, as we expected. Since each subnetwork theoretically corresponds to the con- tributions of a single input feature, we should be able to reconstruct the originalAvalues from each subnetwork. To deriveAfrom each subnetwork, we (1) identified the which column in the subnetwork’sW dec direction has the largest norm and then (2) traced the weights of the network through that path. That is for subnetworkk: j ∗ =argmax j ||W dec j k || 2 (9) ˆa i,j ∗ =W enc i,j ∗ k W dec i,j ∗ k Recall the parameter vectors are normalized to be unit vec- tors so we expect them to be a scalar multiple of the trueA values. As seen in Figure 7, our derivedˆahad a very high correlation to the originalavalues (r 2 = 0.92). 3.3. Higher Rank Circuits 3.3.1. SETUP Because each subnetwork in TMCS traces the path of a single input neuron, the underlying subnetworks should inherently have a rank of 1. In order to test the ability of L3D to learn higher rank circuits, we developed a toy model with inherently higher rank circuits. For this model, we used the same set up as TMCS, but we correlated the sparsities of sets of input features. We used 30 input features, and we filtered our data to ensure that input features 1-5, 6-10, etc, are always active (>0) or inactive (<0) together. In this setup, circuits should always be associated with groups of 5 input features and so should have a rank of 5 (diagram shown in Figure S1). 3.3.2. DECOMPOSITION Although we expect the model to have 6 subnetworks, we used excess parameter tensors (n v = 8) in order to allow more flexibility in learning. We tracked the fraction of inputs for which a subnetwork was used in thetopKrecon- struction (P act ) to identify which were “dead subnetworks”, and reportP act from the last epoch. Furthermore, although we expected the underlying subnetworks to be rank 5, we experimented with using different rank representations to see how well lower-rank parameter directions could repre- sent the model. Interestingly, rank-1 representations of the parameter tensors were able to represent the model nearly as well as rank-5 representations (Figure S2). In Figure 8, we show the decomposition of a rank-3 decomposition. L3D successfully learned a subnetwork corresponding to each of the 5 input feature groups, as well as a number of dead circuits. The higher and lower rank decompositions also learned similar subnetworks (Figure S3). When we trained L3D without these additional subnetworks, the re- construction loss often got caught in local minima. Similar to training sparse autoencoders (Cunningham et al., 2023), having extra degrees of freedom allows for better learning, even if at the end of training the extra subnetworks are never active. 3.4. Toy model with Complex Loss Landscape 3.4.1. SETUP In the previous models, other than the ReLU discontinuity the model’s were linear transformations between inputs and outputs. We should expect their loss landscapes to there- fore be well-behaved, with local attributions being perfectly representative of global attributions (up until the ReLU dis- continuities). However, we wanted to test the limitations of a L3D on a model with a more complex loss landscape, especially when it comes to intervening with a subnetwork. We therefore trained a multi-layer model to predict multiple non-linear functions of input features at once. We trained a GeLU network forX i 7→X 2 i . We used a network with 4 hidden layers of 10 neurons each, and 5 input and output neurons. Once again, the input features are sparse (and range from -1 to 1), incentivizing the toy model to learn circuits in superposition whose interferences will cause minimal errors on the sparse input distribution. We a priori expected the model to have 5 subnetworks, one for each input feature. Although it is less clear what rank the tensors of the underlying circuits should be, there are not inherent reasons to believe subnetworks should be low rank the way there was in the TMS model. 3.4.2. DECOMPOSITION To allow for slightly higher rank subnetworks but still com- press the dimensions of the model, we decomposed our model into 5 rank-2 parameter tensors. Additionally, in- stead of varying rank, we experimented with using differ- ent numbers of subnetworks to represent our model. In the 5-subnetwork decomposition (Figure 9), we found sub- networks tracing the path ofX i 7→X 2 i for each indexi. However, this decomposition had a relatively high recon- struction error of 32%. Much of this was probably because we kept ourtopKhyperparameter constant (atk= 0.1) throughout all our our models for consistency. With only 5 subnetworks, this means that each sample’s reconstruction 7 Figure 6: The subnetworks L3D successfully decomposes the TMCS model into subnetworks computing the contributions of each input feature to the output vector. 0.00.51.01.52.02.53.0 a 0.00 0.02 0.04 0.06 a r 2 = 0.92 Subnetwork 0 1 2 3 4 5 6 7 8 9 Figure 7: We use the L3D subnetworks to derive the values ofAand compare them to the to true coefficients used to train TMCS. We see that they have a very high correlation. will use<1subnetwork on average, limiting the minimum reconstruction error the network can achieve. We also experimented with holding rank constant (we dropped to rank 1 for this) and decomposed the model into different numbers of subnetworks (3, 5, 10, and 15 sub- networks). In our 3-subnetwork decomposition, L3D still learned subnetworks corresponding to single input features, but can of could only represent 3 out of the 5 inputs. As we added more subnetworks, L3D was able to successfully learn more expressive decompositions of the model that reduced reconstruction error (Figure S7). Each decompo- sition continued to learn subnetworks specific to a single input/output index, with the larger decompositions resulting in a few more dead subnetworks as well (Figure S6). 3.4.3. INTERVENTION Intervening on these circuits helps us understand how much local loss landscape is representative of global loss land- scape, particularly when it comes to inactive subnetworks remaining inactive as we move through parameter space. If local loss landscape is truly representative of global loss landscape in this way, then intervening on on a single sub- network should result in only the set of samples that relies on the subnetwork, even if we move very far in that direc- tion. Figure 10 shows our results for these interventions on theX7→X 2 model. Even in this more complex toy model, local loss landscape is a relatively good approximation of the global loss landscape. We can move our model param- eters in a direction of interest and have a large impact on the predictions of the relevant inputs and a minor impact on others. If we perturbed far enough (Figure S4), we did begin to see effects on the predictions of other samples, but the ratio of change in predictions to the relevant samples to those of the irrelevant samples remains very high. Figure 10 shows changes in predictions as we move in a single direction in parameter space. We also wanted to understand how subnetworks might interact with each other as we move through parameter space. In Figure S5 we perturbed multiple subnetworks at once, and measured the new predictions. For the most part, the subnetworks had little inference with each other: the relevant output values for each subnetwork moved relatively independently of each other. 3.5. Real world models Finally, to show the promise of this method to pull out relevant features from real world models, we used L3D to decompose blocks of a language model and computer vision model. These results are primarily qualitative, and were run with minimal compute and little hyperparameter tuning. Our choices for model block, number of subnetworks, and subnetwork rank were relatively arbitrary. These models do not have well-characterized subnetworks the same way our toy models do. To briefly analyze the func- tion of the subnetworks we identified, we looked at the top samples that each subnetwork is relevant to. We computed this metric as described in Section 2.5. We also computed 8 P act = 0.13P act = 0.15P act = 0.00P act = 0.00P act = 0.15P act = 0.16P act = 0.13P act = 0.12 Figure 8: Parameter representations learned by L3D for the high rank circuit decomposition task. Each subnetwork corresponds to a correlated group of feature. The third and fourth subnetworks are “dead” subnetworks that did not make it into the topK selection at all during the final epoch. Figure 9: Subnetworks learned by L3D for theX7→X 2 model. Each subnetwork is relevant to a single input/output index. 1 0 1 2 3 4 v 0 v 1 v 2 v 3 =0.3 v 4 101 1 0 1 2 3 4 101101101101 = 0.3 true X 0 X 1 X 2 X 3 X 4 X 2 i X i Figure 10: The effect of intervening on each subnetwork in theX7→X 2 model. We generated 1000 inputs from the TMS input distribution, intervened on each subnetwork with magnitudeδand measured the change in outputs for each sample. Only the outputs that involve each subnetwork effectively changed. 9 the most affected logits, for each of the top samples (x i ) for each subnetwork (v i ): argmax[abs(∇ δ f(x i ,W 0 +δv j )| δ=0 )](10) Because these models output probabilities, we used KL- divergence as our divergence metric when performing L3D for these models. 3.5.1. LANGUAGEMODEL We decomposed attention block 7 of the tiny-stories-8M model into 100 subnetworks. We used ranks that are ap- proximately 1/10 the original dimensions of the network (see Section B.4), and once again usek=.1. Although L3D can decompose any number of parameter blocks, or all parameters in a model into subnetworks, we limited L3D to a single block to keep memory and compute time low. We chose an attention block because this has been a challeng- ing component of a transformer for SDL to extract features from (Sharkey et al., 2025). We chose a middle layer of the model so that we identify subnetworks that are neither so high-level that they perfectly line up with next-token predic- tion, and not so low-level that they perfectly line up with token id. Table 2 shows the top samples of 10 cherry-picked circuits, and Table C.0.1 shows the top samples for all of the circuits. Although L3D had a relatively high reconstruction error at 40% (potentially due to only using 100 subnetworks), sub- networks seemed relatively interpretable. Even in the full set of circuits, most corresponded to a human-interpretable computation, such as detecting word pairs and phrases, cer- tain parts of grammar, subjects from previous parts of a sentence. We leave it as an exercise to the reader to annotate and interpret each circuit. 3.5.2. COMPUTERVISIONMODEL We decomposed convolutional block 4 of the mobilenet- v3-small model. Once again we choose a middle layer such that the subnetworks we identify are neither involved in low-level computations that would likely require addi- tional pixel attribution methods to interpret, or so high level they perfectly line up with classification. We used similar hyperparameters as in the transformer decomposi- tion, as described in B.4. Once again, we computed the top most affected samples for each circuit. We show the samples for 10 cherry-picked circuits in Figure 11 and for all 100 circuits in Figure C.0.2. Some types of computa- tions include recognition of certain animal faces, colors, backgrounds, and objects. Interestingly, although L3D’s decomposition of mobilenet-v3-small had a lower recon- struction error (23%), many of the subnetworks initially seem somewhat less human-interpretable. We suspect doing pixel attribution may help resolve some of the subnetwork computations as subnetworks might be picking out specific shapes and forms that are not obvious from just viewing the subnetworks most relevant samples at a high level. 10 Id (P act )Input TextTop Logits 0 (0.072) be more careful when eating spicy food. From thatday, day, Monday, side, night too because she helped the bird. From thatday, side, umm, ts, Balls she should have been more careful. From thatday, day, side, cers, ts tummy hurt. From thatday, side, Balls, acas, ters . From thatday, side, acas, cers, Balls 5 (0.107) together.Once upona, an, SEC, irled, clip best friends.Once upona, an, SEC, clip, irled , so they stay colorful and clean.”Once upona, an, orse, ship, ream you for being so persistent, daddy.”Once upona, an, ud, orse, SEC became good friends.Once upona, an, clip, SEC, irled 16 (0.028) it first!” Sara says. ”We want to see the treasure!”Ben, Tom, She, she, Tim . They are not ours to take. They are the sea’s to give.”They, Tom, Ben, Mom, Tim race!” Ben said. ”I bet I can go faster than you!”He, Lily, Mia, , he is not good to touch. Mom said some mushrooms are bad.”But, Mom, They, Ben, Lily chicken too. They are all good for you.”They, Mom, , The, Lily 18 (0.060) . It was your treasure.” Ben shook hishead, izing, Warning, iated, alking . Lily and Ben look at eachother, enlarged, OUT, pping, heit at the shell. They looked at their mom. They looked at eachother, wait, pace, lower, bribe clumsy, Sam,” Tom said, shaking hishead, neck, chin, heads, eyebrows chicken too. They are all good for you.” Tom shook hishead, Warning, FUN, izing, Save 21 (0.094) dad were hurt too. They went to thehospital, doctor, nurse, car, pool They hide the letter under thecouch, bed, sofa, table, slide They could play on theswings, beach, subway, climbers, Safari the old lady talked on thephone, telephone, cellphone, plaza, cafeteria to see who could get the best score. Tim threw theball, balls, basketball, trash, seeds 30 (0.048) She did not see her., and, feet, hand, Mom in the bathtub. She did not hear herMom, voice, mother, big, brother She said to her,, daughter, little, friend, Mom outside. Lily told hermom, ,, grandma, Mom, that night. One day, she told herfriend, friends, Mom, parents, mother 59 (0.023) to sleep.” Tom gave back the jewelry and said, ”Thankyou, background, ptions, mats, react Lily nodded and said, ”Thankyou, opes, ptions, mats, speakers , ”Thankyou, ptions, background, technique, bolts It looked happy. ”Thankyou, ptions, opes, bolts, zel Ben smiled and said, ”Thankyou, ptions, opes, background, bolts 71 (0.080) angry. Lily andBen, Tom, Jill, Mint, Fay ,” Tom said. Lily andTom, itt, est, hy, ippers It had a cut on its leg. Lily andBen, Tom, Mint, Flor, Shawn Anna andBen, iner, ability, astical, sub Lily andBen, Tom, Jack, Mark, Peter 76 (0.411) They like to play with their toys and booksin, and, ,, together, ,” day, Timmy went to play with his friends in the park,, and, with, again, for . Max loved to play with his friends at the park,, every, and, because, with are friends. They like to play in the parkwith, and, every, near, , had a big toy that she really wantedto, ,, and, !, but 86 (0.110) proud of herself for helping her furry friend.Once upon a timethere, at, in, later, it listen to her mom and always be safe.Once upon a timethere, in, at, it, they under her plate or give them to the dog. One dayshe, the, when, they, her friends. They played together every day. One daythe, it, they, Tim, Tom importance of sharing and being kind to his friends.Once upon a time there, at, in, later, with Continued on next page 11 Id (P act )Input TextTop Logits Table 2: For 10 of our favorite subnetworks, we computed the top most affected tokens, in terms of their KL-divergence compared to several reference outputs on the next-token prediction task. For each of the texts, the last token is the token that was found to be the most affected for each subnetwork. For each top token, we also computed the logits with the highest absolute gradients with respect to the subnetworks.. 12 The decomposition for mobilenet-v3-small also had much higher numbers of dead circuits (40%). We suspect adding an auxiliary loss term as in (Gao et al., 2024) might help alleviate this issue as well as improve reconstruction loss further. 4. Discussion L3D is one of the earliest parameter-based decomposition methods. For this reason, we have focused our work on demonstrating the fundamentals of L3D on toy models, and showcasing its promise with more complex models. Here we discuss what we believe are simple improvements to L3D that could enhance its performance and real-world use cases to which to extend L3D. Finally, we discuss unresolved challenges and limitations of L3D. 4.1. Simple Improvements In this work, we did not focus on optimizing L3D, and we chose nearly identical hyperparameters for all of our decompositions. Hyperparameter Choice: For all of our toy model decom- positions, we always chose ourtopKhyperparameter as k= 0.1, even when it was clear that certain toy models should have larger numbers of subnetworks activated per sample than others (For example, theX→X 2 model with 5 inputs and 5 outputs, decomposed into 5 networks, should probably havek≥0.2). Too low ofkchoice is likely re- sponsible for the high reconstruction loss of some of our models. Similarly, we chose the ranks of the subnetworks somewhat arbitrarily. Some preliminary research aims to understand the relationship between rank, compressibility, and interference of subnetworks (H ̈ anni et al., 2024; Bush- naq & Mendel, 2024), and a better understanding of this relationship could help us choose better hyperparameters for L3D. Scaling up: Naturally, the most exciting applications of L3D are with real-world models. While we briefly shared some results on larger models in order to demonstrate L3D’s promise, we by no means did a deep dive into the results. We think L3D can be scaled up to real-world models and can help answer open questions related to the amount of superposition present in different blocks of models, how circuits and features interact with each other and which parts of a model’s architecture are the most over- or under- parameterized. 4.2. Extensions There are a also some higher effort extensions to L3D that may give it more real-world relevance. Finetuning: Our intervention experiments showed promise that subnetworks of L3D could be perturbed in ways that only affect the predictions of relevant samples. As we de- scribe in Section 4.2, this could be taken one step further by finetuning a model on a specific set of parameter direc- tions. Using L3D networks, we could finetune a model on a specific set of parameter directions identified by L3D by freezing the current set of weights and learning an adapter consisting of linear combinations of the subnetworks of choice. This could also benchmark the intervention capabil- ities of L3D versus other mechanistic intervention strategies such as SDL-derived steering vectors. For example, we might use L3D to identify various subnetworks involved in sycophancy, refusal, and other undesired behaviors. After collecting curated data with the goal of finetuning away such behaviors, we could finetune L3D only in the direc- tion of the behavior-related subnetworks and test how well the model achieves our desired output compared to other intervention strategies. Identifying Specific Circuits with Contrastive Pairs: We developed this method as an unsupervised decomposition method, with goals comparable to those of SDL. However, the methods of L3D could be easily modified to use super- vised signals to identify specific circuits of interest. Rather than using gradients of divergence of random pairs, we could decompose gradients of divergence between curated pairs of samples that isolate a behavior of interest, or between outputs of different models on the same sample. 4.3. Challenges Although many of the improvements and extensions of L3D are highly addressable, we think there are some fundamental challenges with parameter-based decomposition methods that may not be easily resolved. Local Attribution: L3D’s algorithm hinges on the some- what surprising phenomenon that local gradient approxima- tions work reasonably well as attribution methods. They clearly work well in the toy models we used for L3D and at least demonstrate promise for the circuits we found in our real world models. However, do they work for all cir- cuits? In our work, we use a randomly selected sample to be our ”reference” output with which to compute di- vergence gradient. By using a randomly selected sample, rather than a single “reference output” such as the mean of the output distribution, we hope that the random noise in the reference sample will average out the effects of any non-convexity in the loss landscape. However, perhaps even in this setup there are parameter directions that are highly non-convex on which it will be difficult to perform local attribution. Quantifying different types of “dark matter” of parameter decomposition by analyzing reconstruction loss (Engels et al., 2025b) could better help us characterize these limitations. 13 Relationship to overparameterized models: Going one step further, we suspect that the reason local attribution methods work so well is because large models are probably overparameterized (Kawaguchi, 2016; Choromanska et al., 2015; Dauphin et al., 2014; Soudry & Hoffer, 2017). Larger models may have wider loss basins, or more degeneracies near their global minima (Keskar et al., 2017; Sagun et al., 2018), making local attribution methods less likely to break down as we move through parameter space. If in the future, a learning algorithm is developed that has fundamentally different limitations that stochastic gradient descent and its relatives, we might lose this property. Moreover, circuit ac- tivations might no longer be sparse. A new learning process might be able to compress subnetworks in such a way that subnetworks have very high levels of interference with each other - removing the degeneracy assumption that underlies L3D. Interpretation of a circuit: Finally, we should address the definition of “circuits”. It is still not well agreed upon what a “feature” is in relation to large networks, and the definition of what should constitute a circuit or subnetwork is even less clear. Is our definition of a circuit - sparsely active subnetworks that can move outputs within the origi- nal output distribution - too restrictive? If there is a circuit that is relevant to every output, a sort of “scaffolding” for more specific circuits - should it be included in the decom- position? If, after identifying the structure of subnetworks, we cannot interpret it beyond a description of its end re- sults, are circuits any more informative than the features they are computing? If parameter decomposition is a viable strategy for understanding and intervening with large net- works, these questions will be important for the mechanistic interpretability community to address. 14 killer whale dugong grey whale electric ray yawl 2 (0.01) killer whale grey whale dugong platypus black stork grey whale yawl killer whale dugong black grouse grey whale letter opener killer whale yawl dugong grey whale killer whale gazelle black grouse yawl grey whale fox squirrel killer whale lion patas killer whale grey whale red-breasted me whiskey jug wire-haired fox killer whale red-breasted me grey whale black grouse hartebeest killer whale red-breasted me badger grey whale skunk grey whale black grouse red-breasted me killer whale sea lion folding chair plate rack cradle rocking chair studio couch 13 (0.02) folding chair rocking chair throne cradle pedestal folding chair rocking chair barber chair cradle jinrikisha folding chair pedestal shoji cradle dining table folding chair rocking chair plate rack cradle muzzle folding chair pedestal barrel guillotine dining table folding chair rocking chair cradle pedestal shoji folding chair dining table pedestal rocking chair cradle folding chair punching bag maraca tripod thresher folding chair rocking chair dining table cradle pedestal dugong killer whale knee pad whiskey jug albatross 35 (0.03) dugong grey whale velvet electric ray ice bear grey whale dugong hammerhead killer whale electric ray grey whale dugong killer whale platypus rock beauty dugong tick eggnog grey whale whiskey jug dugong nipple screen sunscreen lotion dugong killer whale grey whale ice bear albatross grey whale killer whale platypus electric ray ice bear dugong geyser platypus grey whale rock beauty grey whale hammerhead dugong killer whale screen tiger beetle rock python sidewinder green mamba ground beetle 40 (0.19) passenger car moving van mobile home worm fence milk can Granny Smith spaghetti squas lemon jackfruit acorn platypus vine snake axolotl black stork dugong rock beauty spaghetti squas yellow lady's s daisy zucchini platypus mink gazelle fox squirrel otter rock beauty vine snake green mamba tree frog African chamele komondor Dandie Dinmont Sussex spaniel wood rabbit fox squirrel goldfish axolotl rock beauty yellow lady's s strawberry vine snake letter opener rock beauty African chamele green mamba rock beauty spaghetti squas acorn squash chain saw barn spider 43 (0.03) rock beauty lemon goldfish spaghetti squas jackfruit rock beauty frilled lizard platypus squirrel monkey gong rock beauty jellyfish dugong space shuttle corn tiger beetle rock python sidewinder whiptail green mamba rock beauty axolotl goldfish badger tench rock beauty dugong vine snake axolotl yellow lady's s rock beauty yellow lady's s goldfish ocarina axolotl rock beauty dugong axolotl ocarina isopod rock beauty nipple axolotl isopod banded gecko cheetah lynx leopard jaguar wood rabbit 48 (0.11) tiger patas fox squirrel red fox cougar tiger jaguar dhole English foxhoun patas leopard cheetah snow leopard lynx lion tiger lion red wolf red fox lynx tiger lion dhole red wolf jaguar lion chow cougar kit fox dhole jaguar leopard lion snow leopard lynx leopard fox squirrel lynx cheetah jaguar lion fox squirrel dingo kit fox basenji ladybug maraca leaf beetle pool table croquet ball 56 (0.08) goldfish earthstar pinwheel strawberry lesser panda goldfish axolotl rock beauty strawberry yellow lady's s goldfish rock beauty axolotl platypus black grouse face powder ping-pong ball croquet ball maraca knee pad lion fox squirrel dingo Pomeranian wire-haired fox face powder ping-pong ball maraca lemon sunscreen goldfish maraca ocarina rock beauty hen-of-the-wood face powder ping-pong ball maraca digital clock corn lion chow dhole redbone Sussex spaniel tick barn spider isopod long-horned bee ant 70 (0.02) tick ground beetle cockroach ant barn spider ground beetle tiger beetle ant long-horned bee barn spider tick ant ground beetle barn spider cockroach barn spider ant tick frilled lizard chain saw tick ant barn spider isopod long-horned bee tick barn spider chain saw black widow harvestman cockroach ant ground beetle isopod long-horned bee tick barn spider ground beetle scorpion ant ant ground beetle tick cockroach barn spider tiger beetle whiptail rock python sidewinder green mamba 94 (0.22) tick barn spider ant long-horned bee isopod tiger beetle barn spider agama ant tick nematode whistle nipple hook safety pin rock python sidewinder horned viper fox squirrel chain maraca pick tick hair slide safety pin nematode bolo tie ocarina ringneck snake sea snake thunder snake sidewinder nematode horned viper Indian cobra nematode nipple dugong Petri dish oxygen mask lion fox squirrel kit fox dingo Persian cat sea lion polecat dugong black-footed fe otterhound 98 (0.02) colobus cloak prison guillotine space shuttle Japanese spanie neck brace colobus Dandie Dinmont patas ping-pong ball face powder golf ball knee pad spotlight thresher space shuttle amphibian ant airliner Arctic fox kuvasz Samoyed white wolf Great Pyrenees siamang marmoset Sussex spaniel colobus clumber badger indri African elephan colobus tusker ping-pong ball face powder digital clock croquet ball maraca colobus siamang chimpanzee gibbon orangutan Figure 11: For 10 of our favorite subnetworks in the mobilenet-v3-small decomposition, we computed the top most affected samples (images). For each of those samples, we computed which logits had the highest gradient with respect to the subnetwork direction. 15 5. Acknowledgments Thank you to Daniel Filan and Dan Braun for additional comments and feedback. This work was funded by Open Philanthropy and the Machine Learning Alignment and Theory Scholars program (MATS). 6. Code Availability Code for this project can be found at https://github.com/briannachrisman/eigenestimation. 16 References Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety – a review, 2024. URLhttps://arxiv.org/ abs/2404.14082. Braun, D., Bushnaq, L., Heimersheim, S., Mendel, J., and Sharkey, L. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition. 2025. URLhttps://arxiv.org/ abs/2501.14926. Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Tamkin, A., and Carter, S. Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2023. Accessed: 2025-02-17. Bushnaq,L. and Mendel,J.Circuits in superposition:Compressing many small neural cir- cuitsintoone,2024.URLhttps://w.lesswrong.com/posts/roE7SHjFWEoMcGZKd/ circuits-in-superposition-compressing-many-small-neural. Accessed: 2024-03-13. Bushnaq, L., Mendel, J., Heimersheim, S., Braun, D., Goldowsky-Dill, N., H ̈ anni, K., Wu, C., and Hobbhahn, M. Using degeneracy in the loss landscape for mechanistic interpretability, 2024. URLhttps://arxiv.org/abs/2405. 10927. Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders, 2024. URLhttps://arxiv.org/abs/ 2412.06410. Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, p. 192–204. PMLR, 2015. Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023. URLhttps://arxiv.org/abs/2309.08600. Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.Advances in neural information processing systems, 27, 2014. Davies, X., Langosco, L., and Krueger, D. Unifying grokking and double descent, 2023. URLhttps://arxiv.org/ abs/2303.06173. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy models of superposition, 2022. URLhttps://arxiv.org/abs/2209.10652. Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimensionally linear, 2025a. URLhttps://arxiv.org/abs/2405.14860. Engels, J., Riggs, L., and Tegmark, M. Decomposing the dark matter of sparse autoencoders, 2025b. URLhttps: //arxiv.org/abs/2410.14670. Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders, 2024. URLhttps://arxiv.org/abs/2406.04093. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. Hoogland, J., Wang, G., Farrugia-Roberts, M., Carroll, L., Wei, S., and Murfet, D. Loss landscape degeneracy drives stagewise development in transformers, 2025. URLhttps://arxiv.org/abs/2402.02364. H ̈ anni, K., Mendel, J., Vaintrob, D., and Chan, L. Mathematical models of computation in superposition, 2024. URL https://arxiv.org/abs/2408.05451. Kawaguchi, K. Deep learning without poor local minima.Advances in neural information processing systems, 29, 2016. 17 Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima, 2017. URLhttps://arxiv.org/abs/1609.04836. Kram ́ ar, J., Lieberum, T., Shah, R., and Nanda, N. Atp*: An efficient and scalable method for localizing llm behaviour to components, 2024. URLhttps://arxiv.org/abs/2403.00745. Lau, E., Furman, Z., Wang, G., Murfet, D., and Wei, S. The local learning coefficient: A singularity-aware complexity measure, 2024. URLhttps://arxiv.org/abs/2308.12108. Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C. Sparse crosscoders for cross-layer features and model diffing.Transformer Circuits Thread, 2024. Matena, M. and Raffel, C. Npeff: Non-negative per-example fisher factorization, 2023. URLhttps://arxiv.org/ abs/2310.04649. Merullo, J., Eickhoff, C., and Pavlick, E. Talking heads: Understanding inter-layer communication in transformer language models, 2025. URLhttps://arxiv.org/abs/2406.09519. Nanda, N. Attribution patching: Activation patching at industrial scale.URL: https://w. neelnanda. io/mechanistic- interpretability/attribution-patching, 2023. Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. InInternational conference on machine learning, p. 1310–1318. Pmlr, 2013. Sagun, L., Evci, U., Guney, V. U., Dauphin, Y., and Bottou, L. Empirical analysis of the hessian of over-parametrized neural networks, 2018. URLhttps://arxiv.org/abs/1706.04454. Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill, N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garriga-Alonso, A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg, M., Schoots, N., Miller, J., Michaud, E. J., Casper, S., Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland, J., Murfet, D., and McGrath, T. Open problems in mechanistic interpretability, 2025. URLhttps: //arxiv.org/abs/2501.16496. Soudry, D. and Hoffer, E. Exponentially vanishing sub-optimal local minima in multilayer neural networks, 2017. URL https://arxiv.org/abs/1702.05777. Subramani, N., Suresh, N., and Peters, M. E. Extracting latent steering vectors from pretrained language models, 2022. URLhttps://arxiv.org/abs/2205.05124. Syed, A., Rager, C., and Conmy, A. Attribution patching outperforms automated circuit discovery.arXiv preprint arXiv:2310.10348, 2023. Tucker, L. R. Some mathematical notes on three-mode factor analysis.Psychometrika, 31(3):279–311, 1966. Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering, 2024. URLhttps://arxiv.org/abs/2308.10248. Wang, G., Farrugia-Roberts, M., Hoogland, J., Carroll, L., Wei, S., and Murfet, D. Loss landscape geometry reveals stagewise development of transformers. InHigh-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. Watanabe, S. Algebraic information geometry for learning machines with singularities.Advances in neural information processing systems, 13, 2000. Watanabe, S. Algebraic geometry of singular learning machines and symmetry of generalization and training errors. Neurocomputing, 67:198–213, 2005. Watanabe, S. Almost all learning machines are singular. In2007 IEEE Symposium on Foundations of Computational Intelligence, p. 383–388. IEEE, 2007. Wei, S., Murfet, D., Gong, M., Li, H., Gell-Redman, J., and Quella, T. Deep learning is singular, and that’s good.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10473–10486, 2022. 18 A. Definitions A.0.1. DIMENSIONS n s : The number of samples in a batch of inputs n i : The dimensions of a single input vector to a model n o : The dimensions of a single output vector from a model n w : The number of parameters values in a model. n v : The number of subnetworks or parameter directions chosen to decompose a model. A.0.2. MODELSYNTAX X∈R n s ×n i ,x∈R n i : Batch and individual input vectors to a model. y r ∈R n o : A reference output vector W∈R n w ,w∈R: The set of and individual parameter values of a model f:R n s ×n f 7→R n s ×n o : A model mapping a set of input vectors to a set of output vectors. f(X,W): The output of modelfwith parameter valuesWon inputX. f(X,W 0 ) orf(X): The output of modelfwith fixed parameter valuesW 0 .W 0 is the set of learned parameter values from model training. D: Divergence metric between two vectors. Typical divergence metrics are mean-squared error for regression-type outputs, and KL-divergence for probability-type outputs. A.0.3. DECOMPOSITIONSYNTAX V(orV out )∈R n v ×n w ,v(orv out )∈R n w : The set of or individual parameter directions that are used to decompose a model.V out can be used to transform parameter directions in the subnetwork vector space back into the original parameter space of the model. V in ∈R n w ×n v : Transforms the original parameter space of the model into the subnetwork vector space. r: The rank of each component of the decomposition vectors corresponding to tensors in the original model. A.0.4. TRAINING L: The L2 reconstruction loss used to optimizeV in andV out . A.0.5. MEASURING ANDINTERVENTION I(x i ,y j ,v k ) : The impact of subnetworkv k on the divergence between sample outputsf(x i )andy j , or averaged across manyy j reference outputs. δ: A scalar value to moveWin a specific direction. B. Additional Methods B.1. Low-Rank Tensor Representation We use low-rank representations of ourV in andV out , and correspondingly learn low-rank circuits. WhileWis a vector containing all model parameters, these parameters are typically organized into tensors,W=w i i . If our parameters are structured as tensorsW=w i i , each subnetwork or parameter component can be expressed as V in i =v in i i andV out i =v out i i , where each component corresponds to a specific tensor in the original model parameters. To ensure that each of these tensors remains low-rank, we employ theTucker decomposition(Tucker, 1966) (a 19 method for factorizing high-dimensional tensors into a core tensor and a set of factor matrices): The Tucker decomposition decomposes a tensor⊑∈R I 1 ×I 2 ×·×I N into a core tensorGand a set of factor matricesU (n) : ⊑≈G× 1 U (1) × 2 U (2) ·× N U (N) (11) where: -G ∈R R 1 ×R 2 ×·×R N is the core tensor capturing interactions between modes. -U (n) ∈R I n ×R n are the factor matrices, representing a low-rank basis along each mode. -× n denotes the n-mode product of a tensor with a matrix. B.2. Toy Model Training For all of our toy models (except theX7→X 2 model), we generate uniformly random inputs between 0 and 1. For X7→X 2 , we generate uniformly random inputs between -1 and 1. For all toy model data, we use a sparsity value of sparsity=.05. We generate 10000 datapoints and train for 1000 epochs with a batch size of 32. We use an AdamW optimizer with a learning rate of 0.001. B.3. L3D Toy Model Training To train L3D for the toy models, we use the same training distributions as in each toy models. Although optimal hyperparameter values probably depend on the model size, and the rank and number of parameter tensors, we use the same hyperparameters for all of our models. We generate only 1000 datapoints, with a batch size of 32, and train for 1000 epochs. We use an AdamW optimizer with a learning rate of 0.01, and a learning decay rate of .8 every 100 steps. We always use a topK hyperparameter ofk= 0.1. We include all of the model’s parameter tensors, including biases, in the decomposition. B.4. L3D Real World Model Training To decompose tiny-stories-8M, we train L3D using 10000 16-token texts randomly sampled from the tiny-stories dataset. For mobilenet-v3-small, we train L3D using 10000 images samples from CIFAR-100. For both our models, we train for 100 epochs with a learning rate of .005 and a decay rate of .8 every 10 epochs. We computed the top samples using 10000 additional randomly generated images/texts from the same distribution as training, and averaging the contribution of each subnetwork to each sample across 10 reference outputs. For both models, we decompose all parameters involved in our block of interest. We decompose those tensors into tensors 1/10 of each of their original dimensions. For tiny-stories-8M this looks like: transformer.h.4.attn.attention.k_proj.weight: [25, 25] transformer.h.4.attn.attention.v_proj.weight: [25, 25] transformer.h.4.attn.attention.q_proj.weight: [25, 25] transformer.h.4.attn.attention.out_proj.weight: [25, 25] transformer.h.4.attn.attention.out_proj.bias: [25] For mobilenet-v3-small this looks like: features.7.block.0.0.weight: [12, 4, 1, 1] features.7.block.0.1.weight: [12] features.7.block.0.1.bias: [12] features.7.block.1.0.weight: [12, 1, 5, 5] features.7.block.1.1.weight: [12] features.7.block.1.1.bias: [12] features.7.block.2.fc1.weight: [3, 12, 1, 1] features.7.block.2.fc1.bias: [3] features.7.block.2.fc2.weight: [12, 3, 1, 1] features.7.block.2.fc2.bias: [12] features.7.block.3.0.weight: [4, 12, 1, 1] features.7.block.3.1.weight: [4] features.7.block.3.1.bias: [4] 20 C. Supplemental Figures x 1 x 2 x 3 A 1 X A 2 X A 3 X Figure S1: The full architecture of high rank circuit toy model (model C). 21 02004006008001000 Epoch 0.2 0.4 0.6 0.8 1.0 Loss Rank 1 2 3 4 5 6 7 8 Figure S2: Reconstruction Loss vs Rank of the multi-feature/higher rank circuits. 22 Figure S3: Decomposing the toy model of high rank circuits into different numbers of subnetworks. P act = 0.17P act = 0.18P act = 0.15P act = 0.00P act = 0.19P act = 0.07P act = 0.00P act = 0.12P act = 0.00P act = 0.16 Rank 1 (a) Rank-1 Networks P act = 0.13P act = 0.18P act = 0.18P act = 0.00P act = 0.20P act = 0.00P act = 0.16P act = 0.05P act = 0.15P act = 0.00 Rank 2 (b) Rank-2 Networks P act = 0.19P act = 0.00P act = 0.09P act = 0.00P act = 0.00P act = 0.17P act = 0.15P act = 0.15P act = 0.10P act = 0.18 Rank 3 (c) Rank-3 Networks P act = 0.18P act = 0.00P act = 0.09P act = 0.18P act = 0.16P act = 0.00P act = 0.15P act = 0.00P act = 0.16P act = 0.12 Rank 4 (d) Rank-4 Networks P act = 0.00P act = 0.07P act = 0.00P act = 0.13P act = 0.16P act = 0.17P act = 0.19P act = 0.14P act = 0.00P act = 0.18 Rank 5 (e) Rank-5 Networks 23 0 2 4 v 0 v 1 v 2 v 3 =1 v 4 0 2 4 =0.5 0 2 4 0 2 4 0 2 4 101 0 2 4 101101101101 true X 0 X 1 X 2 X 3 X 4 X 2 i X i Figure S4: Effects of intervening on each of the subnetworks of theX7→X 2 model. 24 0 2 4 1 =1 1 =0.5 1 =0.1 1 = 0.1 1 = 0.5 0 =1 1 = 1 0 2 4 0 =0.5 0 2 4 0 =0.1 0 2 4 0 = 0.1 0 2 4 0 = 0.5 101 0 2 4 101101101101101 0 = 1 true X 0 X 1 X 2 X 3 X 4 X 2 i X i Figure S5: Effects of intervening with multiple subnetworks (v 0 on the x-axis,v 1 on the y-axis) at once. 25 Figure S6: Decomposing theX7→X 2 model into different numbers of subnetworks P act : 0.06P act : 0.19P act : 0.09 (a) 3 rank-1 Networks P act : 0.10P act : 0.07P act : 0.05P act : 0.14P act : 0.17 (b) 5 rank-1 Networks P act : 0.11P act : 0.00P act : 0.18P act : 0.06P act : 0.08P act : 0.08P act : 0.15P act : 0.16P act : 0.12P act : 0.09 (c) 10 rank-1 Networks P act : 0.06P act : 0.00P act : 0.24P act : 0.12P act : 0.13P act : 0.00P act : 0.20P act : 0.15P act : 0.19P act : 0.00P act : 0.12P act : 0.00P act : 0.13P act : 0.08P act : 0.12 (d) 15 rank-1 Networks 02004006008001000 Epoch 0.0 0.2 0.4 0.6 0.8 1.0 Loss 3 features 5 features 10 features 15 features Figure S7: Training loss vs. number of subnetworks for theX7→X 2 model 26 C.0.1.TINY-STORIES-8M FULLDECOMPOSITION The full set of subnetworks (withP act >0), most affected samples, and their most affected logits for the tiny-stories-8M decomposition. We list the subnetwork ID andP act , show the most affected samples, and for each sample show the logits with the highest gradient with respect to the subnetwork. Id (P act )Input TextTop Logits 0 (0.072) be more careful when eating spicy food. From thatday, day, Monday, side, night too because she helped the bird. From thatday, side, umm, ts, Balls she should have been more careful. From thatday, day, side, cers, ts tummy hurt. From thatday, side, Balls, acas, ters . From thatday, side, acas, cers, Balls 1 (0.114) me.” Lily smiles and claps herhands, voices, mouths, faces, oves sit next to eachother, other, their, our, ours time, there was a little boy named Timmy.Tim, One, They, It, There on a camping trip. Timmy was very excited! Asthey, their, them, They, theirs saw a cat on a tree. He wanted to befriends, their, animals, our, together 2 (0.036) They are sad. They want to see the treasure.¡—endoftext—¿, They, ”, The, But car, a flower and a star.¡—endoftext—¿, ”, They, , The did not know why.¡—endoftext—¿, The, ”, , They and loud. They did not hear their mom calling them.¡—endoftext—¿, ”, , They, The basket and the knife behind.¡—endoftext—¿, ”, , They, The 3 (0.063) fun day at the park.Once upon a time, there was a boy namedTim, Jack, James, Lily, Alex .”Once upon a time, there was a boy namedTim, Jack, Lily, Ben, James They pretended to be kings andqueens, she, princes, her, She in the future.Once upon a time, there was a big elephant namedEllie, Ell, Daisy, Grace, Lily Once upon a time, there was a boy named Tim.He, Every, She, They, Sue 4 (0.111) , doctor. Thank you, mom. Thank you,, Star, ider, printers, Auto . It looked happy and friendly. ”See, Lily and Ben,, ?”, !”, ?!”, !, panic and cry. Her mom knew itand, ., ,, would, wasn , Mom. Please, canwe, I, pie, soldiers, Hood ’t worry, Timmy., ,”, !, .”, !” 5 (0.107) together.Once upona, an, SEC, irled, clip best friends.Once upona, an, SEC, clip, irled , so they stay colorful and clean.”Once upona, an, orse, ship, ream you for being so persistent, daddy.”Once upona, an, ud, orse, SEC became good friends.Once upona, an, clip, SEC, irled 6 (0.229) to play. They went on the swings and the slide. Lily had somuch, killing, backdrop, doorstep, ocus way there, he got lost. He couldn’t find hisway, results, r, umbers, For noises. Tom had a small car that could go fast and beep. Lilywanted, liked, was, loved, and teddy bear and had a lot offun, adventure, lots, daring, thrilling jump in. They had somuch, satisfying, wr, Ah, izz 7 (0.085) They made a new friend. They were very happy., elegance, effective, ulent, val your dragon.” They all laughed and hugged. They were happy and glad ., unky, utch, aved, cog ice cream. It was cold and sweet. They were very happy., iot, error, angled, uld They are sadand, stack, Figure, rast, wered and milk. Lily was very happyand, ., redible, ulent, arise 8 (0.067) said to the plant, ”We are sorry, plant. We didnot, t, opposite, roll, pe mister,” Tom says. ”We didnot, t, ts, still, steadily worm,” she said. ”I’m sorry, mushroom. I didnot, t, roll, ts, sly marks on the wall too, but Mommy doesnot, .), trade, l, fir The bee was on the apple. It was angry and scared. It didnot, roll, ves, dig, not Continued on next page 27 Id (P act )Input TextTop Logits 9 (0.177) ran to hide behind a tree. She peeked, apped, red, faced, Buddha it in the lock. They push and pull, but nothinghappened, comes, works, is, breaks Mommy will be angry. He says,”, ulating, ver, attered, atter you for the treat!” Spot barked, ked, red, apped, led . Lily petted, ged, led, aced, oled 10 (0.072) that day on., the, she, he, that that day on., the, she, he, that that day on., the, she, he, that a little girl named Lily. She loved to play in the park with her friends ., .), questions, app, ongs named Lily. She loved to play outside in the sun with her friends ., tricks, .’, questions, cycle 11 (0.062) at the shell. They looked at their mom. They looked at each other. They, bits, pper, circuits, uff floor. They are sorry. They do not want to make mom sad.They, rily, acks, spotlight, bits . Lily and Ben look at each other. They are scared.They, acks, over, bits, laz tall man comes to the tree. He has a hat and a coat.He, acks, ogged, itting, ung !” Anna does not listen to Ben. She thinks he is silly.She, ito, ails, ative, acquired 12 (0.063) you,” said Finny sadly. ”Don’t worry, Finny, ster, ur, Duck, armed named Sue. She had a big, red ball. ”Let, Ball, Hi, Can, Wow ”Ben, Ben, grab the stick!” she shouted. ”Give, You, The, That, This football. ”Wow, look at this football!” Ben says. ”It, We, It, it, Mine not listen to Mia. He wanted to win. He didnot, disagree, surrender, yoga, being 13 (0.031) you want some?”L, Anna, Tim, S, M . They hugged Mom and Dad.The, L, M, S, Then say sorry.L, S, She, Anna, He scared too.The, One, L, She, He One, L, The, When, As 14 (0.191) every day. The plant had greenplants, roots, needles, stems, and together in the leaves. The end.¡—endoftext—¿, , , A, From playing, he saw a big hole in thegarden, wall, fence, middle, backyard on walks and helped otherchildren, people, kids, creatures, young but there was none. The sun was getting hotter and the goat was getting thirst ., and, der, y, of 15 (0.178) . Ducky triump, pping, onto, over, inged hill very fast. Tim and Sue laughed and clapping, ap, aps, amb, ink ogged and played, having a lot of fun. As they jogging, olly, ogg, umbled, ogs They both pulled and tugon, ging, ., and, ed named Max., playing, coming, looking, walking 16 (0.028) it first!” Sara says. ”We want to see the treasure!”Ben, Tom, She, she, Tim . They are not ours to take. They are the sea’s to give.”They, Tom, Ben, Mom, Tim race!” Ben said. ”I bet I can go faster than you!”He, Lily, Mia, , he is not good to touch. Mom said some mushrooms are bad.”But, Mom, They, Ben, Lily chicken too. They are all good for you.”They, Mom, , The, Lily 17 (0.152) the park with their bikes. They liked to ride fast and make noises. They saw, heard, met, played, ate up the ball with its beak, ams, ck, umb, arrow You wasted a lot of food and drinks. You haveto, disturbed, wandered, shown, bumped , you can,” Lily and Tom said, nodding. ”But you haveto, disturbed, delayed, pulled, forced eat avocados, they were herbest, friends, new, very, special 18 (0.060) . It was your treasure.” Ben shook hishead, izing, Warning, iated, alking . Lily and Ben look at eachother, enlarged, OUT, pping, heit at the shell. They looked at their mom. They looked at eachother, wait, pace, lower, bribe clumsy, Sam,” Tom said, shaking hishead, neck, chin, heads, eyebrows Continued on next page 28 Id (P act )Input TextTop Logits chicken too. They are all good for you.” Tom shook hishead, Warning, FUN, izing, Save 19 (0.060) !” Anna doesnot, blush, Justin, Alan, Harry . Sara didnot, blush, word, ordering, waitress , you can,” Lily and Tom said, nodding. ”But you haveto, aker, ican, ator, eting want to play anymore. This is too difficult.” But Lily didnot, if, ake, girl, Should ask God to help you eat your soup.” Tom didnot, word, blush, asking, orman 20 (0.105) . It grew new leaves and flowers. Anna and Ben wereamazed, excited, sad, curious, not animals like lions and monkeys. It was so muchbigger, better, that, more, hard jump in. They had so muchenergy, to, that, stuff, time dolls. Lily wasa, not, playing, the, excited bird on a branch. The bird wasblue, sitting, singing, yellow, flying 21 (0.094) dad were hurt too. They went to thehospital, doctor, nurse, car, pool They hide the letter under thecouch, bed, sofa, table, slide They could play on theswings, beach, subway, climbers, Safari the old lady talked on thephone, telephone, cellphone, plaza, cafeteria to see who could get the best score. Tim threw theball, balls, basketball, trash, seeds 22 (0.090) a littlebit, bird, scared, while, too a littlebit, bird, scared, while, too , a littlebird, mouse, dog, bug, red a yummyfood, soup, and, ,, dinner a littlebit, bird, scared, while, too 23 (0.066) clumsy, Sam,” Tom said, shaking hishead, tail, ow, spine, ows loved to play with his toycar, animals, gun, boat, truck playing, he saw a big hole in thefence, wall, tree, garden, corner my. Timmy loved to play with his toycar, gun, hammer, je, boat grabbed her crayon, in, ane, ions, ip 24 (0.063) The end.Once upon a time, there was a little girl named Lily.She, He, Max, Emma, Tim upon a time, there was a little girl named Lily.She, He, Max, Tim, Tom .Once upon a time, there was a little girl named Lily.She, He, Max, Tim, Emma upon a time, there was a little girl named Lily.She, He, Max, Tim, Tom upon a time, there was a little girl named Lily.She, He, Max, Tim, Tom 25 (0.094) .Anna liked to examine things. She likedto, touching, explore, smoot, pile ! Thank you solong, !”, high, hard, fast to go home. His friend asked him what wasgoing, in, happening, inside, he his arm. His mom took him to thehospital, park, store, bathroom, nurse fun that she didn’twant, ., mind, see, notice 26 (0.199) to investigate and found shiny rocks that sparkly, ened, les, le, bled garden. He sneaked, wed, uned, amped, ound shell back. She tried to grab it from Tom’s hand. ”No, Mine, O, Me, Please Lily. She had a cup that she loved to drink juice from everyday, time, week, evening, time Emma heard her sister’s scream and asked, ”Is, Why, Are, Please, Who 27 (0.125) .” They ran backhome, and, ,, together, . ran to hide behind a tree. She peekedbehind, around, her, at, inside ily wanted to touch it anyway. She reachedfor, her, the, it, his . He picked itand, carefully, off, with, out became good friends.Once upon a time, a bird wanted to fly high and, ., ,, like, above 28 (0.065) found you!” It was her friend, Tim.He, ”, They, She, Tim on a camping trip. Timmy was very excited!He, He, was, to, They outside.He, She, , , The , but they were too messy.They, The, , Suddenly, Tim to climb on.He, She, The, , Continued on next page 29 Id (P act )Input TextTop Logits 29 (0.030) told her not to worry and that she would take care., for, and, about, when a big house with a lotto, us, ., er, more even higher!Once upon a time, there was a big, strong robot made ., out, up, from, - played in the garden and took careto, for, ., with, every to play and run all day. One day, Tim found a big bag., in, and, on, with 30 (0.048) She did not see her., and, feet, hand, Mom in the bathtub. She did not hear herMom, voice, mother, big, brother She said to her,, daughter, little, friend, Mom outside. Lily told hermom, ,, grandma, Mom, that night. One day, she told herfriend, friends, Mom, parents, mother 31 (0.062) . She smiles and says,”, attered, ayed, atter, appers Lily nodded and said,”, attered, cher, ayed, atter Mia hugged Ben and said,”, ayed, atter, attered, havoc . She gives each doll a cup and a plate. She says,”, attered, led, ayed, umbled happy to see Anna’s spoon. They say,”, attered, atter, ored, ico 32 (0.083) . Lily wanted to join in on the fun, but her mom toldthem, she, the, Lily, it earlier, but he still wanted to help her. He went over and helpedthe, his, Lily, pick, them very happy. Tim’s mom was proud ofTim, his, them, her, the Mom smiled and hugged them. She gavethe, her, Lily, their, back day, Lily’s mom askedthe, if, him, Lily, them 33 (0.044) be thoughtful and careful when helping others.Once uponthe, time, , first, finding , Monkey always kept his room tidy just like Ellie’s.Once uponthe, time, playing, Lily, . The end.Once uponthe, time, first, then, , and making more pictures together.Once uponthe, time, , first, an with his dad and ride his bike with gears on the clear path.Once upon the, time, first, to, 34 (0.040) Once, The, One, L, Tom Once, The, One, L, Tom Once, The, One, L, Tom Once, The, One, L, Tom Once, The, One, L, Tom 35 (0.070) spider was aboutto, beverages, rene, agons, Spears still sounded bad. He was aboutto, offerings, agons, rig, unky . He didn’t meanto, fullest, custom, destination, idol want to go to the police. They decideto, conclusions, erer, fascination, prod careful notto, iot, plaza, aned, continents 36 (0.091) -cream, and had lots of fun at the park. The end!, ,, of, .”, are gone. The end.”, is, !”, of, result . The end.”, !, was, of, result . The end.”, !, was, of, result the flag wave in the wind. The end!, was, of, .”, , 37 (0.070) !” Lily said. ”Yes, it is,” hermom, aining, ably, irs, irted my didn’t want to share his toys, so hismom, inges, aining, IN, irs fun that she didn’t want to leave. But hermom, aining, lier, iment, inges cereal for breakfast every day. One day, hermom, lier, irs, irting, piece After they finished playing, Timmy went home. Lily’smom, lier, aining, purposes, arl 38 (0.097) were packing, Timmy’s mom reminded him to bring his flash- light. She, They, He, But, say they did not open the box.¡—endoftext—¿, .”, But, They, . She loved to walk on the trail with her dog, Max.They, Max, , The, She back.¡—endoftext—¿, ,”, .”, They, too bear. They tell them that they have to wait for Christmas.¡—endoftext—¿, , ¡, ., They Continued on next page 30 Id (P act )Input TextTop Logits 39 (0.079) didn’tknow, want, like, understand, think didn’tknow, want, like, understand, think didn’tknow, want, like, understand, think sad and didn’tknow, want, understand, care, quit does notlike, know, want, hear, understand 40 (0.192) the rock! Lily was upset and scared. Shedidn, really, questioned, rew, Wow he was very sad. Lilywanted, asked, didn, told, said But we found them here,” Bensays, said, insisted, suggested, wiped , scary fox came into the garden. Bongodidn, was, felt, wanted, did had passed away. Lilywas, felt, didn, went, missed 41 (0.118) and reached for an apple. But she did notsee, wind, Wait, trips, trip !” Sara and Ben are scared. They do not knowwhat, where, moms, sure, shore untied! Timmy didn’tknow, hesitate, hate, doubt, ’ve sad and didn’tknow, knowing, wanting, being, noticing were stuck. Lily started to feel scared and silly. She didn’tknow, knowing, wanting, extra, being 42 (0.113) with his ball. One day,Tim, Benny, Max, Twe, Remy restless. As Timmy rode his bike,he, unison, aining, ainer, centers One day,she, Lily, Tim, Benny, Max wet. One day,Lily, she, Tim, Max, Benny Timmy. One day,Tim, Nem, Remy, Nut, T 43 (0.009) fun day at the park.Once upon a time, therewas, extingu, ixtures, manship, burden that might have something yummy inside.Once upon a time, there was, manship, Shadow, defense, yles a time, therewas, ixtures, accurate, yles, manship truck all day long.Once upon a time, therewas, ixtures, manship, yles, backdrop them disappear again.Once upon a time, therewas, manship, yles, tripod, ixtures 44 (0.085) dough. She put the cookies inthe, her, my, Becky, Mrs to play games with his friends inthe, Christ, elled, ussed, aming He loved to play with his ball inthe, The, Lyn, His, Ray play and run inthe, sect, Christ, oned, elled men and playing inthe, Lyn, Christ, Den, rod 45 (0.035) Once upon a time, there wasan, the, one, two, something Once upon a time, there wasan, the, one, two, something Once upon a time, there wasan, the, one, two, something they lived happily ever after.Once upon a time, there wasan, the, one, another, Lily .”Once upon a time, there wasan, the, Lily, one, something 46 (0.061) his shoe. Timmy was soexcited, proud, sad, surprised, embarrassed mom looked around and found it under the bed. Timmy was so excited, proud, surprised, glad, grateful my was soexcited, proud, sad, surprised, scared was and decided to permit him to play with his skull again. Spot was so excited, grateful, glad, proud, surprised thank you. Lily was soexcited, glad, proud, grateful, sad 47 (0.075) you want.” Tim said,”, ayer, est, ime, over Sue asked. Tim said,”, apper, attered, ime, appers nice. Tom said,”, attered, apper, ilt, iner faucet for the kitchen sink. Mia’s mom said,”, attered, appers, apper, umbled are you sad, Tom?” Tom replied,”, ”, anes, overs, ooters 48 (0.357) up the ball with its beak and bringshim, the, her, back, them to play with cars and balls and blocks. They go to thepark, same, zoo, library, beach It was yellow and black and very pretty. She ranaround, after, outside, and, inside saw a big dog runningaround, in, across, after, up watch where he was going and tripping, ump, umble, umbles, ey Continued on next page 31 Id (P act )Input TextTop Logits 49 (0.193) went to the park with her mom and saw her friends playing hide-and- and, go, tag, ider, pack her mommy said, trying to calm herdown, concentrate, responsibility, downstairs, concentration . Lily and Ben look at eachanother, one, thing, ., toy was too heavy and slow. The bunny got away and the allion, that, the, bunny, of on, they went for walks in the park together and became goodat, and, -, players, siblings 50 (0.082) . Tim saw his friend, a big dog,, ., and, with, called with a smile.Once upon a time, there was a little girlwho, ., called, and, with .Once upon a time, there was a graceful cat., who, called, and, with see the beautiful yellow sunrise.Once upon a time, there was a boy who, and, ., called, with .Once upon a time, there was a little girlwho, ., called, and, with 51 (0.051) my and daddy. One day, while swimming, Timmy, cases, astic, certainty, Noise at the campsite, Timmy, Christ, ISON, arten, Mood he accidentally bumped into the barrel and it started rolling. Tim my, itate, generations, judgments, Staff ’s legs got tired and they stopped to take a break. Timmy, generations, adversary, itate, Long my. Timmy, Forest, itate, oids, Christ 52 (0.078) basket and the knife behind. Dad didnot, warn, scare, poop, rier me.” Lily smiles and clucks, apped, ink, ags, ums to hurt you. Please forgive us.” The plant didnot, Not, prick, Woo, scare had a black cat named Mittens. Mittens was very soft and cuffy, agged, aged, led, owed She sees the letter. It is torn. She sigh., es, and, ing, again 53 (0.101) . Buzzy flew down and said, ”Hello, Hi, Thank, hello, Wow it to Ben. Ben kicks it back to Tom. They havefun, ritz, rer, Absolutely, ream band-aid on it. He gives Lily a sticker and a lily, icks, olly, icked, kin it was time to go home. Timmy went to bed thatnight, afternoon, game, chance, Friday shell back. She tried to grab it from Tom’s hand. ”Give, Hey, Go, O, Come 54 (0.114) at first, but he decided to try it. Nemo and Crabby, bles, iny, bly, as One day, she decided to examine the bathtub, robe, ro, tub, bath her room. She puts the teaspoon in Annaand, ., , ,, ’ me.” Lily smiles and clapped, ucks, ums, s, ink young boy named Tim found a dull, round rock. He picked itand, out, from, with, , 55 (0.048) found you!” It was her friend, Tim.He, ”, They, Tim, It to climb on.He, She, The, , and showed it to her dog.She, , It, The, They was light, so Tim could pull it easily.He, , The, Tim, They had touched the flower.She, He, , The, It 56 (0.085) dough. She put the cookies inthe, a, sacks, Lisa, Sue He saw her onthe, his, their, Wednesday, your back. Then he sees a duck. The duck is swimming ina, an, nature, another, rivers their toys intheir, the, different, another, Mia . One day, she saw a butterfly flying ina, the, her, an, nature 57 (0.077) , Lily wanted to try to lift a heavy frame all byhimself, itself, themselves, yourself, myself dog stopped barking and Timmy felt much better. He gotup, dry, bedroom, soak, mood Mom. They saw a big pond with many ducks and swam, an, immers, ucky, ishes and went outside to eat bythe, itself, his, its, her Max saw a big plane flying in the sky. Max barked excitedly, eyes, p, en, bly 58 (0.112) left and right. They loved marching together., The, One, , They very nice, Mom,” End said., ”, , The, M be careful with fragile things., One, The, ”, Max and they both had a great time chewing on it together. The moral, little, sun, two, next Continued on next page 32 Id (P act )Input TextTop Logits mom was proud of her for being kind and sharing., , The, From, But 59 (0.023) to sleep.” Tom gave back the jewelry and said, ”Thankyou, background, ptions, mats, react Lily nodded and said, ”Thankyou, opes, ptions, mats, speakers , ”Thankyou, ptions, background, technique, bolts It looked happy. ”Thankyou, ptions, opes, bolts, zel Ben smiled and said, ”Thankyou, ptions, opes, background, bolts 60 (0.053) corn move back andforth, appers, unfairly, apper, EST said she didn’t know. Lily looked everywhere for her cup,the, even, her, under, which ? I told you about the cable. You were not wise.I, Next, Now, Do, How to read before she went to bed. Mia looked at the booksheread, books, book, was, r treasure. He hit the ice harder andharder, faster, slower, easier, farther 61 (0.151) too because she helped the bird. From thatmoment, night, ,, time, morning sharing all of their toy tools. From thatmoment, time, ,, afternoon, night forgot about her knee. From thatmoment, night, morning, time, afternoon up on the fridge. From thatnight, moment, morning, ,, time and finally, they found the belt under Tom’s bed. Tom washappy, very, not, surprised, sad 62 (0.029) . She says, ”I, Thank, You, Don, Wow . He ate his celery. He was happy. He said, ”Thank, You, Wow, Pot, Work hugged Lily. ”I, Thank, It, You, Wow They hug mom. They say together. ”Thank, We, Can, I, You Mia hugged Ben and said, ”Thank, You, Don, Wow, Are 63 (0.057) had their wand and their bubbles. They didnot, ann, ales, pered, Lumin had to pick some onions for dinner. Sara didnot, aut, ographs, bags, outlets , cut the bread, and taste the cheese. But she didnot, Play, ooters, Net, bags and loud. They didnot, pered, cher, communities, angles fun. They didnot, orb, iour, cher, recounted 64 (0.179) very scared. She did not know what todo, eat, cook, wash, pack . ”Don’t worry, we’llfind, go, get, fix, clean Sam,” said Tim. ”Do you want toplay, go, race, slide, ride her mom if they couldgo, play, buy, have, make Tom’s faces. ”You two need tolearn, go, find, hurry, clean 65 (0.243) floor. They are sorry. They donot, Wr, vanished, ch, choke might fall in!” Ben didnot, ’t, generation, cled, ographs had to pick some onions for dinner. Sara didnot, wrong, unlucky, uncomfortable, uneasy ask God to help you eat your soup.” Tom didnot, ’t, lier, Winner, lers blue crayon and strike the wall.” Ben doesnot, bags, earnings, Village, lers 66 (0.064) Jack said, ”Sure, that would be great!” The littlegirl, boy, ably, ched, orers red, orange, and yellow colors. One day, a littlegirl, boy, scientists, acity, antly help her whenever she needed it. And the littlegirl, boy, rolled, anted, use was a littlegirl, boy, ations, ators, pots was a littlegirl, boy, ations, ators, pots 67 (0.161) found you!” It was her friend, Tim. Lily giggles, ly, ling, le, showed saw that Lily was suffering because she lostthe, all, a, some, something the bird. They took the bird home and cared forhim, her, the, all, many She touched the rubber duck and felt it squeak. She thought,, maybe, about, for, of ’t want to play with him. She ignoredher, the, them, it, his 68 (0.042) Timmy didn, not, t, never, on my didn, not, t, never, ’s time, Roxy didn, not, t, ‘, ́ my didn, not, t, never, ’s didn, not, t, ’s, . 69 (0.084) clumsy, Sam,” Tom said, shaking hishand, fist, finger, tail, arm Continued on next page 33 Id (P act )Input TextTop Logits the rain. She would jump in all the puddles, rejo, equal, defender, Matthew found you!” It was her friend, Tim. Lily giggled, sacked, decreased, yielded, uted empty. She frowns, ged, outs, ced, fully watch where he was going and tripped, led, sank, ave, annah 70 (0.018) his friends. Oneof, was, sunny, ,, friend friends. Oneof, was, sunny, ,, friend under her plate or give them to the dog. Onenight, of, morning, time, sunny . Oneof, was, ,, morning, is the park with her friends. Oneof, was, ,, night, sunny 71 (0.080) angry. Lily andBen, Tom, Jill, Mint, Fay ,” Tom said. Lily andTom, itt, est, hy, ippers It had a cut on its leg. Lily andBen, Tom, Mint, Flor, Shawn Anna andBen, iner, ability, astical, sub Lily andBen, Tom, Jack, Mark, Peter 72 (0.065) ?” Mom asked. Lily andMax, Tom, Lily, her, Tim angry. Lily andMax, her, Tom, the, Tim grandma. She misses us a lot.” Lily andher, Max, Lily, Mom, mom happy.” Anna andher, Tom, the, Max, Lily Anna andher, Tom, Lily, the, Max 73 (0.065) a time,in, a, the, they, it a time,in, a, the, they, it a time,in, a, the, they, it a time,in, a, the, they, it a time,in, a, the, they, it 74 (0.153) leaves under her feet and tried to climb the icy hill again. Thistime, ines, ans, mong, neys that he needed to be morecomfortable, organized, ., flexible, independent on, Max made sure to watch where he was going and to be more comfortable, flexible, obedient, independent, graceful wife and said, ”I will always provide foryou, ainer, ol, Out, ooked . He loves his sister. He says, ”I am sorry, Anna.I, Will, Sorry, Hi, In 75 (0.147) walking towards him. He was so scared that he didn’t know what to do, see, stir, sound, step didn’t know it would be so noisy.” Lily forgave him and theycontinued, gigg, resumed, repeated, stared very scared. She did not know what todo, think, see, hear, smell to unravel and Timmy and Sally didn’t know what todo, think, say, see, finish for your body.” Benny listened to Ollie’sstory, wise, song, words, voice 76 (0.411) They like to play with their toys and booksin, and, ,, together, ,” day, Timmy went to play with his friends in the park,, and, with, again, for . Max loved to play with his friends at the park,, every, and, because, with are friends. They like to play in the parkwith, and, every, near, , had a big toy that she really wantedto, ,, and, !, but 77 (0.073) Tom felt sad and angry. He wanted to make Lily share. He had an idea, island, tale, islands, kins It’s flying very far away.” Max wigg, add, ags, aded, ailed Then, Lily’s daddy had anidea, kins, ges, bows, ters it. Billy said, ”I have anidea, kins, bows, ters, leen told him about his problem. The rabbit had anidea, kins, bows, ers, ters 78 (0.057) her mommy and daddy. One day, when they went to see the zeod, oise, ric, zag, in hill and into the pond. Timmy and his friends laughed and had so much more, time, that, to, energy Lily’s mom asked her if she wanted to have a fancy teaset, with, ., tea, place you for the treat!” Spot barked, agged, fed, apped, led in a small house, there lived a kind andcompassionate, humble, modest, poor, harmless Continued on next page 34 Id (P act )Input TextTop Logits 79 (0.065) corn move back andthe, it, they, down, he wash it with soap andsoap, put, a, scrub, make to play with their blocks anddolls, their, share, books, have Lily decorated it with sweet frosting andcolorful, candles, glitter, lots, spark jump andplay, have, the, catch, see 80 (0.012) shoes before going outside to play.Once upon aweek, few, while, day, little that might have something yummy inside.Once upon aweek, few, day, while, long pond, happy and clean. The end.Once upon aweek, long, few, beautiful, day to his mom and be careful when playing outside.Once upon aweek, day, few, nice, little be extra careful not to bite anyone again.Once upon afew, week, little, while, long 81 (0.071) she should have been more careful. From thatday, cers, acas, neys, umm too because she helped the bird. From thatday, umm, ts, per, acas up on the fridge. From thatday, ters, anes, acas, ations on her finger to make it feel better. From thatday, saf, circus, concert, lectures sharing all of their toy tools. From thatday, umm, sters, Wings, Balls 82 (0.261) teddy bear. It is soft and brown. Itis, likes, looks, makes, does on the swings and theslides, swings, squirrel, other, sees every day. The plant had greenplants, roots, grass, stems, and see many things inside. There are books, toys, clothes, anda, more, even, games, food finds a small toy car withno, the, three, his, many 83 (0.092) in the future.Once upon a time, there was a big elephant namedEllie, Mighty, George, Harry, Daisy pond. The duck sees the ball and swims, olds, m, ets, ases Spot ran to get it. They both laughed when Spot accidentally knocked over a be aver, ak, ep, aker, at ”I’m sorry. Will you forgive me?” Her friend thought aboutthis, what, the, that, how from the dangerous land.Once upon a time, there was a big dog named Spot, Tom, Buddy, Rex, Bark 84 (0.008) them disappear again.Once upon a time, there was a little girl named Lily, L, Sara, Spirit, Inf upon a time, there was a little girl namedLily, L, Sara, D, Sandy Once upon a time, there was a little girl namedLily, L, Sara, Daisy, Anna ever frightened again.Once upon a time, there was a little girl named Lily, L, Sara, Po, D Once upon a time, there was a little boy namedTom, Tommy, Ben, Sam, Bob 85 (0.073) Anna and Ben are playing with crayons, hers, iers, od, eter The spider was angry and chased after Buzz. Buzz crawled as fast as he could, boy, girl, E, cer to unravel and Timmy and Sally didn’t know what tosay, expect, think, did, use acorn. The moral of theday, lesson, joke, game, lessons up and continued to play games together, but this time, Max made a, the, it, up, his 86 (0.110) proud of herself for helping her furry friend.Once upon a timethere, at, in, later, it listen to her mom and always be safe.Once upon a timethere, in, at, it, they under her plate or give them to the dog. One dayshe, the, when, they, her friends. They played together every day. One daythe, it, they, Tim, Tom importance of sharing and being kind to his friends.Once upon a time there, at, in, later, with 87 (0.126) play with her friends.One, They, She, Yesterday, Do her mommy and daddy.One, They, Yesterday, Do, Grace , Anna was feeling bossy.She, First, Lisa, Jenna, Mark to her bed.She, It, One, When, Every play together in the big green park near their house.One, They, There, Sally, Tommy 88 (0.113) his ball into the goal. Spot ran fast with the ball inthe, its, one, her, front noise. It was a car that zoompast, ing, by, !, across Continued on next page 35 Id (P act )Input TextTop Logits noises. Tom had a small car that could go fast and beloud, fast, very, slow, eps said. ”Deal, Mom. Thank you, Mom. You’rewelcome, very, right, a, good , red ball in the park. He threw it up high and caught it witha, the, ease, two, one 89 (0.190) his ball. He walked andtalked, played, ran, looked, jumped Max saw a big plane flying in the sky. Max barked, ked, ingly, fully, de She sees the letter. It is torn. She sighs, ers, aks, ses, rs ”It’s okay, myloves, sweet, loved, buddy, just her if she shared her cereal with Timmy. Lily said yes,but, excited, after, saying, offering 90 (0.049) dog running after the car. Lily, Lily, Linda, Lena, Rose Emma heard her sister’s scream and asked, ”Lily, Lily, Ben, Anna, recovery lit up with bright lights. Lily, L, Linda, Lily, Rose said, ”Lily, Lily, Ben, Lena, pollen ”Be careful, the edges are sharp!” Lily, Lily, Liam, Ben, Rose 91 (0.076) found you!” It was her friend, Tim. Lily giggled, oured, ingly, iot, connectors lots of fun puddles, as, ocks, led, is to investigate and found shiny rocks that sparkled, edly, ez, lling, rying so pretty and sparkly, Bench, Giants, RO, Sav sprinkles, led, angles, Cam, Crit 92 (0.196) it, but it was too heavy. The barrel rolled all theway, forest, place, jungle, mountains and see the world outside thegate, world, garden, city, forest vanished! Timmy looked all around hishouse, garden, backyard, yard, town my foot hurts. The frame fell on it.” Emma, enny, lee, am, erson is better than fighting. And they all became goodfriends, ls, behold, uld, Int 93 (0.158) to play in the water. He would jump and splash in the big puddle, water, ashes, ail, waves went to the park with her mom and saw herlittle, new, favorite, daughter, toy Ben. He is sad and bored. He misses Ben along, chance, day, time, fun the garden. They liked to observe the bugs and theflowers, plants, trees, bugs, worms When they got home, Lily put on her purple panda, endant, uddle, ears, ail 94 (0.220) and said, ”Yes, I can help you. But first, we havea, something, some, enough, no but Rex blocked his way. ”Leave me alone! I justwanted, wants, like, need, moved , you can,” Lily and Tom said, nodding. ”But you havebeen, a, something, no, too asked Fluffy. ”Yes, I want to come withus, me, the, your, my said to the plant, ”We are sorry, plant. We dida, something, it, our, wrong 95 (0.075) in the park.Once upon a time, there was a little girlnamed, lived, aked, Camer, topics ’l like them at first.Once upon a time, there was a little girlnamed, Camer, irs, unks, orns can always try again tomorrow.Once upon a time, there was a little girl named, Camer, topics, unks, ures on stage too.Once upon a time, there was a little boynamed, orns, osity, topics, unks up when things get hard.Once upon a time, there was a little girl named, Camer, unks, irs, Prepar 96 (0.066) Then, Lily’s daddy had anidea, ising, ID, ep, chairs The dog stopped being frightened and started wagging, agg, ashing, ogging, inking Tom felt sad and angry. He wanted to make Lily share. He had an idea, example, adjective, ID, error didn’t know it would be so noisy.” Lily forgave, apped, aked, aws, understood . Sam wanted to help his friend feel better. Sam had anidea, information, ID, ising, adjective 97 (0.102) had their wand and their bubbles. They didnot, ales, aper, ann, nce sorry, Mia. I wanted to win. I didnot, ator, interacted, rig, alks named Lily. She loved to play with her dolls,but, especially, even, which, so fun. They didnot, orb, aper, iour, nce liked her tank very much and didnot, ales, uffed, plet, uned 98 (0.053) ”You see, Mittens, uffy, uff, bles, ruff The swan nodded and swan, ans, uttered, atted, acked Continued on next page 36 Id (P act )Input TextTop Logits get his acorns, rob, robat, anuts, ockey asked Fluffy, oppy, uff, utter, opsy I’m the last of my family. The other swans, amps, anes, ippers, ooters 99 (0.172) Anna and Ben are playing with crayon, ins, s, ries, els the broken jar and the crumbs on thefloor, table, ground, kitchen, sidewalk went to the circus again.Once upon a time, therewas, named, iced, class, watching grass. They were very happy. But on theway, day, weekend, morning, evening them broke and spilled on thefloor, kitchen, stairs, sidewalk, street C.0.2. MOBILENET-V2-SMALLFULLDECOMPOSITION The most affected tokens (final token in each text) for each of the subnetworks in the mobilenet-v3-small decomposition, and the most affected logits of each sample. 37 tiger beetle barn spider tick agama ant 1 (0.19) tick isopod cockroach long-horned bee ant shoji plate rack window shade cradle chiffonier tick barn spider ground beetle long-horned bee ant tick barn spider isopod ant harvestman nipple hourglass hair spray lotion cocktail shaker tiger beetle ground beetle rock python whiptail sidewinder goldfish rock beauty frilled lizard triceratops electric ray nipple whiskey jug punching bag lotion oil filter face powder sunscreen maraca croquet ball magnetic compas killer whale dugong grey whale electric ray yawl 2 (0.01) killer whale grey whale dugong platypus black stork grey whale yawl killer whale dugong black grouse grey whale letter opener killer whale yawl dugong grey whale killer whale gazelle black grouse yawl grey whale fox squirrel killer whale lion patas killer whale grey whale red-breasted me whiskey jug wire-haired fox killer whale red-breasted me grey whale black grouse hartebeest killer whale red-breasted me badger grey whale skunk grey whale black grouse red-breasted me killer whale sea lion Eskimo dog Siberian husky dhole lion red wolf 4 (0.22) jaguar snow leopard leopard German short-ha fox squirrel wood rabbit fox squirrel hare dhole prairie chicken tiger beetle barn spider ground beetle agama ant Eskimo dog red wolf Siberian husky grey fox dhole tusker sloth bear African elephan Irish water spa curly-coated re barn spider tick isopod chambered nauti tarantula tiger beetle rock python sidewinder green mamba whiptail Arabian camel thresher sorrel hartebeest worm fence isopod platypus polecat fox squirrel mink rock beauty black-and-tan c Irish water spa EntleBucher curly-coated re 6 (0.10) sorrel Arabian camel thresher worm fence monastery African chamele vine snake tree frog green mamba tailed frog ant buckeye snail barn spider ground beetle kit fox fox squirrel coyote grey fox lion skunk colobus black grouse badger sidewinder tiger beetle dung beetle ground beetle ant isopod tick barn spider fiddler crab Dungeness crab rock crab brown bear lion cheetah red fox dhole barn spider isopod tiger beetle sidewinder agama tiger beetle rock python sidewinder whiptail green mamba 7 (0.40) barn spider ground beetle tick long-horned bee isopod tiger beetle barn spider tick agama ant maraca pick tick bolo tie can opener lampshade table lamp chime spotlight bolo tie tick ground beetle cockroach barn spider ant barn spider tick isopod scorpion tarantula tick barn spider chain saw isopod ground beetle hyena fox squirrel Arctic fox lion cheetah barn spider ant tick tiger beetle ground beetle space shuttle projectile missile killer whale albatross 9 (0.41) dugong axolotl ocarina isopod hare dugong killer whale grey whale rock beauty tiger shark dugong killer whale electric ray space shuttle grey whale dugong rock beauty nipple whiskey jug bell pepper grey whale yawl letter opener killer whale Windsor tie dugong platypus grey whale electric ray screen dugong killer whale sea lion grey whale hammerhead dugong can opener grey whale letter opener iron hammerhead rock beauty killer whale electric ray tiger shark ping-pong ball bell pepper corn lemon grocery store 10 (0.49) waffle iron velvet space bar typewriter keyb chain mail eggnog spotlight ocarina table lamp pick tusker Indian elephant African elephan hartebeest gong siamang gorilla guenon chimpanzee patas milk can whiskey jug sunscreen barrel eggnog bolo tie barn spider tick pedestal bearskin cockroach tick ant ground beetle long-horned bee isopod chiton bolo tie maraca frying pan fox squirrel rock python hen-of-the-wood earthstar brambling goldfish rock beauty tench yellow lady's s platypus 11 (0.14) tick ground beetle isopod ant tiger beetle table lamp lampshade spotlight chime bolo tie lion fox squirrel dingo kit fox basenji space bar waffle iron muzzle computer keyboa typewriter keyb red fox kit fox dingo lion coyote goldfish triceratops isopod frilled lizard black grouse goldfish platypus triceratops isopod dugong wood rabbit hare fox squirrel toy terrier wallaby barn spider isopod tick hair slide king crab rock beauty axolotl goldfish badger tench 12 (0.33) rock beauty king penguin maraca nipple whiskey jug rock beauty nipple axolotl isopod marmoset rock beauty frilled lizard tench goldfish eel rock beauty goldfish black grouse axolotl eel lemon spaghetti squas orange consomme gong rock beauty lemon goldfish fox squirrel electric ray rock beauty goldfish spotted salaman ocarina eft rock beauty screen triceratops nipple ocarina rock beauty yellow lady's s jellyfish dugong space shuttle folding chair plate rack cradle rocking chair studio couch 13 (0.02) folding chair rocking chair throne cradle pedestal folding chair rocking chair barber chair cradle jinrikisha folding chair pedestal shoji cradle dining table folding chair rocking chair plate rack cradle muzzle folding chair pedestal barrel guillotine dining table folding chair rocking chair cradle pedestal shoji folding chair dining table pedestal rocking chair cradle folding chair punching bag maraca tripod thresher folding chair rocking chair dining table cradle pedestal tick cockroach ant milk can isopod 14 (0.20) tick barn spider ant isopod harvestman barometer fire screen magnetic compas clog plate rack snow leopard jaguar hen-of-the-wood lynx leopard tiger beetle whiptail rock python sidewinder green mamba African elephan Arabian camel bighorn tusker sorrel African elephan tusker thresher black grouse bighorn snow leopard leopard lynx jaguar cheetah skunk black stork mink Walker hound colobus pedestal table lamp pickelhaube whiskey jug hourglass kit fox lion cheetah lesser panda red wolf 15 (0.59) tiger chambered nauti prairie chicken tiger cat zebra cradle chiffonier shoji desk dining table weasel polecat mink lesser panda indri gorilla siamang guenon patas orangutan rock beauty coral reef puffer goldfish isopod polecat marmoset indri mink magnetic compas timber wolf dhole kit fox grey fox red wolf skunk colobus black grouse badger screw Samoyed grey fox kit fox coyote Arctic fox tiger beetle tick ground beetle rock python sidewinder 18 (0.08) tiger chambered nauti tiger cat zebra prairie chicken bolo tie tick barn spider pedestal triceratops analog clock wall clock stopwatch barometer magnetic compas cockroach ant tick ground beetle long-horned bee tick barn spider ground beetle ant cockroach lawn mower thresher tractor chain saw mobile home ant isopod barn spider ground beetle tiger beetle waffle iron space bar typewriter keyb computer keyboa muzzle banded gecko whiptail sidewinder ocarina fox squirrel tick barn spider long-horned bee ground beetle isopod 19 (0.06) tiger beetle rock python whiptail sidewinder green mamba tiger beetle barn spider tick agama ant thunder snake sidewinder horned viper Indian cobra velvet maraca pick tick can opener safety pin tick snail sidewinder rock python gong goldfish rock beauty axolotl coral reef earthstar fox squirrel platypus earthstar gong indri leopard hen-of-the-wood jaguar cheetah lynx goldfish rock beauty axolotl platypus black grouse fox squirrel platypus earthstar gong indri 21 (0.12) tick ground beetle barn spider ant long-horned bee barn spider tick isopod tarantula wolf spider school bus moving van passenger car trailer truck recreational ve alp valley steel arch brid grey whale dugong timber wolf red wolf barrel milk can tiger tiger beetle rock python sidewinder whiptail green mamba rock beauty daisy goldfish chain saw milk can Eskimo dog red wolf lion dhole tiger hartebeest African elephan lion Rhodesian ridge Indian elephant tick cockroach isopod ant ground beetle 24 (0.26) school bus moving van trailer truck passenger car amphibian cockroach ant tick ground beetle barn spider tiger beetle rock python sidewinder whiptail green mamba snail tick buckeye ringlet croquet ball tick barn spider ground beetle ant isopod tiger beetle ground beetle ant dung beetle tick missile space shuttle airliner projectile geyser red-breasted me barracouta punching bag bearskin patas cockroach whistle isopod tick can opener tiger beetle rock python sidewinder whiptail green mamba 26 (0.06) tick ground beetle barn spider isopod ant tiger beetle barn spider tick agama ant Indian elephant African elephan tusker brown bear Arabian camel moving van trailer truck recreational ve police van mobile home nematode thunder snake sidewinder hook chain whiptail agama frilled lizard alligator lizar vine snake chimpanzee siamang guenon macaque patas Samoyed kuvasz Arctic fox Eskimo dog Great Pyrenees folding chair rocking chair pedestal cradle shoji studio couch fire screen plate rack throne four-poster 28 (0.11) leopard lynx cheetah hen-of-the-wood wood rabbit studio couch plate rack panpipe chiffonier crate isopod bolo tie maraca gong barn spider thresher fox squirrel rotisserie Sussex spaniel redbone wood rabbit hare fox squirrel wallaby ibex studio couch amphibian moving van half track thresher studio couch quilt plate rack barrel redbone rotisserie isopod rock crab American lobste sorrel space bar waffle iron spatula tobacco shop plate rack magnetic compas barometer clog stopwatch fire screen 29 (0.07) rock python nematode chain mail horned viper vine snake tiger velvet nematode tiger cat pedestal thresher triceratops milk can breastplate hog face powder sunscreen nipple ping-pong ball maraca nematode whistle safety pin ocarina chain analog clock stopwatch magnetic compas barometer reel grey whale killer whale hare Cardigan Mexican hairles passenger car moving van bannister trailer truck electric locomo space bar computer keyboa typewriter keyb honeycomb hand-held compu tusker fox squirrel African elephan Indian elephant colobus 31 (0.09) kit fox fox squirrel coyote lion red fox tick frilled lizard isopod revolver ant dugong axolotl ocarina isopod hare tick cockroach ground beetle ant harvestman killer whale yawl grey whale electric ray Arctic fox plate rack patas Japanese spanie safety pin fire screen orangutan chimpanzee capuchin siamang gibbon grey whale ice bear Sealyham terrie dugong space shuttle ice bear Arctic fox hare polecat kit fox dingo Eskimo dog timber wolf coyote Siberian husky 33 (0.02) polecat marmoset indri fox squirrel magnetic compas tick ant cockroach barn spider ground beetle polecat marmoset black-footed fe fox squirrel keeshond fox squirrel wire-haired fox Norwich terrier Australian terr toy terrier jaguar leopard snow leopard lynx wood rabbit rock python sidewinder horned viper fox squirrel chain mink fox squirrel black grouse beaver three-toed slot dugong nipple sunscreen lotion ocarina dugong loggerhead coral reef chambered nauti ice bear space shuttle projectile missile airliner lighter 34 (0.06) fox squirrel earthstar gong platypus indri cheetah snow leopard leopard lynx lion tiger zebra chambered nauti drum tiger cat face powder ping-pong ball maraca croquet ball oil filter lion cheetah lynx cougar tiger cat projectile space shuttle killer whale missile wire-haired fox nematode hook stethoscope muzzle whistle thresher Sussex spaniel rotisserie fox squirrel redbone screwdriver maraca hair slide ballpoint zucchini dugong killer whale knee pad whiskey jug albatross 35 (0.03) dugong grey whale velvet electric ray ice bear grey whale dugong hammerhead killer whale electric ray grey whale dugong killer whale platypus rock beauty dugong tick eggnog grey whale whiskey jug dugong nipple screen sunscreen lotion dugong killer whale grey whale ice bear albatross grey whale killer whale platypus electric ray ice bear dugong geyser platypus grey whale rock beauty grey whale hammerhead dugong killer whale screen rock python sidewinder nematode vine snake horned viper 37 (0.62) grey fox timber wolf red wolf dingo kit fox fox squirrel platypus earthstar gong indri timber wolf coyote fox squirrel snow leopard grey fox ice bear dugong ocarina iron Bedlington terr chimpanzee orangutan siamang guenon Irish water spa chimpanzee patas siamang gorilla gibbon Angora whiskey jug wood rabbit Japanese spanie axolotl projectile space shuttle missile killer whale wire-haired fox banded gecko rock python sidewinder horned viper tailed frog thresher fox squirrel Sussex spaniel rotisserie redbone 39 (0.22) chimpanzee siamang guenon macaque gorilla ground beetle tick barn spider black widow isopod sea lion grey whale dugong electric ray platypus chimpanzee guenon siamang patas langur chimpanzee orangutan siamang guenon patas chain saw lawn mower thresher croquet ball milk can sorrel redbone hog ox boxer chimpanzee siamang patas gorilla guenon Norwegian elkho Eskimo dog Siberian husky Arctic fox keeshond tiger beetle rock python sidewinder green mamba ground beetle 40 (0.19) passenger car moving van mobile home worm fence milk can Granny Smith spaghetti squas lemon jackfruit acorn platypus vine snake axolotl black stork dugong rock beauty spaghetti squas yellow lady's s daisy zucchini platypus mink gazelle fox squirrel otter rock beauty vine snake green mamba tree frog African chamele komondor Dandie Dinmont Sussex spaniel wood rabbit fox squirrel goldfish axolotl rock beauty yellow lady's s strawberry vine snake letter opener rock beauty African chamele green mamba rock beauty spaghetti squas acorn squash chain saw barn spider 43 (0.03) rock beauty lemon goldfish spaghetti squas jackfruit rock beauty frilled lizard platypus squirrel monkey gong rock beauty jellyfish dugong space shuttle corn tiger beetle rock python sidewinder whiptail green mamba rock beauty axolotl goldfish badger tench rock beauty dugong vine snake axolotl yellow lady's s rock beauty yellow lady's s goldfish ocarina axolotl rock beauty dugong axolotl ocarina isopod rock beauty nipple axolotl isopod banded gecko lion lynx cheetah tiger cougar 44 (0.37) barn spider tick isopod tarantula wolf spider prayer rug fire screen book jacket pedestal throne sulphur butterf yellow lady's s ringlet cabbage butterf pick entertainment c thresher lotion moving van screen green mamba vine snake Indian cobra nematode sidewinder motor scooter jinrikisha chain saw moped bicycle-built-f Irish water spa mink otter Sussex spaniel Bedlington terr dugong killer whale hammerhead grey whale tiger shark bell pepper whiskey jug strawberry ocarina maraca reel frying pan barometer loupe analog clock 45 (0.04) folding chair rocking chair pedestal cradle shoji whiptail frilled lizard agama vine snake African chamele folding chair rocking chair plate rack cradle shoji folding chair plate rack cradle chiffonier shoji tick barn spider isopod ant cockroach Angora Persian cat Japanese spanie polecat Windsor tie shoji plate rack window shade cradle chiffonier moving van mobile home minibus rotisserie screen cockroach tick barn spider ant ground beetle tick cockroach ground beetle ant isopod 47 (0.04) trimaran yawl planetarium nipple moving van chiffonier plate rack bookcase barrel cradle folding chair pedestal cradle shoji barrel folding chair cradle plate rack rocking chair park bench table lamp lampshade sunscreen nipple spotlight table lamp spotlight can opener lampshade cocktail shaker shoji barrel lotion milk can pedestal lampshade table lamp spotlight pedestal fire screen chiffonier plate rack china cabinet entertainment c bookcase cheetah lynx leopard jaguar wood rabbit 48 (0.11) tiger patas fox squirrel red fox cougar tiger jaguar dhole English foxhoun patas leopard cheetah snow leopard lynx lion tiger lion red wolf red fox lynx tiger lion dhole red wolf jaguar lion chow cougar kit fox dhole jaguar leopard lion snow leopard lynx leopard fox squirrel lynx cheetah jaguar lion fox squirrel dingo kit fox basenji tiger beetle rock python sidewinder whiptail ground beetle 51 (0.03) tiger beetle barn spider agama ground beetle ant tick ground beetle barn spider long-horned bee isopod ladybug leaf beetle maraca pool table croquet ball chimpanzee gorilla patas siamang guenon ground beetle tick cockroach barn spider ant chimpanzee colobus siamang patas sloth bear patas wallaby platypus Siamese cat Arctic fox honeycomb soccer ball hand-held compu screen computer keyboa rotisserie isopod chiton American lobste Dungeness crab tiger beetle barn spider tick ground beetle ant 53 (0.06) black grouse croquet ball thresher hare Sealyham terrie leopard hen-of-the-wood jaguar snow leopard cheetah grey fox Eskimo dog red wolf Siberian husky dingo fox squirrel kit fox grey fox dhole coyote cheetah leopard lynx wood rabbit jaguar sulphur butterf yellow lady's s cabbage butterf clog pick timber wolf tiger barrel milk can grey fox grey whale dugong brown bear affenpinscher platypus leopard snow leopard jaguar cheetah German short-ha fox squirrel orangutan patas indri titi 55 (0.21) monastery vault fire screen gondola palace tick ground beetle barn spider ant long-horned bee patas chimpanzee siamang gorilla langur school bus moving van passenger car ambulance trailer truck sorrel Walker hound bluetick English foxhoun toy terrier Arabian camel bighorn tusker sorrel African elephan red wolf timber wolf snow leopard grey fox dhole sorrel ox hog bighorn tusker chimpanzee cloak abaya panpipe patas ladybug maraca leaf beetle pool table croquet ball 56 (0.08) goldfish earthstar pinwheel strawberry lesser panda goldfish axolotl rock beauty strawberry yellow lady's s goldfish rock beauty axolotl platypus black grouse face powder ping-pong ball croquet ball maraca knee pad lion fox squirrel dingo Pomeranian wire-haired fox face powder ping-pong ball maraca lemon sunscreen goldfish maraca ocarina rock beauty hen-of-the-wood face powder ping-pong ball maraca digital clock corn lion chow dhole redbone Sussex spaniel tusker African elephan Indian elephant sorrel bison 58 (0.11) tiger beetle ground beetle tick ant green mamba sidewinder rock python vine snake chain chain mail American black sloth bear brown bear lesser panda Irish water spa tusker African elephan Indian elephant sorrel brown bear lion cougar tiger tiger cat lynx grey fox Eskimo dog timber wolf red wolf coyote folding chair stretcher barber chair thresher rocking chair lion dingo dhole red wolf golden retrieve barn spider ant custard apple tick black and gold lion dingo dhole chow Sussex spaniel 59 (0.10) Angora wood rabbit whiskey jug Sealyham terrie Persian cat rock python nematode chain mail chain sidewinder lion fox squirrel dhole beaver patas red wolf grey fox timber wolf dingo Eskimo dog tick barn spider isopod cockroach ant lion brown bear cheetah dhole red fox lion cougar chow kit fox dhole lion fox squirrel kit fox brown bear red fox gorilla patas chimpanzee siamang langur hare wood rabbit fox squirrel toy terrier wallaby 60 (0.31) thresher Sussex spaniel fox squirrel rotisserie redbone tick ground beetle ant cockroach barn spider red wolf timber wolf Eskimo dog Siberian husky malamute tick fiddler crab barn spider Dungeness crab rock crab soccer ball honeycomb space bar computer keyboa hand-held compu fox squirrel mink triceratops Sussex spaniel grey fox barn spider isopod ant tick long-horned bee tiger sorrel redbone orangutan Sussex spaniel thresher amphibian moving van harvester mobile home killer whale grey whale red-breasted me badger dugong 61 (0.34) polecat weasel mink lesser panda black-footed fe killer whale dugong grey whale platypus black stork grey whale fox squirrel killer whale patas lion magnetic compas barometer analog clock clog stopwatch barometer analog clock stopwatch wall clock magnetic compas killer whale black stork red-breasted me gazelle platypus chimpanzee gorilla patas siamang gibbon toy terrier neck brace basenji nipple Sussex spaniel studio couch fire screen upright barrel plate rack snow leopard hen-of-the-wood jaguar tailed frog sidewinder 66 (0.26) lynx leopard snow leopard jaguar cheetah tiger lion tiger cat lynx red wolf leopard lynx hen-of-the-wood Irish water spa cheetah jaguar tiger cheetah lynx leopard lesser panda polecat patas weasel fox squirrel leopard jaguar snow leopard German short-ha cheetah tiger jaguar tiger cat wire-haired fox dhole tiger lynx patas fox squirrel red wolf leopard jaguar tiger gong cheetah tiger beetle barn spider tick agama ant 67 (0.42) red wolf coyote timber wolf kit fox dingo tiger beetle rock python whiptail sidewinder green mamba rock beauty coral reef puffer goldfish anemone fish ant barn spider ground beetle isopod tick isopod ground beetle barn spider tick ant lion cougar cheetah lynx kit fox ground beetle barn spider tick long-horned bee isopod goldfish rock beauty axolotl coral reef earthstar tick barn spider ground beetle ant isopod analog clock wall clock stopwatch magnetic compas barometer 69 (0.04) barn spider tick harvestman ant tarantula red wolf timber wolf Eskimo dog grey fox Siberian husky barometer analog clock wall clock stopwatch magnetic compas fox squirrel muzzle ocarina gong mousetrap barn spider ant tick black and gold garden spider sidewinder rock python horned viper banded gecko bolo tie steel arch brid letter opener plate rack cleaver mountain tent goldfish rock beauty frilled lizard dugong whistle spaghetti squas lemon jackfruit fig croquet ball tick barn spider isopod long-horned bee ant 70 (0.02) tick ground beetle cockroach ant barn spider ground beetle tiger beetle ant long-horned bee barn spider tick ant ground beetle barn spider cockroach barn spider ant tick frilled lizard chain saw tick ant barn spider isopod long-horned bee tick barn spider chain saw black widow harvestman cockroach ant ground beetle isopod long-horned bee tick barn spider ground beetle scorpion ant ant ground beetle tick cockroach barn spider folding chair rocking chair plate rack cradle muzzle 75 (0.02) fox squirrel black grouse grey fox cheetah prairie chicken chimpanzee patas orangutan black-and-tan c gorilla whiptail frilled lizard agama banded gecko vine snake lion fox squirrel dhole beaver patas lynx jaguar Egyptian cat tiger tiger cat hare wood rabbit fox squirrel ibex wallaby brown bear chow Dandie Dinmont otterhound golden retrieve red wolf barrel tiger milk can polecat lion otterhound Lakeland terrie golden retrieve cheetah lotion hair spray sunscreen oil filter whistle 76 (0.07) folding chair rocking chair plate rack cradle muzzle hourglass nipple safety pin can opener whiskey jug timber wolf tiger barrel polecat coyote screen lotion hand-held compu television sunscreen tiger beetle barn spider ant agama tick oil filter sunscreen barrel whistle face powder studio couch cradle plate rack shoji space heater beacon drilling platfo pedestal projectile soap dispenser hair spray sunscreen whistle Band Aid can opener shoji plate rack window shade cradle chiffonier 82 (0.12) school bus moving van trailer truck passenger car amphibian table lamp lampshade spotlight pedestal nipple lotion binder letter opener screen soap dispenser yawl monastery projectile space shuttle pedestal sunscreen ocarina croquet ball lotion nipple pedestal folding chair cradle dining table rocking chair cleaver shoji face powder knee pad slide rule steel arch brid letter opener grey whale mountain tent cleaver passenger car moving van electric locomo lifeboat amphibian tiger chambered nauti zebra prairie chicken tiger cat 83 (0.27) platypus mink gazelle weasel otter passenger car moving van cradle thresher studio couch isopod platypus scorpion Windsor tie cockroach isopod chiton bolo tie maraca frying pan plate rack cradle rotisserie moving van polecat panpipe isopod maraca letter opener whistle brown bear patas polecat otterhound Indian elephant tiger beetle ant whiptail barn spider damselfly otter beaver dugong red-breasted me ocarina folding chair rocking chair cradle throne pedestal 84 (0.58) tick ground beetle barn spider long-horned bee isopod tick fiddler crab barn spider Dungeness crab chain saw tick cockroach ant ground beetle isopod tiger beetle whiptail rock python sidewinder green mamba rock python nematode horned viper chain vine snake tiger beetle barn spider tick agama ant table lamp lampshade milk can espresso maker fire screen tiger wire-haired fox dhole Brabancon griff lion tick cockroach long-horned bee isopod ant analog clock magnetic compas wall clock stopwatch barometer 85 (0.16) thresher Sussex spaniel rotisserie redbone fox squirrel tick isopod ground beetle long-horned bee ant table lamp hatchet spatula pedestal lampshade wall clock barometer analog clock bolo tie maraca analog clock wall clock magnetic compas barometer stopwatch computer keyboa honeycomb space bar typewriter keyb corn pool table dining table bannister ping-pong ball potter's wheel folding chair rocking chair plate rack cradle muzzle folding chair rocking chair barber chair cradle jinrikisha Indian elephant tusker African elephan sorrel ox 87 (0.11) tick barn spider isopod tarantula wolf spider Indian elephant tusker African elephan hartebeest Mexican hairles red wolf timber wolf kit fox coyote grey fox tick stopwatch hair slide chain saw whistle tusker African elephan Indian elephant sorrel bison tusker African elephan triceratops sorrel Mexican hairles Indian elephant African elephan tusker sloth bear curly-coated re lion dingo kit fox red fox red wolf lion fox squirrel dingo kit fox basenji wall clock barometer analog clock stopwatch magnetic compas 88 (0.05) screen hand-held compu lotion television sunscreen folding chair rocking chair muzzle plate rack cradle sunscreen cleaver milk can dishwasher fire screen passenger car tobacco shop moving van streetcar amphibian pedestal hourglass milk can espresso maker nipple studio couch upright pedestal folding chair chiffonier kit fox dingo red fox red wolf basenji moving van passenger car ambulance ocarina trolleybus table lamp lampshade spotlight fire screen chime thresher Sussex spaniel rotisserie fox squirrel redbone 90 (0.02) American black brown bear groenendael sloth bear Sussex spaniel brown bear red wolf Sussex spaniel Norwich terrier otterhound American black brown bear hog mink lesser panda barn spider cardoon strawberry otterhound yellow lady's s mink polecat weasel lesser panda brown bear goldfish rock beauty axolotl torch power drill fox squirrel grey fox wallaby Madagascar cat kit fox redbone sorrel Sussex spaniel Rhodesian ridge EntleBucher barn spider black and gold yellow lady's s garden spider fox squirrel snow leopard hen-of-the-wood leopard jaguar lynx 91 (0.04) timber wolf dingo red wolf coyote dhole leopard jaguar cheetah lynx wood rabbit jaguar cheetah leopard snow leopard lynx tiger tiger cat jaguar dhole wire-haired fox leopard cheetah jaguar German short-ha snow leopard waffle iron abacus velvet space bar chain mail tick ground beetle cockroach ant long-horned bee jaguar tiger fox squirrel leopard tabby table lamp lampshade bolo tie chime pedestal school bus moving van trailer truck passenger car amphibian 92 (0.68) steel arch brid letter opener mountain tent plate rack cleaver passenger car moving van thresher cradle studio couch barn spider croquet ball ant jacamar nipple grey whale letter opener dugong Windsor tie killer whale moving van thresher triceratops chain saw screen African elephan Arabian camel tusker hartebeest muzzle tusker lion African elephan Indian elephant otterhound timber wolf barrel red wolf milk can whiskey jug moving van thresher fire screen planetarium tobacco shop tiger beetle whiptail rock python sidewinder green mamba 94 (0.22) tick barn spider ant long-horned bee isopod tiger beetle barn spider agama ant tick nematode whistle nipple hook safety pin rock python sidewinder horned viper fox squirrel chain maraca pick tick hair slide safety pin nematode bolo tie ocarina ringneck snake sea snake thunder snake sidewinder nematode horned viper Indian cobra nematode nipple dugong Petri dish oxygen mask lion fox squirrel kit fox dingo Persian cat sea lion polecat dugong black-footed fe otterhound 98 (0.02) colobus cloak prison guillotine space shuttle Japanese spanie neck brace colobus Dandie Dinmont patas ping-pong ball face powder golf ball knee pad spotlight thresher space shuttle amphibian ant airliner Arctic fox kuvasz Samoyed white wolf Great Pyrenees siamang marmoset Sussex spaniel colobus clumber badger indri African elephan colobus tusker ping-pong ball face powder digital clock croquet ball maraca colobus siamang chimpanzee gibbon orangutan