Paper deep dive
CircuitTree: Composing Circuits into A Syllable Transcriber
Jiankun Wei, Ewan Dunbar, Gerald Penn
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/12/2026, 5:55:40 AM
Summary
The paper introduces CircuitTree, a method for composing multiple neural network circuits (subnetworks) into a single model to perform complex tasks. Using a greedy, density-scaled composition algorithm and a new data structure called CircuitTree, the authors compose 39 phoneme circuits pruned from HuBERT into a syllable classifier. This approach achieves 64.8% accuracy on open syllable transcription, close to the 66.0% ceiling of independent circuits, while sharing 99.5% of circuit components.
Entities (8)
Relation Signals (7)
HuBERT → sourceof → Phoneme Circuits
confidence 98% · We compose circuits pruned from HuBERT for classifying individual phonemes
Phoneme Circuits → composedinto → Syllable Classifier
confidence 96% · compose circuits pruned from HuBERT ... into a syllable classifier
CircuitTree → enables → Composing N circuits
confidence 95% · We propose a greedy, density-scaled composition algorithm ... then generalize it to N circuits via CircuitTree
CircuitTree → hascomplexity → Theta(N log N)
confidence 94% · CircuitTree, a new data structure built on a Θ(N log N) tree-compose schedule.
Density-Scaled Composition → overcomes → Naive Union Limitations
confidence 92% · We propose a greedy, density-scaled composition algorithm that overcomes limitations of previous attempts to compose circuits by naive union
Naive Union → suffersfrom → Density Domination
confidence 90% · We attribute this performance degradation to two key issues: density domination and interference.
Naive Union → suffersfrom → Interference
confidence 90% · overlaying two independently optimized masks introduces circuit interference
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The goal of circuit discovery is to decompose a monolithic neural network into tiny modular subnetworks which are interpretable in the sense that each does an identifiable subtask. We develop a practical method for composing arbitrarily many circuits into a single model. We propose a greedy, density-scaled composition algorithm that overcomes limitations of previous attempts to compose circuits by naive union, then generalize it to $N$ circuits via CircuitTree, a new data structure built on a $\Theta(N\log N)$ tree-compose schedule. We compose circuits pruned from HuBERT for classifying individual phonemes into a syllable classifier, a capability none of the 39 phoneme circuits has individually. Chaining CircuitTrees into a hierarchy-of-circuits pipeline, we transcribe open syllables at 64.8\% accuracy, within 1.2 points of a 66.0\% ceiling that shares no circuitry at all, with 99.5\% of circuit components shared.
Tags
Links
- Source: https://arxiv.org/abs/2610.12520v1
- Canonical: https://arxiv.org/abs/2610.12520v1
Trouble viewing inline? Open PDF directly →
Full Text
22,693 characters extracted from source content.
Expand or collapse full text
CIRCUITTREE: COMPOSING CIRCUITS INTO A SYLLABLE TRANSCRIBER Jiankun Wei 1 , Ewan Dunbar 2 , Gerald Penn 3 123 Univerity of Toronto ABSTRACT The goal of circuit discovery is to decompose a monolithic neural network into tiny modular subnetworks which are in- terpretable in the sense that each does an identifiable sub- task. We develop a practical method for composing arbitrar- ily many circuits into a single model. We propose a greedy, density-scaled composition algorithm that overcomes limita- tions of previous attempts to compose circuits by naive union, then generalize it to N circuits via CircuitTree, a new data structure built on a Θ(N logN ) tree-compose schedule. We compose circuits pruned from HuBERT for classifying indi- vidual phonemes into a syllable classifier, a capability none of the 39 phoneme circuits has individually. Chaining Circuit- Trees into a hierarchy-of-circuits pipeline, we transcribe open syllables at 64.8% accuracy, within 1.2 points of a 66.0% ceil- ing that shares no circuitry at all, with 99.5% of circuit com- ponents shared. Index Terms— circuit discovery, mechanistic inter- pretability, model merging, modularity, HuBERT 1. INTRODUCTION Many researchers has proposed to view modern multi-task neural network models as the composition of large numbers of sub-networks (circuits) with independent capabilities. [1– 6]. Early work identified circuits manually [7]; automated methods soon followed [8], reducing circuit discovery to a process of choosing a metric and a dataset to define a be- haviour, and patching activations until the responsible com- ponents emerge. While much research has proposed circuit discovery as a tool for deepening our understanding of black-box models, the ability to decompose models into small, modular sub- networks has practical implications. Researchers have en- visioned circuit discovery as a way to “re-engineer” a net- work by decomposing the monolithic model into modular, Lego-like pieces, each responsible for a single task [1, 2, 6, 9, 10]. The practical payoff of this decomposition is mod- ularity. Once primitive circuits are identified, they can be joined to solve complex tasks without having to rediscover every possible combination from scratch. Merullo, Eickhoff, and Pavlick [3] found that a circuit discovered for one task in This work was supported by [embargoed for anonymous review]. Fig. 1: Circuit discovery of weight and edge circuits. GPT-2 is substantially reused for a related one, showing that a small hand-designed patch to four attention heads is suffi- cient to change the model’s behaviour. Mondorf et al. [1] went further, posing general circuit composability as a funda- mental open question and combining circuits for ten string- edit operations through mask-level set operations. However, their experiments were limited to synthetic toy models gener- ated via Tracr [11]. Most recently, Yu et al. [2] discovered that composing two independently discovered circuits pruned from GPT-2 for different tasks by simple mask union largely preserves functional performance. This is a compelling vision, but the evidence for it re- mains preliminary. Existing composition results stop at a pair of circuits, or at best a handful, and success is measured by whether the composed circuit still performs the individual tasks it was pruned on. Can primitive circuits can be com- posed at scale, and into new capabilities? We answer both halves of this question: composition survives the jump from a hand-verified pair to dozens of independently-discovered cir- cuits merged at once, and the resulting shared circuitry is able to do a more complex task that none of the 39 source circuits were individually pruned to solve. We do this by composing circuits pruned from an English phoneme classifier based on HuBERT, about as elementary a building block as a spoken language offers, into an syllable classifier. The outline of this paper is as follows. First, we iden- tify failures of plain mask union (density domination, inter- ference) and design a greedy, density-scaled algorithm that composes a pair of circuits. Second, we generalize this to N circuits with the help of a new data structure, CIRCUITTREE. arXiv:2610.12520v1 [cs.LG] 7 Oct 2026 Third, we compose the resulting vowel and consonant CIR- CUITTREEs into a hierarchy-of-circuits pipeline with 99.5% of matrices shared across phonemes. The resulting pipeline transcribes English CV and VC syllables, a task the individ- ual circuits were not trained for, and which the composition algorithm does not optimize for either. We reach 64.8% ac- curacy against the ceiling of 66.0% obtained by running the circuits independently. 2. BACKGROUND ON CIRCUIT DISCOVERY Circuit-discovery methods split along what they prune. Weight-level methods learn a differentiable mask directly over model parameters, e.g., via hard-concrete L 0 relax- ations [12] or movement pruning [13], yielding a sparse sub- network but leaving the dense computation graph untouched. Edge-level methods instead prune the residual-stream com- putation graph itself while keeping weights dense: ACDC searches over activation patches [8], and Edge Pruning [14] learns a differentiable mask over edges with a straight- through relaxation. DiscoGP [2] is a circuit discovery al- gorithm that does both weight and edge pruning. 3. EXPERIMENTAL SETUP Dataset. We train and evaluate on the Articulatory Index corpus [15], originally collected by [16], which contains pro- nunciations of 10,329 CV and VC monosyllables produced by 20 English speakers (12 male, 8 female). The data for each phoneme is split into train, validation, and test sets. Re- sults reported are for test. Thresholds for ensemble classifiers described in Section 4 are tuned on the validation set. All accept/reject decisions in Algorithm 1 and its N -way exten- sion use tolerance ε = 0.02 absolute F1; scale factors (Eq. 1) are clipped to [0.05, 5.0]; the syllable type gate (§6) is itself a DISCOGP circuit under the same protocol. Following [6], we prune 15 vowel and 24 consonant DISCOGP circuits from HuBERT-base [17] with a frozen mean-pooled linear head trained on the binary task of iden- tifying whether a given vowel or consonant in present the syllable. Below, we combine these binary circuits into en- semble multi-class classifiers as described in Section 4. The resulting weight circuits consist of 144 logit tensors (12 layers×Q,K,V,O,MLP-in, MLP-out×weight,bias). Edge circuits are 49 logit tensors, one per downstream node, each holding one logit per eligible upstream head or MLP. 4. PAIRWISE CIRCUIT COMPOSITION We start by developing algorithms for composing two phonemes’ circuits p i ,p j . We combine pairs of binary classifiers into three-way classifiers p i ,p j , other.Each binary classifier for phoneme p i is associated with a threshold τ i , tuned by validation set. When combining two classifiers, whichever phoneme’s confidence surpasses its threshold the most will be the prediction. If neither does, we predict “other”. In this setting, the classifiers being combined can be run indepen- dently, or can be composed into a single system with shared weights and edges. We first test a naive composition strategy by taking the element-wise union of weights and edges present in either cir- cuit. As shown in Figure 2, the independent, unmerged clas- sifiers (Figure 2a) substantially outperform this naive union across all vowel phoneme pairs (Figure 2b). We attribute this performance degradation to two key issues: density domina- tion and interference. By density domination we refer to the issue that circuits can have very different weight densities (2–3% for some phonemes, 15–20% for others). Simple union is dominated by the denser circuit, so the composition behaves like the dense circuit. We resolve this by scaling each mask in both circuits, or, in the general case, in all N circuits. For a shared key of the underlying model k, let d 1 (k),...,d N (k) denote the fraction of active entries in each phoneme p 1 ,...,p N ’s mask. We calculate each phoneme’s scale factor at key k σ i (k) (which have the same shape as circuit masks at key k) as follows: σ i (k) = clip t(k) d i (k) , [s min ,s max ] (1) where t(k) = N Y n=1 d n (k) 1/N ,(2) is the geometric mean of the densities. In following MODEL FORWARD() method where N cir- cuits are applied to the HuBERT model, we first element-wise union (naive union) the N circuits and select the kept parame- ter/edge connection, then scale them by element-wise mulpil- ication of the scale factor σ i (k). Even at matched density, overlaying two independently optimized masks introduces circuit interference: the presence of a given weight or edge may be crucial to the behaviour of one circuit but detrimental to the other. As the objects we are combining are binary masks, the techniques developed by [18] for the case of model combination by weight averaging do not apply. We instead propose a greedy algorithm to merge a subset of 90% the matrices that do not decrease the F1 score for each vowel more than some threshold. Algorithm 1 sorts the computation graph topologically and visits every key exactly once, proposing to take the union of the two circuits’ masks for the given matrix, which we apply twice to the weight mask, then to the edge mask, only if both circuits’ binary F1 stays within ε of its independent (unmerged) performance. Figure 2c shows the performance is comparable to inde- pendent (unmerged) performance (Figure 2a). æəaʊaɪɝɛɪɔɪʊɑɔeɪ i oʊu æ ə aʊ aɪ ɝ ɛ ɪ ɔɪ ʊ ɑ ɔ eɪ i oʊ u 0.780.910.920.930.850.860.930.730.790.850.870.890.850.91 0.780.810.810.840.730.770.840.660.690.750.770.790.760.82 0.910.810.930.950.840.880.950.760.810.870.890.910.880.93 0.920.810.930.960.850.890.970.770.820.870.900.920.880.94 0.930.840.950.960.860.910.980.800.840.900.910.940.900.97 0.850.730.840.850.860.800.870.690.720.790.800.830.780.85 0.860.770.880.890.910.800.920.730.770.830.870.900.830.90 0.930.840.950.970.980.870.920.810.840.900.920.940.910.96 0.730.660.760.770.800.690.730.810.650.710.720.750.740.77 0.790.690.810.820.840.720.770.840.650.750.780.790.760.82 0.850.750.870.870.900.790.830.900.710.750.830.850.810.88 0.870.770.890.900.910.800.870.920.720.780.830.890.840.90 0.890.790.910.920.940.830.900.940.750.790.850.890.860.93 0.850.760.880.880.900.780.830.910.740.760.810.840.860.89 0.910.820.930.940.970.850.900.960.770.820.880.900.930.89 (a) Independent circuits æəaʊaɪɝɛɪɔɪʊɑɔeɪ i oʊu æ ə aʊ aɪ ɝ ɛ ɪ ɔɪ ʊ ɑ ɔ eɪ i oʊ u 0.590.480.630.330.380.560.290.490.570.600.670.400.700.59 0.590.350.450.650.340.600.560.460.450.330.590.780.410.52 0.480.350.330.390.310.750.450.340.420.310.470.720.520.48 0.630.450.330.390.310.600.780.340.430.450.320.730.400.69 0.330.650.390.390.330.560.820.740.470.380.590.730.460.63 0.380.340.310.310.330.390.440.300.310.470.310.390.560.45 0.560.600.750.600.560.390.750.610.580.520.310.690.500.53 0.290.560.450.780.820.440.750.700.440.350.520.490.550.60 0.490.460.340.340.740.300.610.700.630.660.370.700.620.63 0.570.450.420.430.470.310.580.440.630.520.620.630.330.63 0.600.330.310.450.380.470.520.350.660.520.540.710.380.47 0.670.590.470.320.590.310.310.520.370.620.540.530.360.60 0.400.780.720.730.730.390.690.490.700.630.710.530.680.82 0.700.410.520.400.460.560.500.550.620.330.380.360.680.59 0.590.520.480.690.630.450.530.600.630.630.470.600.820.59 (b) Naive union æəaʊaɪɝɛɪɔɪʊɑɔeɪ i oʊu æ ə aʊ aɪ ɝ ɛ ɪ ɔɪ ʊ ɑ ɔ eɪ i oʊ u 0.790.890.930.910.820.870.940.750.780.840.880.910.830.90 0.790.840.860.850.740.790.860.680.730.760.810.850.830.82 0.890.840.950.950.840.880.940.830.790.870.880.930.870.92 0.930.860.950.940.880.890.940.840.830.860.900.950.860.94 0.910.850.950.940.860.880.950.830.820.870.930.950.900.93 0.820.740.840.880.860.790.850.730.720.770.810.850.820.86 0.870.790.880.890.880.790.900.820.770.830.800.880.850.90 0.940.860.940.940.950.850.900.870.840.890.910.930.910.97 0.750.680.830.840.830.730.820.870.710.750.690.830.840.72 0.780.730.790.830.820.720.770.840.710.770.770.810.760.81 0.840.760.870.860.870.770.830.890.750.770.840.880.830.88 0.880.810.880.900.930.810.800.910.690.770.840.920.860.89 0.910.850.930.950.950.850.880.930.830.810.880.920.890.92 0.830.830.870.860.900.820.850.910.840.760.830.860.890.91 0.900.820.920.940.930.860.900.970.720.810.880.890.920.91 0.0 0.2 0.4 0.6 0.8 1.0 Macro F1 (c) Circuit Composition (ours) Fig. 2: Pairwise macro-F1 heatmaps across all 15 vowels under three composition strategies: (a) naive union, (b) unmerged union, and (c) merged circuits. Algorithm 1 Merge Clusters Require: circuitsC = C n N n=1 , each C n = M n (k) k∈K a set of|keys| matrices; thresholdsτ n N n=1 ; tolerance ε 1: K ← TOPOLOGICAL SORT(keys) 2: compute σ n (k)∀k ∈ K, ∀n∈ [N ] via Eq. 1 3: f 0 n ← MODEL FORWARD(C n ,∅,τ n ) ∀n∈ [N ] 4: S ←∅; Σ←∅ 5: for each k ∈ K do 6: S ′ ← S∪k 7: ˆ C ←C with masks unioned according to S ′ 8:Σ← σ i (k) 9: f n ← MODEL FORWARD( ˆ C n , Σ,τ n ) ∀n∈ [N ] 10:if f n ≥ f 0 n − ε for all n∈ [N ] then 11:S ← S ′ ▷ accepted 12:end if 13: end for 14: return S, b C n | S N n=1 , Σ| S Algorithm 2 Linear schedule 1: function LINEAR(C 1 , . . . , C n ) 2: T ← C 1 3:for v = 2 to n do T ← MERGECLUSTERS(T, C v ) 4:end for 5:return T 6: end function 5. TREE-MERGE SCHEDULES AND CIRCUITTREE DATA STRUCTURE Composing all 15 vowels (or 24 consonants) into a single module still raises two further questions that a single invo- cation of Algorithm 1 cannot answer on its own. Assuming that we combine circuits iteratively, by adding one phoneme at a time, in what order should circuits be combined? And, when each new phoneme is added, how do we carry forward information about sharing validated at previous iterations? The simplest strategy composes N phonemes one at a ScheduleAccuracy(%)Density(%) Unmerged82.31.13 Linear (Θ(kn 2 ))77.91.11 Balanced (Θ(kn logn))81.81.10 Kruskal merge82.21.10 HuBERT Frozen [6]86.4100 Table 1: 15-way vowel accuracy of the N -way merge sched- ules vs. Full HuBERT model, with post-merge circuit density. Algorithm 3 Balanced schedule 1: function BALANCED(C 1 , . . . , C n ) 2: Q← [C 1 , . . . , C n ] 3:while|Q| > 1 do 4:Q ′ ← [ ] 5:for i = 0, 2, 4, . . . while i+1 <|Q| do 6:Q ′ .append MERGECLUSTERS(Q[i], Q[i+1]) 7:end for 8:if|Q| odd then Q ′ .append(Q[last]) 9:end if▷ carried forward 10:Q← Q ′ 11:end while 12:return Q[0] 13: end function time (Algorithm 2). The trouble is that step i merges a sin- gle new member into a cluster that already holds i−1 mem- bers, so MERGECLUSTERS (Algorithm 1) must re-validate all i resulting members against every one of the k keys, costing ki MODELFORWARD calls. These costs accumulate over the whole chain: C lin = k n X i=2 i = k n(n+1) 2 − 1 = Θ(kn 2 ).(3) A more efficient strategy avoids ever growing one clus- ter far larger than its merge partner. Instead, singletons are merged into pairs, pairs into fours, and so on (Algorithm 3). At round r = 0,..., log 2 n−1, every cluster has size 2 r , and there are n/2 r+1 merges in that round, each invoking MERGECLUSTERS on 2· 2 r members at a cost of k · 2 r+1 MODELFORWARD calls. The result is that every round costs the same, kn, regardless of r: k· n 2 r+1 |z merges in round r ·2 r+1 | z members validated / merge = k n.(4) Summing this constant cost, kn, over the log 2 n rounds: C bal = k n log 2 n = Θ(kn logn).(5) Both linear and balanced share a limitation: both are blind schedules, choosing merge order purely from cluster size and never from how similar two phonemes actually are. We in- clude a third schedule as an upper bound on achievable per- formance. Here, the distance between two phonemes is de- fined by the merge ratio from §4, so that pairs sharing lit- tle of their masks are placed far apart. Kruskal’s algorithm is then applied repeatedly, always composing the two closest clusters that have not yet been joined. This schedule’s com- plexity ranges from Θ(kn logn) in the best case to Θ(kn 2 ) in the worst case. As Table 1 shows, the balanced schedule matches the performance of Kruskal merging while remain- ing Θ(kn logn), making it preferred in practice. 5.1. The CIRCUITTREE primitive MERGECLUSTERS operates on nodes of a CIRCUITTREE: a binary tree whose leaves hold a phoneme’s original circuit, and whose internal nodes record only which keys are shared among their two children’s members at that level (Figure 3). To see why this structure is necessary, consider the nat- ural alternative: re-union the raw masks of all current mem- bers every round, and fall back to fully private masks when any member’s F1 drops too far. This memoryless approach has a serious flaw. If a later round failed by an unrelated third phoneme on some key k, the naive approach discards all shar- ing on k, including sharing between two phonemes that had already validated it several rounds earlier. CIRCUITTREE avoids this by never forgetting a validated result. Instead of recomputing sharing from scratch, it re- solves each member’s effective mask for key k by walking down the tree and taking the first (i.e. closest-to-root) level at which that key was successfully shared, falling back to the member’s own leaf if it never was: RESOLVE(T ,m,k) = U T (k)k ∈ merged(T ) M m (k)at a leaf RESOLVE(org(T , m), m, k) else (6) Scale factors are accumulated along the same path. Because sharing is monotone non-decreasing along any root-to-leaf walk, a failed merge at the root can never undo a key that two phonemes already agreed to share three levels down. v 0 v 1 v 2 v 3 v 0 , v 1 sharesk 1 , k 3 v 2 , v 3 sharesk 1 , k 2 v 0 , v 1 , v 2 , v 3 sharesk 1 only Fig. 3: A CIRCUITTREE over four phonemes. k 1 validates at every level. k 3 only validated inside v 0 ,v 1 : the root-level merge failed for k 3 , but v 0 ,v 1 still use their shared value for it via Eq. 6, rather than reverting to private masks. Table 2: CV/VC-syllable transcription accuracy. SystemAccuracy (%) Merge Ratio (%) AllPartial Unmerged66.00.00.0 CIRCUITTREE64.830.399.5 6. HIERARCHY-OF-CIRCUITS TRANSCRIBER We now compose the merged vowel and consonant CIRCUIT- TREE modules into a syllable classifier. The pipeline has two stages. First, a separately-pruned DISCOGP circuit acts as a gate: it determines the syllable’s consonant/vowel order (e.g. CV vs. VC). Second, this order routes the input to the corre- sponding vowel or consonant CIRCUITTREE module, which resolves the specific phoneme. The two stages are combined by slotting the predicted phonemes into the given template. Table 2 contains our central result. Collapsing 39 in- dependent, binary phoneme circuits into just two shared CIRCUITTREE modules costs only 1.2 points of accuracy (66.0% → 64.8%), while 99.5% of keys are partially shared and 30.3% are shared across all phonemes. In other words, the vast majority of each phoneme’s circuitry is recovered from a common pool rather than stored independently, at a small and controlled cost to accuracy. 7. CONCLUSION This paper has demonstrated circuit composition at scale: a density-scaled, interference-aware merge algorithm, gen- eralized through CIRCUITTREE to compose many circuits at once. Composing primitive circuits for the vowels and consonants of English into a working transcriber shows that primitive circuits can be joined well beyond a handful, and into a capability none of them were individually pruned for. We believe this method is promising for much more complex problems, both in speech and in other domains. 8. COMPLIANCE WITH ETHICAL STANDARDS This study was conducted retrospectively using the Articula- tory Index corpus, a pre-existing, publicly available speech dataset distributed by the Linguistic Data Consortium. No new data involving human or animal subjects were collected as part of this work. As this research involved secondary anal- ysis of an existing, licensed open dataset rather than the col- lection of new human subject data, no additional ethical ap- proval was required. 9. REFERENCES [1] P. Mondorf, S. Wold, and B. Plank, “Circuit compo- sitions: Exploring modular structures in transformer- based language models,” in Proc. ACL, 2025. [2] Lei Yu et al., “Sheaf discovery with joint computa- tion graph pruning and flexible granularity,” in Proc. EMNLP, Nov. 2025, p. 8833–8848. [3] J. Merullo, C. Eickhoff, and E. Pavlick, “Circuit compo- nent reuse across tasks in transformer language models,” in Proc. ICLR, 2024. [4] Xi Chen et al., “All circuits lead to rome: Rethinking functional anisotropy in circuit and sheaf discovery for LLMs,” in Proc. ICML, 2026, Proceedings of Machine Learning Research. [5] Pratim Chowdhary,Peter Chin,and Deepernab Chakrabarty, “K-MSHC: Unmasking minimally suffi- cient head circuits in large language models with exper- iments on syntactic classification tasks,” arXiv preprint arXiv:2505.12268, 2025. [6] Wei et al., “Sparse weight and edge circuit discovery in transformer-based acoustic models,” IEEE Spoken Lan- guage Technology Workshop 2026, 2026. [7] Kevin Ro Wang et al., “Interpretability in the wild: A circuit for indirect object identification in GPT-2 small,” in Proc. ICLR, 2022. [8] Arthur Conmy et al., “Towards automated circuit dis- covery for mechanistic interpretability,” in NeurIPS. 2023, vol. 37, p. 16318–16352, Curran Associates Inc. [9] R. Pan and H. Rajan, “On decomposing a deep neural network into modules,” in Proc. ESEC/FSE, 2020, p. 889–900. [10] H. Kingetsu, K. Kobayashi, and T. Suzuki, “Neural net- work module decomposition and recomposition,” arXiv preprint arXiv:2112.13208, 2021. [11] D. Lindner, J. Kram ́ ar, S. Farquhar, M. Rahtz, T. Mc- Grath, and V. Mikulik, “Tracr: Compiled transformers as a laboratory for interpretability,” in NeurIPS, 2023. [12] C. Louizos, M. Welling, and D. P. Kingma, “Learning sparse neural networks through L 0 regularization,” in Proc. ICLR, 2018. [13] V. Sanh, T. Wolf, and A. M. Rush, “Movement pruning: Adaptive sparsity by fine-tuning,” in NeurIPS, 2020. [14] A. Bhaskar, A. Wettig, D. Friedman, and D. Chen, “Finding transformer circuits with edge pruning,” in NeurIPS, 2024. [15] Thomas Schatz et al.,“Articulation Index LSCP (LDC2015S12),” Linguistic Data Consortium, 2015. [16] P. Fousek, P. Svojanovsky, F. Grezl, and H. Hermansky, “New nonsense syllables database analyses and prelimi- nary asr experiments,” in Proc. ICSLP, 2004, p. 2004– 29. [17] Wei-Ning Hsu et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE TASLP, vol. 29, p. 3451–3460, 2021. [18] P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “TIES-merging: Resolving interference when merging models,” in NeurIPS, 2023.