Paper deep dive
Circuit Claims Depend on What Is Extracted and How It Is Compared
Yang Sheng, Jie Fu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/22/2026, 3:15:37 AM
Summary
This paper argues that circuit extraction claims are under-determined because they depend heavily on methodological choices regarding which circuit object is reported (e.g., core vs. extended graph) and how circuits are compared (e.g., exact edge overlap vs. attention head sets). Using a synthetic Lean tactic-prediction benchmark, the authors demonstrate that while exact component-to-component edge overlap is low and sensitive to these choices, coarser summaries like selected attention head sets remain stable. They further find that accuracy gains from reinforcement learning on compositional proofs correlate with the addition of structural components beyond atomic circuits. The paper concludes by proposing a standardized reporting practice for circuit-extraction studies.
Entities (8)
Relation Signals (6)
Circuit Extraction â dependson â Reporting Choices
confidence 95% ¡ the claim it supports depends on which circuit is reported and how two circuits are compared
Lean Tactic-Prediction Benchmark â usedfor â Circuit Extraction Stability Analysis
confidence 95% ¡ We make this concrete in a synthetic Lean tactic-prediction benchmark
Exact Edge Overlap â issensitiveto â Extraction Choices
confidence 92% ¡ Exact component-to-component edge overlap is low and sensitive to these choices
Attention Head Set Overlap â isstableacross â Extraction Choices
confidence 90% ¡ two coarser summaries stay stable: the set of selected attention heads
RL Accuracy Gains â correlateswith â Circuit Structure Complexity
confidence 88% ¡ The largest accuracy gains from RL on compositional proofs come with the most structure beyond the atomic circuits
Query/Key Representation â affects â Graph Size
confidence 85% ¡ representing each attention head with separate query and key nodes rather than one merged node... changes graph sizes only slightly
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior. We argue that this reading is under-determined: preserving behavior does not single out one circuit, because the claim it supports depends on which circuit is reported and how two circuits are compared. We make this concrete in a synthetic Lean tactic-prediction benchmark -- predicting the next step of a proof -- where fixed proof rules with randomized surface form let differences between extracted circuits be attributed to these choices rather than to the task. Across dense and weight-sparse checkpoints (most weights constrained to zero) of the same transformer, evaluated on atomic (single-rule) and compositional (multi-rule) proofs, we vary which extracted object is reported (a compact prediction-preserving circuit, a broader graph that also keeps surrounding read, write, and routing structure, or the smallest subgraph meeting a post-ablation loss threshold), and whether each attention head's query and key are represented jointly or separately. Exact component-to-component edge overlap is low and sensitive to these choices, at times dropping to a random baseline, while two coarser summaries stay stable: the set of selected attention heads, and the circuit-size ranking of conditions that differ in which supervised checkpoint initializes reinforcement learning (RL). The largest accuracy gains from RL on compositional proofs come with the most structure beyond the atomic circuits. A circuit-level claim is therefore well defined only once one states which circuit is reported, the pruning threshold used to extract it, and the level at which circuits are compared. We distill these requirements into a reporting practice for circuit-extraction studies.
Tags
Links
- Source: https://arxiv.org/abs/2607.18921v1
- Canonical: https://arxiv.org/abs/2607.18921v1
Trouble viewing inline? Open PDF directly â
Full Text
91,453 characters extracted from source content.
Expand or collapse full text
Circuit Claims Depend on What Is Extracted and How It Is Compared Yang Sheng 1,2 Jie Fu 3 1 Fudan University 2 Shanghai Innovation Institute 3 IQuest Research Correspondence: yann.sheng123@gmail.com; jie.fu@iquestlab.com Code:§ github.com/Stepuuu/circuit-extraction-stability Abstract Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior. We argue that this reading is under- determined: preserving behavior does not single out one circuit, because the claim it supports depends on which circuit is reported and how two circuits are compared. We make this concrete in a synthetic Lean tactic-prediction benchmarkâpredicting the next step of a proofâwhere fixed proof rules with randomized surface form let differences between extracted circuits be attributed to these choices rather than to the task. Across checkpoints spanning dense and several weight-sparse levels (most weights constrained to zero) of the same transformer, evaluated on atomic (single- rule) and compositional (multi-rule) proofs, we vary which extracted object is reported (a compact prediction-preserving circuit, a broader graph that also retains the surrounding read, write, and routing structure, or the smallest subgraph whose post-ablation loss stays under a chosen threshold), and whether each attention headâs query and key are represented jointly or separately. Some descriptions stay stable under this variation while others change: the overlap of exact, component- to-component edges is low and sensitive to these choices, at times dropping to the level of a random baseline, while two coarser summaries stay stableâthe set of selected attention heads, and the circuit-size ranking of conditions that differ in which supervised checkpoint initializes reinforcement learning (RL). Across those same conditions, the largest exact-match accuracy gains from RL on compositional proofs come with the most structure beyond the atomic circuits. A circuit-level claim is therefore well defined only once one states which circuit is reported, the pruning threshold used to extract it, and the level at which circuits are compared; otherwise these unstated choices, not behavior alone, decide it. We distill these requirements into a reporting practice for circuit-extraction studies. 1 Introduction Understanding which parts of a neural network implement a behavior is a central goal of mechanistic interpretability [Elhage et al., 2021]. One common operationalization is circuit extraction: identifying a small set of components whose presence is enough to preserve or explain a target behavior under ablation [De Cao et al., 2022, Bhaskar et al., 2024, Yu et al., 2025]. Circuit analyses have produced concrete explanations in language models, synthetic algorithmic settings, and sparse feature representations [Wang et al., 2023, Conmy et al., 2023, Marks et al., 2025, Dunefsky et al., 2024]. However, every circuit analysis is shaped by multiple methodological decisions: which extracted graph is reported (a compact prediction-preserving circuit versus a broader graph that also retains surrounding read, write, or routing structure), which ablation or reference distribution defines behavior preservation, whether each attention head is represented by one merged query/key node or by separate query and key nodes, and whether two circuits are compared as exact component-to-component edge Preprint. arXiv:2607.18921v1 [cs.LG] 21 Jul 2026 A Dense/sparse: which description is reproducible? Heads exceed baseline; exact edges do not consistently. Attention-head set Jaccard@10 success Attention-head set Jaccard@10 near-miss Exact-edge Jaccard@10 success Exact-edge Jaccard@10 near-miss B Q/K + loss budget: do rankings change? All three graphs preserve the RL-condition ordering. Core circuit Support graph Fixed-loss pruning RL-condition ordering 3/3 max gap ⤠0.055 C Behavior recovery: what else is recruited? Largest gains are accompanied by the largest bridge fraction. 0.00.20.40.60.81.0 Jaccard@10 0.666 0.554 0.163 0.142 open diamond = random top-k baseline 0.000.020.040.06 Q/K node-fraction gap 0.008 / 0.010 0.003 / 0.013 0.015 / 0.024 0102030 compositional gain (percentage points) 0.0 0.1 0.2 0.3 bridge fraction atomic-only composition-only circle = moderatesquare = extensive Curriculum moderate +24.64 gain bridge fraction 0.343 Curriculum extensive +31.27 gain bridge fraction 0.161 Figure 1: Overview of the three findings. A: at the primary dense-versus-75%-sparse comparison point, the set of selected attention heads overlaps much more across the dense and75%weight-sparse RL checkpoints than the exact component-to- component edge lists do; the head-level overlap is also far above a random top-kbaseline, while the exact-edge overlap is not consistently above it (Table 18 extends this to other sparse anchors and matched-sparsity cross-seed pairs). B: representing each attention head with separate query and key nodes rather than one merged node, with each graphâs selection rule held fixed, leaves unchanged how the three graphs rank the RL initialization conditions by graph size; the same ranking holds across the tested range of loss thresholds for the pruning graph. C: within each expanded-distribution family, the RL condition with the largest compositional-task accuracy gain also has the largest fraction of compositional-task circuit nodes outside the matched atomic-task circuits; held-out surface and depth shifts limit the scope. lists or as coarser attention-head sets. Some of these decisions are discussed explicitly in prior work; others are typically implicit. These choices can change whether two checkpoints appear to share a circuit. For example, in our dense-versus-sparse extractions, the same checkpoint pair has much higher overlap when circuits are compared as sets of selected attention heads than as exact edge lists (Figure 1A; quantified in Section 4.1). In many natural-language tasks, the effect of these choices is hard to separate from genuine differences in mechanism: the behavior can mix lexical cues, syntax, surface format, and several latent subskills, and the atomic pieces are not specified by construction. A change under another ablation rule, reference distribution, or graph-selection rule may reflect either mechanism or surface cue. In indirect-object identification [Wang et al., 2023], for instance, the corrupted reference is itself a design choiceâone corruption might swap the two names, another might alter the sentence templateâ and different choices can make different heads appear necessary, with behavior alone unable to say whether the difference is mechanistic or an artifact of what each reference leaves untouched. Recent validation work shows that such choices can change faithfulness scores and reported circuit claims in practice [Miller et al., 2024, Shi et al., 2024, Tigges et al., 2024]. The question is therefore not only whether a circuit preserves behavior, but how robust that claim is to the choice of which circuit is reported and how circuits are compared. We study this problem in a synthetic, rule-generated Lean tactic-prediction benchmark. Each input is a short proof state (available hypotheses plus a goal), and the target label is the next tactic, a Lean command that advances the proof. In the examples below,h p : pmeans that hypothesish p proves propositionp(and declarations such asp,q,d : Propdeclare propositions);â˘separates the available hypotheses from the goal to prove: Atomic AND p,q,d : Prop; h p : p, h q : q, h d : d⢠p⧠q Target tactic: exact And.intro h_p h_q Compositional AND-OR p,q,r,d : Prop; h p : p, h q : q, h d : d⢠(p⧠q)⨠r Target tactic: exact Or.inl (And.intro h_p h_q) The atomic AND example asks the model to provepâ§qfrom proofs ofpandq; the target tactic builds that conjunction withAnd.intro. The compositional AND-OR example asks for(pâ§q)â¨r: because no proof ofris provided, the target tactic chooses the left side of the disjunction withOr.inland then uses the same conjunction construction inside it. The premiseh d : dis an unused distractor. The atomic and compositional task structures are known by construction, and surface names, distractors, and sampled examples are randomized while the proof rules remain fixed. This controlled setting lets us vary the reported circuit and the way circuits are compared without confounding from changes in 2 the task family. We vary four factors in this setting: (i) model sparsity, (i) whether each attention head is represented with a single merged node for query and key or with separate query and key nodes, (i) the allowed post-ablation loss for the pruning graph, and (iv) which supervised checkpoint initializes reinforcement learning (RL). Figure 1 summarizes the three comparisons, and Table 1 maps each claim to its metric and evidence. Two of these findings concern stabilityâwhich conclusions hold up across the extraction and comparison choicesâand the third connects circuit structure to behavior. (1) Dense and75%weight- sparse checkpoints have substantial overlap in their selected attention-head sets but only weak overlap in their exact component-to-component edge lists. Whether two checkpoints appear to share a circuit therefore depends on whether the comparison is made at the attention-head level or the exact-edge level. (2) Representing each attention head with separate query and key nodes rather than a single merged one changes graph sizes only slightly; for the pruning graph, the same ordering of RL initialization conditions also persists across the tested loss-threshold range. (3) Finally, across RL runs initialized from different supervised checkpoints, the runs with the largest accuracy gains on compositional tasks also have the largest fraction of compositional-task circuit nodes that are absent from the matched atomic-task circuitsâstructure the circuit adds beyond reusing the atomic-task circuits. Section 4 gives the full evidence. 2 Related Work Mechanistic interpretability has developed residual-stream circuit formalisms [Elhage et al., 2021], targeted case studies [Wang et al., 2023], automated discovery pipelines [Conmy et al., 2023], path patching [Goldowsky-Dill et al., 2023], scalable attribution localization [KramĂĄr et al., 2024], and learned feature circuits [Dunefsky et al., 2024]. Closest to the question here, causal-abstraction work shows that the same network can admit multiple valid higher-level descriptions [Geiger et al., 2025], faithfulness evaluations show sensitivity to validation and extraction choice [Miller et al., 2024, Shi et al., 2024], and circuit-stability work finds that algorithms and component types can persist across training and scale even when individual attention-head implementations change [Tigges et al., 2024]. Recent work argues that a single task in an LLM can be supported by multiple structurally distinct faithful circuits or sheaves, using an overlap penalty to discover low-overlap solutions [Chen et al., 2026]. Fine-tuning studies provide a related positive case in which entity-tracking circuits persist across training stages [Prakash et al., 2024]. These strands describe one underlying phenomenon under several names: the non-robustness of faithfulness metrics and circuit estimates to ablation, validation, or pipeline choices [Miller et al., 2024, Shi et al., 2024, MĂŠloux et al., 2025b]; the non-identifiability of circuits that equally preserve a target behavior [Geiger et al., 2025, Chen et al., 2026, MĂŠloux et al., 2025a]; disagreement among the explanations recovered for one model [Krishna et al., 2022]; and, conversely, the consistency of coarser circuit structure across training and scale [Tigges et al., 2024]. Those studies ask whether mechanisms persist across training, scale, or fine-tuning, or whether alternative faithful mechanisms can be found; our work asks a different question: when benchmark and checkpoints are held fixed, which claims about the extracted circuitâfor example, whether two checkpoints share a circuit, or whether one RL condition selects more bridge componentsâchange when the extracted graph changes, when query/key attention support is merged or separated, when the post-ablation loss threshold changes, or when the comparison level changes. Our experiments also draw on sparse-model and pruning-based interpretability. Differentiable masking and pruning methods can isolate small causal subsets of model components [De Cao et al., 2022, Bhaskar et al., 2024, Yu et al., 2025], while benchmark environments such as Tracr, InterpBench, and the Mechanistic Interpretability Benchmark (MIB) make exact recovery and faithfulness easier to test [Lindner et al., 2023, Gupta et al., 2024, Mueller et al., 2025]. Recent sparse-model and sparse-feature-circuit work suggests that sparsity in model structure or learned representations can make mechanisms more inspectable [Gao et al., 2025, Draye et al., 2025, Marks et al., 2025]. For the compositional comparison, Mondorf et al. [2025] compare circuits for modular string-edit operations and test reuse/composition through circuit set operations; by contrast, our experiments vary which graph is extracted and whether query and key sides of attention are merged or separated while checkpoints stay fixed. 3 Controlled compositional benchmarks motivate the benchmark design. Synthetic compositional- generalization benchmarks test whether learned behaviors transfer across rule-composed inputs [Lake and Baroni, 2018, Keysers et al., 2020, Kim and Linzen, 2020, Ruis et al., 2020, Hupkes et al., 2020, Clark et al., 2020, Tafjord et al., 2021, Yang et al., 2023], and behavioral testing work shows how surface-form controls can change evaluation conclusions [Ribeiro et al., 2020, Wu et al., 2023]. Our synthetic Lean benchmark borrows this controlled-generation principle but applies it to circuit extraction rather than behavioral evaluation: the proof-state rules and the atomic-versus-compositional decomposition are fixed and known by construction, while surface form and reasoning depth vary. This known decomposition is what lets us attribute differences between extracted circuits to the extraction and comparison choices rather than to uncontrolled changes in the taskâthe confound that makes such attribution hard in natural-language settings. 3 Experimental Setup 3.1 Benchmark and Training Distributions We construct a benchmark of nine synthetic Lean tactic-prediction tasks: four atomic and five compositional (Table 6). Each example is generated dynamically from fixed propositional proof rules, with randomized proposition and hypothesis names plus distractor premises, and the model predicts the first tactic that advances the proof state. As a concrete instance, an atomic example might ask the model to provep⧠qfromh p : p,h q : q, and a distractorh d : d, where the target tactic is exact And.intro h_p h_q; a compositional example might ask the model to prove(p⧠q)⨠r from the same three hypotheses, where the target tactic isexact Or.inl (And.intro h_p h_q). The compositional goal introduces a new propositionrfor which no hypothesis is given, so the proof must use the left disjunct. The synthetic Lean family is chosen for the three properties that a controlled circuit comparison requires. First, the proof-state generation rules are fixed, so atomic and compositional structures vary independently of surface form. Second, randomized names, distractors, and dynamically generated examples reduce fixed-string memorization, though the boundary tests below show that shared format cues can still matter. Third, because examples are dynamically generated rather than drawn from a fixed test set, a single test draw could over- or under-state accuracy by chance; therefore, we report behavioral results on two independently sampled test sets of100examples per task, labeled Suite 1 and Suite 2. To obtain supervised checkpoints that differ only in the breadth of their training distribution, we train on a narrow baseline distribution and two expanded distributions that share the same proof-state generation rules while varying surface form. The moderately expanded distribution adds distractor- heavy, goal-first, and long-name variants; the extensively expanded distribution additionally includes compact and relabeled formats. These checkpoints become the initialization conditions for the RL comparison, and because only the surface form changes while the proof-state rules stay fixed, differences across themâand across the circuits extracted from themâreflect training-distribution breadth rather than a change in the task. This generator design combines controlled compositional generation with held-out surface variation [Lake and Baroni, 2018, Keysers et al., 2020, Kim and Linzen, 2020, Ruis et al., 2020, Hupkes et al., 2020, Ribeiro et al., 2020, Wu et al., 2023] and the use of controlled benchmarks for interpretability evaluation [Lindner et al., 2023, Gupta et al., 2024, Mueller et al., 2025]. We instantiate these ideas in a proof-oriented setting related to natural-language proof generation and Lean theorem proving [Tafjord et al., 2021, Yang et al., 2023]. We use Lean tactics as supervision targets because the label is an exact tactic string, giving an unambiguous supervision signal and a hard exact-match metric for both behavioral evaluation and ablation-based circuit selection. Because examples are generated dynamically, supervised data volume is specified by token budget rather than by a fixed corpus size: each supervised run consumes5B tokens drawn from one of these distributions, and each RL run uses 500composition-only prompts per task with8generations per prompt over10epochs (full settings in Table 3). 4 Dense25%50%75%90% Weight sparsity 60 70 80 90 100 Mean accuracy (%) Dense anchor75% anchor Supervised Reinforcement-refined (A) Baseline accuracy across sparsity levels. Dense25%50%75%90% Weight sparsity 50 55 60 65 70 75 Mean accuracy (%) 75% sparsity focus Baseline narrow Moderately expanded Extensively expanded (B) Supervised accuracy after distribution expansion. Figure 2: Supervised accuracy across sparsity levels. A: the baseline training distribution across dense and four weight-sparse regimes; the highest-accuracy checkpoint and the checkpoint used for detailed graph extraction need not coincide. B: at the same sparsity points, expanded training distributions reduce supervised compositional accuracy, especially at the 75% weight-sparse graph-extraction checkpoint. 3.2 Model Families and Training Stages We train weight-sparse models for two reasons: weight sparsity yields smaller, more inspectable circuits, which makes circuit extraction and comparison tractable [Gao et al., 2025], and the level of sparsity is itself one of the factors whose effect on circuit claims we measure. Our weight-sparse training follows the top-kconvention of that work:x%weight-sparse meansx%of weights are constrained to zero during training. All experiments use a single 8-layer transformer architecture (d model = 2048,n head = 128,d head = 16,d mlp = 8192, context length128,âź 415.7M parameters), trained on our Lean tactic-prediction benchmark at five sparsity levels: dense, 25%, 50%, 75%, and 90%weight-sparse. This range is large enough to learn the task family across dense, weight-sparse, supervised, and reinforcement learning (RL) settings, yet small enough for repeated extraction, ablation, and seed/sparsity comparisons. Three independent seeds are trained at every sparsity level. Our experiments are organized as a grid that crosses the three training-data families introduced aboveâ baseline, moderately expanded, and extensively expandedâwith three reinforcement-learning (RL) initialization conditions. The three RL initialization conditions in each expanded-distribution family differ only in their upstream supervised checkpoint: Curriculum starts from the full curriculum-trained checkpoint, Atomic-only starts from a checkpoint trained on atomic tasks only, and Composition-only starts from a checkpoint trained on compositional tasks only. The prompt mix used for Group Relative Policy Optimization (GRPO; Shao et al., 2024) remains composition-only across all three initialization conditions; the condition labels refer to upstream checkpoint identity, not to the RL prompt distribution. Detailed evidence coverage is in Table 4. Behavioral comparisons over the full sparsity sweep and detailed graph-level analyses use different parts of the grid. The full-sweep behavioral comparisons use all five sparsity levels, whereas exact- edge and extracted-graph analyses use the dense checkpoint and the checkpoint at75%weight sparsity as fixed comparison points. Dense serves as the unpruned reference;75%weight sparsity is a pre-specified graph-extraction point, chosen to balance graph readability, object size, and reuse of one dense-vs-weight-sparse comparison across graph-level analyses. Figure 2 situates these checkpoints within the full five-sparsity behavioral sweep:75%maintains strong supervised and RL accuracy in the baseline family, while expanded training distributions reduce supervised compositional accuracy at the same sparsity points. Apart from this checkpoint choice, training and checkpoint-selection protocols are held fixed across the compared families: supervised runs share one 5B-token curriculum recipe, and the narrow baseline RL family and all expanded-distribution RL initialization conditions share the same GRPO refinement and model-selection rule [Shao et al., 2024]. Table 3 gives the shared training protocol. 3.3 Circuit Objects, Query/Key Support, and Comparisons We report three extracted circuit objects: the core circuit, a compact prediction-preserving subgraph; the extended support graph, a wider graph that also keeps surrounding read, write, and routing components; and the fixed-loss pruning graph, the smallest subgraph whose ablated loss stays below a target threshold. They differ in which components they keep and how they are selected, so that the same sensitivity question can be checked across multiple extracted graphs. The objects combine 5 two extraction criteria from prior work with a broader reporting view, and share a common node vocabularyâresidual-stream reads and writes, attention query/key/value/write channels, and MLP read, neuron, and write componentsâadapted from the residual-stream circuit framing of Elhage et al. [2021] and the fine-grained node families used in Gao et al. [2025]. Throughout, the support of a node denotes the set of upstream components in this extraction graph whose retained activations feed that nodeâfor example, the residual-stream reads that feed an attention headâs query, key, or value channel at a given token position, or the MLP and attention writes that carry information back into the residual stream. An extracted circuit is the pair of selected nodes and the support edges linking them. Below, a change in how the circuit is extracted means changing one of selection rule, criterion, or query/key node choice while holding the checkpoint, task set, and relevant loss threshold fixed. Core circuit. The core circuit is a compact extracted subgraph selected because, after the other candidate components are ablated, the remaining components still preserve the target tactic prediction. This object instantiates the prediction-preserving learned-mask extraction line of De Cao et al. [2022], Bhaskar et al. [2024], Yu et al. [2025] over the node vocabulary above. We adapt this line in two places: (i) the primary selection criterion is exact-match target-tactic accuracy after ablation, with the masked cross-entropy plusÎť â 0 penalty used only as a tiebreaker; (i) the mask objective is supervised on Lean tactic tokens. Concrete hyperparameters are in Table 5. Extended support graph. The extended support graph is built on the same learned-mask machinery, node vocabulary, and prediction-preserving extraction line as the core circuit [De Cao et al., 2022, Bhaskar et al., 2024, Yu et al., 2025], but reports a wider selected graph: additional read, write, and routing components around the same prediction-preserving behavior. This wider graph is a deliberately broader reporting view rather than a separate extraction criterion; it tests whether claims that hold for the compact core also hold when surrounding support structure is included. Fixed-loss pruning graph. The loss-constrained pruning subgraph instantiates the loss-budget circuit-selection line of Conmy et al. [2023], Bhaskar et al. [2024], and is the closest analogue in our setup of the task-specific pruning step in Gao et al. [2025]. After optimizing a continuous top-k pruning-score objective, the procedure selects the smallest discrete circuit whose ablated loss stays below a target thresholdĎ, i.e.k â = mink :L abl (C k )⤠Ď. We adapt this line with a two-stage continuous-then-discrete top-kselection for our Lean tactic-prediction loss. For readability, this object is called the fixed-loss pruning graph below. The main coupled-vs-factorized comparison uses Ď = 0.12; a loss-budget sweep applies the fitted pruning-score ranking at additionalĎvalues to test whether the ordering conclusion depends on this pruning threshold. In addition to the three objects, we compare two ways to represent query-side and key-side support for attention heads. Prior circuit-extraction and localization methods choose a graph granularity before searching or patching circuits [Conmy et al., 2023, Goldowsky-Dill et al., 2023, Bhaskar et al., 2024]; here we hold the extraction settings fixed and vary only this query/key node choice. For an attention headHat layerâ, its query-side support is the set of upstream components feedingHâs query projection at the target token position, and its key-side support is the analogous set feeding Hâs key projection (typically read from earlier positions). Coupled Q/K, the merged query/key representation, uses a sharedattn_qknode family for query-side and key-side support. Factorized Q/K, the separated representation, keeps them as separateattn_qandattn_knode families, while keeping the read, value, write, MLP-read, MLP-neuron, and MLP-write node families explicit. The set of possible nodes and edges can therefore change even when the same coarse read-to-write route is present. For the core circuit and extended support graph, the learned-mask optimization recipe and the selection rule are held fixed across the coupled and factorized versions. Table 5 summarizes the concrete extraction recipes, selection criteria, and loss or accuracy criteria. How claims are compared. Each comparison tracks whether the qualitative takeaway stays the same after the circuit object or query/key support choice changes: whether the ordering or sign stays the same, for example, whether one RL initialization condition still has the larger node fraction, or whether overlap between attention-head sets still exceeds exact-edge overlap. Comparison rule. Comparison is object-specific. The three circuit objects differ in node types, edge types, and extraction objectives by design, so claims are compared separately for each object. 6 3.4 Metrics and Statistics Four classes of metrics support the comparisons. Behavior. Behavioral accuracy reports overall accuracy on both evaluation suites together with atomic-task and compositional-task means, following the exact-match accuracy used in compositional- generalization and proof-prediction benchmarks [Lake and Baroni, 2018, Hupkes et al., 2020, Yang et al., 2023]. When a single summary is useful, the mean accuracy is Acc mean = 1 2 Acc suite1 + Acc suite2 . Exact-edge overlap and routing-head overlap. We use the standard Jaccard index for set overlap [Jaccard, 1901], which recent interpretability work also uses to compare discovered circuits [Tigges et al., 2024, MĂŠloux et al., 2025b]: the size of the intersection divided by the size of the union, so0 means no shared selected items and1means identical selected sets. In our Jaccard@10 summaries, each set is first restricted to the top10selected entries for that metric. Exact-edge overlap is computed from paired dense/weight-sparse comparisons over16matched task-object entries in each split. For relation type r âstructural, semantic, routing, J (r) k (A,B) = |Top (r) k (A)⊠Top (r) k (B)| |Top (r) k (A)⪠Top (r) k (B)| , withk = 10throughout. To form the structural and semantic relations, we map each active support edge to a bundle ID before taking top-ksets. A structural bundle ID groups active support edges by layer, source membership category, target membership category, and source/target node family. A membership category records how a selected component relates to the atomic circuits for the same compositional task: for example, whether it is shared with an atomic circuit, reused from an atomic circuit, or selected mainly in the compositional circuit. Semantic overlap coarsens those source/target categories into shared-with-atomic, reused-atomic, compositional-only, and other classes before computing Jaccard. The routing relation is separate: a routing-head set is the set of attention heads selected by the top-krouting edges, grouping by layer and head ID before computing Jaccard. Semantic overlap is reported for completeness; in our experiments it is numerically indistinguishable from structural overlap. The exact-edge comparison uses the same pre-specified dense and75%weight-sparse RL checkpoints, matching entries by task, extracted object, and selected prompt. The success split contains entries whose selected prompt is solved by both compared checkpoints. The near-miss split is drawn from non-success prompts that are closest to the correct tactic output under a tactic-aware token and exact-match score (Appendix C). The reportedn = 16per split counts the matched task-object entries for which both endpoints have scored structural and routing bundles. Node fractions and extraction gaps. Compositional-task node fraction is the mean selected-node count divided by the full object-specific node universe over the five compositional tasks, with the universe defined by the given object and query/key support choice. The coupled-vs-factorized Q/K fraction gap compares the two versions of the same object within the same family-condition cell, â frac = |f coup â f fact |. Because each fraction is normalized by that objectâs own node universe, these are within-object summaries rather than common-scale effect sizes across objects. Motivated by circuit-set comparisons of compositional circuits [Mondorf et al., 2025], we define bridge fraction per task asBridgeFrac(t) = C t \ S i C (i) t /|C t | , whereC t is the circuit extracted from the composite task andC (i) t are the corresponding atomic-task circuits. The reported bridge fraction is the mean of BridgeFrac(t) over the five compositional tasks. Aggregation, uncertainty, and controls. Full sparsity sweeps report means across three seeds and visualize per-seed scatter. For exact-edge overlap, 95% bootstrap confidence intervals [Efron, 1979]âwhich recent work recommends for reporting uncertainty on circuit estimates [MĂŠloux et al., 2025b]âare computed over the 16 matched task-object entries in each split. The RL initialization- condition plots aggregate means over five sparsity levels and three seeds per family-condition cell. We use three negative-control conditions, aggregated over the same five compositional tasks: matched atomic keeps the union of the atomic-task circuits that define the compositional task, mismatched atomic keeps a low-overlap atomic union from the same checkpoint, and random averages size-, layer-, and layer-family-matched random controls. 7 Table 1: Summary of the main findings. Metric definitions are given in Section 3.4; full numerical values, confidence intervals, random-baseline checks, and controls are reported in the cited figures and tables. Node- fraction gaps are within-object summaries; Ď = 0.12 is the main loss-budget point. FindingKey resultEvidenceInterpretation / scope Routing-head sets overlap more than exact edges Routing-head overlap is high on both success and near-miss splits (Jaccard@10 0.67 and 0.55), while exact-edge overlap stays low (0.16 and 0.14). Random-baseline checks are in the cited tables. Figure 3; Tables 11, 12, 18 Dense vs.75%weight-sparse RL comparison; near-miss exact-edge overlap is not detectably above random (p = 0.106). Object-level summaries preserve RL ordering At Ď = 0.12, curriculum coupled-vs-factorized node-fraction gaps are at most 0.024 (fixed-loss pruning graph, extensively expanded family) across the three objects; all pairwise RL-condition orderings agree (3/3) across objects, families, and tested loss budgets. Figure 5; Tables 13, 14, 15, 16 Object-level selected-node summaries, not exact-edge identity; sweep checks pruning- threshold dependence. Compositional RL gains track bridge fraction Curriculum RL improves compositional accuracy by+24.64points in the moderately expanded family and +31.27 points in the extensively expanded family. The same two curriculum cells have the largest core bridge fractions (0.343 and 0.161), while other RL conditions are at most 0.054. Figure 6; Tables 10, 13 Atomic-only RL gains⤠6points and composition-only gains14â18points; bridge fraction is a directional association driven by the two curriculum cells, not a causal claim. 0.00.20.40.60.8 Jaccard@10 Routing-head Jaccard Exact-edge Jaccard 0.67 0.55 0.16 0.14 successnear-miss Figure 3: Routing-head sets are more reproducible than exact edge lists under the same extraction method for dense and75%weight-sparse RL checkpoints. The head-level summary groups top routing edges by attention- head identity before computing Jaccard@10. Intervals are 95% bootstrap intervals over matched task-object entries; semantic Jaccard@10 matches structural Jaccard@10 in this comparison and is reported in Table 11. 4 Results We report two controlled comparisons of stability and a third that asks whether the resulting circuits relate to behavior. First, we compare dense and weight-sparse checkpoints under the same extraction method to identify which summary of the extracted graph is reproducible. Second, we represent the query and key sides of attention separately rather than merged, revealing which conclusions survive that split. Finally, we relate compositional-task RL gains across initialization conditions to core-circuit bridge fraction. Table 1 gives the compact quantitative summary. 4.1 Routing-Head Sets Are More Reproducible Than Exact Edges With the extraction method fixed, stability is visible at the coarser attention-head level rather than in the exact edge list. Figure 3 shows that exact edge lists overlap weakly across dense and weight-sparse checkpoints, but the attention heads selected by top routing edges overlap substantially more. Figure 3 reports the pre-specified dense-versus-75%-sparse comparison used throughout this paper for graph- level analysis; Table 18 extends it to other sparse anchors and matched-sparsity cross-seed pairs. On n = 16matched task-object entries in each split, structural Jaccard@10 is only0.163on the success split and0.142on the near-miss split, and semantic coarsening is numerically indistinguishable in this comparison (Table 11). Grouping top routing edges by attention head raises Jaccard@10 to0.666on the success split and0.554on the near-miss split, with clearly separated bootstrap intervals. Against random top-kdraws from the observed candidate set for each metric, the head-level Jaccard@10 remains far above chance (0.666vs.0.199on the success split,0.554vs.0.185on the near-miss split; Table 12). On the near-miss split, exact-edge (structural) Jaccard@10 is statistically indistinguishable from chance (0.142vs. a0.117random top-kbaseline,p = 0.106), whereas head-level overlap clearly exceeds its own baseline. The reproducible signal across paired checkpoint comparisons 8 Q K Ă V V V Ă / / / Q K Ă V V V V Ă / / / state_0: h A : A h B : B ⢠(A AND B) OR C attn read : 7.15 Q111.10 +0.05 K99.14 +0.10 dense route : Q72.14 / K99.0 channel râv vâw V43.3 +0.017 +0.604 V104.11 â0.048 â0.459 V115.9 â0.034 â0.540 V94.14 â0.019 â0.561 dense route : V111.7 / V99.8 / V17.11 writes Or.inl (And.intro h A h B ) mw7.15 â N7418, N1720, N1918 (MLP neurons: â0.58, â0.59, â0.60) dense route : N6770 / N4670 / N1549 h A : Ah B : B (A AND B) OR C Or.inl And.intro 1 Late Goal-State Read 2 QueryâKey Selector 3 Value Mixture 4 Late Lexicalizer Figure 4: Representative75%weight-sparse RL case for AND-OR. Arrows denote the direction of information flow. The recurring pattern is a coarse read-to-write path: a late-position readâa compact query/key selector (the attention pattern that decides what to attend to)âa small group of value-vector components carrying signed contributionsâthe attention write back into the residual streamâdownstream MLP components that raise the target tactic-token logits. What recurs across the dense and weight-sparse checkpoints is this functional path, not an exact edge-by-edge match. is therefore a coarser set of attention heads selected by the routing edges, not the literal edge list. Matched-sparsity cross-seed comparisons provide additional support (Table 18). Takeaway. Dense and75%weight-sparse checkpoints overlap strongly in routing-head sets (Jaccard@100.666on the success split and0.554on the near-miss split), but only weakly in exact component-to-component edge lists (0.163 and 0.142). Figure 4 shows a representative75%weight-sparse RL AND-OR success case from this paired comparison, with the paired dense/weight-sparse node-level case in Figure 11. The shared structure is a recurring read/write path: a compact query-key selector routes a small value-vector component group intoattn_write:7,15, and downstream MLP components increase the final tactic-token logits for Or.inl and And.intro. Aggregate evidence comes from Figure 3 and the supplement. 4.2 Across Query/Key Choices and Loss Budgets, the Three Graphs Preserve the Same Orderings The second comparison holds the checkpoint condition fixed and varies two extraction choices in turn: whether the query/key representation is coupled or factorized, and the post-ablation loss budget that selects the fixed-loss pruning graph. Under the query/key split, all three graphs keep the same ranking of the RL initialization conditions by graph size; under the loss-budget sweep, the fixed-loss pruning graph keeps that ranking too. Holding the loss budget fixed atĎ = 0.12, the coupled-vs-factorized query/key gap stays small for every family-condition cell. For the curriculum-initialized condition, mean absolute gaps in the moderately and extensively expanded distributions are0.008and0.010for the core circuit,0.003 and0.013for the extended support graph, and0.015and0.024for the fixed-loss pruning graph (Table 13); across all within-object family-condition summaries, gaps are at most 0.024. 9 support edge selected route Core circuit Coarse read-to-write route. preserved Extended support graph Fixed-loss pruning graph read QKV writeMLP read QKV writeMLPread Q K V writeMLP read QKV writeMLPread Q K V writeMLP Coupled Q/K:shared QK nodeFactorized Q/K:split Q/K nodes Coupled Q/K:shared QK nodeFactorized Q/K:split Q/K nodes Figure 5: Representative extraction under coupled (merged) versus factorized (separate) query/key support at the main loss budget on an RL checkpoint at75%weight sparsity. This schematic illustrates the object-level pattern at one cell; the aggregate selected-node fractions and RL-condition orderings in Tables 13â15 confirm that the orderings are preserved at this pruning threshold. Table 16 extends this check across the tested range of loss budgets, withĎ = 0.12as the main reporting point. The full graph counterpart of this exemplar is Figure 10. These small gaps do not change the RL-condition orderings. After averaging within each family- condition cell over five sparsity levels and three seeds, the coupled and factorized versions agree on all three pairwise condition rankings for every object in both expanded distributions (Table 14). A bootstrap over sparsity-seed units gives the same qualitative result: the observed agreement is3/3for every object and family, with the largest pairwise flip probability (0.232) appearing in the extensively expanded fixed-loss pruning comparison between the curriculum-initialized and composition-only conditions (Table 15). Figure 5 illustrates one representative case; Figure 10 gives the full graph-level counterpart for that exemplar. We hold the fitted pruning-score ranking fixed and sweepĎ â0.04, 0.08, 0.12, 0.16, 0.20. For the fixed-loss pruning graph, the coupled-vs-factorized curriculum gap stays modest across the full range (at most0.055under the family-level aggregation used for Table 16), and pairwise RL-condition ordering agreement is3/3in every family at every testedĎ. Changing the loss budget changes graph size and target-loss satisfaction rate, but not the ordering conclusion in this fixed-score sweep (Table 16). Takeaway. Neither extraction choice changes how the three RL initialization conditions rank by graph size. Splitting each attention head into separate query and key nodes (rather than one merged node) barely changes the size of any of the three graph objects and keeps that ranking for all three; sweeping the pruning loss budget likewise keeps it for the fixed-loss pruning graph. 4.3The Largest Compositional Gains Are Accompanied by the Largest Core-Circuit Bridge Fraction Having established which descriptions are stable, we now connect them to behavior. Relating behavioral improvement to the structure of the extracted core circuit, we find that curriculum- initialized RL produces both the largest compositional-task gains and the largest core-circuit bridge fractionâthe share of compositional-task circuit nodes that lie outside the union of the corresponding atomic-task circuits. Behavioral gains.RL initialized from the curriculum-trained supervised checkpoint improves both expanded-distribution families: averaged over five sparsity levels on Suite 1, compositional accuracy rises by+24.64points (43.83â 68.47) in the moderately expanded family and by+31.27points (46.73 â 78.00) in the extensively expanded family, while overall accuracy rises from58.19to 10 â40â2002040 RL-SFT gain (points) +34.0 +23.2 +22.5 +24.6 +18.9 +31.2 +32.3 +22.6 +37.1 +33.1 AtomicCompositional Dense 25% 50% 75% 90% Dense 25% 50% 75% 90% Moderately expanded Extensively expanded (A) Compositional gains. 0.10.20.3 Bridge fraction ModerateExtensive Curriculum Atomic-only Composition- only (B) Bridge fraction. 0.000.010.020.03 Q/K node-fraction gap Curriculum Atomic-only Composition-only Curriculum Atomic-only Composition-only max 0.024 Moderately expanded Extensively expanded CoreSupportFixed-loss (C) Query/key gap. Figure 6: Compositional gains align with core-circuit bridge fraction. A: curriculum-initialized RL gains concentrate on compositional tasks. B: bridge fraction is the share of compositional-task core-circuit nodes outside matched atomic-task circuits. C: query/key gaps remain small and preserve RL-condition orderings (Tables 13, 16); expanded-grid endpoint accuracies are reported in Table 10 and Figure 7. 73.47in the moderately expanded family and to76.95in the extensively expanded family (Figure 6; Table 10). Bridge fraction. The same curriculum-initialized conditions have the largest core-circuit bridge fraction. Atomic-only and composition-only initialization conditions remain weaker behaviorally (their RL Suite 1 mean accuracies are at most33.37; Table 10), while bridge fraction is0.343in the moderately expanded curriculum cell and0.161in the extensively expanded curriculum cell, versus ⤠0.054in the other RL conditions (Figure 6; Table 13). Behavioral recovery on compositional tasks is therefore accompanied by additional core-circuit nodes outside the matched atomic-task circuitsâan observed association between behavior and core-circuit structure. Each cell averages over five sparsity levels and three seeds, but the association rests on only the two curriculum cells, which are highest on both compositional accuracy gain and bridge fraction. We therefore read it as a directional pattern across conditions rather than a measured effect size. Takeaway. The RL conditions with the largest compositional-task accuracy gains also have the largest bridge fractions, but this is an observed association between behavior and core-circuit structure, not a causal claim. Appendix H gives boundary and ablation controls that delimit scope and separate atomic-source effects from shared format cues. 5 Conclusion Two comparisons expose different kinds of stability in the same circuit-extraction setting. The routing-head set recurs across dense and weight-sparse checkpoints, even when exact edge lists do not, and only the head-level overlap is consistently above a random top-kbaseline. For each of the three graphsâthe core circuit, the extended support graph, and the fixed-loss pruning graphâthe number of selected nodes changes little under coupled versus factorized Q/K representations; for the fixed-loss pruning graph, the ranking of the RL initialization conditions by graph size is also preserved across the tested loss-threshold range. Finally, across the same conditions, the largest compositional-task gains are accompanied by the largest fraction of compositional-task circuit nodes that lie outside the matched atomic-task circuits. Together, these findings isolate a methodological source of ambiguity in circuit extraction. A single extracted graph can support different claims at different levels of description: an exact edge list and the set of attention heads those edges touch can disagree about whether two checkpoints share a circuit, and object-level summaries can be reproducible across choices that an exact edge list is not. The empirical claim supported by a circuit extraction therefore depends on what is extracted and how the extracted graph is represented, pruned, and compared; left unstated, these choices, not behavior alone, decide it. Reporting practice.These results suggest a reporting practice for circuit-extraction studies: state the reported graph, the extraction rule and pruning threshold, the query/key node representation, the 11 comparison level supporting the claim, and any boundary or negative-control checks that bound the interpretation. Scope. The synthetic Lean family provides fixed proof-state rules and atomic and compositional structures known by construction for auditing circuit extraction in a controlled setting. Extending the same comparisons to broader benchmarks, deeper or larger models, and other architectures is the natural next step; Appendix J gives the full discussion. References Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-0587. Xi Chen, Mingyu Jin, Jingcheng Niu, Yutong Yin, Jinman Zhao, Bangwei Guo, Dimitris N. Metaxas, Zhaoran Wang, Yutao Yue, and Gerald Penn. All circuits lead to rome: Rethinking functional anisotropy in circuit and sheaf discovery for LLMs. arXiv preprint arXiv:2605.12671, 2026. URL https://arxiv.org/abs/2605.12671. Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3882â3890, 2020. doi: 10.24963/ijcai.2020/537. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and AdriĂ Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, 2023. Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 16â27, 2022. doi: 10.18653/v1/2022. blackboxnlp-1.2. Florent Draye, Anson Lei, Hsiao-Ru Pan, Ingmar Posner, and Bernhard SchĂślkopf. Sparse attention post-training for mechanistic interpretability. arXiv preprint arXiv:2512.05865, 2025. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Processing Systems, volume 37, pages 24375â24410, 2024. doi: 10.52202/079017-0768. Bradley Efron. Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7(1):1â26, 1979. doi: 10.1214/aos/1176344552. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/framework/index.html. Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V Govande, Bowen Baker, and Dan Mossing. Weight-sparse transformers have interpretable circuits. arXiv preprint arXiv:2511.13653, 2025. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83):1â64, 2025. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023. Rohan Gupta, IvĂĄn Arcuschin, Thomas Kwa, and AdriĂ Garriga-Alonso. InterpBench: Semi- synthetic transformers for evaluating mechanistic interpretability techniques. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-2950. 12 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. URL https://arxiv.org/abs/2203.15556. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757â795, 2020. doi: 10.1613/jair.1.11674. Paul Jaccard. Ătude comparative de la distribution florale dans une portion des alpes et des jura. Bulletin de la SociĂŠtĂŠ Vaudoise des Sciences Naturelles, 37:547â579, 1901. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations, 2020. Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087â9105, 2020. doi: 10.18653/v1/2020.emnlp-main.731. JĂĄnos KramĂĄr, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP*: An efficient and scalable method for localizing LLM behaviour to components. arXiv preprint arXiv:2403.00745, 2024. Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitionerâs perspective. arXiv preprint arXiv:2202.01602, 2022. URL https://arxiv.org/abs/2202.01602. Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2873â2882, 2018. David Lindner, JĂĄnos KramĂĄr, Sebastian Farquhar, Matthew Rahtz, Tom McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. In Advances in Neural Information Processing Systems, 2023. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In International Conference on Learning Representations, 2025. Maxime MĂŠloux, Silviu Maniu, François Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? arXiv preprint arXiv:2502.20914, 2025a. URL https://arxiv.org/abs/2502.20914. Maxime MĂŠloux, François Portet, and Maxime Peyrard. Mechanistic interpretability as statistical estimation: A variance analysis. arXiv preprint arXiv:2510.00845, 2025b. URLhttps://arxiv. org/abs/2510.00845. Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/ forum?id=zSf8PJyQb2. Philipp Mondorf, Sondre Wold, and Barbara Plank. Circuit compositions: Exploring modular structures in transformer-based language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14934â14955, 2025. doi: 10.18653/v1/2025.acl-long.727. 13 Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, IvĂĄn Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. MIB: A mechanistic interpretability benchmark. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 45069â45108, 2025. Neel Nanda, Josh Engels, Arthur Conmy, Senthooran Rajamanoharan, Bilal Chughtai, Callum McDougall, JĂĄnos KramĂĄr, and Lewis Smith. A pragmatic vision for interpretability. Align- ment Forum, 2025. URLhttps://w.alignmentforum.org/posts/StENzDcD3kpfGJssR/ a-pragmatic-vision-for-interpretability. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2402.14811. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902â4912, 2020. doi: 10.18653/v1/2020. acl-main.442. Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A bench- mark for systematic generalization in grounded language understanding. In Advances in Neural Information Processing Systems, 2020. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300. Claudia Shi, Nicolas Beltran-Velez, Achille Nazaret, Carolina Jiang Zheng, AdriĂ Garriga- Alonso, Andrew Jesson, Maggie Makar, and David M. Blei.Hypothesis testing the cir- cuit hypothesis in LLMs.In Advances in Neural Information Processing Systems, vol- ume 37, 2024.URLhttps://proceedings.neurips.c/paper_files/paper/2024/ hash/abccb8a90b30d45b948360ba41f5a20f-Abstract-Conference.html. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621â3634, 2021. doi: 10.18653/v1/2021.findings-acl.317. Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. LLM circuit analyses are consistent across training and scale. In Advances in Neural Information Processing Systems, volume 37, 2024. doi: 10.52202/079017-1287. URLhttps://proceedings.neurips.c/paper_files/ paper/2024/hash/47c7edadfee365b394b2a3bd416048da-Abstract-Conference.html. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, 2023.URLhttps: //openreview.net/forum?id=NpsVSN6o4ul. Zhengxuan Wu, Christopher D. Manning, and Christopher Potts. ReCOGS: How incidental details of a logical form overshadow an evaluation of semantic interpretation. Transactions of the Association for Computational Linguistics, 11:1719â1733, 2023. doi: 10.1162/tacl_a_00623. Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. Lei Yu, Jingcheng Niu, Zining Zhu, Xi Chen, and Gerald Penn. Sheaf discovery with joint computation graph pruning and flexible granularity. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8822â8837. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.emnlp-main.446. URLhttps://aclanthology. org/2025.emnlp-main.446/. 14 A Term and Abbreviation Reference Table 2: Definitions of paper-specific analysis terms. TermDefinition Suite 1 / Suite 2Two independently sampled evaluation suites generated from the same proof-state generator and task distribution. In-family evaluationEvaluation on examples sampled from the same proof-state generator and task distribution as the corresponding training/evaluation family, excluding the held-out surface and depth shifts. Core circuitCompact extracted subgraph whose retained components preserve the target tactic prediction when other candidate components are ablated. Extended support graphBroader extracted graph that retains additional read, write, and routing components around the same prediction-preserving behavior, beyond the minimal prediction-preserving components. Fixed-loss pruning graphSmallest top-kdiscrete circuit, according to the learned pruning ranking, whose post-ablation loss is at or below the target loss threshold Ď . Coupled Q/KRepresentation in which query-side and key-side attention support for a head are merged into one attn_qk node type. Factorized Q/KRepresentation in which query-side and key-side attention support are represented separately as attn_q and attn_k node types. Exact-edge overlapJaccard@10 overlap between the exact component-to-component support edges selected for two matched checkpoints. Routing edgeSelected support edge in the routing relation. For head-level comparisons, routing edges are grouped by the attention head they pass through. Routing-head setSet of attention-head IDs obtained by grouping the top-krouting edges by layer and head, ignoring the exact routing-edge endpoints. Random top-k baselineChance baseline that draws top-ksets with the observed set sizes from the observed candidate set for the same metric. Node fractionSelected-node count divided by the full node universe for the given graph object and query/key representation. Bridge fractionFraction of compositional-task core-circuit nodes that are outside the union of the corresponding atomic-task circuits. Near-miss splitMatched prompts not solved by both compared checkpoints, selected by the tactic-aware near-miss score defined in Appendix C. B Training Recipes and Checkpoint Selection This section summarizes the shared optimization and model-selection protocols used for the super- vised and reinforcement families, making the main comparisons matched training comparisons. Table 3: Training and checkpoint-selection recipes for the model families. StageScopeFixed recipeSelection / evaluation Supervised familyBaseline narrow family and the two expanded-distribution SFT sweeps Shared 8-layer transformer architecture (d model = 2048, n head = 128, d head = 16, d mlp = 8192;â 415.7M params), 5B tokens, batch size 512, 1000-step warmup, constant3Ă 10 â4 learning rate; curriculum schedule simple-only [0, 0.3), gradual mix [0.3, 0.6), full task set [0.6, 1.0] Eval every 2000 steps with 50 samples per task; save the best checkpoint by mean Suite 1/Suite 2 eval accuracy; final reporting uses that checkpoint on two independent 100-sample test suites Reinforcement family Baseline RL sweep and expanded-distribution RL initialization-condition grid GRPO with composition-only prompts, 500 training samples per task, 8 generations per prompt, temperature 1.0, learning rate 10 â5 , β = 0.1, batch size 8 with gradient accumulation 4, and 10 epochs; RL condition labels refer to upstream checkpoint identity (Curriculum, Atomic-only, Composition-only) rather than to prompt mix Two 50-sample eval suites with early 5-step and later 50-step cadence; the best validation checkpoint is selected by mean Suite 1/Suite 2 eval accuracy; final reporting uses the same two 100-sample test suites 15 Table 4: Evidence coverage for the main claims and scope checks. Evidence blockCoveragePurpose Narrow-family behavior (baseline SFT + RL) 5 sparsitiesĂ 3 seedsBaseline sparsity and training-stage trends (SFT vs. RL) under the baseline narrow generator family Narrow-family graph-level extracted-graph analysis dense and75%weight-sparse comparison points, plus supplementary edge-overlap comparisons across all weight-sparse comparison points and cross-seed pairs Main edge-overlap comparison, supplementary checks, and paired dense/weight-sparse bundle com- parisons Expanded-distribution supervised behavior 2 familiesĂ 5 sparsitiesĂ 3 seeds Supervised effects of training-distribution expan- sion Expanded-distribution RL behavior2 familiesĂ 3 RL initialization conditionsĂ 5 sparsitiesĂ 3 seeds Evidence for post-SFT accuracy gains across all sparsity and seed conditions Expanded-distribution RL extracted-graph analysis 2 familiesĂ 3 RL initialization conditionsĂ 5 sparsitiesĂ 3 seedsĂ 2 versions Condition-level bridge comparisons and post-RL object-change comparisons Boundary conditions4 regimesĂ 2 sparsitiesScope calibration under format shift, distractor shift, and deeper OOD compositions C Circuit Extraction Protocols and Representative-Case Selection Table 5 summarizes the extraction configurations used for the three object families. The core circuit and extended support graph are both learned-mask extractions that save the mask with the best exact-match target-tactic accuracy after ablation, with the masked-objective value used as a tiebreaker. The loss-constrained pruning subgraph follows the exact-pruning pipeline instead: after optimizing a continuous top-kmask objective, the smallest discrete circuit whose ablated loss stays below the target-loss budget is selected. This distinction matters for interpretation: the loss-constrained pruning subgraph has a different extraction objective and a different operating constraint from the compact core circuit. The representative AND-OR success schematic in Figures 4, 5, and 11 follows a fixed representative- case selection protocol. For that task, the protocol evaluates a pre-specified multi-checkpoint pool and summarizes solved cases by target-tactic exact-match score after ablation, membership-category mass from the overlap analysis, highest-mass layer, graph size, and sample length across the core circuit, extended support graph, and loss-constrained pruning subgraph. It then selects the solved case closest to the pool mean after z-scoring these summary features. This protocol chooses illustrative schematics; aggregate evidence comes from the tables. Near-miss split scoring.The exact-edge comparison candidate pool is the pre-generated evaluation pool used for dense/weight-sparse representative-case selection: each candidate prompt has the same prompt ID evaluated at both checkpoints. After matching by task, extracted object, prompt ID, and availability of scored bundles at both endpoints, the near-miss split keeps the top quartile of non-success matched entries by a tactic-aware near-miss score,0.5times mean tactic-line token accuracy plus 0.3 times first-tactic exact-match rate plus 0.2 times output-token accuracy. Table 1 in the main text gives the compact quantitative summary. The tables that follow provide the underlying evidence blocks, extraction recipes, and fuller quantitative details for each finding family. D Benchmark Tasks and Evaluation Splits Table 7 makes the task format concrete. Displayed names are simplified for readability; generated samples randomize proposition names, hypothesis names, order, and distractor premises. Suite 1 and Suite 2 are two independently generated test suites. Reporting both reduces the risk of overstating a single draw from the dynamic generator. 16 Table 5: Circuit-extraction recipes for the three object families. For the core circuit and extended support graph, coupled Q/K and factorized Q/K use the same selection rule but different query/key node choices. For the loss-constrained pruning subgraph, both expanded-distribution sweeps use the same target-loss budget (Ď = 0.12); the two setups differ only in whether the query/key representation is coupled or factorized. Object familyQuery/key setup / extractor Optimization recipeDiscrete object selection Core circuitCore-circuit mask extractor under both query/key setups Learned hard-concrete mask, Îť â 0 = 0.05, 200 epochs, threshold 0.5, tactic-token supervision, 100 optimization samples per task, 200 reference samples Save the mask with best exact-match target-tactic accuracy after ablation, with CE+Îť â 0 tiebreak; binarize at threshold 0.5 Extended support graphCoupled-Q/K support extractor; factorized-Q/K support extractor Learned read/q/k/v/write mask, Îť â 0 = 0.05, 150 epochs, threshold 0.5, tactic-token supervision, 100 optimization samples per task, 200 reference samples Save the mask with best exact-match target-tactic accuracy after ablation, with CE+Îť â 0 tiebreak; binarize at threshold 0.5 Loss-constrained pruning subgraph Coupled-Q/K exact-pruning sweep; factorized-Q/K exact-pruning sweep Exact-pruning objective with k-penalty coefficient 3Ă 10 â5 , 150 epochs, learning rate 3Ă 10 â3 , weight decay 10 â3 , grad clip 1.0, 100 optimization samples per task, 200 reference samples After optimization, choose the smallest top-k circuit whose ablated loss stays below the target-loss budget (Ď = 0.12 for both expanded-distribution sweeps) Table 6: Atomic and compositional benchmark task families. FamilyTaskGoal pattern AtomicANDp⧠q AtomicOR-L / OR-Rp⨠q AtomicIDp CompositeAND-OR(p⧠q)⨠r CompositeOR-AND(p⨠q)⧠r CompositeAND-OR-AND(p⧠q)⨠(r⧠s) CompositeNESTED-AND(p⧠q)⧠r CompositeNESTED-OR(p⨠q)⨠r 17 Table 7: Canonicalized task examples. Each row shows the proof state given to the model and the target first tactic string; h d : d is an unused distractor premise. FamilyTaskExample proof stateTarget tactic AtomicANDp,q,d : Prop h p : p, h q : q, h d : d ⢠p⧠q exact And.intro h_p h_q AtomicOR-Lp,q,d : Prop h p : p, h d : d ⢠p⨠q exact Or.inl h_p AtomicIDp,d : Prop h p : p, h d : d ⢠p exact h_p CompositeAND-ORp,q,r,d : Prop h p : p, h q : q, h d : d ⢠(p⧠q)⨠r exact Or.inl (And.intro h_p h_q) CompositeOR-ANDp,q,r,d : Prop h p : p, h r : r, h d : d ⢠(p⨠q)⧠r exact And.intro (Or.inl h_p) h_r CompositeNESTED-ORp,q,r,d : Prop h p : p, h d : d ⢠(p⨠q)⨠r exact Or.inl (Or.inl h_p) 18 E Supplementary Behavioral Tables Table 8: Baseline-distribution family across sparsity. Means are computed over three seeds. SparsitySFT Suite 1SFT Suite 2RL Suite 1RL Suite 2 Dense59.2259.4193.9393.37 25%-sparse67.1966.0493.8993.52 50%-sparse66.5668.0098.8999.15 75%-sparse68.4871.3096.4496.67 90%-sparse74.4174.0098.3798.26 Table 9: Expanded-distribution supervised families across sparsity. Means are computed over three seeds. SparsityModerate Suite 1Moderate Suite 2Extensive Suite 1Extensive Suite 2 Dense59.2658.1157.0455.85 25%-sparse55.9655.7063.4163.93 50%-sparse54.3354.0052.6354.37 75%-sparse60.5660.5256.4156.96 90%-sparse60.8560.5661.4860.11 Table 10: Expanded-distribution RL condition summary across all sparsity and seed settings, reporting Suite 1 and compositional-task accuracy means. Means are aggregated over 5 sparsity levelsĂ3 seeds within each family-condition cell. Condition labels denote the upstream supervised checkpoint used to initialize the same composition-only GRPO refinement. FamilyConditionSFT Suite 1 RL Suite 1 â Suite 1 SFT comp. RL comp. â comp. Moderately expandedCurriculum58.1973.4715.2843.8368.4724.64 Moderately expandedAtomic-only55.4229.94-25.480.004.274.27 Moderately expandedComposition-only36.3629.74-6.6236.3650.5614.20 Extensively expandedCurriculum58.1976.9518.7646.7378.0031.27 Extensively expandedAtomic-only50.1228.29-21.830.005.595.59 Extensively expandedComposition-only39.6933.37-6.3239.6957.8018.11 60708090 Mean accuracy (%) +24.9 +20.1 +16.2 +14.6 -0.4 +25.2 +20.6 +19.6 +22.1 +1.7 SFTRL Dense 25% 50% 75% 90% Dense 25% 50% 75% 90% Moderately expanded Extensively expanded Figure 7: Expanded-grid RL endpoint accuracies for the moderately and extensively expanded distributions. Bars compare the upstream supervised checkpoint with the RL-refined endpoint across sparsity levels; annotations report the corresponding RLâSFT accuracy changes. Table 10 gives the family-condition means used in the main comparison. F Supplementary Extracted-Graph Tables This section provides the numerical counterparts to the main extracted-graph findings. Table 11 records the exact dense-versus-weight-sparse overlap values, Table 12 adds a random top-kbaseline using the observed candidate set, Table 13 records the condition-by-condition object statistics 19 underlying Figure 6, and Tables 14â15 summarize how much the RL-condition ordering itself is preserved across coupled Q/K and factorized Q/K. Table 11: Exact-edge overlap between the dense RL checkpoint and the RL checkpoint at75%weight sparsity. Confidence intervals are 95% bootstrap intervals over matched task-object entries. Semantic overlap is reported for completeness; in these comparisons it is numerically indistinguishable from structural overlap. SplitStructural Jaccard@10Semantic Jaccard@10Routing-head-set Jaccard@10 Success0.163 [0.135, 0.195]0.163 [0.135, 0.194]0.666 [0.586, 0.748] Near-miss0.142 [0.111, 0.177]0.142 [0.109, 0.177]0.554 [0.479, 0.631] Table 12: Random top-kbaseline for the dense-vs-75%-weight-sparse edge-overlap comparison. For each split and metric, the random baseline draws top-ksets with the observed set sizes from the observed candidate set for that metric. Thepcolumn reports the Monte Carlo probability that a random mean is at least the observed mean. Routing-head-set values remain far above this conservative baseline. SplitMetricObservedRandom mean Random 95% intervalpCand. set SuccessStructural0.1630.126[0.087, 0.168]0.04037 SuccessSemantic0.1630.126[0.088, 0.168]0.03937 SuccessRouting-head sets 0.6660.199[0.156, 0.245]< 0.00131 Near-missStructural0.1420.117[0.080, 0.158]0.10639 Near-missSemantic0.1420.117[0.080, 0.158]0.10839 Near-missRouting-head sets 0.5540.185[0.143, 0.230]< 0.00133 Table 13: Extracted-graph statistics by RL initialization condition across expanded-distribution families. Means are aggregated over 5 sparsity levelsĂ3 seeds within each family-condition cell; condition labels denote the upstream supervised checkpoint used to initialize the same composition-only GRPO refinement. Core fraction and core bridge fraction use the coupled-Q/K core circuit. The three gap columns report mean absolute node-fraction differences between coupled Q/K and factorized Q/K for each object. Fractions are computed as selected-node counts divided by each objectâs node universe, so these columns are within-object summaries instead of common-scale effect sizes across objects. FamilyCondition Core fraction Core bridge fraction Core gap Support gap Loss-constr. gap Moderately expanded Curriculum0.5140.3430.0080.0030.015 Moderately expanded Atomic-only0.7730.0540.0010.0060.016 Moderately expanded Composition-only0.7210.0210.0080.0080.011 Extensively expanded Curriculum0.6740.1610.0100.0130.024 Extensively expanded Atomic-only0.7760.0230.0010.0050.015 Extensively expanded Composition-only0.7100.0280.0040.0140.016 Table 14: RL initialization-condition ordering agreement between coupled Q/K and factorized Q/K across expanded-distribution families.For each family-object combination, conditions are ordered by mean compositional-task node fraction after aggregating over 5 sparsity levelsĂ3 seeds within each family-condition cell. â3/3â means that all three pairwise condition comparisons agree between versions; lower numbers would indicate that fewer pairwise condition comparisons agree. FamilyCore circuitExtended supportLoss-constrained Moderately expanded3/33/33/3 Extensively expanded3/33/33/3 20 Table 15: Bootstrap check for the RL initialization-condition ordering result. Each bootstrap sample resamples the 15 sparsity-seed units within each family-condition cell. The last two columns report the probability that coupled Q/K and factorized Q/K disagree on the two comparisons involving the curriculum-initialized condition. FamilyObjectObserved agreement Flip prob. Curric. vs Atomic Flip prob. Curric. vs Comp. Moderately expandedCore circuit3/30.0000.000 Moderately expandedExtended support3/30.0000.000 Moderately expandedLoss-constrained3/30.0050.095 Extensively expandedCore circuit3/30.0000.099 Extensively expandedExtended support3/30.0000.027 Extensively expandedLoss-constrained3/30.0030.232 Table 16: Loss-budget sweep for the fixed-loss pruning graph. For each family, loss budget, and curriculum- initialized condition, the coupled-vs-factorized query/key gap is computed by averaging node fractions over seeds and compositional tasks within each sparsity level, taking the coupled-vs-factorized absolute difference at each sparsity, then averaging those five sparsity-level gaps. Pairwise RL-condition ordering agreement between coupled and factorized is reported at five tested loss budgets. The main reporting point used in Section 4.2 and Tables 13â15 isĎ = 0.12. Across the tested range, gaps stay small in absolute terms and ordering agreement stays at 3/3 for both expanded distributions. Family / QuantityĎ = 0.04Ď = 0.08Ď = 0.12Ď = 0.16Ď = 0.20 Mod. exp. curric. gap0.0500.0300.0150.0170.013 Ext. exp. curric. gap0.0550.0200.0240.0130.013 Mod. exp. ordering agr.3/33/33/33/33/3 Ext. exp. ordering agr.3/33/33/33/33/3 Table 17: Pruning-threshold diagnostics for the loss-budget sweep. As the allowed post-ablation loss increases, the selected fixed-loss pruning graph becomes smaller and satisfies the target-loss constraint more often. These aggregate diagnostics are computed from the same loss-budget sweep as Table 16 and serve as a pruning-threshold check, not as an additional finding. Loss budget ĎTarget-loss satisfaction rate Mean node fractionMedian node fractionMean selected size k 0.040.2990.3620.30362,173 0.080.6270.2760.23647,403 0.120.7110.2460.21342,232 0.160.7380.2400.21141,195 0.200.7630.2360.20940,566 G Supplementary Edge-Overlap Comparisons Across Sparsity and Seed The main text uses the dense-versus-75%-weight-sparse fine-grained edge-overlap comparison as the primary comparison point. Table 18 and Figure 8 extend the comparison without reusing that primary anchor: routing-head-set Jaccard@10 remains above structural Jaccard@10 at the other weight-sparse anchors and in matched-sparsity cross-seed comparisons. The strongest additional support comes from matched-sparsity cross-seed comparisons, where each row averages three seed pairs. Cross-sparsity dense-versus-weight-sparse comparisons show the same ranking and serve as weight-sparse single-pair checks rather than separate frequency estimates. 21 Table 18: Supplementary fine-grained edge-overlap comparisons beyond the main dense and75%weight-sparse comparison point. Cross-sparsity rows compare dense against the non-primary weight-sparse anchors within the matched task-object entries; each row summarizes one matched compare pair (n = 1). Cross-seed rows compare matched-sparsity pairs across seeds; each row averages three seed pairs (n = 3). Values are mean Jaccard@10; n counts compare pairs rather than task-object entries. SettingSplitStructuralSemanticRoutingn Cross-sparsity, 25%-sparseSuccess0.2870.2870.6081 Cross-sparsity, 50%-sparseSuccess0.3260.3260.6321 Cross-sparsity, 90%-sparseSuccess0.1670.1670.6261 Cross-sparsity, 25%-sparseNear-miss0.2850.2850.6391 Cross-sparsity, 50%-sparseNear-miss0.3240.3240.6391 Cross-sparsity, 90%-sparseNear-miss0.1580.1580.6391 Cross-seed, DenseSuccess0.3630.3630.5633 Cross-seed, 25%-sparseSuccess0.4110.4110.8043 Cross-seed, 50%-sparseSuccess0.5250.5250.8683 Cross-seed, 75%-sparseSuccess0.5490.5490.7883 Cross-seed, 90%-sparseSuccess0.3810.3810.9533 Cross-seed, DenseNear-miss0.3670.3670.5283 Cross-seed, 25%-sparseNear-miss0.4210.4210.8323 Cross-seed, 50%-sparseNear-miss0.5330.5330.8353 Cross-seed, 75%-sparseNear-miss0.5510.5510.7853 Cross-seed, 90%-sparseNear-miss0.4000.4000.9463 0.00.20.40.60.81.0 Mean Jaccard@10 25% 50% 90% Anchor 0.2870.608 0.3260.632 0.1670.626 Success cases 0.00.20.40.60.81.0 Mean Jaccard@10 0.2850.639 0.3240.639 0.1580.639 Near-miss cases StructuralRouting (A) Cross-sparsity dense-vs-weight-sparse comparisons. 0.00.20.40.60.81.0 Mean Jaccard@10 Dense 25% 50% 75% 90% Anchor 0.3630.563 0.4110.804 0.5250.868 0.5490.788 0.3810.953 Success cases 0.00.20.40.60.81.0 Mean Jaccard@10 0.3670.528 0.4210.832 0.5330.835 0.5510.785 0.4000.946 Near-miss cases StructuralRouting (B) Matched-sparsity cross-seed comparisons. Figure 8: Exact-edge supplement beyond the main dense and75%weight-sparse comparison point. Routing- head-set Jaccard@10 remains higher than structural Jaccard@10 across both cross-sparsity and cross-seed comparisons; the cross-seed panel provides the strongest additional support. Semantic overlap exactly matches structural overlap throughout this supplement. 22 H Boundary Conditions and Negative Controls These tables provide behavior-level scope and control evidence in raw numeric form. Boundary tests calibrate the scope of the in-family findings reported in Section 4: the curriculum-initialized condition (labeled Curriculum + RL in Table 19) reaches95.79%in-family accuracy but only0.51%on the distractor shift,1.30%on the relabel shift,2.92%on the goal-first shift, and0.42%on the deeper compositional-depth check (OOD depth; Figure 9, Table 19). The extraction findings characterize the in-family proof-state generator and should not be interpreted as evidence of OOD transfer or broader compositional generalization. Within this in-family setting, matched atomic unions retain more target predictions after ablation than mismatched atomic unions, while mismatched unions remain well above random controls (Table 20). The matched-vs-mismatched gap indicates that the correct atomic source matters; the mismatched-vs-random gap is consistent with shared format/task-family cues contributing. In- family Distr. heavy Re- label Goal- first OOD depth Curriculum SFT Curriculum + RL Atomic-only + RL Composition-only + RL 63.20.41.31.30.0 95.80.51.32.90.4 42.93.92.26.59.0 45.51.66.70.41.9 Accuracy (%) (A) Boundary-condition behavior. Dense75%Dense75%Dense75% Matched atomic Mismatched atomic Random CoreSupportPruned 0.930.920.860.880.940.96 0.770.720.720.680.890.83 0.300.330.110.090.130.09 0.0 0.5 1.0 Retention (B) Target predictions retained by atomic-control circuits. Figure 9: Boundary tests and controls for the narrow-family regimes at the dense and75%weight-sparse comparison points. A: the strongest baseline-family behavior does not carry over to the evaluated held- out surface/depth shifts. B: matched atomic unions preserve the most target predictions after ablation, and mismatched atomic unions remain well above random controls. The matched-vs-mismatched gap indicates that the correct atomic source matters; the mismatched-vs-random gap is consistent with shared format/task-family cues contributing. Table 19: Boundary-condition summary for narrow-family regimes at the dense and75%weight-sparse comparison points. Values are regime means in percent. Condition labels denote the upstream checkpoint used before the same RL refinement: âAtomic-only + RLâ starts from the atomic-only supervised checkpoint, and âComposition-only + RLâ starts from the composition-only supervised checkpoint. The in-family column is the standard evaluation; Distractor / Relabel / Goal-first are held-out surface or format shifts; OOD depth is deeper compositional extrapolation. RegimeIn-familyDistractorRelabelGoal-firstOOD depth Curriculum SFT63.240.371.301.340.00 Curriculum + RL95.790.511.302.920.42 Atomic-only + RL42.873.892.226.488.96 Composition-only + RL45.461.626.670.421.88 Table 20: Negative-control target-prediction retention after ablation, aggregated over five compositional tasks and three seeds. âMatched atomicâ keeps the union of the atomic-task circuits that define the compositional task; âmismatched atomicâ keeps a low-overlap atomic union from the same checkpoint; ârandomâ averages size-, layer-, and layer-family-matched random controls. Object / CheckpointMatched atomicMismatched atomic Random Matched âmismatch Mismatch ârandom Core circuit / Dense0.9260.7740.2980.1520.476 Core circuit / 75%-sparse0.9200.7220.3320.1980.390 Extended support graph / Dense0.8590.7190.1100.1400.609 Extended support graph / 75%-sparse0.8840.6800.0950.2040.585 Loss-constrained pruning subgraph / Dense 0.9370.8930.1330.0440.760 Loss-constrained pruning subgraph / 75%-sparse 0.9560.8300.0920.1260.737 23 Representative extraction graphs Same 75% sparse RL checkpoint; columns are extraction objects, rows are coupled Q/K vs. factorized Q/K. Coupled Q/K Single QK node. Factorized Q/K Q/K split. 16 nodes ¡ 36 edges Core circuit Shared across both Q/K representations L0L1L2L3L4L5L6L7 Attn MLP coordination-heavy 56 nodes ¡ 146 edges Read/write support L0L1L2L3L4L5L6L7 Read QK V Write MLP in MLP MLP out mixed support ¡ 2 atomic-specific 55 nodes ¡ 136 edges L0L1L2L3L4L5L6L7 Read QK V Write MLP in MLP MLP out shared-support dominant 64 nodes ¡ 160 edges Read/write support L0L1L2L3L4L5L6L7 Read Q K V Write MLP in MLP MLP out mixed support 62 nodes ¡ 146 edges L0L1L2L3L4L5L6L7 Read Q K V Write MLP in MLP MLP out shared-support dominant shared support coordination-heavy atomic-specific support top-weight projected edge Extended support graph Fixed-loss pruning graph Extended support graph Fixed-loss pruning graph loss-constrained; Ď=0.12 loss-constrained; Ď=0.12 Figure 10: Graph-level counterpart to Figure 5. Columns compare the core circuit, extended support graph, and loss-constrained pruning subgraph; rows compare coupled Q/K and factorized Q/K on the same representative checkpoint. L0L1L2L3L4L5L6L7 Dense RL 39 nodes ¡ 34 edges shared read/write path; localized MLP-side residue 75%-sparse RL 42 nodes ¡ 34 edges Q/K/V carrier support; coord.-heavy MLP support early read middle selectorlate attn write support type composition Dense39 nodes 75%-sparse42 nodes pathway composition Denseattn 30MLP 9 75%-sparseattn 18MLP 24 Matched dense/sparse routing support Same read -> selector -> write organization; endpoint-specific support helps account for exact-edge mismatch. shared supportcoordination-heavyatomic-specific Figure 11: Paired dense/weight-sparse routing support for the representative AND-OR success case discussed in Section 4.1. Both the dense and75%weight-sparse RL endpoints preserve the same coarse early-readâ middle-selectorâlate-attention-write organization, but the supporting components differ: the dense endpoint places more selected support on MLP-side components, whereas the weight-sparse endpoint places more selected support on query/key/value attention components and on components outside the corresponding atomic-task circuits. The lower bars summarize support-category and attention-vs-MLP composition for this exemplar. Thus the figure illustrates why routing-head support can remain aligned while exact edge lists mismatch; aggregate overlap evidence is reported in Figure 3. I Additional Schematic Examples The next two figures play different supporting roles. Figure 10 preserves the full graph diagram for the same representative checkpoint summarized in Figure 5. Figure 11 provides the paired dense/weight-sparse routing comparison for the representative success case discussed in Section 4.1. 24 J Limitations and Broader Impact Benchmark and model scope. The synthetic Lean benchmark and the single 8-layer, 415.7M- parameter transformer family form a deliberately controlled setting: explicit atomic and compositional proof steps, dynamic names and distractors, two independently generated evaluation suites, and matched training recipes are precisely what enable controlled circuit comparison, while the model is large enough for nontrivial sparse and RL behavior yet tractable enough for repeated extraction and seed/sparsity comparisons. The shared proof-state generation rules may also make format-level cues shared across tasks easier to observe than in more heterogeneous families. Validating the same sensitivity to extraction choices at larger scales, in other architectures, and across more heterogeneous benchmarks is the natural next step. The full sparsity sweep in Figure 2 also provides context for the graph-extraction anchors: under the fixed training recipe used here, accuracy is not monotonic in weight sparsity. Graph evidence and pruning thresholds. The detailed graph evidence in the main text remains local to a small set of chosen checkpoints: exact-edge overlap covers16 + 16matched task-object entries, and the graph-level analysis focuses on the pre-specified dense and75%weight-sparse checkpoints with representative exemplars rather than frequency estimates over a larger set of extracted graphs. Table 18 and Figure 8 extend the overlap checks across all weight-sparse checkpoints and cross-seed pairs. The loss-constrained pruning subgraph also depends on its pruning threshold, since its discrete object is selected under an explicit loss budget; the main reporting point isĎ = 0.12, and Table 16 extends the coupled-vs-factorized comparison to four additional loss budgets. Object- level node fractions reported in the mechanism-stage tables were verified from the selected component lists. Boundary tests and negative controls. Boundary tests in Appendix H measure behavior under shift, and circuit-level stability under shift remains a separate validation target. Negative controls show that matched atomic controls preserve more target predictions after ablation than mismatched controls, while mismatched controls remain well above random; this pattern is consistent with shared format/task-family cues contributing. The controlled comparisons isolate sensitivity to extraction choices, with broader behavioral transfer left for future validation. Scope of circuit-style claims. Beyond the specific choices audited here, recent field-level assess- ments argue that fully reverse-engineering the internal computation of large models may be unrealistic, and that interpretability should instead target partial understanding that is validated on downstream tasks [Nanda et al., 2025]. Our results do not resolve this debate, but they are consistent with its premise: even in a controlled setting where the task structure is known by construction, the exact edge-list level of description is the least reproducible one, while coarser summaries stay stable. The reporting practice recommended in the conclusion is compatible with either reading: it asks only that a circuit-level claim state the level of description it rests on. Broader impact. The setting consists of controlled transformers trained on synthetic Lean tactic- prediction tasks, not deployed systems. The practical benefit is methodological: studies that name which graph is reported, how it is extracted, and which comparison supports a mechanistic conclusion are better positioned to produce reproducible and comparable circuit analyses. The corresponding indirect risk is over-interpretation: findings from a small synthetic benchmark, or weakly validated mechanistic claims based on a single extraction recipe, could mislead downstream users if treated as guidance for deployed large models. The reporting practice recommended in the conclusion is intended to mitigate this risk, and extending the controlled comparison to deployed-model inter- pretability would require validation across larger models, longer contexts, and more heterogeneous tasks. K Reproducibility Package and Compute Resources We release the source code and reproduction scripts athttps://github.com/Stepuuu/ circuit-extraction-stability. The package is designed to support reproduction of the main experimental pipeline and contains the synthetic data-generation and tokenizer scripts, supervised and GRPO training entry points, task-level evaluation scripts, circuit-extraction scripts for the core 25 and factorized query/key graph objects, fixed-loss pruning sweep code, a minimal runtime module, a requirements file, and representative commands. Generated datasets, checkpoints, logs, and result artifacts can be produced by the released scripts. The reported experiments do not redistribute third-party datasets or pretrained models as experimental assets. The released training and analysis scripts run supervised, GRPO, and circuit-analysis workloads as independent GPU-accelerated jobs. For the dominant supervised recipe in Table 3, a 5B-token run of the shared 8-layer, 415.7M-parameter transformer corresponds to approximately1.2Ă 10 19 training FLOPs under the standard dense-transformer6NDestimate [Kaplan et al., 2020, Hoffmann et al., 2022]. The paper and supplemental package specify the model size, token budgets, batch sizes, gradient-accumulation settings, sequence length, evaluation cadence, and RL sampling settings that determine the training and analysis workload scale. 26