Paper deep dive
Causal Mechanism Reduction: Mechanism Replacement for Neural Network Pruning and Abstraction
Amir Asiaee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 6:34:20 AM
Summary
The paper introduces Causal Mechanism Reduction (CMR), a framework that treats neural networks as deterministic structural causal models to replace internal mechanisms with constants or affine functions. This process compiles into smaller dense networks and induces reduced causal models verifiable via interchange interventions. The authors derive a unified second-order replacement-risk objective that encompasses methods like Variance-Based Pruning (VBP) and Logit-Distortion Scoring. A key contribution is proving that Logit-Distortion is invariant to positive-scaling reparameterizations, unlike VBP, which fails this invariance test. Empirical results show CMR variants are competitive with VBP in accuracy but superior in invariance and interchange-intervention fidelity.
Entities (9)
Relation Signals (8)
Logit-Distortion Scoring → isinvariantto → Positive-Scaling Reparameterization
confidence 95% · logit-distortion score is exactly invariant
Variance-Based Pruning → isnotinvariantto → Positive-Scaling Reparameterization
confidence 95% · VBP's kept set collapses to chance-level overlap ... VBP is not invariant to this transformation
Causal Mechanism Reduction → uses → Structural Causal Model
confidence 95% · CMR treats a trained network as a deterministic structural causal model
Causal Mechanism Reduction → enables → Bias Folding
confidence 90% · These replacements compile exactly into smaller dense networks by bias and weight folding
Causal Mechanism Reduction → includesasspecialcase → Logit-Distortion Scoring
confidence 90% · special cases recover ... logit-distortion scoring
Causal Mechanism Reduction → includesasspecialcase → Variance-Based Pruning
confidence 90% · special cases recover mean replacement, variance-based pruning (VBP)
DeiT-Tiny → isevaluatedon → ImageNet-100
confidence 90% · matched fine-tuning of DeiT-Tiny on ImageNet-100
Causal Mechanism Reduction → verifiesvia → Interchange Intervention
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Which internal mechanisms of a neural network can be replaced while preserving the computation it performs? Structured pruning asks for smaller deployable networks; causal abstraction asks for high-level models that commute with interventions. We introduce causal mechanism reduction (CMR), a framework that treats a trained network as a deterministic structural causal model and replaces selected internal variables by constants or affine functions of retained variables. These replacements compile exactly into smaller dense networks by bias and weight folding, and induce reduced causal models testable with interchange interventions. We derive a unified second-order replacement-risk objective whose special cases recover mean replacement, variance-based pruning (VBP), logit-distortion scoring, and affine neuron merging, together with a margin-based certificate linking logit distortion to interchange-intervention agreement. The framework also exposes a basic invariance requirement: functionally identical ReLU networks should induce the same reduction. Under exact positive-scaling reparameterizations, VBP's kept set collapses to chance-level overlap while the logit-distortion score is exactly invariant. Empirically, CMR variants are competitive with VBP under matched fine-tuning of DeiT-Tiny on ImageNet-100; the clearer separation appears in the invariance and interchange tests, where the logit-distortion score preserves kept sets and consistently improves distributional fidelity. CMR thus gives pruning, compilation, and causal-abstraction verification a common object to optimize and verify.
Tags
Links
- Source: https://arxiv.org/abs/2602.24266v2
- Canonical: https://arxiv.org/abs/2602.24266v2
Trouble viewing inline? Open PDF directly →
Full Text
105,254 characters extracted from source content.
Expand or collapse full text
Causal Mechanism Reduction: Mechanism Replacement for Neural Network Pruning and Abstraction Amir Asiaee Vanderbilt University Medical Center Nashville, TN amir.asiaeetaheri@vumc.org Abstract Which internal mechanisms of a neural network can be replaced while preserving the computation it performs? This question has two versions: structured pruning asks for smaller deployable networks, while causal abstraction asks for high-level models that commute with interventions. We introduce causal mechanism reduction (CMR), a framework for mechanism replacement that treats a trained network as a deterministic structural causal model and replaces selected internal variables by constants or affine functions of retained variables. These replacements compile exactly into smaller dense networks through bias and weight folding, and they induce reduced causal models that can be tested with interchange interventions. We derive a unified second-order replacement-risk objective with special cases that recover mean replacement, variance-based pruning, logit-distortion scoring, and affine neuron merging. A margin-based certificate connects logit distortion to interchange-intervention agreement, giving a formal link between compression error and abstraction fidelity. The framework also identifies a basic invariance requirement for mechanism scores: functionally identical ReLU networks should induce the same reduction. In exact positive-scaling reparameterizations, variance-based pruning violates this requirement, with kept-set Jaccard collapsing to the chance level for the budget (about one third), while our logit-distortion score is exactly invariant. In experiments, CMR variants are competitive with VBP under matched fine-tuning of DeiT-Tiny on ImageNet-100, where final-block FFN pruning is largely recovered by fine-tuning. The clearer separation appears in the invariance and interchange tests: CMR-Logit preserves kept sets under reparameterization and consistently improves distributional fidelity under interchange interventions, with small, directionally consistent interchange-accuracy gains. CMR thus provides a common object for pruning, compilation, and causal-abstraction verification. 1 Introduction Neural networks are often simplified after training. In compression, the simplified model should be smaller and faster without losing predictive behavior. In causal abstraction and mechanistic interpretability, the simplified model should preserve how internal variables support behavior under interventions. These goals are usually studied separately, but they share a basic question: which internal mechanisms can be replaced without changing what the network does? The common operation is mechanism replacement. A pruning method may replace a hidden unit by a constant and fold the resulting contribution into the next bias; an abstraction method may keep a subset of internal variables and ask whether interventions on those variables commute with the original computation. In this view, structured pruning and causal abstraction differ less in the operation they perform than in the fidelity metric they demand: deployable task behavior for pruning, interchange-intervention agreement for abstraction. We formalize this operation as causal mechanism reduction. Given a trained feedforward network viewed as a deterministic structural causal model (SCM) [31], we replace selected units by constants or by affine functions of retained units. The same reduced object has two readings. Computationally, it compiles exactly into a smaller dense network through bias and weight folding. Causally, it defines an explicit reduced SCM whose state map is the projection onto retained activations and whose fidelity can be checked under interchange interventions [13, 2]. This is not only a change of vocabulary. A unified second-order replacement-risk theorem (Theorem˜1) shows that mean replacement, variance-based pruning (VBP) [3], logit-distortion scoring, and affine neuron merging [20] are all special cases of the same local quadratic objective. A margin-based interchange-fidelity certificate (Theorem˜2) then connects one tractable surrogate, expected squared logit distortion, to class-level agreement under interchange interventions. The same replacement score can therefore be read both as a compression criterion and as a proposal for a verifiable causal abstraction. The formalism also reveals a concrete failure mode of an existing pruning heuristic. ReLU networks admit exact function-preserving rescalings: multiplying a hidden unit by a positive scalar and dividing its outgoing weights by the same scalar leaves the network function unchanged. A behavioral reduction criterion should not identify different mechanisms in two such networks. VBP is not invariant to this transformation because activation variance changes with scale; logit distortion is invariant. In our stress test, exact ReLU scaling reparameterizations on [0.01,100][0.01,100] drive VBP’s kept-set Jaccard to chance-level overlap (0.350.35 against a random-subset floor of ≈1/3≈ 1/3) across functionally identical networks, while CMR-Logit remains at Jaccard 1.01.0, with downstream effects on interchange-intervention agreement and distributional fidelity. Empirically, the results are deliberately two-sided. On DeiT-Tiny final-block FFN pruning after transfer to ImageNet-100, all methods land within 0.70.7 top-1 points of the 84.6284.62 baseline after matched fine-tuning (most cells slightly above it), including random pruning; this indicates that the CMR variants are competitive with VBP in a modern transformer setting, but also shows that top-1 accuracy is saturated in this protocol. The clearest method signal appears in the invariance stress test, while interchange-intervention experiments on CIFAR-10 ConvNet and ResNet-56 / CIFAR-100 verify that compiled reductions can be evaluated as approximate causal abstractions, not only as compressed predictors. Contributions. • A unified replacement primitive. We define constant and affine mechanism replacements that compile into smaller dense networks and induce explicit reduced SCMs that can be verified under interchange interventions. • Theory connecting pruning scores to abstraction fidelity. We prove a unified replacement-risk theorem that recovers VBP, Logit-MSE / CMR-Logit, and affine weighted-least-squares (WLS) scoring as special cases, and a margin-based certificate connecting logit distortion to interchange-intervention agreement. • Evidence for when the abstraction lens matters. We show a large reparameterization-invariance failure for VBP, verify compiled reductions under interchange interventions, and compare CMR variants with VBP, magnitude, random, and DepGraph baselines across DeiT-Tiny / ImageNet-100, CIFAR-10 ConvNet, and ResNet-56 / CIFAR-100. Notation and roadmap. Section˜2 places the work relative to causal abstraction, structured pruning, and second-order compression. Section˜3 defines mechanism replacement for deterministic neural SCMs, including interventional risk and exact compilation. Section˜4 presents the replacement-risk theorem, the interchange-fidelity certificate, and the invariance result. Section˜5 reports the invariance stress test, modern transformer benchmark, and interchange-fidelity study. Section˜6 discusses limitations. 2 Related Work and Positioning Causal abstraction and mechanistic interpretability. Causal abstraction asks when a high-level structural causal model (SCM) commutes with a low-level one under interventions [32, 2]; approximate abstraction allows graded mismatch between low- and high-level models [1]. Interchange interventions turn this criterion into a neural-network test by swapping internal states and measuring interchange-intervention accuracy (IIA) [13]. Interchange-intervention training shows that behavioral accuracy can miss mechanistic failures [14]. Later work studies distributed alignment search and frames causal abstraction as a formal language for mechanistic interpretability [15, 12]. Here, abstraction is constructive: we search over restricted mechanism replacements that produce an explicit reduced SCM and verify the compiled model under interchange interventions. Alignment-map complexity. Recent critiques sharpen the role of the abstraction map. With sufficiently expressive nonlinear alignment maps, high IIA can be uninformative even for randomly initialized networks [36]. CMR avoids this failure mode by fixing the maps: the state map is the fixed projection onto retained activations, the intervention map is the identity on retained coordinates, and the reported IIA is tied to the margin certificate of Theorem˜2 rather than to a learned alignment. Structured and activation-statistic pruning. Structured pruning removes units, channels, heads, or blocks, yielding smaller dense computations rather than sparse masks. Classical compression pipelines combine pruning with quantization and coding [16]; structured CNN methods remove filters or channels using magnitude, sparsity-inducing gates, or Taylor criteria [25, 26, 19, 30]. Variance-based pruning (VBP) removes low-variance MLP units and folds mean activations into downstream biases [3]; mean replacement, neuron merging, dependency-aware pruning, and submodular structured pruning use related selection and replacement principles [8, 20, 9, 6]. CMR treats these operations as instances of mechanism replacement: constant replacement recovers mean folding and VBP under stationarity and uniform curvature, while affine replacement connects to neuron merging and soft interventions in causal models [28]. Beyond a new score, this yields a shared objective that exposes when an activation statistic is invariant, when it is fragile, and when the reduced network should be read as an approximate abstraction. Second-order and modern transformer pruning. Classical second-order pruning methods such as Optimal Brain Damage and Optimal Brain Surgeon use Taylor expansions in weight space [24, 17]. Modern variants scale this idea by approximating or inverting curvature for large networks, including WoodFisher, OBC/GPTQ/SparseGPT, and Hessian-diagonal tooling [34, 11, 10, 5, 7]. Recent transformer and LLM pruning methods such as Wanda, LLM-Pruner, fast BERT pruning, and X-Pruner provide strong practical baselines for weight, channel, or block removal [35, 27, 22, 40]; attention-head pruning shows that many transformer heads can be removed with small behavioral loss [29, 39]. CMR targets a different object: it scores activation mechanisms and their replacements, compiles the selected interventions by bias or weight folding, and gives the resulting dense network an explicit interventional semantics. 3 Problem Setup: Mechanism Replacement 3.1 Networks as deterministic SCMs Let fθ:→ℝqf_θ:X ^q be a trained feedforward network with fixed parameters θ. Given a task loss ℓ:ℝq×→ℝ≥0 :R^q×Y _≥ 0 and a calibration set cal=(xs,ys)s=1nD_cal=\(x_s,y_s)\_s=1^n, define L(θ):=n−1∑s=1nℓ(fθ(xs),ys)L(θ):=n^-1 _s=1^n (f_θ(x_s),y_s). (We overload ℓ as the task loss and as a layer index; the loss always carries arguments, and layer indices appear as sub- or superscripts.) For layer ℓ∈[L] ∈[L] with width dℓd_ , let a(ℓ)(x)∈ℝdℓa^( )(x) ^d_ denote the post-nonlinearity activation vector and let a(0)(x):=xa^(0)(x):=x. The calibration activations are A(ℓ)∈ℝn×dℓA^( ) ^n× d_ with As,j(ℓ):=aj(ℓ)(xs)A^( )_s,j:=a^( )_j(x_s). For a unit j, write j:=A:,j(ℓ)a_j:=A^( )_:,j, with layer superscripts omitted when clear; let a¯j:=n−1n⊤j a_j:=n^-11_n a_j and Var[j]:=n−1‖j−a¯jn‖22Var[a_j]:=n^-1\|a_j- a_j1_n\|_2^2. A feedforward network is also viewed here as a deterministic SCM over its internal activations. The exogenous input is X, the endogenous variables are the activations aj(ℓ)ℓ,j\a_j^( )\_ ,j, and the structural equations are the forward computations z(ℓ)=W(ℓ)a(ℓ−1)+b(ℓ)z^( )=W^( )a^( -1)+b^( ) and a(ℓ)=σ(ℓ)(z(ℓ))a^( )=σ^( )(z^( )). This SCM view is not a claim that the network recovers exogenous real-world causes. It supplies intervention semantics for a deterministic computation graph. 3.2 Replacement, risk, and compilation CMR modifies this SCM by replacing selected internal mechanisms with the trained weights held fixed. For a unit j in layer ℓ , the constant replacement do(aj(ℓ):=c)do(a_j^( ):=c) severs the incoming edges to that unit and sends the constant c to all downstream consumers. More generally, fix a retained set K⊆[dℓ]K [d_ ] and a replaced set S=[dℓ]∖KS=[d_ ] K. A replacement class Φ maps retained activations to replacements for ASA_S: constants use ϕ(As,K)=cφ(A_s,K)=c, while affine replacements use ϕ(As,K)=β+BAs,Kφ(A_s,K)=β+BA_s,K or a restricted parent subset. For a single constant intervention, let fθdo(ℓ,j:=c)f_θ^do( ,j:=c) be the intervened network and define Lℓ,j(c):=n−1∑s=1nℓ(fθdo(ℓ,j:=c)(xs),ys)L_ ,j(c):=n^-1 _s=1^n (f_θ^do( ,j:=c)(x_s),y_s). The corresponding effect is ΔLℓ,j(c):=Lℓ,j(c)−L(θ) L_ ,j(c):=L_ ,j(c)-L(θ). Structured compression can therefore be posed as a constrained intervention-selection problem: minℐ⊆[L]×[d∙],|ℐ|=kc(ℓ,j)(ℓ,j)∈ℐLdo(ℐ,). _ subarraycI [L]×[d_ ],\;|I|=k\\ \c_( ,j)\_( ,j) subarray L^do(I,c). (1) The exact objective is expensive to search directly, so Section˜4 derives the local replacement-risk proxy used for scoring. Proposition 1 (Bias-folding equivalence). Consider layer ℓ with output a(ℓ)∈ℝdℓa^( ) ^d_ , followed by an affine transformation u=Wa(ℓ)+bu=Wa^( )+b, where W∈ℝm×dℓW ^m× d_ and b∈ℝmb ^m. Suppose unit j is clamped to constant c. Define W′:=W:, ∈ℝm×(dℓ−1)W :=W_:, j ^m×(d_ -1) and b′:=b+cW:,jb :=b+cW_:,j. Then, for all a (ℓ)a^( )_ j, Wa(ℓ)+b|aj(ℓ)=c=W′a (ℓ)+b′Wa^( )+b|_a^( )_j=c=W a^( )_ j+b . For a set S of constant replacements, the same folding accumulates as b′=b+∑j∈ScjW:,jb =b+ _j∈ Sc_jW_:,j and W′=W:, =W_:, S. Affine replacements fold analogously by redistributing the replaced unit’s outgoing column onto the parent columns and bias. Thus CMR can be read in two ways: as a logical intervention in the neural SCM and as a smaller dense network with no runtime masking (Figure˜1). All proofs and longer derivations for the main-text statements are collected in Appendix A. 4 Theory: Replacement Risk, Fidelity, and Invariance 4.1 Unified replacement risk The central score is a local approximation to the loss incurred by replacing mechanisms at a fixed layer. For a replaced set S and retained set K=[dℓ]∖SK=[d_ ] S, write the replacement perturbation on sample s as δs:=ϕ(As,K)−As,S _s:=φ(A_s,K)-A_s,S. Theorem 1 (Unified replacement-risk decomposition). Fix a layer ℓ , a retained set K⊆[dℓ]K [d_ ], a replaced set S=[dℓ]∖KS=[d_ ] K, a replacement class Φ (constants, affine functions of AKA_K, or learned mechanisms), and a discrepancy d on logits or task loss that is twice differentiable at the observed activations. The second-order local replacement-risk proxy has the quadratic form QS(ϕ)=s[gs,S⊤δs+12δs⊤Hs,Sδs],δs:=ϕ(As,K)−As,S.Q_S(φ)\;=\;E_s\! [\,g_s,S _s+ 12\, _s H_s,S\, _s\, ], _s\;:=\;φ(A_s,K)-A_s,S. Under block-diagonal or diagonal curvature, QSQ_S decomposes across replaced mechanisms, yielding independent per-unit or per-group scores. The constant, affine, VBP [3], logit-distortion, and neuron-merging [20] cases follow as named special cases. For single-unit constant replacement As,j(ℓ)↦cA^( )_s,j c, the proxy is minimized by cℓ,j⋆=⊤j−n⊤n⊤,c _ ,j= h a_j-1_n g1_n h, (2) provided n⊤>01_n h>0. Under samplewise gradient stationarity (gs=0g_s=0 for all s) and uniform positive curvature, this reduces to mean replacement and ranking by activation variance, recovering VBP. Under squared logit distortion with downstream logits affine in unit j, the same theorem gives c⋆=a¯jc = a_j and sℓ,jlogit=Var[j]‖W:,j‖22s^logit_ ,j=Var[a_j]\,\|W_:,j\|_2^2. For affine replacement, the minimizer is a curvature-weighted least-squares fit; for block-diagonal curvature, group scores add and a size-k replacement set is selected by the bottom-k scores. Table˜1 summarizes these cases and the corresponding methods. The assumptions are local: the network is fixed, the computation graph is deterministic and feedforward, replacements are constants or affine functions of retained variables, downstream consumers are affine when we claim exact folding, and diagonal or block-diagonal curvature is used for scoring rather than for compilation or verification. 4.2 Interchange-fidelity certificate Replacement risk is a cheap candidate score; causal-abstraction verification asks whether the reduced model commutes with the original under interchange interventions [13, 2]. For an interchange intervention I on retained coordinates, the state map τ is projection onto retained activations and the intervention map ω applies the corresponding low-level intervention. The following certificate links expected logit distortion to IIA when the low-level intervened model has margin. Theorem 2 (Margin-based interchange-fidelity certificate). Let zLI(x)z_L^I(x) denote the low-level network’s logit vector under interchange intervention I, and let zHω(I)(x)z_H^ω(I)(x) denote the compiled high-level model’s logits under the corresponding intervention ω(I)ω(I). Define the interchange margin as mI(x):=zL,y(x)I(x)−maxy′≠y(x)zL,y′I(x)m_I(x):=z_L,y(x)^I(x)- _y ≠ y(x)z_L,y ^I(x), where y(x):=argmaxyzL,yI(x)y(x):= *arg\,max_yz_L,y^I(x). Then for every ϵ>0ε>0, Pr[y^Hω(I)(x)≠y^LI(x)]≤Pr[mI(x)≤2ϵ]+Pr[‖zHω(I)(x)−zLI(x)‖∞>ϵ]. [ y_H^ω(I)(x)≠ y_L^I(x) ]\;≤\; [m_I(x)≤ 2ε ]\;+\; [\|z_H^ω(I)(x)-z_L^I(x)\|_∞>ε ]. By Markov’s inequality, the second term is bounded by [‖zHω(I)(x)−zLI(x)‖∞2]/ϵ2E [\|z_H^ω(I)(x)-z_L^I(x)\|_∞^2 ]/ε^2, which is upper-bounded by the expected squared logit distortion D2D_2 of the compiled model under the verification distribution. The CMR-Logit score estimates D2D_2 when the calibration distribution matches the interchange marginal on the replaced units and cross-unit distortion terms are accounted for; the experiments evaluate the certificate with the empirically measured joint D2D_2. Equivalently, if D2D_2 is the unnormalized expected squared logit distortion and M(t):=Pr[mI(x)≤t]M(t):= [m_I(x)≤ t], then for every ϵ>0ε>0, IIA≥1−M(2ϵ)−D2/ϵ2IIA≥ 1-M(2ε)-D_2/ε^2. When mI(x)≥γm_I(x)≥γ almost surely, this gives IIA≥1−4D2/γ2IIA≥ 1-4D_2/γ^2. The margin term is the unavoidable failure mode: near decision boundaries, no small logit-distortion guarantee can certify class agreement. 4.3 Reparameterization invariance ReLU networks admit exact function-preserving symmetries: for any s>0s>0, multiplying a hidden unit’s activations by s and dividing its outgoing weights by s leaves the network function unchanged. Variance is not invariant to this transformation; expected squared logit distortion is invariant. Proposition 2 (ReLU scaling invariance of CMR-Logit). Let fθf_θ be a feedforward ReLU network, and let j be a hidden unit in layer ℓ with post-ReLU activation aj(x)a_j(x) and outgoing weight column Wℓ+1,:,jW_ +1,:,j. For any s>0s>0, scale the incoming weights and bias of unit j by s, so that its activation becomes aj′(x)=saj(x)a _j(x)=sa_j(x) for all x, and scale its outgoing column by Wℓ+1,:,j′=s−1Wℓ+1,:,jW _ +1,:,j=s^-1W_ +1,:,j. Then the network function is unchanged for every input x. Under this transformation, Var[aj′]⋅‖Wℓ+1,:,j′‖22=Var[aj]⋅‖Wℓ+1,:,j‖22,Var[a _j]·\|W _ +1,:,j\|_2^2=Var[a_j]·\|W_ +1,:,j\|_2^2, so the CMR-Logit ranking is preserved exactly, up to global normalization constants. By contrast, Var[aj′]=s2Var[aj]Var[a _j]=s^2Var[a_j]; with independent positive scalings across units, variance-based rankings can be changed arbitrarily among units with nonzero variance. 5 Experiments We organize the empirical evaluation around three claims: (1) coordinate-invariance separates methods on functionally identical networks (Section˜5.1); (2) CMR is competitive on a modern transformer pruning benchmark, where matched fine-tuning makes top-1 accuracy a saturated recovery metric (Section˜5.2); and (3) the compiled reductions behave as approximate abstractions under interchange interventions (Section˜5.3). We close with diagnostics that show when local scores are reliable and when retraining or recomputation dominates the ranking signal. Methods. We evaluate three variants of CMR (CMR-Const, CMR-Logit, CMR-Affine) and compare them with variance-based pruning [3], magnitude pruning, and the structural-pruning baseline DepGraph [9]. A random selector serves as an unstructured recovery reference. Table˜1 maps the methods and related baselines to special cases of Theorem˜1. Compute. DeiT-Tiny experiments use a single NVIDIA RTX 4500 Ada (24 GB VRAM, CUDA 13.0); all reported throughput numbers are measured on this device. ConvNet and ResNet experiments use cached checkpoints and run on CPU, which is sufficient for these smaller workloads and ensures the interchange-intervention budget is held constant across methods within each experiment (R=2000R=2000 swaps per cell for verification and the off-diagonal diagnostic, R=1000R=1000 for the affine ablation, R=500R=500 for the calibration-shift study). Metrics. Compression experiments report retained fraction ρ, top-1/top-5 accuracy after any matched fine-tuning, parameters, multiply-accumulate operations (MACs), and measured throughput. Abstraction experiments report interchange-intervention accuracy (IIA; higher is better) and KL divergence between the original and reduced intervention logits (lower is better). The invariance stress test reports the Jaccard overlap between the kept sets selected before and after an exact function-preserving reparameterization. Figure 1: Mechanism replacement overview. A low-level network is viewed as a deterministic SCM over activations. A retained set K defines the projection τ, while replaced units S are assigned constant or affine mechanisms that fold into a smaller dense network. We evaluate both compiled task behavior and commutativity under interchange interventions. Table 1: Known structured-pruning and abstraction-discovery methods as special cases of the unified replacement-risk objective (Theorem˜1). Replacement objective × replacement class × score formula × method recovered. Replacement objective Replacement class Score formula Method recovered uniform curvature, samplewise stationarity constant Var(aj)Var(a_j) mean replacement [8] / VBP [3] squared logit distortion constant Var(aj)‖W:,j‖22Var(a_j)\,\|W_:,j\|_2^2 Logit-MSE / CMR-Logit supervised CE local risk constant curvature-weighted score CMR-Const supervised / logit local risk affine weighted residual CMR-Affine, neuron merging [20] next-layer reconstruction affine / full set reconstruction residual submodular pruning [6] 5.1 Reparameterization-invariance stress test Proposition˜2 predicts that CMR-Logit rankings are unchanged under exact ReLU positive scaling, while VBP rankings depend on the absolute activation scale. This is the controlled setting in which method choice matters most: the paired networks compute the same function, so a changed kept set reflects coordinate dependence rather than behavioral structure. The two cached models are a small convolutional CIFAR-10 classifier with a 256256-unit penultimate representation and a ResNet-56 trained on CIFAR-100 whose post-GAP representation has 6464 channels; both are scored at the penultimate-representation–head interface. For each seed, unit-wise scalings sj∼LogUniform(smin,smax)s_j (s_ ,s_ ) are applied as aj↦sjaja_j s_ja_j, W:,j↦W:,j/sjW_:,j W_:,j/s_j at the representation–head interface, i.e., the transformation that an exact ReLU rescaling of unit j induces on the cached activations and head weights; the maximum logit difference between original and rescaled models is below 10−510^-5 in every cell, a numerical sanity check on the identity. Each cell keeps half the units (128128 of 256256 on the ConvNet, 3232 of 6464 on ResNet-56), with ten scaling draws over five (ConvNet) and three (ResNet-56) trained checkpoints. On the CIFAR-10 ConvNet [21] at [0.01,100][0.01,100], CMR-Logit has kept-set Jaccard 1.000±0.0001.000± 0.000 across ten seeds while VBP has Jaccard 0.346±0.0210.346± 0.021 (Table˜2; Figure˜2), statistically indistinguishable from the chance floor: the expected Jaccard of two independent random half-subsets is ≈1/3≈ 1/3, and the measured random selector gives 0.333±0.0300.333± 0.030. At the narrower [0.1,10][0.1,10], CMR-Logit remains exactly invariant and VBP remains scale-dependent. The same pattern appears on ResNet-56 [18] / CIFAR-100, where CMR-Logit is again exactly invariant and VBP has Jaccard 0.374±0.0370.374± 0.037 at [0.01,100][0.01,100] (random floor 0.356±0.0920.356± 0.092), so under the strongest scaling range VBP’s kept-set overlap falls to chance level. This is not merely label instability: at keep =128=128 on the CIFAR-10 ConvNet under the strongest scaling range, the downstream CMR-Logit IIA gap over VBP is +0.131+0.131 and the KL gap is −0.362-0.362. Table 2: Kept-set stability under exact ReLU reparameterization. Jaccard overlap is measured between the kept set selected on the original network and the kept set selected on a functionally identical positively-scaled network (mean ± sd over ten seeds); bold marks the best Jaccard in each row. Each kept set retains half the units; the chance floor for two independent random half-subsets is ≈1/3≈ 1/3 (measured random selector: 0.333±0.0300.333± 0.030 ConvNet, 0.356±0.0920.356± 0.092 ResNet-56). Model Scale range CMR-Logit VBP magnitude CIFAR-10 ConvNet [0.01,100][0.01,100] 1.000±0.0001.000± 0.000 0.346±0.0210.346± 0.021 0.359±0.0270.359± 0.027 CIFAR-10 ConvNet [0.1,10][0.1,10] 1.000±0.0001.000± 0.000 0.371±0.0240.371± 0.024 0.378±0.0270.378± 0.027 ResNet-56 / CIFAR-100 [0.01,100][0.01,100] 1.000±0.0001.000± 0.000 0.374±0.0370.374± 0.037 0.352±0.0490.352± 0.049 ResNet-56 / CIFAR-100 [0.1,10][0.1,10] 1.000±0.0001.000± 0.000 0.389±0.0380.389± 0.038 0.361±0.0520.361± 0.052 Figure 2: Reparameterization stress test. CMR-Logit is invariant under exact positive scaling, while VBP and magnitude rankings change substantially across functionally identical networks. Curves pool three model populations (CIFAR-10 ConvNet, ResNet-56 / CIFAR-100, and an untrained ResNet-20 control), so plotted values differ from the per-model rows of Table˜2; the random selector’s spread reflects independent re-draws, not scale dependence. 5.2 Modern pruning benchmark We initialize DeiT-Tiny from ImageNet-1K pretrained weights [38] and fine-tune it for five epochs on ImageNet-100 [37] (the standard 100-class subset of ImageNet-1K [33], ≈127≈ 127K train / 55K val) to a baseline of top-1 / top-5 84.62/97.3284.62/97.32. We then prune the final transformer block’s feed-forward (FFN) intermediate units at keep fractions 0.75,0.50,0.25\0.75,0.50,0.25\ using each scoring method and apply a matched ten-epoch fine-tune (identical optimizer, schedule, and initialization from the same baseline checkpoint; mini-batch order varies per cell). Scores are computed from class-token activations of the final-block FFN on a fixed calibration set of n=1024n=1024 images, using the block’s down-projection output as a local logit surrogate for the CMR variants; the pruned projections act on all tokens at deployment. Each cell is a single seed. Throughput is measured in half precision over five timed sweeps of 10241024 validation images after two warmup sweeps; evaluation uses a direct 224×224224× 224 resize, applied identically to baseline and pruned models. After fine-tuning, all six methods finish within 0.70.7 top-1 points of the baseline (Tables˜3 and 3(a); pruned models 5.325.32–5.475.47M parameters, 1.041.04–1.061.06G MACs, throughput 7.47.4–7.87.8K img/s on the RTX 4500 Ada). Before fine-tuning, the picture is sharper: CMR-Logit, VBP, magnitude, and random preserve baseline accuracy zero-shot (top-1 83.283.2–84.684.6), CMR-Const drops moderately (71.671.6–76.176.1), and CMR-Affine collapses to near-chance (2.62.6–4.44.4); the matched fine-tune recovers every method, which is precisely why the post-fine-tuning grid saturates. This makes the benchmark a recovery and deployment-viability test rather than a statistically reliable ranking test. Among non-random selectors, CMR-Logit is highest at keep =0.75=0.75 (85.1685.16 vs. VBP 84.5084.50), while CMR-Affine is highest at keep =0.50=0.50 and 0.250.25 (85.0485.04 and 85.1685.16, against VBP’s 85.0085.00 and 84.7084.70). CMR-Const also slightly exceeds VBP at the most aggressive setting (84.9684.96 vs. 84.7084.70). Random pruning followed by matched fine-tuning is competitive, including the best single cell in the grid (85.3085.30 at keep =0.25=0.25), suggesting that the final DeiT-Tiny FFN has enough redundant expansion capacity for ten epochs of fine-tuning to recover from many reasonable selections. We therefore use ImageNet-100 to show that CMR variants remain competitive with VBP on a modern transformer, and use the invariance stress test (Section˜5.1) for the clear method-choice separation. Table 3: ImageNet-100 top-1 after matched fine-tuning. DeiT-Tiny final-block FFN pruning at retained fraction ρ; baseline top-1 is 84.6284.62; bold marks the best top-1 in each column. Method ρ=0.75ρ=0.75 ρ=0.50ρ=0.50 ρ=0.25ρ=0.25 CMR-Logit 85.1685.16 84.7884.78 84.8884.88 CMR-Affine 84.7084.70 85.0485.04 85.1685.16 CMR-Const 84.6684.66 84.6684.66 84.9684.96 VBP 84.5084.50 85.0085.00 84.7084.70 magnitude 84.7084.70 84.7884.78 84.6284.62 random 84.6884.68 84.8284.82 85.3085.30 On cached ResNet-56 / CIFAR-100 without fine-tuning, CMR-Logit and VBP prune the post-GAP representation; DepGraph physically removes stage-three residual channels and propagates through to the classifier, with channels ranked by classifier-input column norm and the dependency graph used for propagation (single seed, no fine-tuning). At keep 0.75/0.50/0.250.75/0.50/0.25, CMR-Logit top-1 is 0.670/0.534/0.2840.670/0.534/0.284, VBP 0.672/0.526/0.2860.672/0.526/0.286, DepGraph 0.253/0.065/0.0200.253/0.065/0.020. DepGraph gives larger parameter and MAC reductions but at a different deployment budget; a matched-budget fine-tuned comparison under DepGraph’s native group-norm importance is left to future work. 5.3 Interventional self-abstraction verification We verify compiled reductions under R=2000R=2000 Bernoulli interchange interventions at p=0.5p=0.5 on cached CIFAR-10 ConvNet and ResNet-56 / CIFAR-100 checkpoints (Tables˜4 and 3(b); five seeds per cell, over five ConvNet and three ResNet-56 trained checkpoints). On the CIFAR-10 ConvNet, CMR-Logit lies on the interchange-fidelity / compression frontier with IIA/KL 0.858/0.0820.858/0.082 at keep 0.750.75, 0.675/0.4310.675/0.431 at keep 0.500.50, and 0.483/1.0340.483/1.034 at keep 0.250.25. VBP is close behind on cell means at the first two keep fractions, though the paired per-seed IIA deltas are small (+0.007±0.016+0.007± 0.016 at keep 0.750.75, +0.009±0.017+0.009± 0.017 at keep 0.500.50) and not individually significant, and ResNet-56 / CIFAR-100 shows the same small CMR-Logit edge at keep 0.750.75 and 0.500.50. A powered MNIST [23] baseline with ten-seed paired bootstrap CIs (Appendix B.2) matches this pattern qualitatively: the IIA edge is small and not significant, while the KL edge is significant; a Boolean-circuit sanity check (Appendix B.3) confirms CMR-Logit recovers the ground-truth compositional structure. CMR-Const shows a useful failure mode for local risk scores. When each replaced unit’s optimal constant is fitted independently, simultaneously folding the constants ignores cross-unit curvature interactions. Jointly refitting the constants over the replaced block at compile time mitigates the worst interaction; the corrected CMR-Const IIA on the CIFAR-10 ConvNet is 0.720/0.582/0.4550.720/0.582/0.455 at keep 0.75/0.50/0.250.75/0.50/0.25. This is below CMR-Logit but above the uncorrected constant-replacement failure mode, and it matches the off-diagonal curvature diagnostic in Section˜5.4. CMR-Affine is unstable in this no-fine-tuning regime (ConvNet IIA 0.388/0.282/0.2170.388/0.282/0.217, below the random selector’s 0.723/0.538/0.3780.723/0.538/0.378); we quantify and discuss this in Section˜6. Table 4: Interventional verification summary. Each cell reports IIA/KL for compiled reductions, means over five seeds; higher IIA and lower KL are better. Bold marks per-model, per-column winners for each metric. Model Method ρ=0.75ρ=0.75 ρ=0.50ρ=0.50 ρ=0.25ρ=0.25 CIFAR-10 ConvNet CMR-Logit 0.858/0.0820.858/0.082 0.675/0.4310.675/0.431 0.483/1.0340.483/1.034 CIFAR-10 ConvNet VBP 0.850/0.0870.850/0.087 0.666/0.4470.666/0.447 0.456/1.0770.456/1.077 CIFAR-10 ConvNet CMR-Const 0.720/0.3560.720/0.356 0.582/0.7300.582/0.730 0.455/1.1850.455/1.185 ResNet-56 / CIFAR-100 CMR-Logit 0.628/0.6970.628/0.697 0.369/1.8420.369/1.842 0.159/3.2320.159/3.232 ResNet-56 / CIFAR-100 VBP 0.620/0.7140.620/0.714 0.361/1.8800.361/1.880 0.152/3.2240.152/3.224 ResNet-56 / CIFAR-100 CMR-Const 0.538/0.9440.538/0.944 0.303/2.2070.303/2.207 0.128/3.4360.128/3.436 (a) ImageNet-100 pruning. Top-1 after matched fine-tuning versus MACs; labels mark the FFN keep fraction. (b) Interventional verification. IIA and KL under interchange interventions; the x-axis is representation reduction 1−ρ1-ρ. Curves pool the CIFAR-10 ConvNet, ResNet-56 / CIFAR-100, and an untrained ResNet-20 control; per-model values are in Table˜4. The CMR-Affine curve shows the no-fine-tuning instability discussed in Section˜6. Figure 3: Pruning and abstraction behavior. The ImageNet-100 benchmark establishes modern-transformer recovery under matched fine-tuning, while interchange verification tests whether compiled reductions behave as approximate high-level causal models. 5.4 Ablations and diagnostics The ablations isolate three sources of variation that could otherwise be mistaken for method effects. First, calibration sensitivity is modest for CMR-Logit. Under a class-subset calibration shift (calibrate on classes 0–44, evaluate on 1,0241,024 held-out CIFAR-10 test images under R=500R=500 interchange interventions), CMR-Logit has KL 0.472/0.4800.472/0.480 at calibration sizes n=500/2000n=500/2000, compared with 0.616/0.5990.616/0.599 for VBP. CMR-Affine is more data-dependent in the cached full-distribution sweep: at keep 128128, test accuracy rises from 86.3786.37 at n=500n=500 to 87.2587.25 at n=10,000n=10,000, explaining why affine replacement is fragile in small calibration regimes; on ImageNet-100 the compiled affine model itself collapses zero-shot, and its recovery there is attributable to the matched fine-tune (Section˜5.2). Second, affine parent selection is not the bottleneck in the small ConvNet setting. Pearson, output-weight-aware Pearson, and random parent sets have nearly identical mean IIA at keep 0.50.5 (Appendix B.1), while larger parent sets without sufficient regularization can worsen KL. Third, the local one-shot approximation is stressed by cross-unit curvature: the mean off-diagonal Hessian-mass ratio across random 32-unit blocks of the 256-dimensional penultimate CIFAR-10 ConvNet representation is 0.8490.849. In that setting, iterative recomputation of the CMR-Const scores improves IIA from 0.5840.584 to 0.6140.614 at keep 0.50.5 and reduces KL from 0.7100.710 to 0.5970.597. Appendix B.5 reports the corresponding score, compile, and verify wall-clock breakdown. 6 Discussion and Limitations The empirical results distinguish three questions that are often conflated in pruning papers. ImageNet-100 asks whether mechanism replacement remains viable in a modern transformer under matched fine-tuning; it does, but the top-1 metric is saturated enough that method rankings are not reliable. The reparameterization stress test asks whether a score respects the function computed by the network rather than the coordinates used to represent it; here the separation is sharp. Interchange verification asks whether the compiled reduction behaves like an approximate high-level causal model; this is where the abstraction interpretation becomes testable rather than metaphorical. This perspective also clarifies CMR’s contribution. The primitive is not “delete a unit”, but “replace a mechanism and measure the behavioral cost.” Different choices of replacement class and discrepancy recover variance-based pruning, logit-distortion scoring, constant replacement, affine merging, and constructive causal abstraction as points in the same design space. The benefit is that the assumptions behind each score become explicit and can be tested against invariance, compilation, and interchange fidelity; no single score dominates every benchmark. Locality and curvature. The unified replacement-risk theorem of Theorem˜1 is stated for a single layer or block. Multi-layer iterative reduction requires recomputing activations after each replacement and re-scoring downstream layers; our off-diagonal diagnostic shows why this matters. The diagonal or block-diagonal curvature approximation makes scoring cheap and additive, but cross-unit curvature matters. Iterative recomputation is the natural extension when the one-shot approximation is too coarse, and our diagnostic results show that it can improve interchange fidelity. Replacement capacity. Affine replacement is more expressive than constant replacement, but the extra capacity is not automatically useful. On small cached vision models it can overfit the calibration signal or optimize a metric that does not translate into low interchange KL. With matched fine-tuning, as in the DeiT-Tiny / ImageNet-100 experiment, the same replacement class can recover and become competitive, though the recovery there is attributable to the fine-tune itself, which also rescues random pruning, and the compiled affine model is near-chance zero-shot. We therefore view CMR-Affine as a data- and metric-dependent option rather than a uniformly preferred selector. Causal semantics. The causal model in this paper is a computational structural causal model over neural activations. Its interventions are interventions on the network computation graph, and its abstraction claim is commutativity under a fixed state map and corresponding interchange interventions [2, 13]. We do not claim that the reduced model recovers exogenous real-world causal variables. It is a reduced neural SCM whose variables may support interpretation, but whose formal guarantee concerns computational behavior. 7 Conclusion We introduced causal mechanism reduction, a mechanism-replacement view of neural network reduction that unifies structured pruning and constructive causal abstraction. The unified replacement-risk theorem recovers several existing scores as special cases, the folding results make the reductions deployable as smaller dense networks, and the margin-based certificate connects logit distortion to interchange fidelity. Empirically, the main lesson is that coordinate dependence is not a cosmetic issue: functionally identical ReLU networks can induce different variance-based reductions, while logit-distortion scoring is invariant and preserves better interchange behavior. Mechanism replacement therefore offers both compression and causal-abstraction research a common object to optimize, compile, and verify. References [1] S. Beckers, F. Eberhardt, and J. Y. Halpern (2020) Approximate causal abstraction. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, Proceedings of Machine Learning Research, Vol. 115, p. 606–615. Cited by: §2. [2] S. Beckers and J. Y. Halpern (2019) Abstracting causal models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, p. 2678–2685. External Links: Document Cited by: §1, §2, §4.2, §6. [3] U. Berisha, J. Mehnert, and A. P. Condurache (2025) Variance-based pruning for accelerating and compressing trained networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §A.2.2, §1, §2, §5, Table 1, Theorem 1. [4] M. Cogswell, F. Ahmed, R. Girshick, L. Zitnick, and D. Batra (2015) Reducing overfitting in deep networks by decorrelating representations. arXiv preprint arXiv:1511.06068. Cited by: §A.3.2. [5] F. Dangel, F. Künstner, and P. Hennig (2020) BackPACK: packing more into backprop. In International Conference on Learning Representations (ICLR), Cited by: §2. [6] M. El Halabi, S. Srinivas, and S. Lacoste-Julien (2022) Data-efficient structured pruning via submodular optimization. In Advances in Neural Information Processing Systems 35 (NeurIPS), Cited by: §2, Table 1. [7] M. Elsayed, H. Farrahi, F. Dangel, and A. R. Mahmood (2024) Revisiting scalable Hessian diagonal approximations for applications in reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235, p. 12448–12468. Cited by: §2. [8] U. Evci, N. Le Roux, P. Castro, and L. Bottou (2018) Mean replacement pruning. Note: OpenReview submission to ICLR 2019 Cited by: §2, Table 1. [9] G. Fang, X. Ma, M. Song, M. B. Mi, and X. Wang (2023) DepGraph: towards any structural pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16091–16101. Cited by: §2, §5. [10] E. Frantar and D. Alistarh (2023) SparseGPT: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 202, p. 10323–10337. Cited by: §2. [11] E. Frantar, S. P. Singh, and D. Alistarh (2022) Optimal brain compression: a framework for accurate post-training quantization and pruning. In Advances in Neural Information Processing Systems 35 (NeurIPS), Cited by: §2. [12] A. Geiger, D. Ibeling, A. Zur, M. Chaudhary, S. Chauhan, J. Huang, A. Arora, Z. Wu, N. D. Goodman, C. Potts, and T. Icard (2025) Causal abstraction: a theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research 26 (83), p. 1–64. Cited by: §2. [13] A. Geiger, H. Lu, T. Icard, and C. Potts (2021) Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, Cited by: §1, §2, §4.2, §6. [14] A. Geiger, Z. Wu, H. Lu, J. Rozner, E. Kreiss, T. Icard, N. D. Goodman, and C. Potts (2022) Inducing causal structure for interpretable neural networks. In Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 162, p. 7324–7338. Cited by: §2. [15] A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman (2024) Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning (CLeaR), Proceedings of Machine Learning Research, Vol. 236, p. 160–187. Cited by: §2. [16] S. Han, H. Mao, and W. J. Dally (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Representations (ICLR), Cited by: §2. [17] B. Hassibi and D. G. Stork (1992) Second order derivatives for network pruning: optimal brain surgeon. In Advances in Neural Information Processing Systems 5 (NeurIPS), p. 164–171. Cited by: §A.3.2, §2. [18] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. External Links: Document Cited by: §5.1. [19] Y. He, X. Zhang, and J. Sun (2017) Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision (ICCV), p. 1398–1406. External Links: Document Cited by: §2. [20] W. Kim, S. Kim, M. Park, and G. Jeon (2020) Neuron merging: compensating for pruned neurons. In Advances in Neural Information Processing Systems 33 (NeurIPS), Cited by: §1, §2, Table 1, Theorem 1. [21] A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report University of Toronto. Cited by: §5.1. [22] W. Kwon, S. Kim, M. W. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami (2022) A fast post-training pruning framework for transformers. In Advances in Neural Information Processing Systems, Cited by: §2. [23] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), p. 2278–2324. External Links: Document Cited by: §5.3. [24] Y. LeCun, J. S. Denker, and S. A. Solla (1989) Optimal brain damage. In Advances in Neural Information Processing Systems 2 (NeurIPS), p. 598–605. Cited by: §A.3.2, §2. [25] H. Li, A. Kadav, I. Đurđanović, H. Samet, and H. P. Graf (2017) Pruning filters for efficient convnets. In 5th International Conference on Learning Representations (ICLR), Cited by: §2. [26] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang (2017) Learning efficient convolutional networks through network slimming. In IEEE International Conference on Computer Vision (ICCV), p. 2755–2763. External Links: Document Cited by: §2. [27] X. Ma, G. Fang, and X. Wang (2023) LLM-Pruner: on the structural pruning of large language models. Advances in Neural Information Processing Systems. Cited by: §2. [28] R. Massidda, A. Geiger, T. Icard, and D. Bacciu (2023) Causal abstraction with soft interventions. In Proceedings of the Second Conference on Causal Learning and Reasoning (CLeaR), Proceedings of Machine Learning Research, Vol. 213, p. 68–87. Cited by: §2. [29] P. Michel, O. Levy, and G. Neubig (2019) Are sixteen heads really better than one?. In Advances in Neural Information Processing Systems 32 (NeurIPS), p. 14014–14024. Cited by: §2, Remark 6. [30] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz (2017) Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations (ICLR), Cited by: §2. [31] J. Pearl (2009) Causality: models, reasoning, and inference. 2 edition, Cambridge University Press. Cited by: §1. [32] P. K. Rubenstein, S. Weichwald, S. Bongers, J. M. Mooij, D. Janzing, M. Grosse-Wentrup, and B. Schölkopf (2017) Causal consistency of structural equation models. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence (UAI), Cited by: §2. [33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei (2015) ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115 (3), p. 211–252. External Links: Document Cited by: §5.2. [34] S. P. Singh and D. Alistarh (2020) WoodFisher: efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems 33 (NeurIPS), Cited by: §2. [35] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter (2024) A simple and effective pruning approach for large language models. In International Conference on Learning Representations (ICLR), Cited by: §2. [36] D. Sutter, J. Minder, T. Hofmann, and T. Pimentel (2025) The non-linear representation dilemma: is causal abstraction enough for mechanistic interpretability?. In Advances in Neural Information Processing Systems 38 (NeurIPS), Note: Spotlight Cited by: §2. [37] Y. Tian, D. Krishnan, and P. Isola (2020) Contrastive multiview coding. In European Conference on Computer Vision (ECCV), External Links: 1906.05849 Cited by: §5.2. [38] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021) Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 139, p. 10347–10357. Cited by: §5.2. [39] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov (2019-07) Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), p. 5797–5808. External Links: Document Cited by: §2, Remark 6. [40] L. Yu and W. Xiang (2023) X-Pruner: eXplainable pruning for vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24355–24363. Cited by: §2. Appendix A Proofs and Extended Derivations Roadmap. This appendix separates the load-bearing derivations from optional extensions. Appendix A.1 derives the single-unit quadratic proxy, Appendix A.2 recovers mean replacement and VBP as special cases, Appendix A.3 gives the multi-unit additivity and off-diagonal diagnostic, and Appendix A.4 records the Taylor-remainder controls. Appendix A.5 then gives the proofs for the main-text theorem, certificate, folding claim, and invariance claim. Appendix A.6 is an optional grouped mechanism extension for attention heads; it is not used by the main experiments. A.1 Quadratic Proxy for Interventional Risk Exactly evaluating the interventional risk Lℓ,j(c)L_ ,j(c) requires a forward pass through the modified network for each candidate constant c. We now derive a computationally efficient second-order approximation that admits closed-form optimization over c and yields an interpretable unit importance score. A.1.1 Single-Unit Constant Intervention Fix layer ℓ and unit j. The hard intervention do(aj(ℓ):=c)do(a^( )_j:=c) replaces the stochastic activation vector j∈ℝna_j ^n (across calibration samples) with the constant vector cnc1_n. Define the induced perturbation: (c):=cn−j∈ℝn. δ(c)\;:=\;c1_n-a_j\;∈\;R^n. (3) Samplewise sensitivity. Let Ls:=ℓ(fθ(xs),ys)L_s:= (f_θ(x_s),y_s) denote the loss on sample s. We define the per-sample gradient and curvature with respect to the scalar activation As,j(ℓ)A^( )_s,j: gs:=∂Ls∂As,j(ℓ),hs:=∂2Ls∂(As,j(ℓ))2,g_s\;:=\; ∂ L_s∂ A^( )_s,j, h_s\;:=\; ∂^2L_s∂(A^( )_s,j)^2, (4) and collect these into vectors ,∈ℝng,h ^n. Since the empirical risk decomposes as a sum over independent samples, the Hessian of L(θ)L(θ) with respect to the activation column ja_j is diagonal: ∂2L∂j∂j⊤=1nDiag(). ∂^2L _j _j \;=\; 1nDiag(h). (5) Second-order proxy. Expanding Lℓ,j(c)L_ ,j(c) to second order around the observed activations yields: Δℓ,j(c):=Lℓ,j(c)−L(θ)≈1n⊤(c)+12n(c)⊤Diag()(c). _ ,j(c)\;:=\;L_ ,j(c)-L(θ)\;≈\; 1ng δ(c)\;+\; 12n\, δ(c) Diag(h)\, δ(c). (6) This quadratic proxy is exact when the loss is quadratic in the activations; otherwise, the approximation error is controlled by the magnitude of third-order derivatives (Appendix A.4). A.1.2 Optimal Constant and Importance Score We now derive closed-form expressions for the optimal intervention constant and the resulting minimal loss increase. Proposition 3 (Optimal intervention constant). Assume n⊤>01_n h>0 (positive total curvature). The unique minimizer of the quadratic proxy (6) over c∈ℝc is: cℓ,j⋆=⊤j−n⊤n⊤=∑s=1nhs⋅As,j(ℓ)∑s=1nhs⏟curvature-weighted mean−∑s=1ngs∑s=1nhs⏟gradient correction.c _ ,j\;=\; h a_j-1_n g1_n h\;=\; _s=1^nh_s· A^( )_s,j _s=1^nh_s_curvature-weighted mean\;-\; _s=1^ng_s _s=1^nh_s_gradient correction. (7) Proof. Define the scaled objective Q(c):=n⋅Δℓ,j(c)Q(c):=n· _ ,j(c). Substituting (c)=cn−j δ(c)=c1_n-a_j: Q(c)=⊤(cn−j)+12(cn−j)⊤Diag()(cn−j).Q(c)\;=\;g (c1_n-a_j)+ 12(c1_n-a_j) Diag(h)(c1_n-a_j). The first-order condition Q′(c)=0Q (c)=0 gives: n⊤+c⋅n⊤−⊤j=0⟹c⋆=⊤j−n⊤n⊤.1_n g+c·1_n h-h a_j=0 c = h a_j-1_n g1_n h. The second derivative Q′(c)=n⊤>0Q (c)=1_n h>0 confirms this is a minimum. ∎ Proposition 4 (Unit importance score). The minimized proxy loss increase defines the importance score: sℓ,j:=minc∈ℝΔℓ,j(c)=Δℓ,j(cℓ,j⋆).s_ ,j\;:=\; _c _ ,j(c)\;=\; _ ,j(c _ ,j). (8) This admits the closed form: sℓ,j=12nj⊤Diag()j−1n⊤j−(⊤j−n⊤)22n⋅n⊤. s_ ,j\;=\; 12na_j Diag(h)a_j\;-\; 1ng a_j\;-\; (h a_j-1_n g)^22n·1_n h. (9) Proof. Substitute cℓ,j⋆c _ ,j from (7) into (6). Writing ⋆:=c⋆n−j δ :=c 1_n-a_j and using the identity (c⋆)2n⊤=(⊤j−n⊤)2/(n⊤)(c )^21_n h=(h a_j-1_n g)^2/(1_n h), algebraic simplification yields (9). ∎ Remark 1 (Interpretation). The score sℓ,js_ ,j quantifies the irreducible loss degradation from pruning unit j: the best-case impact after optimally choosing the replacement constant. Units with low scores are prime candidates for removal. Selection rule. For a target sparsity of k units within layer ℓ , a greedy one-shot strategy selects the k units with smallest scores sℓ,jj=1dℓ\s_ ,j\_j=1^d_ and applies their corresponding optimal constants cℓ,j⋆\c _ ,j\. A.2 Recovering Known Scores We now show that several established pruning heuristics emerge as special cases of our interventional framework under specific scoring assumptions. These assumptions are sub-cases of the unified assumption set in Section˜4; they are used to rank candidate replacements, not to justify the exact compilation step. A.2.1 Mean Replacement Pruning The simplest structured pruning heuristic replaces each pruned unit’s activation with its empirical mean a¯j:=1nn⊤j a_j:= 1n1_n a_j. We show this is optimal under natural stationarity conditions. Lemma 1 (Optimality of mean replacement). Suppose the following conditions hold for unit (ℓ,j)( ,j): (i) Samplewise gradient stationarity: gs=0g_s=0 for all s∈[n]s∈[n] (every per-sample gradient vanishes at the observed activations); (i) Uniform curvature: hs=αh_s=α for all s∈[n]s∈[n], for some α>0α>0. Then cℓ,j⋆=a¯jc _ ,j= a_j, and the importance score simplifies to: sℓ,j=α2⋅Var[j],whereVar[j]:=1n∑s=1n(As,j(ℓ)−a¯j)2.s_ ,j\;=\; α2·Var[a_j], [a_j]:= 1n _s=1^n (A^( )_s,j- a_j )^2. (10) Proof. Under conditions (i)–(i), equation (7) reduces to: c⋆=α⋅n⊤j−0α⋅n=1nn⊤j=a¯j.c = α·1_n a_j-0α· n= 1n1_n a_j= a_j. With =g=0 the linear term of the proxy vanishes identically, so the proxy (6) becomes Δℓ,j(c)≈α2n‖(c)‖22 _ ,j(c)≈ α2n\| δ(c)\|_2^2, a pure quadratic in c. Minimizing over c at c=a¯jc= a_j yields sℓ,j=α2n∑s(As,j(ℓ)−a¯j)2=α2Var[j]s_ ,j= α2n _s(A^( )_s,j- a_j)^2= α2Var[a_j]. ∎ Remark 2 (When do the conditions hold?). Condition (i) holds exactly at a critical point of the empirical risk with respect to the unit’s activations (per-sample activations are independent coordinates, so a critical point forces every gsg_s to vanish), and approximately for well-trained networks where per-sample gradients are small. Condition (i) is reasonable for losses with approximately constant curvature (e.g., squared error) or as a first-order approximation when curvature variation is small. Under the weaker mean-stationarity condition n⊤=01_n g=0, the minimizer is still c⋆=a¯jc = a_j, but the minimized score acquires the unit-dependent correction −n−1⊤j-n^-1g a_j, so ranking by variance is recovered only when this correction vanishes (e.g., g empirically uncorrelated with ja_j). A.2.2 Variance-Based Pruning as a Limiting Case We now establish a formal equivalence between our framework and variance-based pruning (VBP) [3]. Theorem 3 (Recovery of VBP). Under the conditions of Lemma˜1, ranking units by the interventional importance score sℓ,js_ ,j is equivalent to ranking by activation variance Var[j]Var[a_j]. Consequently, variance-based pruning with mean replacement is recovered as a special case of the interventional proxy (6). Proof. By Lemma˜1, sℓ,j=α2Var[j]s_ ,j= α2Var[a_j] where α>0α>0 is constant across units within a layer. Since ranking is invariant to positive affine transformations, rank(sℓ,j)=rank(Var[j])rank(s_ ,j)=rank(Var[a_j]). ∎ Interpretation. VBP prunes units with low activation variance: units whose outputs are nearly constant across inputs and thus carry little information. Our causal perspective reveals that this is optimal precisely when the loss landscape has uniform curvature: low-variance units can be replaced by their mean with minimal interventional effect. A.2.3 Beyond Uniform Curvature: The General Case When curvature varies across samples, the optimal intervention constant departs from the simple mean. Decomposing (7): cℓ,j⋆=a¯j(h)⏟curvature-weightedmean−g¯h¯⏟gradientcorrection,c _ ,j\;=\; a_j^(h)_ subarrayccurvature-weighted\\ mean subarray\;-\; g h_ subarraycgradient\\ correction subarray, (11) where we define the curvature-weighted statistics: a¯j(h):=∑shsAs,j(ℓ)∑shs,g¯:=1n∑sgs,h¯:=1n∑shs. a_j^(h):= _sh_sA^( )_s,j _sh_s, g:= 1n _sg_s, h:= 1n _sh_s. This reveals two sources of departure from mean replacement: (a) Curvature weighting: Samples with higher curvature hsh_s (steeper local loss landscape) contribute more to the optimal constant, prioritizing accurate reconstruction on “sensitive” inputs. (b) Gradient correction: Nonzero average gradient g¯ g shifts the optimal constant away from the weighted mean, exploiting first-order structure to reduce loss. Remark 3 (Computational cost). Computing cℓ,j⋆c _ ,j and sℓ,js_ ,j for all units in layer ℓ requires O(n⋅dℓ)O(n· d_ ) operations given precomputed gradients and Hessian diagonals, linear in both sample count and layer width. A.2.4 Explicit Remainder Bounds for ReLU Networks Proposition˜6 bounds the Taylor approximation error in terms of the Hessian Lipschitz constant ρ. For ReLU networks, we can derive an explicit expression for ρ in terms of architectural parameters. Assumption 1 (Bounded ReLU network). Consider an L-layer ReLU network with: (A1) Weight matrices satisfying ‖W(ℓ)‖2≤Mℓ\|W^( )\|_2≤ M_ and ‖W(ℓ)‖F≤Fℓ\|W^( )\|_F≤ F_ (A2) Inputs bounded as ‖x‖2≤R\|x\|_2≤ R (A3) Loss function ℓ with bounded third derivative in the logit vector: the operator (injective) norm of the third-derivative tensor of ℓ with respect to the logits is at most κ (for cross-entropy with bounded logits, κ=O(1)κ=O(1)) (A4) The intervention segment considered remains inside one ReLU activation-pattern region; segments that cross region boundaries incur an additional Hessian-jump contribution not tracked here. Theorem 4 (Local Hessian Lipschitz constant for ReLU networks). Under ˜1, the interventional risk Lℓ,j(c)L_ ,j(c) for unit j in layer ℓ has Hessian Lipschitz constant bounded by: ρℓ,j≤κ⋅(∏m=ℓ+2LMm)3⋅∥W:,j(ℓ+1)∥23, _ ,j\;≤\;κ· ( _m= +2^LM_m )^3·\|W^( +1)_:,j\|_2^3, (12) with the empty product equal to 11 when ℓ+1=L +1=L. Proof. The loss as a function of aj(ℓ)a^( )_j composes: (i) the downstream network fℓ+1:Lf_ +1:L mapping aj(ℓ)a^( )_j to output, and (i) the loss function ℓ . Step 1: Downstream Jacobian. For ReLU networks, the Jacobian of layer m output w.r.t. layer ℓ activation is: ∂a(m)∂aj(ℓ)=(∏m′=ℓ+1mD(m′)W(m′))ej=(∏m′=ℓ+2mD(m′)W(m′))D(ℓ+1)W:,j(ℓ+1), ∂ a^(m)∂ a^( )_j= ( _m = +1^mD^(m )W^(m ) )e_j\;=\; ( _m = +2^mD^(m )W^(m ) )D^( +1)W^( +1)_:,j, where D(m′)=diag([z(m′)>0])D^(m )=diag(1[z^(m )>0]) is the ReLU gradient (diagonal, entries in 0,1\0,1\). Thus, since the selector eje_j extracts column j of W(ℓ+1)W^( +1): ‖∂a(m)∂aj(ℓ)‖2≤‖W:,j(ℓ+1)‖2⋅∏m′=ℓ+2m‖W(m′)‖2=‖W:,j(ℓ+1)‖2⋅∏m′=ℓ+2mMm′. \| ∂ a^(m)∂ a^( )_j \|_2≤\|W^( +1)_:,j\|_2· _m = +2^m\|W^(m )\|_2=\|W^( +1)_:,j\|_2· _m = +2^mM_m . Step 2: Chain rule for third derivative. Within a fixed activation-pattern region the downstream map aj(ℓ)↦fθa^( )_j f_θ is affine, so its own second and third derivatives vanish and the chain rule leaves exactly one term: writing J:=∂fθ/∂aj(ℓ)∈ℝqJ:=∂ f_θ/∂ a^( )_j ^q, ∂3L∂(aj(ℓ))3=∑p,q′,r∂3ℓ∂zp∂zq′∂zrJpJq′Jr,|∂3L∂(aj(ℓ))3|≤κ‖J‖23. ∂^3L∂(a^( )_j)^3= _p,q ,r ∂^3 ∂ z_p∂ z_q ∂ z_rJ_pJ_q J_r, | ∂^3L∂(a^( )_j)^3 |≤κ\|J\|_2^3. Step 3: Lipschitz constant. Within a fixed ReLU activation-pattern region, the downstream map is affine in aj(ℓ)a^( )_j, so the Hessian varies only through the smooth loss derivative. On such a region, applying Step 1 with m=Lm=L: ρ=sup|∂3L∂(aj(ℓ))3|≤κ⋅(∏m=ℓ+2LMm)3⋅‖W:,j(ℓ+1)‖23.ρ= | ∂^3L∂(a^( )_j)^3 |≤κ· ( _m= +2^LM_m )^3·\|W^( +1)_:,j\|_2^3. This is (12). ∎ Corollary 1 (Layer-wise remainder scaling). For unit j in layer ℓ with optimal intervention cj⋆c _j, under ˜1, the Taylor remainder satisfies: |R3|≤κ6n(∏m=ℓ+2LMm)3‖W:,j(ℓ+1)‖23⋅(n[(cj⋆−a¯j)2+Var[j]])3/2.|R_3|\;≤\; κ6n ( _m= +2^LM_m )^3\|W^( +1)_:,j\|_2^3· (n [(c _j- a_j)^2+Var[a_j] ] )^3/2. (13) The bracket reduces to Var[j]Var[a_j] exactly when cj⋆=a¯jc _j= a_j, which holds for CMR-Logit and under the samplewise-stationarity, uniform-curvature conditions of Lemma˜1. Remark 4 (Depth dependence). The bound (12) grows exponentially with depth (L−ℓ)(L- ) due to the product of spectral norms. This suggests: (a) The quadratic proxy is more accurate for units in later layers (smaller L−ℓL- ). (b) Networks with spectral normalization (Mℓ=1M_ =1) have depth-independent bounds. Practical computation. For a trained network, ‖W(ℓ)‖2\|W^( )\|_2 can be computed via power iteration, and κ depends on the loss (e.g., κ=O(1)κ=O(1) for cross-entropy with bounded logits). This yields a checkable bound on proxy accuracy. A.3 Multi-Unit Interventions and the Off-Diagonal Diagnostic Thus far, we have analyzed single-unit interventions. In practice, compression requires selecting and intervening on multiple units simultaneously. We now extend the quadratic proxy to this setting and characterize when optimal selection decomposes into independent per-unit decisions. A.3.1 Joint Quadratic Proxy Let S⊆[dℓ]S [d_ ] denote a subset of units to prune, with intervention constants S=(cj)j∈S∈ℝ|S|c_S=(c_j)_j∈ S ^|S|. Define the stacked perturbation vector: S(S):=vec(cjn−jj∈S)∈ℝn|S|, δ_S(c_S)\;:=\;vec (\c_j1_n-a_j\_j∈ S )\;∈\;R^n|S|, where vec(⋅)vec(·) concatenates the per-unit perturbations. The joint second-order expansion is: Δℓ,S(S)≈1nS⊤S(S)+12nS(S)⊤HSS(S), _ ,S(c_S)\;≈\; 1n\,g_S δ_S(c_S)\;+\; 12n\, δ_S(c_S) H_S\, δ_S(c_S), (14) where S∈ℝn|S|g_S ^n|S| is the stacked gradient and HS∈ℝn|S|×n|S|H_S ^n|S|× n|S| is the Hessian with respect to the stacked activation matrix A:,S(ℓ)A^( )_:,S. The coupling challenge. The Hessian HSH_S generally contains off-diagonal blocks HjkH_jk capturing interactions between units j and k. These cross-unit couplings arise from shared downstream paths and make joint optimization over (S,S)(S,c_S) computationally intractable for large layers. A.3.2 Decoupling via Diagonal Curvature For scoring, a standard approximation in second-order pruning [24, 17] drops cross-unit curvature interactions: HS≈blockdiag(Diag(j))j∈S.H_S\;≈\;blockdiag (Diag(h_j) )_j∈ S. (15) Under this approximation, each unit’s contribution to the loss is locally independent. Theorem 5 (Additivity and optimal selection). Assume the scoring Hessian is replaced by the block-diagonal structure (15). Then: (i) Additivity: The joint proxy decomposes as a sum of single-unit proxies: Δℓ,S(S)≈∑j∈SΔℓ,j(cj). _ ,S(c_S)\;≈\; _j∈ S _ ,j(c_j). (16) (i) Separable optimization: The optimal constants minimize independently: minS∈ℝ|S|Δℓ,S(S)=∑j∈Smincj∈ℝΔℓ,j(cj)=∑j∈Ssℓ,j. _c_S ^|S| _ ,S(c_S)\;=\; _j∈ S _c_j _ ,j(c_j)\;=\; _j∈ Ss_ ,j. (17) (i) Greedy optimality: For a budget of k units, the subset minimizing the proxy is: S⋆=argminS⊆[dℓ],|S|=k∑j∈Ssℓ,j=bottom-k(sℓ,jj=1dℓ).S \;=\; *arg\,min_S [d_ ],\,|S|=k _j∈ Ss_ ,j\;=\;bottom-k (\s_ ,j\_j=1^d_ ). (18) Proof. (i) Under (15), both the linear term S⊤Sg_S δ_S and the quadratic form S⊤HSS δ_S H_S δ_S decompose across units, yielding (16). (i) With no cross-unit terms, minimization over Sc_S separates into |S||S| independent scalar minimizations, each solved by Proposition˜3. (i) Since the minimized objective is a sum of independent scalar scores, the optimal size-k subset consists of the k smallest summands. ∎ Remark 5 (Computational complexity). Under diagonal curvature, selecting k units from a layer of width dℓd_ requires: (a) O(n⋅dℓ)O(n· d_ ) to compute all scores sℓ,j\s_ ,j\, and (b) O(dℓlogk)O(d_ k) to extract the bottom-k via a partial sort. This is linear in layer size, tractable even for wide layers. Proposition 5 (Off-diagonal curvature diagnostic). Let B be a block of b candidate units, let HB∈ℝb×bH_B ^b× b be the curvature matrix used for a block-level quadratic score, and write HB=DB+EBH_B=D_B+E_B with DB:=diag(HB)D_B:=diag(H_B) and EB:=HB−DBE_B:=H_B-D_B. Define ρoff(B):=‖EB‖F‖HB‖F, _off(B):= \|E_B\|_F\|H_B\|_F, with ρoff(B)=0 _off(B)=0 when HB=0H_B=0. For any block perturbation δB∈ℝb _B ^b, the additivity error made by using DBD_B instead of HBH_B satisfies |12δB⊤HBδB−12δB⊤DBδB|≤12ρoff(B)‖HB‖F‖δB‖22. | 12 _B H_B _B- 12 _B D_B _B |\;≤\; 12\, _off(B)\,\|H_B\|_F\,\| _B\|_2^2. If additionally HBH_B is positive semidefinite (as for Gauss–Newton or Fisher curvature) and HB⪰μBIH_B _BI on the span of the considered perturbations, with μB>0 _B>0, then |12δB⊤HBδB−12δB⊤DBδB| | 12 _B H_B _B- 12 _B D_B _B | ≤ρoff(B)κF(B)(12δB⊤HBδB), ≤ _off(B)\, _F(B)\, ( 12 _B H_B _B ), κF(B) _F(B) :=‖HB‖FμB≤bλmax(HB)μB. := \|H_B\|_F _B≤ b\, _ (H_B) _B. Proof. The two quadratic scores differ only in the off-diagonal term: 12δB⊤HBδB−12δB⊤DBδB=12δB⊤EBδB. 12 _B H_B _B- 12 _B D_B _B= 12 _B E_B _B. By Cauchy–Schwarz and ‖EB‖2≤‖EB‖F\|E_B\|_2≤\|E_B\|_F, |12δB⊤EBδB|≤12‖EB‖2‖δB‖22≤12‖EB‖F‖δB‖22=12ρoff(B)‖HB‖F‖δB‖22. | 12 _B E_B _B |≤ 12\|E_B\|_2\| _B\|_2^2≤ 12\|E_B\|_F\| _B\|_2^2= 12 _off(B)\|H_B\|_F\| _B\|_2^2. If HB⪰μBIH_B _BI on the relevant span, then ‖δB‖22≤μB−1δB⊤HBδB\| _B\|_2^2≤ _B^-1 _B H_B _B, which gives the relative bound. The final inequality follows from ‖HB‖F≤bλmax(HB)\|H_B\|_F≤ b _ (H_B) for a positive semidefinite b×b× b matrix. ∎ When is diagonal curvature plausible? The approximation (15) is exact for the quadratic score when the mixed second derivatives between replaced units vanish. For an affine readout with squared logit distortion, this corresponds to orthogonal output-weight columns for the replaced units. More generally, it is an approximation whose quality is measured by Proposition˜5; weakly correlated unit sensitivities, sometimes encouraged by decorrelating regularizers [4], make the diagnostic small. A.4 Third-Order Remainder Bound The quadratic proxy (6) truncates a Taylor expansion at second order. We now quantify the approximation error under standard smoothness conditions. Proposition 6 (Third-order remainder bound). For sample s, write ℓs(u):=ℓ(fθdo(ℓ,j:=u)(xs),ys) _s(u):= (f_θ^do( ,j:=u)(x_s),y_s ) for the per-sample loss as a function of the intervened activation. Suppose each ℓs _s has ρ-Lipschitz second derivative on the segment [min(As,j(ℓ),c),max(As,j(ℓ),c)][ (A^( )_s,j,c), (A^( )_s,j,c)]: |ℓs′(u)−ℓs′(v)|≤ρ|u−v|,∀u,v in the segment,∀s∈[n]. | _s (u)- _s (v) |\;≤\;ρ|u-v|, ∀ u,v in the segment,\;∀ s∈[n]. Then the proxy error satisfies: |Δℓ,j(c)−(1n⊤(c)+12n(c)⊤Diag()(c))|≤ρ6n∥(c)∥23. | _ ,j(c)- ( 1ng δ(c)+ 12n δ(c) Diag(h) δ(c) ) |\;≤\; ρ6n\,\| δ(c)\|_2^3. (19) Proof. Apply the integral form of Taylor’s remainder to each sample around its observed activation As,j(ℓ)A^( )_s,j, with δs(c)=c−As,j(ℓ) _s(c)=c-A^( )_s,j: ℓs(c)=ℓs(As,j(ℓ))+gsδs(c)+12hsδs(c)2+R3,s,|R3,s|≤ρ6|δs(c)|3, _s(c)= _s(A^( )_s,j)+g_s\, _s(c)+ 12h_s\, _s(c)^2+R_3,s, |R_3,s|≤ ρ6\,| _s(c)|^3, by the per-sample Lipschitz condition. Averaging over samples, the proxy error is n−1∑s|R3,s|≤ρ6n∑s|δs(c)|3=ρ6n‖(c)‖33≤ρ6n‖(c)‖23n^-1 _s|R_3,s|≤ ρ6n _s| _s(c)|^3= ρ6n\| δ(c)\|_3^3≤ ρ6n\| δ(c)\|_2^3, using ‖x‖3≤‖x‖2\|x\|_3≤\|x\|_2. This is (19). ∎ Corollary 2 (Perturbation norm at the optimal constant). For every constant c, the perturbation norm satisfies the identity ‖(c)‖22=n[(c−a¯j)2+Var[j]]\| δ(c)\|_2^2=n [(c- a_j)^2+Var[a_j] ], minimized at c=a¯jc= a_j with value nVar[j]nVar[a_j]. At the optimal intervention cℓ,j⋆c _ ,j, therefore, ‖(c⋆)‖2=n[(cℓ,j⋆−a¯j)2+Var[j]]\| δ(c )\|_2= n[(c _ ,j- a_j)^2+Var[a_j]], which equals nVar[j] nVar[a_j] exactly when cℓ,j⋆=a¯jc _ ,j= a_j (CMR-Logit; samplewise stationarity with uniform curvature). Thus for low-variance units (prime pruning candidates) whose optimal constants stay near the mean, the cubic remainder is small. Practical implications. The bound (19) suggests that the quadratic proxy is most accurate precisely for the units we wish to prune: those with low activation variance and hence small ‖(c⋆)‖2\| δ(c )\|_2. For high-variance units (which we retain), proxy accuracy matters less since they are not candidates for removal. A.5 Proofs for Main Theory and Named Special Cases Proof of Theorem˜2. Proof. Fix an input and interchange intervention pair (x,I)(x,I), and abbreviate zL:=zLI(x),zH:=zHω(I)(x),Δz:=zH−zL,y:=y^LI(x).z_L:=z_L^I(x), z_H:=z_H^ω(I)(x), z:=z_H-z_L, y:= y_L^I(x). Suppose mI(x)>2ϵm_I(x)>2ε and ‖Δz‖∞≤ϵ\| z\|_∞≤ε. For every y′≠y ≠ y, zH,y−zH,y′=(zL,y−zL,y′)+(Δzy−Δzy′)≥mI(x)−|Δzy|−|Δzy′|>2ϵ−2ϵ=0.z_H,y-z_H,y =(z_L,y-z_L,y )+( z_y- z_y )≥ m_I(x)-| z_y|-| z_y |>2ε-2ε=0. Thus y remains the unique top class for the high-level intervened model, so disagreement is impossible on the event mI(x)>2ϵ∩‖Δz‖∞≤ϵ\m_I(x)>2ε\∩\\| z\|_∞≤ε\. Equivalently, y^Hω(I)(x)≠y^LI(x)⊆mI(x)≤2ϵ∪‖Δz‖∞>ϵ.\ y_H^ω(I)(x)≠ y_L^I(x)\ \m_I(x)≤ 2ε\∪\\| z\|_∞>ε\. Taking probabilities over the verification distribution on (x,I)(x,I) and applying the union bound gives the stated event inequality. For the second term, Markov’s inequality applied to the nonnegative random variable ‖Δz‖∞2\| z\|_∞^2 yields Pr[‖Δz‖∞>ϵ]≤[‖Δz‖∞2]ϵ2. [\| z\|_∞>ε]≤ E[\| z\|_∞^2]ε^2. Since ‖Δz‖∞2≤‖Δz‖22≤q‖Δz‖∞2\| z\|_∞^2≤\| z\|_2^2≤ q\| z\|_∞^2, the unnormalized squared logit-distortion objective D2:=[‖zHω(I)(x)−zLI(x)‖22]D_2:=E[\|z_H^ω(I)(x)-z_L^I(x)\|_2^2] gives Pr[‖Δz‖∞>ϵ]≤D2ϵ2. [\| z\|_∞>ε]≤ D_2ε^2. If CMR-Logit is reported as the per-logit mean D¯2:=q−1D2 D_2:=q^-1D_2, the bound is qD¯2/ϵ2q D_2/ε^2. ∎ IIA consequence of the margin certificate. Proof. The first display is the theorem plus the Markov bound. Optimizing over ϵε gives the variational certificate. If mI(x)≥γm_I(x)≥γ almost surely, then M(2ϵ)=0M(2ε)=0 for every ϵ<γ/2ε<γ/2; take the limit as ϵε increases to γ/2γ/2. ∎ Proof of Proposition˜1. Proof. Decompose the affine consumer by isolating column j: Wa(ℓ)+b=W:,jaj(ℓ)+W:, a (ℓ)+b.Wa^( )+b=W_:,ja^( )_j+W_:, ja^( )_ j+b. Under the intervention aj(ℓ)=ca^( )_j=c, the first term is the constant cW:,jcW_:,j, which absorbs into the bias. This gives W:, a (ℓ)+(b+cW:,j)W_:, ja^( )_ j+(b+cW_:,j). ∎ Proof of Theorem˜1. Proof. For a calibration sample s, freeze the retained activations As,KA_s,K and view the downstream discrepancy as a function of the coordinates to be replaced: rs(u):=s(Fℓ→L(As,K,u)),u∈ℝ|S|,r_s(u):=D_s\! (F_ → L(A_s,K,u) ), u ^|S|, where Fℓ→LF_ → L denotes the deterministic downstream computation from layer ℓ to logits or to the scalar task loss, depending on the choice of d. For logit distortion, s(z)=d(z,Fℓ→L(As,K,As,S))D_s(z)=d(z,F_ → L(A_s,K,A_s,S)); for supervised task loss, s(z)=ℓ(z,ys)D_s(z)= (z,y_s). Define gs,S:=∇rs(As,S),Hs,S:=∇2rs(As,S).g_s,S:=∇ r_s(A_s,S), H_s,S:=∇^2r_s(A_s,S). For any replacement ϕ∈Φφ∈ , write δs=ϕ(As,K)−As,S _s=φ(A_s,K)-A_s,S. Taylor’s theorem gives rs(As,S+δs)−rs(As,S)=gs,S⊤δs+12δs⊤Hs,Sδs+Rs(δs),r_s(A_s,S+ _s)-r_s(A_s,S)=g_s,S _s+ 12 _s H_s,S _s+R_s( _s), where RsR_s is the third-order remainder. Dropping RsR_s and averaging over s gives the displayed quadratic proxy QS(ϕ)Q_S(φ). If the downstream map is affine in ASA_S and d is a quadratic logit distortion, the remainder is zero and the expression is exact. Now let G be a partition of S into replacement mechanisms, for example single units or predefined groups. If Hs,S=blockdiag(Hs,G)G∈H_s,S=blockdiag(H_s,G)_G and δs=(δs,G)G∈ _s=( _s,G)_G , then gs,S⊤δs+12δs⊤Hs,Sδs=∑G∈(gs,G⊤δs,G+12δs,G⊤Hs,Gδs,G).g_s,S _s+ 12 _s H_s,S _s= _G (g_s,G _s,G+ 12 _s,G H_s,G _s,G ). Averaging over samples yields QS(ϕ)=∑G∈QG(ϕG)Q_S(φ)= _G Q_G( _G). Diagonal curvature is the special case in which every group has size one. Positive semidefiniteness is only needed to interpret the resulting quadratic scores as convex scoring objectives and to ensure uniqueness when the curvature is positive on the replacement subspace. ∎ Optimal constant. Proof. With S=jS=\j\, the perturbation is δs=c−As,j(ℓ) _s=c-A_s,j^( ), the sample gradient is gsg_s, and the sample curvature is hsh_s. The proxy QjQ_\j\ is exactly the scalar quadratic in (6). Its first-order condition and positive total curvature condition are those of Proposition˜3. ∎ VBP recovery. Proof. Under samplewise stationarity and uniform curvature, (2) gives cℓ,j⋆=a¯jc _ ,j= a_j. Substituting this constant into the proxy gives sℓ,j=(α/2)Var[j]s_ ,j=(α/2)Var[a_j] by Lemma˜1. Since α/2α/2 is positive and common across units in the layer, the ranking agrees with activation variance, which is precisely Theorem˜3. ∎ CMR-Logit score. Proof. Assume the map from the layer-ℓ activations to the logits is affine in unit j with coefficient column W:,jW_:,j (exact at the last layer, or within a fixed ReLU activation-pattern region; elsewhere the score is the corresponding next-layer surrogate). Then the logit perturbation induced by replacing unit j with c is Δzs(c)=W:,j(c−As,j(ℓ)). z_s(c)=W_:,j(c-A^( )_s,j). Thus the squared logit distortion is ‖Δzs(c)‖22=‖W:,j‖22(c−As,j(ℓ))2.\| z_s(c)\|_2^2=\|W_:,j\|_2^2(c-A^( )_s,j)^2. At the original activation the first derivative of this discrepancy is zero, and the curvature with respect to As,j(ℓ)A^( )_s,j is the sample-independent scalar 2‖W:,j‖222\|W_:,j\|_2^2. Therefore (2) gives c⋆=a¯jc = a_j. Averaging the minimized distortion over samples gives ‖W:,j‖22n−1∑s(As,j(ℓ)−a¯j)2\|W_:,j\|_2^2n^-1 _s(A^( )_s,j- a_j)^2, which is the displayed score. ∎ Affine WLS score and folding. Proof. The affine special case of Theorem˜1 for a single replaced unit j with parent set P⊆KP K minimizes, over θ=(θ0,θP)θ=( _0, _P), the curvature-weighted ridge objective QP(θ)=⊤(Pθ−)+12(Pθ−)⊤D(Pθ−)+λ2‖θ‖22,Q_P(θ)\;=\;g ( _Pθ-a)+ 12\,( _Pθ-a) D\,( _Pθ-a)+ λ2\,\|θ\|_2^2, where P:=[n,A:,P] _P:=[1_n,A_:,P] is the design matrix, :=ja:=a_j, D:=Diag()D:=Diag(h), and λ≥0λ≥ 0 is an optional ridge parameter. Differentiate QPQ_P with respect to θ: ∇θ=P⊤+P⊤D(Pθ−)+λθ. _θ= _P g+ _P D( _Pθ-a)+λθ. Setting this gradient to zero gives the normal equations. When |P|=0|P|=0, the design matrix is n1_n, and the scalar normal equation gives the constant in (2), with the usual ridge modification when λ>0λ>0. For folding, write the next affine consumer as u=WKaK+Wjaj+bu=W_Ka_K+W_ja_j+b. Replacing aja_j by θ0+θP⊤aP _0+ _P a_P gives u=WKaK+WjθP⊤aP+(b+Wjθ0)u=W_Ka_K+W_j _P a_P+(b+W_j _0), so the parent columns and bias can be updated exactly as in Proposition˜1. ∎ Multi-unit additivity. Proof. The block-diagonal part of Theorem˜1 gives a sum of independent group objectives. For single-unit blocks, each summand is the single-unit score sℓ,js_ ,j from Proposition˜4. Minimizing a sum of independent scalar scores subject only to the cardinality constraint |S|=k|S|=k selects the k smallest scores, as stated in Theorem˜5. ∎ Proof of Proposition˜2. Proof. Positive homogeneity of ReLU gives ReLU(su)=sReLU(u)ReLU(su)=s\,ReLU(u) for every s>0s>0, so scaling the incoming affine parameters of unit j by s changes its activation from aj(x)a_j(x) to saj(x)sa_j(x). The contribution of this unit to the next preactivation is unchanged because (s−1Wℓ+1,:,j)(saj(x))=Wℓ+1,:,jaj(x)(s^-1W_ +1,:,j)(sa_j(x))=W_ +1,:,ja_j(x). All other units and parameters are fixed, hence every subsequent activation and the final function fθ(x)f_θ(x) are unchanged. The variance and norm transform as Var[saj]=s2Var[aj]Var[sa_j]=s^2Var[a_j] and ‖s−1Wℓ+1,:,j‖22=s−2‖Wℓ+1,:,j‖22\|s^-1W_ +1,:,j\|_2^2=s^-2\|W_ +1,:,j\|_2^2, proving the product invariance. Applying different scalings to different units leaves all CMR-Logit scores fixed but multiplies VBP scores by arbitrary positive factors, so any strict ordering of nonzero-variance units can be realized by a suitable choice of scalings. ∎ A.6 Optional Extension: Multi-Head Attention This subsection is an optional extension of the mechanism-replacement formalism, not a load-bearing claim for the main experiments. Modern transformer architectures rely on multi-head attention (MHA) as a core computational primitive; the same intervention logic can treat attention heads as grouped mechanisms for structured pruning. A.6.1 Attention as a Structural Causal Model Multi-head attention mechanism. Consider an MHA layer with H heads operating on input X∈ℝT×dX ^T× d (sequence length T, embedding dimension d). Each head h∈[H]h∈[H] computes: headh(X)=softmax(XWhQ(XWhK)⊤dk)XWhV∈ℝT×dv,head_h(X)\;=\;softmax\! ( XW^Q_h(XW^K_h) d_k )XW^V_h\;∈\;R^T× d_v, (20) where WhQ,WhK∈ℝd×dkW^Q_h,W^K_h ^d× d_k and WhV∈ℝd×dvW^V_h ^d× d_v are the query, key, and value projections for head h. The heads are concatenated and projected: MHA(X)=Concat(head1,…,headH)WO=∑h=1Hheadh(X)WhO,MHA(X)\;=\;Concat(head_1,…,head_H)W^O\;=\; _h=1^Hhead_h(X)W^O_h, (21) where WO∈ℝHdv×dW^O ^Hd_v× d and WhO∈ℝdv×dW^O_h ^d_v× d denotes the block of WOW^O corresponding to head h. SCM structure. The MHA layer induces an SCM where: • Exogenous: Input embeddings X∼X _X • Endogenous: Head outputs headh(X)h=1H\head_h(X)\_h=1^H and attention patterns Ah(X)h=1H\A_h(X)\_h=1^H • Structural equations: Defined by (20)–(21) Heads contribute additively to the layer output via (21), which keeps the intervention analysis tractable. A.6.2 Hard Interventions on Attention Heads Definition 1 (Head intervention). For head h, a constant intervention do(headh:=C)do(head_h:=C) replaces the head’s output with a fixed matrix C∈ℝT×dvC ^T× d_v, severing its dependence on X: MHAdo(h:=C)(X)=CWhO+∑h′≠hheadh′(X)Wh′O.MHA^do(h:=C)(X)\;=\;CW^O_h+ _h ≠ hhead_h (X)W^O_h . (22) The zero intervention do(headh:=0)do(head_h:=0) corresponds to head removal. Vectorized notation. For a calibration set of n sequences, let h∈ℝn×T×dvH_h ^n× T× d_v denote the stacked head outputs. Flattening to h:=vec(h)∈ℝnTdvh_h:=vec(H_h) ^nTd_v, the intervention do(headh:=C)do(head_h:=C) induces perturbation: h(C):= 1n⊗vec(C)−h∈ℝnTdv. δ_h(C)\;:=\;1_n (C)-h_h\;∈\;R^nTd_v. (23) A.6.3 Quadratic Proxy for Head Removal Proposition 7 (Head importance score). Define the per-sample, per-position, per-dimension gradient and curvature: gs,t,i:=∂Ls∂[headh]t,i,ηs,t,i:=∂2Ls∂[headh]t,i2,g_s,t,i:= ∂ L_s∂[head_h]_t,i, _s,t,i:= ∂^2L_s∂[head_h]_t,i^2, (24) with vectorized forms h,h∈ℝnTdvg_h, η_h ^nTd_v. Under the diagonal scoring approximation, the quadratic proxy for intervening on head h with constant C is: Δh(C)≈1nh⊤h(C)+12nh(C)⊤Diag(h)h(C). _h(C)\;≈\; 1ng_h δ_h(C)+ 12n δ_h(C) Diag( η_h) δ_h(C). (25) For head removal (pruning), the natural intervention is C=0C=0: Proposition 8 (Head removal score). The interventional risk increase from removing head h (setting C=0C=0) under the quadratic proxy is: shremove:=Δh(0)≈−1nh⊤h+12nh⊤Diag(h)h. s_h^remove\;:=\; _h(0)\;≈\;- 1ng_h h_h+ 12nh_h Diag( η_h)h_h. (26) Under samplewise gradient stationarity (h=g_h=0) and uniform curvature (ηs,t,i=α _s,t,i=α), this simplifies to: shremove=α2⋅1n‖h‖22=α2⋅s[‖headh(Xs)‖F2].s_h^remove\;=\; α2· 1n\|h_h\|_2^2\;=\; α2·E_s\! [\|head_h(X_s)\|_F^2 ]. (27) Proof. Substituting C=0C=0 into (25) with h(0)=−h δ_h(0)=-h_h yields (26). Under the stated conditions, the first term −1nh⊤h- 1ng_h h_h vanishes (samplewise gradient stationarity; note ⊤h=01 g_h=0 alone would not suffice, since the perturbation −h-h_h is sample-dependent) and the second term becomes α2n‖h‖22 α2n\|h_h\|_2^2. ∎ Remark 6 (Interpretation). Equation (27) reveals that under idealized conditions, head importance is proportional to the expected squared Frobenius norm of its output: heads producing larger-magnitude outputs are more important. This is consistent with empirical observations that “dead” heads (near-zero output) can be safely pruned [39, 29]. A.6.4 Optimal Constant Intervention for Heads Rather than removing a head entirely, we can replace it with an optimal constant matrix. Proposition 9 (Optimal head constant). Assume ∑s=1nηs,t,i>0 _s=1^n _s,t,i>0 for every position–dimension pair (t,i)(t,i). The optimal constant intervention C⋆∈ℝT×dvC ^T× d_v minimizing (25) satisfies, elementwise: Ct,i⋆=∑s=1nηs,t,i⋅[headh(Xs)]t,i−∑s=1ngs,t,i∑s=1nηs,t,i.C _t,i\;=\; _s=1^n _s,t,i·[head_h(X_s)]_t,i- _s=1^ng_s,t,i _s=1^n _s,t,i. (28) Under uniform curvature and per-coordinate gradient stationarity (∑sgs,t,i=0 _sg_s,t,i=0), Ct,i⋆=1n∑s=1n[headh(Xs)]t,iC _t,i= 1n _s=1^n[head_h(X_s)]_t,i, the sample mean at each position. Proof. The proxy (25) is separable across the TdvTd_v output dimensions under the diagonal scoring approximation. Each dimension reduces to the scalar problem of Proposition˜3. ∎ A.6.5 Bias Folding for Attention Heads Proposition 10 (Head removal via output projection folding). To implement do(headh:=C)do(head_h:=C), modify the MHA layer as follows: (i) Remove head h’s parameters (WhQ,WhK,WhV)(W^Q_h,W^K_h,W^V_h) and the corresponding block WhOW^O_h. (i) For a position-constant intervention C=Tc⊤C=1_Tc with c∈ℝdvc ^d_v, add the constant contribution to the downstream bias: b′:=b+(WhO)⊤cb :=b+(W^O_h) c. A position-dependent C contributes the T×dT× d matrix CWhOCW^O_h, which is not a bias vector; implementing it requires a fixed sequence length T and a stored T×dT× d additive buffer. In particular, the position-dependent optimum C⋆C of Proposition˜9 is foldable only after averaging over positions. For head removal (C=0C=0), simply delete the head’s parameters with no bias update. Computational savings. Removing k heads from an H-head MHA layer reduces: • Query/Key/Value projections: (H−k)/H(H-k)/H of original FLOPs • Attention computation: (H−k)/H(H-k)/H of original FLOPs • Output projection: (H−k)/H(H-k)/H of original parameters and FLOPs The savings are linear in the number of pruned heads. Appendix B Additional Experimental Results B.1 Ablations and diagnostics (full) Affine parent selection. On CIFAR-10 ConvNet at keep 0.50.5, the mean IIA over parent selectors is 0.2640.264 for Pearson, 0.2650.265 for output-weight-aware Pearson, and 0.2660.266 for random; random additionally has lower KL. Selector identity has no measurable effect on interchange agreement under the corrected affine WLS solve, and the supervised affine variant is dominated by CMR-Logit at the same budget. This is consistent with the affine corollary providing extra capacity rather than a strong inductive bias. Calibration size and shift. CMR-Affine accuracy at keep 128128 improves from 0.8640.864 to 0.8720.872 as calibration size grows from 500500 to 1000010000. Under a class-subset calibration shift (calibrate on classes 0–44 only; evaluate on 1,0241,024 held-out test images spanning all classes, under R=500R=500 interchange interventions), CMR-Logit KL is 0.472/0.4800.472/0.480 at n=500/2000n=500/2000, against 0.616/0.5990.616/0.599 for VBP; the logit-distortion criterion therefore transfers better across calibration distributions than the variance criterion. Off-diagonal curvature diagnostic. The mean off-diagonal Frobenius ratio ‖HB−diag(HB)‖F/‖HB‖F=0.849\|H_B-diag(H_B)\|_F/\|H_B\|_F=0.849 on random CIFAR-10 ConvNet blocks (5050 random 3232-unit blocks of the 256256-dimensional penultimate representation per seed, three seeds, n=2000n=2000 calibration samples); cross-unit interaction is therefore substantial at the layer scale tested. Proposition˜5’s relative error bound is therefore not tight in this regime; iterative recompute of the CMR-Const scores (a two-step 256→192→128256→ 192→ 128 schedule), which avoids the additivity assumption by re-scoring after each prune step, outperforms one-shot scoring at keep 0.50.5 (IIA 0.6140.614 versus 0.5840.584, KL 0.5970.597 versus 0.7100.710). B.2 Powered MNIST baseline with paired tests In the standard (non-reparameterized) setting, CMR-Logit has a small but consistent edge over VBP on a two-hidden-layer 784784–512512–512512–1010 MNIST MLP, pruning the 512512-unit penultimate layer, evaluated under the same R=2000R=2000 Bernoulli interchange protocol over ten seeds. In the table, CMR-Logit appears under its predecessor name Logit-MSE, and cwvar denotes the curvature-weighted constant-replacement score of Proposition˜4, the predecessor of CMR-Const. Table˜5 summarizes mean ± standard deviation; paired bootstrap CIs at keep =384=384 give ΔIIA(CMR-Logit−VBP)=+0.0007 (CMR-Logit-VBP)=+0.0007 [−0.0037,+0.0052][-0.0037,+0.0052], not significant; at keep =256=256, ΔIIA=+0.0056 =+0.0056 [−0.00085,+0.0123][-0.00085,+0.0123], also not significant, but ΔKL=−0.031 =-0.031 [−0.042,−0.019][-0.042,-0.019] is significant (lower KL is better). Keep Method Test Acc. IIA (p=0.5p=0.5) KL (p=0.5p=0.5) 384 Logit-MSE 0.981±0.0020.981± 0.002 0.952±0.0060.952± 0.006 0.035±0.0070.035± 0.007 VBP 0.981±0.0020.981± 0.002 0.951±0.0040.951± 0.004 0.038±0.0070.038± 0.007 cwvar 0.981±0.0020.981± 0.002 0.930±0.0110.930± 0.011 0.084±0.0230.084± 0.023 256 Logit-MSE 0.980±0.0010.980± 0.001 0.782±0.0060.782± 0.006 0.578±0.0200.578± 0.020 VBP 0.979±0.0020.979± 0.002 0.776±0.0100.776± 0.010 0.609±0.0260.609± 0.026 cwvar 0.980±0.0020.980± 0.002 0.732±0.0120.732± 0.012 0.861±0.0870.861± 0.087 Table 5: MNIST baseline summary (mean ± std over seeds 0–9). The qualitative interpretation is consistent with the small CMR-Logit edge reported on the CIFAR-10 ConvNet in Section˜5.3: the abstraction lens does not provide a large IIA win in the standard setting, but is consistently better on logit fidelity. The reparameterization stress test (Section˜5.1) is where the gap becomes large, not the standard setting. B.3 Boolean circuit sanity check We test the discovery procedure on a controlled three-layer Boolean MLP trained to fit y=XOR(AND(x1,x2),OR(x3,x4))y=XOR(AND(x_1,x_2),OR(x_3,x_4)) from ∈0,18x∈\0,1\^8, where the four irrelevant input coordinates x5,…,x8x_5,…,x_8 are independent Bernoulli(1/2)(1/2). The trained MLP has two hidden layers of 6464 units each, with pruning targeting the 6464-unit penultimate layer; the calibration set is n=2000n=2000 inputs drawn from a 4,0964,096-point dataset with an 80/2080/20 train/test split (Table˜6). Six seeds, R=2000R=2000 Bernoulli swaps at p=0.5p=0.5. Table 6: Boolean circuit interchange-fidelity at keep =32=32 and 1616 (out of 64). CMR-Logit / Logit-MSE matches VBP on IIA and wins on KL at keep =32=32, and is essentially tied at keep =16=16; both clearly beat random. Keep Method Test Acc. IIA (p=0.5p=0.5) KL (p=0.5p=0.5) 32 Logit-MSE 1.000±0.0011.000± 0.001 0.927±0.0200.927± 0.020 0.172±0.0920.172± 0.092 VBP 0.989±0.0240.989± 0.024 0.923±0.0250.923± 0.025 0.234±0.1490.234± 0.149 cwvar 0.981±0.0420.981± 0.042 0.864±0.0250.864± 0.025 0.527±0.2510.527± 0.251 Random 0.921±0.0750.921± 0.075 0.794±0.0580.794± 0.058 1.013±0.3541.013± 0.354 16 Logit-MSE 0.987±0.0290.987± 0.029 0.814±0.0290.814± 0.029 0.806±0.2020.806± 0.202 VBP 0.951±0.0500.951± 0.050 0.827±0.0330.827± 0.033 0.790±0.2120.790± 0.212 cwvar 0.928±0.0670.928± 0.067 0.752±0.0190.752± 0.019 1.234±0.2161.234± 0.216 Random 0.768±0.1140.768± 0.114 0.710±0.0670.710± 0.067 1.509±0.5591.509± 0.559 CMR-Logit (= Logit-MSE in predecessor notation) and VBP are essentially tied on this controlled task, both with high IIA at keep =32=32 (∼0.93 0.93). CMR-Logit attains lower KL (0.172 vs 0.234) and higher Test accuracy (1.000 vs 0.989) at keep =32=32. At more aggressive keep =16=16, VBP edges out CMR-Logit by 0.013 IIA, while CMR-Logit retains a Test accuracy advantage. Random pruning loses substantially at both budgets, confirming that the task is non-trivial. B.4 Affine vs. constant comparison at aggressive sparsity The affine corollary of Theorem˜1 adds capacity but is metric-dependent: it can improve interchange agreement at aggressive budgets while degrading KL fidelity. Table˜7 reports paired-test deltas Δ(affine−constant) (affine-constant) over ten seeds with R=2000R=2000 swaps, on the predecessor MNIST setup at keep ∈64,128∈\64,128\ and parent count r∈4,16r∈\4,16\ with two ridge levels. Table 7: Affine −- constant deltas at keep ∈64,128∈\64,128\ on MNIST with paired bootstrap 95% CIs and seed counts. Positive Δ means affine is larger; for IIA and Test acc., positive is better; for KL, positive is worse. keep r ridge metric Δ (95% CI) seeds p 64 4 10−410^-4 IIA (p=0.5p=0.5) 0.0490 [0.0338, 0.0653] 10 0.000 64 4 10−410^-4 KL (p=0.5p=0.5) 1.2275 [1.1261, 1.3275] 10 0.000 64 4 10−410^-4 acc 0.0564 [0.0315, 0.0855] 10 0.004 64 4 10−210^-2 IIA (p=0.5p=0.5) 0.0519 [0.0378, 0.0668] 10 0.000 64 4 10−210^-2 KL (p=0.5p=0.5) 0.8086 [0.7022, 0.9076] 10 0.000 64 4 10−210^-2 acc 0.0564 [0.0320, 0.0847] 10 0.003 64 16 10−410^-4 IIA (p=0.5p=0.5) 0.0387 [0.0207, 0.0571] 10 0.004 64 16 10−410^-4 KL (p=0.5p=0.5) 1.9373 [1.7361, 2.1020] 10 0.000 64 16 10−410^-4 acc 0.0588 [0.0334, 0.0895] 10 0.004 64 16 10−210^-2 IIA (p=0.5p=0.5) 0.0506 [0.0359, 0.0664] 10 0.000 64 16 10−210^-2 KL (p=0.5p=0.5) 1.1775 [1.0215, 1.3148] 10 0.000 64 16 10−210^-2 acc 0.0588 [0.0334, 0.0893] 10 0.004 128 4 10−410^-4 IIA (p=0.5p=0.5) 0.0130 [0.0075, 0.0184] 10 0.002 128 4 10−410^-4 KL (p=0.5p=0.5) 0.6867 [0.6483, 0.7243] 10 0.000 128 4 10−410^-4 acc 0.0048 [0.0022, 0.0076] 10 0.008 128 4 10−210^-2 IIA (p=0.5p=0.5) 0.0148 [0.0098, 0.0199] 10 0.000 128 4 10−210^-2 KL (p=0.5p=0.5) 0.4629 [0.4019, 0.5224] 10 0.000 128 4 10−210^-2 acc 0.0051 [0.0025, 0.0079] 10 0.007 128 16 10−410^-4 IIA (p=0.5p=0.5) 0.0112 [0.0036, 0.0189] 10 0.025 128 16 10−410^-4 KL (p=0.5p=0.5) 0.9836 [0.9079, 1.0482] 10 0.000 128 16 10−410^-4 acc 0.0052 [0.0025, 0.0081] 10 0.008 128 16 10−210^-2 IIA (p=0.5p=0.5) 0.0138 [0.0076, 0.0199] 10 0.002 128 16 10−210^-2 KL (p=0.5p=0.5) 0.6082 [0.5335, 0.6795] 10 0.000 128 16 10−210^-2 acc 0.0054 [0.0027, 0.0082] 10 0.005 At keep =64=64 with r=16r=16 and ridge 10−210^-2, the affine variant gains ΔIIA=+0.0506 =+0.0506 [+0.0359,+0.0664][+0.0359,+0.0664] and ΔTest acc.=+0.0588 acc.=+0.0588 [+0.0334,+0.0893][+0.0334,+0.0893], but pays ΔKL=+1.18 =+1.18 [+1.02,+1.31][+1.02,+1.31] (worse). The trade-off is sharp: aggressive structured pruning benefits from affine capacity on class-level metrics, but the higher-capacity replacement also amplifies distributional distortion. This is the empirical version of the metric dependence flagged in Section˜6. B.5 Wall-clock timing breakdown Table˜8 decomposes the discovery procedure into score, compile, and verify stages on the predecessor MNIST setup, averaged over seeds with 95% bootstrap CIs. Table 8: CMR stage timings on MNIST, mean ± standard deviation with 95% bootstrap CIs. Score is a single forward pass over the calibration set; compile is the bias / weight folding step; verify is one R=2000R=2000 interchange-intervention pass over precomputed penultimate features (only the compiled head is re-evaluated per swap). keep step variant mean ± std 95% CI 128 compile affine 0.3151±0.02800.3151± 0.0280 s [0.2990, 0.3336] s 256 compile affine 0.2007±0.01640.2007± 0.0164 s [0.1915, 0.2120] s 128 compile const 0.0000841±0.00000550.0000841± 0.0000055 s [0.0000807, 0.0000875] s 256 compile const 0.0000751±0.00000790.0000751± 0.0000079 s [0.0000703, 0.0000800] s all score logit_mse 0.1844±0.04550.1844± 0.0455 s [0.1645, 0.2163] s 128 verify affine 0.004165±0.0003680.004165± 0.000368 s [0.003944, 0.004399] s 256 verify affine 0.005842±0.0011130.005842± 0.001113 s [0.005237, 0.006600] s 128 verify const 0.006269±0.0013890.006269± 0.001389 s [0.005558, 0.007218] s 256 verify const 0.007750±0.0008310.007750± 0.000831 s [0.007222, 0.008256] s Scoring is a one-pass operation independent of replacement class (∼0.18 0.18 s for n=2000n=2000). Constant compilation is essentially free (<0.1<0.1 ms). Affine compilation requires solving the curvature-weighted normal equations from the affine special case of Theorem˜1 and runs in 0.20–0.32 s; the result is amortized across all subsequent verify passes. At this last-layer budget, scoring and affine compilation dominate total wall-clock; verification is cheap (<8<8 ms for R=2000R=2000) because the interchange swaps re-evaluate only the compiled head on precomputed penultimate features. Appendix C Broader Impacts This work is primarily foundational. A positive impact is that mechanism replacement can make neural-network reduction more inspectable: reduced models are compiled artifacts whose behavior can be checked under interchange interventions, rather than only smaller networks with similar accuracy. A potential negative impact is overinterpretation: a reduced computational SCM may be mistaken for a real-world causal explanation or used as evidence of safety in settings where the validation distribution is too narrow. We therefore emphasize the locality of the assumptions, the fixed computational meaning of the state map, and the need for task-specific validation before deployment.