Paper deep dive
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Jiangfeng Chen, Xinyu Wang, Tianshuo Yan, Hanwei Wu, Xiao-Wen Chang, Yang Zhang, Lei Ding
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/2/2026, 9:34:52 AM
Summary
The paper introduces FTC (Sequential Functional Structured Tucker Compression), a post-training compression framework for Large Language Model (LLM) attention mechanisms. Unlike independent matrix approximation methods, FTC jointly exploits the shared structure among Query, Key, and Value (Q/K/V) projections using a Tucker tensor representation while adapting to the representation shift caused by earlier compression steps. It employs an activation-induced metric derived from the current compressed student model to fit shared Q/K/V factors and handles the output projection separately. FTC allocates storage budgets by jointly optimizing Tucker ranks and sparse-core capacity, avoiding the need for fine-tuning or gradient-based recovery. Evaluations on seven decoder-only LLMs (6B to 32B parameters) show that FTC achieves lower WikiText-2 perplexity and better downstream task performance compared to existing methods, particularly under aggressive compression.
Entities (8)
Relation Signals (6)
FTC → appliesto → LLM
confidence 98% · Post-training compression of LLM attention... Across seven decoder-only LLMs... FTC achieves...
FTC → uses → Tucker Compression
confidence 95% · We propose FTC, a sequential structured compression framework... jointly exploiting the native Q/K/V head structure... exploit shared structure through Tucker representations
FTC → achievesbetterperformanceon → WikiText-2
confidence 94% · FTC achieves the lowest WikiText-2 perplexity among the compared methods
FTC → optimizes → Q/K/V Projections
confidence 92% · FTC... jointly exploits the native Q/K/V head structure... fit the query, key, and value projections jointly
FTC → handlesseparately → Output Projection
confidence 90% · The output projection is handled separately to account for the changed post-attention representation.
FTC → uses → Activation-induced metric
confidence 89% · FTC constructs an activation-induced metric from the current compressed model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
Tags
Links
- Source: https://arxiv.org/abs/2610.00717v1
- Canonical: https://arxiv.org/abs/2610.00717v1
Trouble viewing inline? Open PDF directly →
Full Text
123,858 characters extracted from source content.
Expand or collapse full text
Sequential Functional Structured Tucker Compression for Large Language Model Attentions Jiangfeng Chen1,∗ Xinyu Wang2,3,∗ Tianshuo Yan4 Hanwei Wu3,5 Xiao-Wen Chang2 Yang Zhang1 Lei Ding1,† 1University of Manitoba 2McGill University 3Simpleway 4The University of Hong Kong 5McMaster University *Equal contribution †Corresponding author: lei.ding@umanitoba.ca Abstract Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose Ftc, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. Ftc requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, Ftc achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale. 1 Introduction Large language models (LLMs) remain costly at inference time because Transformer blocks repeatedly apply large dense projection matrices, creating substantial memory and computational overhead on resource-constrained hardware. A common approach is to approximate each projection independently with low-rank factors (Yuan et al., 2023; Wang et al., 2025c). This can miss redundancy between projections: the query, key, and value projections read the same hidden input and may use overlapping feature directions. Although independent low-rank factorizations reduce the cost of each projection, they may still spend a fixed budget repeatedly representing feature directions shared across Q/K/V. A joint tensor representation can instead represent these common directions through shared factors, leaving more of the budget to capture the differences across projections and heads. Previous methods like TensorLLM and LeSTD exploit shared structure through Tucker representations (Gu et al., 2025; Li et al., 2026). The remaining challenge is to share useful feature structure without overly restricting the representation available to individual heads. Sharing structure across Q/K/V is not sufficient by itself, since the features worth preserving depend on how approximation error is evaluated.Activation-aware low-rank methods use calibration activations for this purpose (Yuan et al., 2023; Wang et al., 2025c), yet the representations seen by later layers change as compression proceeds. A shared subspace fitted to the original activations may therefore become misaligned with the current compressed model. Moreover, the output projection operates on the post-attention representation, so it requires a separate approximation metric. These observations motivate a sequential framework that adapts the Q/K/V representation to the current compressed model while treating the output projection separately. We propose FTC, a sequential Functional structured Tucker Compression framework for attention projections. At each layer, FTC constructs an activation-induced metric from the current compressed model and uses it to fit the query, key, and value projections jointly. The factors along the model and within-head feature dimensions are shared across projections, while each head slice retains its own core coefficients. Leaving the head-slice mode uncompressed preserves head-specific coefficients and naturally supports both multi-head attention (MHA) and grouped-query attention (GQA) (Ainslie et al., 2023). We also balance the head slices during factor estimation to keep differences in projection magnitude from dominating the shared fit. Sharing feature directions does not require each head to use every possible combination of them. A dense Tucker core nevertheless stores a coefficient for every pair of retained input and within-head directions. We instead allow each head slice to retain a subset of these interactions. The Tucker ranks and the total number of retained coefficients are chosen together, counting the factors, core values, and support metadata in the budget. This provides a choice between fewer shared directions with a denser core and more directions with a sparser core. We then prune the core gradually, refitting the factors and surviving coefficients between stages. The output projection requires a different treatment because it acts on the attention representation rather than on the common input to the query, key, and value projections. After compressing these projections, we recompute the attention features. Using these updated features, we construct a calibrated target for the output projection by matching the original model’s attention-branch outputs with regularization toward the pretrained weights, yielding a closed-form linear solution.. The calibrated target is then compressed under a separate activation-induced metric, and the updated block provides the inputs for compressing the next layer. We evaluate FTC across seven decoder-only LLMs spanning MHA and GQA architectures, with the largest gains appearing under aggressive compression. A matched per-matrix baseline helps separate the effect of the joint structured representation from that of the activation-induced metric and sequential calibration. Our contributions are as follows: • We develop FTC, which jointly fits shared Q/K/V Tucker factors under an activation-induced metric from the current compressed model, while preserving the native MHA/GQA head structure. The output projection is calibrated separately in closed form using updated attention features and teacher branch outputs. • We jointly allocate Tucker ranks and sparse-core capacity under a storage budget accounting for factors, retained coefficients, and support metadata. The resulting allocation balances subspace and sparsification errors, followed by gradual core pruning with intermediate refitting. • Across seven decoder-only LLMs from 6B to 32B, FTC achieves lower WikiText-2 perplexity and higher aggregate downstream accuracy under aggressive compression on all five GQA models, without fine-tuning or gradient-based recovery. A matched per-matrix control further supports the benefit of the structured representation. 2 Related Work Activation-aware low-rank compression. Recent low-rank compression methods extend beyond weight-space reconstruction by incorporating model inputs or parameter importance into the approximation objective. Fisher-weighted SVD uses parameter sensitivity (Hsu et al., 2022), while ASVD and SVD-LLM exploit activation information to improve low-rank approximation (Yuan et al., 2023; Wang et al., 2025c). Later work further considers rank allocation, truncation-error compensation, and numerical stability (Wang et al., 2025b; Li et al., 2025; Parkina and Rakhuba, 2025). Ftc differs by constructing its activation-induced metric from the current compressed model and using it to fit a joint head-wise Q/K/V representation rather than independent projection matrices. Shared and tensorized representations. Shared and tensorized compression methods seek to exploit redundancy across projections or layers through common low-dimensional structure. Basis Sharing uses shared low-rank bases with layer-specific coefficients (Wang et al., 2025a), while TensorLLM and LeSTD model attention projections with Tucker-based shared factors (Gu et al., 2025; Li et al., 2026). Other work explores tensor-train and sparse tensor representations for model compression (Xu et al., 2023; Solgi et al., 2025), as well as joint Q/K/V factorization (Wang et al., 2025d). Recent studies also indicate that overly restrictive shared subspaces may not capture the heterogeneous structure of modern LLM representations (Zagitov et al., 2026). FTC addresses this issue by sharing Tucker factors across Q/K/V and retaining head-specific core coefficients, balancing common structure with head-specific flexibility. Sequential correction and sparse allocation. Sequential compression and error correction have been explored in prior work. SparseGPT performs layerwise activation-aware pruning (Frantar and Alistarh, 2023), while MPIFA and A-SVD introduce reconstruction or refinement steps to reduce accumulated compression error (Zhao et al., 2025; Sinha and Fleuret, 2026). FTC instead uses the current student activations to define the activation-induced metric for the joint Q/K/V tensor approximation, and compresses the output projection separately using the updated attention representation. Pruning methods such as Wanda and SliceGPT remove individual weights or structured hidden dimensions (Sun et al., 2024; Ashkboos et al., 2024), while LoRAP assigns low-rank approximation and structured pruning to different Transformer sublayers (Li et al., 2024). Global low-rank-plus-sparse allocation has also been studied across layers (Zhou et al., 2026). In contrast, FTC couples the Tucker factor ranks and the number of retained latent-core coefficients within each attention layer. Core sparsity selects rank-one interactions between shared directions rather than producing zeros in the reconstructed weight matrices; practical acceleration therefore requires a kernel that exploits the latent representation. Hardware-supported patterns such as 2:4 provide one possible deployment route (Mishra et al., 2021). 3 Function-Driven Structured Compression We compress the attention blocks sequentially using representations generated by the current compressed model. Figure 1 summarizes the overall framework, and the following subsections develop its activation-induced metric, structured Q/K/V approximation, and output-projection calibration. Figure 1: Overview of FTC. At each layer, FTC builds an activation-induced metric from the current student representations, jointly compresses the Q/K/V projections with a shared Tucker representation, and allocates the layer budget between Tucker ranks and sparse-core capacity. The core is then refined progressively, while WOW_O is calibrated and compressed separately using the updated attention representation. The resulting student representation is passed to the next layer, forming a sequential compression process. 3.1 Preliminaries and Problem Formulation Consider the l-th Transformer block and let XlS∈ℝN×dmodelX_l^S ^N× d_model denote the hidden representations produced by the current compressed student model, where the superscript SS denotes the student stream and N is the number of tokens. Since the preceding layers have already been compressed, XlSX_l^S reflects the representation encountered by the current student at layer l. Let hq,lh_q,l and hkv,lh_kv,l denote the numbers of query and key/value heads, respectively. We stack the corresponding head-wise Q/K/V projection matrices along the third mode to form the joint tensor l∈ℝdmodel×dhead×ml,ml=hq,l+2hkv,l.W_l ^d_model× d_head× m_l, m_l=h_q,l+2h_kv,l. The third mode contains all query-head slices together with the native key- and value-head slices. Hence, in GQA and MQA, shared K/V heads are included only once rather than being replicated to match the number of query heads. This joint representation exposes the shared feature structure across the Q/K/V projections while retaining their native head organization. Since all Q/K/V projections operate on the same hidden representation, we compress them jointly while sharing the first two Tucker factors (Kolda and Bader, 2009): ^l=l×1U1,l×2U2,l, W_l=G_l×_1U_1,l×_2U_2,l, where U1,l∈ℝdmodel×R1,lU_1,l ^d_model× R_1,l, U2,l∈ℝdhead×R2,lU_2,l ^d_head× R_2,l, and l∈ℝR1,l×R2,l×mlG_l ^R_1,l× R_2,l× m_l is a latent core tensor. Sharing U1,lU_1,l and U2,lU_2,l exploits common feature subspaces across Q/K/V, while leaving the third mode uncompressed. Moreover, the output projection WO,lW_O,l is compressed separately due to its distinct input representation; implementation details are provided in Appendix B. For Q/K/V, we do not directly regress to teacher outputs. Instead, we measure the effect of a weight perturbation on the current student activation stream. For a projection perturbation ΔWl W_l, 1N‖XlSΔWl‖F2=tr(ΔWl⊤Cs,lΔWl),Cs,l=1N(XlS)⊤XlS, 1N \|X_l^S W_l \|_F^2=tr ( W_l C_s,l W_l ), C_s,l= 1N(X_l^S) X_l^S, where Cs,lC_s,l is the empirical second-moment matrix of the current student representations. We therefore define the regularized activation-induced metric Hε,l=Cs,l+εlt¯lI,εl>0,t¯l=tr(Cs,l)/dmodel,H_ ,l=C_s,l+ _l t_lI, _l>0, t_l=tr(C_s,l)/d_model, where t¯l t_l is the mean diagonal value of Cs,lC_s,l, which normalizes the damping scale across layers, and εl _l controls the regularization strength. For the joint Q/K/V tensor, the resulting structured approximation problem takes the form min^l∈ℳl(Bl)‖^l−l‖Hε,l2, _ W_l _l(B_l) \| W_l-W_l \|_H_ ,l^2, (1) where ℳl(Bl)M_l(B_l) is the family of such structured representations whose total storage cost satisfies a layerwise compression budget BlB_l, including the shared factors, retained core values, and associated index metadata. Since Hε,lH_ ,l is constructed from the current student representations, the approximation metric is updated sequentially as compression proceeds. Equation (1) defines the activation-induced approximation objective. We first convert this objective exactly into a Euclidean reconstruction problem by metric whitening. We then introduce slice balancing to prevent high-energy Q/K/V head projections from dominating the shared Tucker factors. 3.2 Sequential Activation-Induced Tensor Approximation We now transform the activation-metric problem in Eq. (1) into a standard Euclidean tensor approximation. Since Hε,l≻0H_ ,l 0, let Hε,l=Ll⊤LlH_ ,l=L_l L_l be its Cholesky factorization. For any tensor ∈ℝdmodel×dhead×mlA ^d_model× d_head× m_l, the weighted norm satisfies ‖Hε,l2=‖×1Ll‖F2.\|A\|_H_ ,l^2=\|A×_1L_l\|_F^2. We therefore define the metric-whitened tensor ~l=l×1Ll. W_l=W_l×_1L_l. For the Tucker representation ^l=l×1U1,l×2U2,l, W_l=G_l×_1U_1,l×_2U_2,l, setting U~1,l=LlU1,l U_1,l=L_lU_1,l gives ^l×1Ll=l×1U~1,l×2U2,l. W_l×_1L_l=G_l×_1 U_1,l×_2U_2,l. Hence, Eq. (1) is equivalently written as min‖~l−l×1U~1,l×2U2,l‖F2, \| W_l-G_l×_1 U_1,l×_2U_2,l \|_F^2, with the layerwise storage budget BlB_l. After compression, the mode-1 factor is mapped back by U1,l=Ll−1U~1,lU_1,l=L_l^-1 U_1,l. Thus, metric whitening exactly converts the activation-induced objective into a Euclidean tensor approximation problem. The following subsection introduces a separate slice-level reweighting to balance the contributions of Q/K/V head projections with different magnitudes. 3.3 Shared Tucker Representation with Slice Balancing After metric whitening, the activation-induced objective becomes a Euclidean reconstruction problem. We further balance the Q/K/V head slices when estimating the shared Tucker factors, so that high-energy slices do not dominate the shared subspaces. Specifically, let W~l,j:=~l(:,:,j),sj,l:=∥W~l,j∥F,wj,l:=sj,l−2, W_l,j:= W_l(:,:,j), s_j,l:=\| W_l,j\|_F, w_j,l:=s_j,l^-2, for j=1,…,mlj=1,…,m_l. We therefore consider minl,U~1,l,U2,l∑j=1mlwj,l‖W~l,j−U~1,l(:,:,j)U2,l⊤‖F2,s.t.U~1,l⊤U~1,l=I,U2,l⊤U2,l=I. _G_l, U_1,l,U_2,l _j=1^m_lw_j,l \| W_l,j- U_1,lG_l(:,:,j)U_2,l \|_F^2, .t. U_1,l U_1,l=I, U_2,l U_2,l=I. (2) Since wj,l=‖W~l,j‖F−2w_j,l=\| W_l,j\|_F^-2, each term measures the reconstruction error relative to the energy of the corresponding slice. Thus, each Q/K/V slice contributes according to its relative approximation error rather than its absolute magnitude. To characterize the shared model-dimension subspace, we temporarily omit the mode-22 compression and consider the reduced problem min∑j=1mlB1,l,…,Bml,lwj,l‖W~l,j−U~lBj,l‖F2,s.t.U~l⊤U~l=I. _ subarraycB_1,l,…,B_m_l,l subarray _j=1^m_lw_j,l \| W_l,j- U_lB_j,l \|_F^2, .t. U_l U_l=I. (3) For fixed U~l U_l, the optimal coefficient matrices are Bj,l⋆=U~l⊤W~l,j.B_j,l = U_l W_l,j. Consequently, the optimal rank-R1,lR_1,l shared column subspace is spanned by the leading left singular vectors of W~cat,l=[w1,lW~l,1⋯wml,lW~l,ml]. W_cat,l= bmatrix w_1,l W_l,1&·s& w_m_l,l W_l,m_l bmatrix. Hence, the shared mode-11 factor captures the dominant directions jointly supported by the normalized Q/K/V head projections. A formal proof is given in Appendix D. We extend this shared-subspace principle to the first two modes through the Tucker representation ~^l=l×1U~1,l×2U2,l, W_l=G_l×_1 U_1,l×_2U_2,l, while leaving the third mode uncompressed. Thus, the first two factors are shared across Q/K/V head slices, whereas each core slice retains its own coefficients. The next subsection jointly allocates the layerwise storage budget between the Tucker ranks and the retained sparse-core coefficients. 3.4 Budget-Coupled Rank–Sparsity Allocation We allocate the layerwise storage budget among the shared Tucker factors, the retained latent-core coefficients, and their support metadata. For a candidate rank pair l=(R1,l,R2,l)R_l=(R_1,l,R_2,l), define Pfac(l)=dmodelR1,l+dheadR2,l,Nl(l)=R1,lR2,lml.P_fac(R_l)=d_modelR_1,l+d_headR_2,l, N_l(R_l)=R_1,lR_2,lm_l. The sparse-core support is stored as a bit-packed binary mask ℳl∈0,1R1,l×R2,l×mlM_l∈\0,1\^R_1,l× R_2,l× m_l with ‖ℳl‖0=Kl\|M_l\|_0=K_l, requiring one bit per latent-core location. Only the KlK_l surviving core coefficients are stored, in packed form. If each stored value uses bvb_v bits, the total storage cost is Slbit(l,Kl)=bv[Pfac(l)+Kl]+Nl(l).S^bit_l(R_l,K_l)=b_v [P_fac(R_l)+K_l ]+N_l(R_l). Let Bl=Blbit/bvB_l=B_l^bit/b_v denote the layer budget in bvb_v-bit scalar equivalents. The largest admissible core cardinality is Kl(l)=minNl(l),⌊Bl−Pfac(l)−Nl(l)bv⌋,Pfac(l)+Nl(l)bv≤Bl.K_l(R_l)= \N_l(R_l), B_l-P_fac(R_l)- N_l(R_l)b_v \, P_fac(R_l)+ N_l(R_l)b_v≤ B_l. To evaluate candidate rank pairs, we absorb the slice-balancing weights into the whitened tensor, ¯l(:,:,j)=wj,l~l(:,:,j),j=1,…,ml. W_l(:,:,j)= w_j,l\, W_l(:,:,j), j=1,…,m_l. For fixed orthonormal Tucker factors, the projection core is ¯l⋆=¯l×1U~1,l⊤×2U2,l⊤, G_l = W_l×_1 U_1,l ×_2U_2,l , with corresponding subspace approximation error Esub(l)=‖¯l−¯l⋆×1U~1,l×2U2,l‖F2.E_sub(R_l)= \| W_l- G_l ×_1 U_1,l×_2U_2,l \|_F^2. Given a core cardinality KlK_l, let Kl(¯l⋆)P_K_l( G_l ) retain the KlK_l largest-magnitude coefficients of the projection core. The corresponding one-step sparsification error is Dsp(l,Kl)=‖¯l⋆−Kl(¯l⋆)‖F2.D_sp(R_l,K_l)= \| G_l -P_K_l ( G_l ) \|_F^2. For fixed orthonormal Tucker factors and their projection core, the subspace residual is orthogonal to the represented Tucker subspace. Consequently, the error further decomposes exactly as Estruct(l,Kl)=Esub(l)+Dsp(l,Kl).E_struct(R_l,K_l)=E_sub(R_l)+D_sp(R_l,K_l). (4) This decomposition exposes the budget trade-off: increasing the Tucker ranks reduces the subspace approximation error, but consumes more storage in the shared factors and support mask, potentially leaving fewer coefficients for the core. We therefore optimize the Tucker ranks and core cardinality jointly under the layerwise storage budget. The resulting problem admits the following reduction: Proposition 3.1 (Budget-coupled rank–sparsity allocation). For each feasible rank pair l∈ℛlR_l _l, let Kl(l)K_l(R_l) be the largest core cardinality allowed by the storage budget. Since Dsp(l,Kl)D_sp(R_l,K_l) is non-increasing in KlK_l, the joint surrogate problem minl,KlEsub(l)+βDsp(l,Kl),β∈(0,1], _R_l,K_l\;E_sub(R_l)+β D_sp(R_l,K_l), β∈(0,1], subject to the layerwise storage budget, reduces to l⋆=argminl∈ℛlEsub(l)+βDsp(l,Kl(l)),Kl⋆=Kl(l⋆).R_l = _R_l _lE_sub(R_l)+β D_sp (R_l,K_l(R_l) ), K_l =K_l(R_l ). Here, β discounts the one-step sparsification cost to account for the subsequent prune–refit refinement. After allocation, the selected core is gradually pruned to Kl⋆K_l with support-constrained refitting; details are given in Appendix E. 3.5 Separate Compression of the Output Projection The output projection is treated separately since it operates on the post-attention representation rather than on the block input XlSX_l^S shared by Q/K/V. After compressing Q/K/V, we recompute the student attention representation ZlSZ_l^S and use it to construct a dedicated activation-induced metric for WOW_O. When output-projection calibration is enabled, we additionally use the corresponding teacher representation ZlTZ_l^T to define a target adapted to the compressed attention features. Specifically, we solve minW1N‖ZlSW−ZlTWO,l‖F2+μo‖W−WO,l‖F2, _W 1N \|Z_l^SW-Z_l^TW_O,l \|_F^2+ _o \|W-W_O,l \|_F^2, which balances recovery of the teacher attention-branch output against deviation from the pretrained output projection. This quadratic problem has a closed-form solution, denoted by WO,lcalW_O,l^cal. When calibration is disabled, we simply use the original WO,lW_O,l as the compression target. After replacing WO,lW_O,l with its compressed counterpart, the updated block produces Xl+1SX_l+1^S, which is used to construct the metric for the next layer. The procedure therefore continues sequentially through the model. Full calibration and implementation details are provided in Appendix B. 4 Experiments We evaluate our Ftc across decoder-only LLMs of different scales and attention architectures. We report perplexity and downstream performance, followed by ablations and analyses of the main components. Below we first introduce the models, baselines and protocol of our experiments: Models. We evaluate on two MHA models (GPT-J-6B, Llama-2-13B) and five modern GQA models (Mistral-7B-v0.3, Llama-3-8B, Qwen2.5-7B, Qwen3-8B-Base with QK-norm, and Qwen3-32B). Among them, Qwen3-32B is further used to test scale and architectural generalization. Baselines. We compare Ftc with five published baselines: enh-LeSTD, our GQA-capable port of LeSTD using the same head-concat tensorization (Li et al., 2026); TLLM-gen, a TensorLLM-style data-free dense Tucker baseline (Gu et al., 2025); MPIFA (Zhao et al., 2025); SVD-LLM (Wang et al., 2025c); and Basis Sharing (Wang et al., 2025a). Seq-SVD is our per-matrix control under the same functional metric and sequential calibration as Ftc, rather than a published baseline. Protocol. Ftc uses 256 WikiText-2 calibration sequences of 2,048 tokens (seed 42), while data-dependent baselines follow their published 128-sequence configurations; Table 17 quantifies this difference. All methods use the same evaluation windows and parameter-count accounting. Following LeSTD, ρ denotes the retained storage ratio; smaller ρ means stronger compression. The detailed storage accounting is provided in Appendix A.1. All main-table Ftc results use one fixed recipe across models and ratios: the budget-allocation parameter β=0.5β=0.5, the teacher-Gram anchoring parameter a=0.5a=0.5, no QKV teacher map, a closed-form teacher map for WOW_O with regularization parameter μ=1μ=1, the gradual core-pruning rate α=0.02α=0.02, two refits, 256 calibration windows, and slice-wise normalization. No fine-tuning or gradient-based recovery is used. 4.1 Perplexity across three model generations We first evaluate model-level degradation using WikiText-2 perplexity across all seven models and compression ratios. Table 1 reports two representative operating points, while Figure 2 shows the full compression sweep. Table 1: WikiText-2 perplexity at ρ∈0.6,0.2ρ∈\0.6,0.2\ (dense ppl in parentheses; best per row in bold). All methods use the same realised compression ratio. Here, “enh-L”, “TLLM”, and “SVD” abbreviate enh-LeSTD, TLLM-gen, and SVD-LLM, respectively. ρ=0.6ρ=0.6 ρ=0.2ρ=0.2 Model Ftc Seq-SVD enh-L TLLM MPIFA SVD Basis Ftc Seq-SVD enh-L TLLM MPIFA SVD Basis GPT-J-6B (8.86) 9.30 9.47 9.93 9.68 9.46 10.26 10.16 13.60 14.84 17.47 19.22 17.24 25.04 23.02 Llama-2-13B (4.89) 5.11 5.27 5.13 5.52 5.26 5.68 5.57 7.52 9.00 9.79 13.01 10.21 14.20 12.77 Mistral-7B (5.32) 5.44 5.69 5.58 6.07 5.63 5.92 5.75 7.33 12.10 146.31 18.41 15.92 25.40 16.18 Llama-3-8B (6.14) 6.64 7.03 6.82 8.70 7.26 8.02 7.60 10.44 19.46 20.84 29.85 27.02 48.09 35.60 Qwen2.5-7B (6.85) 7.20 7.98 7.76 34.10 7.84 8.56 7.82 12.55 150.93 106.59 188.22 7791 255.23 140.65 Qwen3-8B (7.00) 7.36 7.65 8.20 9.12 8.17 9.24 9.18 10.70 23.66 27.08 45.45 22.23 75.77 46.68 Qwen3-32B (7.61) 7.77 7.98 7.84 8.56 8.40 9.23 8.70 9.22 22.97 15.23 29.71 26.75 66.29 29.63 Figure 2: WikiText-2 perplexity versus retained attention-projection storage ratio across seven models. Dashed lines denote the corresponding dense-model perplexity. The results reveal three consistent trends across model families and compression levels. (1) Aggressive compression magnifies the performance gap. The gap widens sharply as compression becomes more aggressive. At ρ=0.2ρ=0.2, with WikiText-2 calibration, Ftc achieves the best perplexity on all five modern GQA models and remains within 1.211.21–1.83×1.83× dense WikiText-2 perplexity. In contrast, SVD-LLM reaches 4.84.8–37.3×37.3× dense perplexity. At ρ=0.2ρ=0.2, Ftc reduces perplexity by 3939–88%88\% relative to the best competing baseline on each GQA model. (2) Joint tensorization provides substantial gains beyond the metric and sequential calibration. Seq-SVD serves as a controlled per-matrix counterpart to Ftc, sharing the same activation-induced metric and sequential calibration while factorizing each projection independently. At ρ=0.2ρ=0.2, Ftc achieves 3939–92%92\% lower perplexity across the five GQA models, highlighting the substantial contribution of joint tensorization beyond the activation-induced metric and calibration alone. (3) The advantage persists at 32B scale. On Qwen3-32B, FTC reaches 9.229.22 perplexity at ρ=0.2ρ=0.2, or 1.21×1.21× the dense perplexity, compared with 15.2315.23 for the strongest competing baseline (39.5%39.5\% lower). On Qwen3-8B, the corresponding reduction is 51.9%51.9\% (10.7010.70 vs. 22.2322.23). Thus, FTC retains the lowest perplexity at both scales, although the relative margin is smaller on the 32B model. 4.2 Downstream tasks We next examine whether the perplexity improvements translate to downstream performance. Table 2 reports avg6 and 5-shot GSM8K accuracy at representative storage ratios, while Figure 3 shows performance retention across the full compression sweep. Table 2: Downstream performance on modern GQA models at ρ∈0.6,0.4,0.2ρ∈\0.6,0.4,0.2\. avg6 is the mean normalized accuracy over six multiple-choice tasks; GSM8K is 5-shot exact match. Dense results are shown in the row labels; full per-task results are in Appendix A.3. Ftc Seq-SVD enh-LeSTD TLLM-gen MPIFA SVD-LLM Basis Model ρ a6 G8K a6 G8K a6 G8K a6 G8K a6 G8K a6 G8K a6 G8K Mistral-7B (0.740 / 0.285) 0.6 0.726 0.24 0.684 0.17 0.707 0.15 0.654 0.10 0.695 0.20 0.667 0.17 0.688 0.20 0.4 0.689 0.10 0.569 0.01 0.625 0.01 0.501 0.01 0.552 0.02 0.505 0.02 0.549 0.03 0.2 0.512 0.01 0.359 0.00 0.320 0.00 0.342 0.01 0.314 0.01 0.307 0.01 0.294 0.00 Llama-3-8B (0.735 / 0.435) 0.6 0.716 0.29 0.642 0.09 0.694 0.20 0.597 0.04 0.639 0.01 0.575 0.02 0.615 0.04 0.4 0.668 0.10 0.510 0.04 0.601 0.04 0.447 0.03 0.448 0.02 0.410 0.04 0.474 0.03 0.2 0.488 0.02 0.345 0.01 0.357 0.01 0.327 0.01 0.306 0.00 0.308 0.03 0.315 0.01 Qwen2.5-7B (0.717 / 0.820) 0.6 0.689 0.71 0.631 0.45 0.658 0.49 0.326 0.01 0.666 0.53 0.624 0.42 0.669 0.54 0.4 0.659 0.45 0.332 0.01 0.490 0.01 0.300 0.01 0.456 0.04 0.398 0.02 0.522 0.09 0.2 0.405 0.01 0.305 0.01 0.298 0.00 0.297 0.01 0.289 0.03 0.297 0.01 0.296 0.01 Qwen3-8B (0.725 / 0.855) 0.6 0.707 0.73 0.668 0.46 0.644 0.54 0.586 0.23 0.670 0.44 0.601 0.20 0.661 0.20 0.4 0.624 0.10 0.561 0.09 0.529 0.07 0.432 0.01 0.522 0.08 0.425 0.01 0.561 0.04 0.2 0.500 0.01 0.342 0.01 0.329 0.01 0.304 0.00 0.358 0.00 0.322 0.01 0.335 0.01 Qwen3-32B (0.743 / 0.885) 0.6 0.733 0.77 0.730 0.64 0.717 0.77 0.687 0.70 0.722 0.71 0.684 0.67 0.701 0.60 0.4 0.733 0.75 0.679 0.55 0.682 0.51 0.618 0.21 0.629 0.41 0.525 0.04 0.657 0.44 0.2 0.671 0.28 0.375 0.00 0.460 0.01 0.355 0.00 0.372 0.01 0.335 0.00 0.369 0.01 Figure 3: Downstream retention versus retained storage ratio on five GQA models. Top: avg6 relative to dense; bottom: 5-shot GSM8K relative to dense. The dashed line marks 10% GSM8K retention. The downstream results reveal two main patterns across tasks, model scales, and compression levels. (1) Generative reasoning is more sensitive to compression than multiple-choice performance. At ρ=0.4ρ=0.4, Ftc retains 8686–93%93\% of the dense avg6 on the four 7–8B models, whereas its GSM8K retention ranges from only 1212–55%55\%. This gap is substantially smaller at larger scale: Qwen3-32B retains 98.7%98.7\% of its dense avg6 and 84.7%84.7\% of its dense GSM8K accuracy at the same compression ratio. These results indicate that reasoning-intensive generation degrades earlier under compression, while increased model scale provides considerably greater robustness. (2) Performance differences become more pronounced under aggressive compression. At ρ=0.2ρ=0.2, Ftc exceeds the best published baseline by 10.710.7–17.017.0 percentage points in avg6 across the four 7–8B models, and by 21.121.1 points on Qwen3-32B. Thus, even when GSM8K has collapsed to near-zero accuracy on smaller models, the multiple-choice aggregate continues to reveal substantial differences in the quality retained by different compression schemes. 4.3 Attribution and ablations We next isolate the contribution of the main components of FTC through controlled ablations under the same realized compression budget. Table 3 reports the resulting WikiText-2 perplexity after removing or replacing one component at a time. Table 3: Component ablations on WikiText-2. “full” denotes the final fixed head-concat recipe; each column removes or replaces one component at the same realized storage ratio and reports perplexity with the relative change in parentheses. Relative changes are computed against the matching full-recipe result in the same row. Model ρ full w/o whitening w/o slice-wise norm w/o seq. calibration w/o anchoring K/V repeat-expand dense core no joint tensor w/o WOW_O target Mistral-7B 0.6 5.440 5.56 (+2.2%) 5.50 (+1.1%) 5.46 (+0.4%) 5.44 (+0.0%) 5.53 (+1.6%) 6.37 (+17.2%) 5.69 (+4.7%) 5.46 (+0.4%) Mistral-7B 0.2 7.334 13.19 (+79.8%) 9.33 (+27.2%) 13.24 (+80.6%) 7.45 (+1.5%) 10.41 (+41.9%) 7.65 (+4.3%) 12.10 (+65.0%) 10.15 (+38.4%) Llama-3-8B 0.6 6.640 6.73 (+1.4%) 7.01 (+5.5%) 6.94 (+4.5%) 6.60 (-0.6%) 6.87 (+3.4%) 8.45 (+27.3%) 7.03 (+5.9%) 6.94 (+4.5%) Llama-3-8B 0.2 10.440 20.56 (+96.9%) 19.77 (+89.4%) 33.65 (+222.4%) 10.87 (+4.1%) 17.18 (+64.6%) 11.19 (+7.2%) 19.46 (+86.4%) 29.50 (+182.6%) Qwen2.5-7B 0.6 7.197 7.77 (+7.9%) 7.17 (-0.3%) 7.24 (+0.5%) 7.24 (+0.5%) 7.24 (+0.6%) 12.71 (+76.6%) 7.98 (+10.9%) 7.19 (-0.1%) Qwen2.5-7B 0.2 12.548 221.15 (+1662.4%) 74.34 (+492.4%) 21.36 (+70.2%) 14.36 (+14.5%) 12.88 (+2.6%) 26.89 (+114.3%) 150.93 (+1102.8%) 12.88 (+2.6%) Qwen3-8B 0.6 7.362 8.22 (+11.7%) 7.35 (-0.2%) 7.52 (+2.2%) 7.42 (+0.8%) 7.52 (+2.2%) 9.28 (+26.1%) 7.65 (+3.9%) 7.54 (+2.5%) Qwen3-8B 0.2 10.700 32.28 (+201.7%) 11.96 (+11.8%) 59.68 (+457.7%) 11.35 (+6.1%) 14.53 (+35.8%) 11.63 (+8.7%) 23.66 (+121.2%) 15.24 (+42.4%) The ablation results reveal three main patterns in how the individual components contribute across compression levels and model architectures. (1) Metric whitening, sequential calibration, and joint tensorization become increasingly important as the compression budget tightens. At ρ=0.6ρ=0.6, removing whitening, sequential calibration, or joint tensorization increases perplexity by at most 11.7%11.7\%. At ρ=0.2ρ=0.2, the corresponding penalties rise to 79.879.8–1662.4%1662.4\%, 70.270.2–457.7%457.7\%, and 65.065.0–1102.8%1102.8\%, respectively. These results show that activation-induced metric, adaptation to the evolving student stream, and joint Q/K/V modeling become increasingly important as the compression budget tightens. (2) The contributions of the remaining components vary more substantially across models. At ρ=0.2ρ=0.2, removing anchoring changes perplexity by only 1.51.5–14.5%14.5\%, whereas removing slice-wise normalization costs 11.811.8–492.4%492.4\%. Replacing the sparse core with its matched-budget dense counterpart incurs only 4.34.3–8.7%8.7\% on three models but 114.3%114.3\% on Qwen2.5-7B. The WOW_O teacher target is beneficial on all four models, with its removal increasing perplexity by 2.62.6–182.6%182.6\%. (3) Fixed head-concat is consistently preferred to K/V repeat-expansion. At ρ=0.2ρ=0.2, replacing head-concat with K/V repeat-expansion increases perplexity by 41.9%41.9\% on Mistral-7B, 64.6%64.6\% on Llama-3-8B, 2.6%2.6\% on Qwen2.5-7B, and 35.8%35.8\% on Qwen3-8B. The magnitude varies substantially across architectures, but the native head-concat construction is preferred on all four GQA models. Additional teacher-map and hyperparameter studies are reported in Appendix A.4. 5 Conclusion We proposed Ftc, a structured attention-compression framework combining an activation-induced metric, sequential calibration, joint Q/K/V tensorization, and budget-coupled rank–sparsity allocation. Across MHA and GQA models from 6B to 32B, Ftc achieves consistently lower perplexity and stronger downstream retention than the compared methods, without fine-tuning or gradient-based recovery. Ablations further show that metric whitening, sequential calibration, and joint tensorization become increasingly important as the compression budget tightens. References Ainslie et al. (2023) J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai GQA: training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, p. 4895–4901. External Links: Link Cited by: §1. Ashkboos et al. (2024) S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman SliceGPT: compress large language models by deleting rows and columns. In ICLR, External Links: Link Cited by: §2. Frantar and Alistarh (2023) E. Frantar and D. Alistarh SparseGPT: massive language models can be accurately pruned in one-shot. In ICML, p. 10323–10337. External Links: Link Cited by: §2. Gu et al. (2025) Y. Gu, W. Zhou, G. Iacovides, and D. Mandic TensorLLM: tensorising multi-head attention for enhanced reasoning and compression in LLMs. In IJCNN, External Links: Link Cited by: §1, §2, §4. Hsu et al. (2022) Y. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, and H. Jin Language model compression with weighted low-rank factorization. In ICLR, External Links: Link Cited by: §2. Kolda and Bader (2009) T. G. Kolda and B. W. Bader Tensor decompositions and applications. SIAM review, p. 455–500. Cited by: §3.1. Li et al. (2024) G. Li, Y. Tang, and W. Zhang LoRAP: transformer sub-layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695. External Links: Link Cited by: §2. Li et al. (2026) Y. Li, Z. Guo, M. Yin, and B. Li LeSTD: LLM compression via learning-based sparse tensor decomposition. In ICLR, External Links: Link Cited by: §1, §2, §4. Li et al. (2025) Z. Li, M. Xia, J. Zhang, Z. Hui, H. Qin, L. Kong, Y. Zhang, and X. Yang AdaSVD: adaptive singular value decomposition for large language models. arXiv preprint arXiv:2502.01403. External Links: Link Cited by: §2. Mishra et al. (2021) A. Mishra, J. A. Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, and P. Micikevicius Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378. External Links: Link Cited by: §2. Parkina and Rakhuba (2025) U. Parkina and M. Rakhuba COALA: numerically stable and efficient framework for context-aware low-rank approximation. arXiv preprint arXiv:2507.07580. External Links: Link Cited by: §2. Sinha and Fleuret (2026) A. K. Sinha and F. Fleuret A-SVD: anchored and adaptive SVD for large language model compression. arXiv preprint arXiv:2604.02119. External Links: Link Cited by: §2. Solgi et al. (2025) R. Solgi, K. Zhen, R. V. Swaminathan, N. Susanj, A. Mouchtaris, S. Kunzmann, and Z. Zhang Saten: sparse augmented tensor networks for post-training compression of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 23674–23683. External Links: Link Cited by: §2. Sun et al. (2024) M. Sun, Z. Liu, A. Bair, and J. Z. Kolter A simple and effective pruning approach for large language models. In ICLR, External Links: Link Cited by: §2. Wang et al. (2025a) J. Wang, Y. Chen, I. Lin, B. Li, and G. L. Zhang Basis sharing: cross-layer parameter sharing for large language model compression. In ICLR, p. 13386–13404. External Links: Link Cited by: §2, §4. Wang et al. (2025b) X. Wang, S. Alam, Z. Wan, H. Shen, and M. Zhang SVD-LLM v2: optimizing singular value truncation for large language model compression. In NAACL, p. 4287–4296. External Links: Link Cited by: §2. Wang et al. (2025c) X. Wang, Y. Zheng, Z. Wan, and M. Zhang Svd-llm: truncation-aware singular value decomposition for large language model compression. In ICLR, p. 19299–19319. External Links: Link Cited by: §1, §1, §2, §4. Wang et al. (2025d) Y. Wang, H. Wang, and S. Q. Zhang QSVD: efficient low-rank approximation for unified query-key-value weight compression in low-precision vision-language models. arXiv preprint arXiv:2510.16292. External Links: Link Cited by: §2. Xu et al. (2023) M. Xu, Y. L. Xu, and D. P. Mandic TensorGPT: efficient compression of large language models based on tensor-train decomposition. arXiv preprint arXiv:2307.00526. External Links: Link Cited by: §2. Yuan et al. (2023) Z. Yuan, Y. Shang, Y. Song, D. Yang, Q. Wu, Y. Yan, and G. Sun Asvd: activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. Cited by: §1, §1, §2. Zagitov et al. (2026) A. Zagitov, A. Miasnikov, M. Krutikov, V. Aletov, G. Molodtsov, N. Bashirov, A. Tsedenov, and A. Beznosikov Rethinking the role of tensor decompositions in post-training LLM compression. arXiv preprint arXiv:2606.03465. External Links: Link Cited by: §2. Zhao et al. (2025) J. Zhao, Y. Zhang, and C. V. Cannistraci Pivoting factorization: a compact meta low-rank representation of sparsity for efficient inference in large language models. arXiv preprint arXiv:2501.19090. External Links: Link Cited by: §2, §4. Zhou et al. (2026) C. Zhou, Q. Qiao, Y. Zhou, Y. Wu, S. Weng, W. Zhang, and C. Jin Large language model compression with global rank and sparsity optimization. arXiv preprint arXiv:2505.03801. External Links: Link Cited by: §2. AI Use Statement In preparing this work, we used generative AI tools to obtain feedback on the research methodology and experimental design, review mathematical derivations for potential errors and inconsistencies, and assist with the interpretation of experimental results. We also used these tools to draft and revise portions of the manuscript, improve its readability and organization, identify relevant literature, and prepare a draft workflow diagram. We have reviewed all AI-assisted material retained in this work. Specifically, we independently checked the mathematical arguments, verified references against the original sources, and cross-checked numerical statements against our experimental records. We revised AI-assisted text and figures to ensure that they accurately describe the methods and results. We take full responsibility for the final content of this work, including all text, claims, and artifacts produced with the assistance of generative AI. Appendix A Experimental details and full results A.1 Reproduction protocol and accounting LeSTD reproduction. We reproduce LeSTD from its published description and first verify the implementation against its reported GPT-J results. Native LeSTD is not directly applicable to GQA because its per-head factorization assumes one-to-one Q/K/V head structure. We therefore use enh-LeSTD, a GQA-compatible port that adopts the same native head-concat tensorization as Ftc while retaining LeSTD’s data-free objective. This isolates the compression objective from the tensorization required for GQA support. Accounting. We define ρ as the retained storage ratio of the compressed attention projections. For Q/K/V, the accounting includes the shared Tucker factors, packed nonzero core values, and the bit-packed support mask; WO,lW_O,l is compressed and counted separately. Thus, sparse-core metadata is included in ρ. Let sattns_attn denote the fraction of dense model storage occupied by the compressed attention projections. Since all dense weights use the same precision, this equals their parameter fraction. Leaving the remaining model unchanged, the corresponding whole-model storage reduction is (1−ρ)sattn.(1-ρ)s_attn. Table 4 reports this conversion. Table 4: Storage accounting at each attention-projection storage budget ρ. Realized storage is determined by the selected layerwise Tucker ranks (R1,l,R2,l)(R_1,l,R_2,l), sparse-core cardinality KlK_l, output-projection rank rO,lr_O,l, and bit-packed support mask. Whole-model storage removed (%) Model Attn. share ρ=.8ρ=.8 ρ=.6ρ=.6 ρ=.4ρ=.4 ρ=.2ρ=.2 GPT-J-6B 31.1% 6.2 12.4 18.7 24.9 Llama-2-13B 32.2% 6.4 12.9 19.3 25.8 Mistral-7B 18.5% 3.7 7.4 11.1 14.8 Llama-3-8B 16.7% 3.3 6.7 10.0 13.4 Qwen2.5-7B 10.8% 2.2 4.3 6.5 8.6 Qwen3-8B 18.4% 3.7 7.4 11.0 14.7 Calibration and evaluation. Ftc uses 256 calibration sequences of 2,048 tokens (WikiText-2 train, seed 42) on every model; the data-dependent baselines use the 128 sequences of their published configurations, and Table 17 quantifies the effect of this difference. Perplexity on WikiText-2 test with stride-2048 windows; PTB and C4 in App. A.2. Tasks: 1,000 examples per task (983 for MathQA, 200 for GSM8K), acc_norm for multiple choice, exact-match accuracy for 5-shot GSM8K (batch 8, greedy). Large-model specifics. Qwen3-32B does not fit one 40GB card in bf16, so compression runs layer-at-a-time with the weights on CPU (only the embedding, the shared rotary table and one decoder layer are resident) and task evaluation shards the model across two cards with CPU spill. Nothing else changes: same recipe, same 256 calibration windows, same accounting. Compression cost. Table 5 reports the wall-clock time of the complete Ftc pipeline. On a single A100-40GB, the 7–8B models require 16–39 minutes per compression setting, GPT-J-6B requires 21–38 minutes, and Llama-2-13B requires 39–166 minutes. Qwen3-32B requires 96–170 minutes with CPU offloading and layer-wise compression. No gradient-based optimization is used; the dominant costs are Gram eigendecomposition and HOOI/refit iterations. Table 5: Ftc compression wall-clock time (minutes) per model–ratio configuration. Columns correspond to ρ=0.8/0.6/0.4/0.2ρ=0.8/0.6/0.4/0.2. All models use the fixed head-concat recipe on a single A100-40GB GPU; Qwen3-32B additionally uses CPU offloading and is compressed one decoder layer at a time. Model mode 0.8 0.6 0.4 0.2 GPT-J-6B 1 card 38 34 26 21 Llama-2-13B 1 card 166 93 75 39 Mistral-7B 1 card 33 30 26 26 Llama-3-8B 1 card 34 29 27 25 Qwen2.5-7B 1 card 21 20 16 16 Qwen3-8B 1 card 39 33 28 27 Qwen3-32B 1 card 170 137 106 96 A.2 Full perplexity tables Table 6: Full WikiText-2 perplexity results over all models and ρ∈0.8,0.6,0.4,0.2ρ∈\0.8,0.6,0.4,0.2\. Best results in each row are highlighted. Model ρ Ftc Seq-SVD enh-LeSTD TLLM-gen MPIFA SVD-LLM Basis GPT-J-6B 0.8 9.04 9.14 8.99 9.24 9.04 9.60 9.61 0.6 9.30 9.47 9.93 9.68 9.46 10.26 10.16 0.4 10.03 10.46 10.31 11.05 10.68 12.16 11.80 0.2 13.60 14.84 17.47 19.22 17.24 25.04 23.02 Llama-2-13B 0.8 4.98 5.06 4.92 5.13 5.00 5.31 5.28 0.6 5.11 5.27 5.13 5.52 5.26 5.68 5.57 0.4 5.52 5.87 5.72 6.75 6.05 6.72 6.42 0.2 7.52 9.00 9.79 13.01 10.21 14.20 12.77 Mistral-7B 0.8 5.36 5.43 5.40 5.56 5.39 5.56 5.51 0.6 5.44 5.69 5.58 6.07 5.63 5.92 5.75 0.4 5.70 6.54 6.29 8.18 6.63 7.34 6.66 0.2 7.33 12.10 146.31 18.41 15.92 25.40 16.18 Llama-3-8B 0.8 6.36 6.50 6.36 7.16 6.47 6.97 6.88 0.6 6.64 7.03 6.82 8.70 7.26 8.02 7.60 0.4 7.31 8.75 8.28 12.72 9.70 12.61 10.47 0.2 10.44 19.46 20.84 29.85 27.02 48.09 35.60 Qwen2.5-7B 0.8 6.99 7.21 7.17 7.61 7.16 7.50 7.28 0.6 7.20 7.98 7.76 34.10 7.84 8.56 7.82 0.4 7.84 19.96 10.60 84.22 11.60 15.23 10.77 0.2 12.55 150.93 106.59 188.22 7791 255.23 140.65 Qwen3-8B 0.8 7.12 7.25 7.27 7.87 8.03 8.87 8.84 0.6 7.36 7.65 8.20 9.12 8.17 9.24 9.18 0.4 8.48 8.82 10.17 14.04 9.63 15.59 11.11 0.2 10.70 23.66 27.08 45.45 22.23 75.77 46.68 Qwen3-32B 0.8 7.65 7.73 7.66 7.97 7.99 8.67 8.31 0.6 7.77 7.98 7.84 8.56 8.40 9.23 8.70 0.4 8.08 8.86 8.66 10.58 9.90 15.13 9.60 0.2 9.22 22.97 15.23 29.71 26.75 66.29 29.63 Table 7: PTB (left block) and C4 (right block) perplexity, method order as in Table 1 (F = Ftc, qS = Seq-SVD, eL = enh-LeSTD, TG = TLLM-gen, MP = MPIFA, SL = SVD-LLM, BS = Basis). Mistral’s high PTB values reflect its tokenizer/domain mismatch and affect all methods equally. PTB C4 Model ρ F qS eL TG MP SL BS F qS eL TG MP SL BS GPT-J-6B 0.8 16.33 16.80 16.28 17.03 16.30 17.45 17.42 13.62 14.12 13.29 14.06 13.62 14.84 14.83 0.6 17.20 18.39 17.85 18.43 17.46 19.74 19.44 14.33 15.24 14.18 15.09 14.82 16.45 16.27 0.4 20.34 25.08 21.93 25.25 24.20 30.19 28.64 16.61 18.80 16.94 18.75 18.69 21.89 21.26 0.2 50.52 67.11 82.87 101.12 88.92 162.55 115.96 29.12 34.64 40.79 42.92 42.27 65.53 59.42 Llama-2-13B 0.8 52.27 61.58 53.14 54.14 56.42 61.75 60.82 7.05 7.60 7.01 7.45 7.04 8.22 8.14 0.6 56.73 75.51 58.10 62.43 65.56 79.16 78.79 7.74 8.79 7.73 8.73 8.47 9.93 9.64 0.4 118.70 118.43 76.41 100.17 136.49 163.98 140.54 9.39 11.19 9.84 12.20 11.76 14.47 13.39 0.2 226.93 244.05 207.72 249.05 286.09 487.70 478.87 16.53 21.84 22.85 30.64 28.65 43.23 38.24 Mistral-7B 0.8 39.73 38.55 38.46 42.97 38.26 39.49 40.60 8.67 9.07 8.75 9.20 8.79 9.30 9.12 0.6 43.16 43.52 42.85 55.49 41.47 48.17 46.79 8.91 10.20 9.29 10.64 9.78 10.76 10.11 0.4 50.17 65.37 63.35 120.65 71.57 97.88 77.40 9.71 13.41 11.47 16.46 14.00 17.23 14.10 0.2 114.66 233.04 1202 487.88 299.66 479.05 350.76 15.91 34.86 562.15 49.10 51.15 82.38 54.43 Llama-3-8B 0.8 11.70 12.47 11.72 13.51 12.03 13.47 13.05 10.09 14.06 10.12 12.58 10.57 20.88 11.82 0.6 12.44 14.54 13.13 17.76 14.70 17.51 15.92 11.01 20.44 11.71 16.67 20.53 38.23 15.03 0.4 15.43 23.04 19.24 36.09 29.62 47.80 34.60 13.16 23.36 16.62 29.47 31.70 54.62 31.45 0.2 50.71 107.02 111.94 253.19 128.53 264.41 217.80 29.18 90.63 69.35 103.12 132.05 253.66 180.43 Qwen2.5-7B 0.8 13.28 14.51 13.96 15.24 13.62 14.89 14.10 11.97 15.49 12.48 13.38 13.78 15.62 12.69 0.6 14.26 18.97 17.19 129.63 16.93 19.39 16.79 12.50 21.10 13.81 95.39 19.00 22.11 16.06 0.4 18.20 110.39 39.29 586.17 43.70 76.62 36.03 15.42 138.98 27.67 306.41 43.86 70.26 33.25 0.2 89.42 2112 2477 5202 55759 3251 1760 73.37 1106 732.57 1263 26390 1406 786.30 Qwen3-8B 0.8 12.94 13.77 13.47 15.65 14.90 16.91 16.41 12.09 12.77 12.48 13.92 13.85 15.38 15.13 0.6 13.58 15.51 17.08 21.98 16.89 19.69 18.26 12.72 14.21 14.76 17.59 15.18 17.61 16.70 0.4 16.41 22.19 31.72 68.02 26.57 48.73 27.38 14.78 19.16 22.21 35.16 22.44 38.76 25.39 0.2 33.02 106.03 264.11 636.55 103.14 806.93 204.73 23.73 69.32 85.84 145.38 64.87 249.44 133.10 Qwen3-32B 0.8 13.49 14.10 13.54 14.27 14.15 16.53 15.37 12.47 12.98 12.51 13.06 13.02 14.45 13.63 0.6 13.99 15.53 14.27 16.12 16.76 20.48 17.70 12.79 14.25 12.99 14.36 15.05 17.70 15.30 0.4 15.14 24.28 18.27 26.99 39.58 71.73 24.43 13.68 18.74 15.18 19.35 23.69 37.84 19.81 0.2 24.38 220.91 132.24 455.35 299.63 600.37 291.75 19.26 88.34 43.53 113.39 113.11 213.61 90.84 A.3 Full downstream grid Table 9 lists every task row measured: per-task acc_norm for the six base tasks, MathQA, GSM8K, and avg6, for every (model, method, ratio) cell on disk — the modern-model grid (4 models × 4–6 methods × 4 ratios), the MHA models, and the Qwen3.5 hybrid rows. “–” marks configurations that are not applicable (diverged native LeSTD at ρ=0.2ρ=0.2). Table 8: Complete downstream grid, MHA models (acc_norm; GSM8K exact match). Model Method ρ PIQA ARC-e ARC-c HSwag WinoG LAMB MathQA GSM8K avg6 GPT-J-6B dense 1.0 0.756 0.616 0.367 0.654 0.654 0.678 0.281 0.015 0.621 Ftc 0.8 0.747 0.607 0.358 0.642 0.659 0.665 0.269 0.010 0.613 Ftc 0.6 0.742 0.589 0.333 0.626 0.640 0.658 0.264 0.005 0.598 Ftc 0.4 0.735 0.577 0.317 0.567 0.618 0.573 0.263 0.025 0.564 Ftc 0.2 0.684 0.464 0.270 0.438 0.575 0.255 0.258 0.000 0.448 enh-LeSTD 0.8 0.758 0.620 0.362 0.649 0.651 0.682 0.270 0.010 0.620 enh-LeSTD 0.6 0.751 0.596 0.337 0.616 0.623 0.686 0.269 0.030 0.602 enh-LeSTD 0.4 0.728 0.544 0.304 0.561 0.608 0.499 0.248 0.010 0.541 enh-LeSTD 0.2 0.669 0.448 0.272 0.405 0.570 0.149 0.237 0.000 0.419 TLLM-gen 0.8 0.744 0.577 0.348 0.626 0.637 0.629 0.272 0.020 0.594 TLLM-gen 0.6 0.730 0.565 0.324 0.592 0.623 0.621 0.271 0.020 0.576 TLLM-gen 0.2 0.660 0.412 0.254 0.387 0.556 0.146 0.221 0.000 0.403 MPIFA 0.8 0.754 0.601 0.352 0.631 0.657 0.684 0.277 0.045 0.613 MPIFA 0.6 0.747 0.584 0.338 0.594 0.641 0.689 0.272 0.010 0.599 MPIFA 0.4 0.724 0.556 0.299 0.533 0.619 0.576 0.273 0.010 0.551 MPIFA 0.2 0.615 0.427 0.270 0.358 0.586 0.239 0.249 0.010 0.416 SVD-LLM 0.8 0.739 0.592 0.336 0.600 0.667 0.696 0.274 0.025 0.605 SVD-LLM 0.6 0.736 0.562 0.314 0.567 0.646 0.674 0.270 0.015 0.583 SVD-LLM 0.4 0.703 0.537 0.285 0.499 0.622 0.519 0.260 0.035 0.527 SVD-LLM 0.2 0.601 0.400 0.262 0.334 0.567 0.152 0.233 0.000 0.386 Basis Sharing 0.8 0.744 0.597 0.339 0.601 0.658 0.684 0.272 0.025 0.604 Basis Sharing 0.6 0.737 0.579 0.319 0.574 0.648 0.662 0.272 0.010 0.586 Basis Sharing 0.4 0.709 0.554 0.292 0.504 0.634 0.530 0.257 0.025 0.537 Basis Sharing 0.2 0.601 0.412 0.264 0.348 0.568 0.162 0.241 0.010 0.393 Llama-2-13B dense 1.0 0.793 0.777 0.482 0.789 0.720 0.777 0.329 0.205 0.723 Ftc 0.8 0.791 0.763 0.469 0.784 0.719 0.764 0.314 0.190 0.715 Ftc 0.6 0.787 0.766 0.465 0.768 0.694 0.738 0.295 0.115 0.703 Ftc 0.4 0.768 0.726 0.416 0.715 0.690 0.647 0.273 0.030 0.660 Ftc 0.2 0.693 0.565 0.321 0.520 0.618 0.263 0.234 0.010 0.497 enh-LeSTD 0.8 0.793 0.767 0.475 0.782 0.723 0.750 0.315 0.195 0.715 enh-LeSTD 0.6 0.780 0.768 0.452 0.765 0.712 0.710 0.300 0.100 0.698 enh-LeSTD 0.4 0.763 0.708 0.416 0.697 0.681 0.574 0.273 0.040 0.640 enh-LeSTD 0.2 0.646 0.453 0.270 0.438 0.573 0.145 0.226 0.005 0.421 TLLM-gen 0.8 0.781 0.752 0.454 0.758 0.708 0.717 0.303 0.170 0.695 TLLM-gen 0.6 0.762 0.727 0.412 0.708 0.691 0.643 0.279 0.075 0.657 TLLM-gen 0.4 0.697 0.586 0.337 0.603 0.636 0.430 0.247 0.015 0.548 TLLM-gen 0.2 0.606 0.393 0.249 0.380 0.556 0.049 0.222 0.000 0.372 MPIFA 0.8 0.785 0.766 0.468 0.773 0.734 0.763 0.325 0.200 0.715 MPIFA 0.6 0.772 0.749 0.454 0.732 0.712 0.720 0.296 0.155 0.690 MPIFA 0.4 0.730 0.627 0.366 0.640 0.680 0.532 0.249 0.010 0.596 MPIFA 0.2 0.600 0.407 0.247 0.360 0.574 0.097 0.240 0.015 0.381 SVD-LLM 0.8 0.774 0.732 0.465 0.736 0.724 0.721 0.301 0.120 0.692 SVD-LLM 0.6 0.755 0.680 0.404 0.681 0.705 0.659 0.280 0.045 0.647 SVD-LLM 0.4 0.713 0.601 0.335 0.599 0.661 0.433 0.256 0.035 0.557 SVD-LLM 0.2 0.577 0.396 0.233 0.339 0.551 0.073 0.229 0.005 0.361 Basis Sharing 0.8 0.782 0.745 0.465 0.746 0.720 0.715 0.295 0.135 0.696 Basis Sharing 0.6 0.762 0.700 0.437 0.713 0.717 0.665 0.285 0.075 0.666 Basis Sharing 0.4 0.723 0.623 0.359 0.622 0.669 0.476 0.261 0.025 0.579 Basis Sharing 0.2 0.587 0.421 0.253 0.367 0.550 0.087 0.232 0.005 0.378 Table 9: Complete downstream grid, GQA and hybrid models. Model Method ρ PIQA ARC-e ARC-c HSwag WinoG LAMB MathQA GSM8K avg6 Mistral-7B dense 1.0 0.814 0.807 0.530 0.796 0.740 0.753 0.367 0.285 0.740 Ftc 0.8 0.805 0.812 0.521 0.779 0.745 0.737 0.345 0.235 0.733 Ftc 0.6 0.805 0.811 0.506 0.765 0.741 0.728 0.336 0.240 0.726 Ftc 0.4 0.795 0.781 0.474 0.726 0.701 0.655 0.303 0.105 0.689 Ftc 0.2 0.720 0.595 0.321 0.555 0.619 0.263 0.247 0.015 0.512 Seq-SVD 0.6 0.781 0.759 0.450 0.711 0.716 0.690 0.320 0.175 0.684 Seq-SVD 0.4 0.706 0.603 0.337 0.582 0.661 0.524 0.287 0.010 0.569 Seq-SVD 0.2 0.545 0.339 0.243 0.343 0.562 0.121 0.220 0.000 0.359 enh-LeSTD 0.8 0.805 0.807 0.509 0.774 0.736 0.738 0.345 0.260 0.728 enh-LeSTD 0.6 0.794 0.796 0.479 0.745 0.733 0.697 0.334 0.155 0.707 enh-LeSTD 0.4 0.770 0.709 0.404 0.668 0.672 0.526 0.287 0.005 0.625 enh-LeSTD 0.2 0.524 0.300 0.252 0.295 0.535 0.011 0.213 0.000 0.320 TLLM-gen 0.8 0.798 0.790 0.484 0.748 0.725 0.711 0.331 0.190 0.709 TLLM-gen 0.6 0.770 0.738 0.426 0.699 0.695 0.594 0.314 0.095 0.654 TLLM-gen 0.4 0.694 0.540 0.320 0.552 0.632 0.270 0.242 0.010 0.501 TLLM-gen 0.2 0.566 0.337 0.259 0.335 0.537 0.016 0.209 0.015 0.342 MPIFA 0.8 0.803 0.789 0.511 0.781 0.733 0.737 0.360 0.260 0.726 MPIFA 0.6 0.791 0.755 0.466 0.723 0.734 0.699 0.342 0.200 0.695 MPIFA 0.4 0.710 0.581 0.320 0.539 0.653 0.508 0.276 0.020 0.552 MPIFA 0.2 0.527 0.312 0.230 0.281 0.512 0.022 0.222 0.005 0.314 SVD-LLM 0.8 0.797 0.783 0.486 0.755 0.728 0.705 0.355 0.230 0.709 SVD-LLM 0.6 0.776 0.735 0.435 0.677 0.719 0.660 0.312 0.170 0.667 SVD-LLM 0.4 0.681 0.542 0.305 0.472 0.633 0.394 0.257 0.020 0.505 SVD-LLM 0.2 0.516 0.291 0.231 0.294 0.506 0.007 0.223 0.005 0.307 Basis Sharing 0.8 0.803 0.777 0.502 0.757 0.742 0.706 0.360 0.305 0.715 Basis Sharing 0.6 0.790 0.775 0.469 0.709 0.723 0.664 0.314 0.195 0.688 Basis Sharing 0.4 0.727 0.599 0.352 0.550 0.614 0.450 0.274 0.025 0.549 Basis Sharing 0.2 0.517 0.251 0.251 0.240 0.506 0.001 0.207 0.000 0.294 Llama-3-8B dense 1.0 0.805 0.790 0.531 0.796 0.738 0.750 0.401 0.435 0.735 Ftc 0.8 0.794 0.801 0.535 0.779 0.733 0.735 0.370 0.395 0.729 Ftc 0.6 0.788 0.786 0.501 0.766 0.740 0.717 0.326 0.290 0.716 Ftc 0.4 0.778 0.730 0.438 0.730 0.700 0.634 0.302 0.100 0.668 Ftc 0.2 0.702 0.522 0.308 0.521 0.596 0.282 0.241 0.020 0.488 Seq-SVD 0.6 0.739 0.702 0.422 0.684 0.716 0.590 0.345 0.085 0.642 Seq-SVD 0.4 0.683 0.467 0.315 0.518 0.641 0.436 0.257 0.035 0.510 Seq-SVD 0.2 0.541 0.312 0.234 0.329 0.541 0.113 0.221 0.005 0.345 enh-LeSTD 0.8 0.786 0.796 0.531 0.773 0.731 0.713 0.371 0.345 0.722 enh-LeSTD 0.6 0.782 0.784 0.486 0.754 0.716 0.641 0.330 0.195 0.694 enh-LeSTD 0.4 0.739 0.694 0.409 0.658 0.650 0.458 0.287 0.040 0.601 enh-LeSTD 0.2 0.588 0.365 0.249 0.358 0.536 0.044 0.226 0.005 0.357 TLLM-gen 0.8 0.767 0.760 0.483 0.727 0.715 0.647 0.313 0.070 0.683 TLLM-gen 0.6 0.731 0.657 0.394 0.641 0.660 0.498 0.267 0.035 0.597 TLLM-gen 0.4 0.671 0.472 0.289 0.489 0.584 0.175 0.241 0.025 0.447 TLLM-gen 0.2 0.565 0.328 0.235 0.314 0.507 0.015 0.215 0.010 0.327 MPIFA 0.8 0.793 0.771 0.506 0.760 0.725 0.683 0.375 0.360 0.706 MPIFA 0.6 0.741 0.711 0.441 0.664 0.709 0.570 0.342 0.005 0.639 MPIFA 0.4 0.622 0.421 0.278 0.439 0.608 0.323 0.242 0.020 0.448 MPIFA 0.2 0.517 0.277 0.222 0.277 0.522 0.019 0.199 0.000 0.306 SVD-LLM 0.8 0.770 0.738 0.461 0.733 0.724 0.589 0.357 0.040 0.669 SVD-LLM 0.6 0.720 0.626 0.381 0.613 0.682 0.427 0.317 0.020 0.575 SVD-LLM 0.4 0.610 0.389 0.266 0.403 0.578 0.215 0.243 0.040 0.410 SVD-LLM 0.2 0.523 0.279 0.242 0.277 0.501 0.027 0.221 0.030 0.308 Basis Sharing 0.8 0.791 0.720 0.458 0.732 0.715 0.619 0.343 0.330 0.672 Basis Sharing 0.6 0.742 0.637 0.394 0.680 0.686 0.550 0.340 0.045 0.615 Basis Sharing 0.4 0.653 0.448 0.294 0.498 0.626 0.325 0.254 0.025 0.474 Basis Sharing 0.2 0.527 0.288 0.234 0.278 0.535 0.027 0.219 0.015 0.315 Qwen2.5-7B dense 1.0 0.791 0.769 0.511 0.781 0.730 0.720 0.446 0.820 0.717 Ftc 0.8 0.796 0.797 0.511 0.780 0.714 0.690 0.439 0.780 0.715 Ftc 0.6 0.790 0.727 0.492 0.770 0.707 0.650 0.451 0.715 0.689 Ftc 0.4 0.785 0.732 0.486 0.742 0.676 0.535 0.395 0.445 0.659 Ftc 0.2 0.614 0.410 0.258 0.423 0.565 0.163 0.227 0.015 0.405 Seq-SVD 0.6 0.745 0.690 0.442 0.688 0.707 0.514 0.359 0.445 0.631 Seq-SVD 0.4 0.539 0.297 0.209 0.322 0.528 0.095 0.237 0.010 0.332 Seq-SVD 0.2 0.543 0.264 0.242 0.255 0.524 0.000 0.237 0.010 0.305 enh-LeSTD 0.8 0.787 0.777 0.497 0.767 0.709 0.639 0.426 0.705 0.696 enh-LeSTD 0.6 0.779 0.727 0.474 0.733 0.693 0.543 0.419 0.495 0.658 enh-LeSTD 0.4 0.673 0.533 0.322 0.551 0.622 0.240 0.306 0.015 0.490 enh-LeSTD 0.2 0.526 0.268 0.247 0.258 0.491 0.000 0.220 0.000 0.298 TLLM-gen 0.8 0.782 0.776 0.507 0.750 0.693 0.521 0.398 0.645 0.671 TLLM-gen 0.6 0.531 0.321 0.243 0.313 0.528 0.021 0.281 0.010 0.326 TLLM-gen 0.4 0.531 0.263 0.247 0.281 0.481 0.000 0.238 0.015 0.300 TLLM-gen 0.2 0.518 0.251 0.242 0.258 0.514 0.000 0.191 0.005 0.297 MPIFA 0.8 0.784 0.752 0.487 0.762 0.708 0.692 0.464 0.715 0.698 MPIFA 0.6 0.748 0.764 0.475 0.709 0.726 0.575 0.423 0.525 0.666 MPIFA 0.4 0.605 0.476 0.299 0.446 0.614 0.296 0.274 0.035 0.456 MPIFA 0.2 0.514 0.235 0.235 0.252 0.496 0.000 0.192 0.025 0.289 SVD-LLM 0.8 0.771 0.749 0.481 0.741 0.716 0.646 0.448 0.680 0.684 SVD-LLM 0.6 0.732 0.707 0.434 0.646 0.684 0.538 0.378 0.420 0.624 SVD-LLM 0.4 0.565 0.390 0.265 0.391 0.542 0.232 0.260 0.020 0.398 SVD-LLM 0.2 0.516 0.259 0.243 0.260 0.500 0.004 0.234 0.015 0.297 Basis Sharing 0.8 0.796 0.800 0.529 0.765 0.715 0.688 0.443 0.720 0.716 Basis Sharing 0.6 0.776 0.742 0.472 0.723 0.711 0.589 0.410 0.535 0.669 Basis Sharing 0.4 0.662 0.582 0.374 0.550 0.602 0.364 0.294 0.090 0.522 Basis Sharing 0.2 0.538 0.267 0.219 0.250 0.494 0.009 0.239 0.010 0.296 Table 10: Complete downstream grid, GQA and hybrid models (continued). Model Method ρ PIQA ARC-e ARC-c HSwag WinoG LAMB MathQA GSM8K avg6 Qwen3-8B dense 1.0 0.786 0.801 0.561 0.774 0.724 0.704 0.562 0.855 0.725 Ftc 0.8 0.788 0.832 0.583 0.763 0.735 0.664 0.563 0.765 0.728 Ftc 0.6 0.781 0.793 0.564 0.750 0.715 0.640 0.515 0.735 0.707 Ftc 0.4 0.730 0.641 0.441 0.712 0.670 0.553 0.316 0.105 0.624 Ftc 0.2 0.690 0.550 0.324 0.550 0.582 0.305 0.247 0.005 0.500 Seq-SVD 0.6 0.765 0.712 0.487 0.718 0.699 0.627 0.423 0.455 0.668 Seq-SVD 0.4 0.707 0.567 0.373 0.606 0.642 0.472 0.284 0.085 0.561 Seq-SVD 0.2 0.557 0.320 0.209 0.305 0.534 0.129 0.245 0.005 0.342 enh-LeSTD 0.8 0.789 0.831 0.590 0.753 0.718 0.617 0.547 0.790 0.716 enh-LeSTD 0.6 0.773 0.760 0.493 0.684 0.664 0.489 0.454 0.540 0.644 enh-LeSTD 0.4 0.724 0.609 0.388 0.558 0.604 0.294 0.307 0.070 0.529 enh-LeSTD 0.2 0.555 0.348 0.238 0.305 0.514 0.016 0.215 0.005 0.329 TLLM-gen 0.8 0.777 0.761 0.499 0.706 0.700 0.519 0.468 0.635 0.660 TLLM-gen 0.6 0.749 0.671 0.444 0.629 0.647 0.373 0.393 0.235 0.586 TLLM-gen 0.4 0.663 0.495 0.320 0.454 0.550 0.109 0.286 0.010 0.432 TLLM-gen 0.2 0.532 0.297 0.215 0.282 0.500 0.001 0.210 0.000 0.304 MPIFA 0.8 0.791 0.773 0.570 0.780 0.714 0.626 0.484 0.695 0.709 MPIFA 0.6 0.773 0.722 0.505 0.740 0.714 0.564 0.450 0.440 0.670 MPIFA 0.4 0.684 0.531 0.330 0.536 0.644 0.406 0.279 0.080 0.522 MPIFA 0.2 0.554 0.333 0.217 0.311 0.533 0.199 0.225 0.000 0.358 SVD-LLM 0.8 0.777 0.752 0.529 0.762 0.715 0.542 0.436 0.335 0.679 SVD-LLM 0.6 0.750 0.577 0.416 0.690 0.688 0.488 0.373 0.200 0.601 SVD-LLM 0.4 0.624 0.446 0.255 0.393 0.555 0.278 0.246 0.010 0.425 SVD-LLM 0.2 0.520 0.302 0.214 0.276 0.511 0.106 0.226 0.005 0.322 Basis Sharing 0.8 0.774 0.769 0.566 0.765 0.716 0.561 0.458 0.430 0.692 Basis Sharing 0.6 0.775 0.746 0.531 0.724 0.707 0.482 0.406 0.200 0.661 Basis Sharing 0.4 0.737 0.587 0.402 0.632 0.638 0.369 0.330 0.045 0.561 Basis Sharing 0.2 0.544 0.289 0.207 0.306 0.529 0.137 0.239 0.005 0.335 Qwen3-32B dense 1.0 0.802 0.829 0.608 0.818 0.729 0.671 0.611 0.885 0.743 Ftc 0.6 0.801 0.815 0.594 0.811 0.723 0.656 0.536 0.770 0.733 Ftc 0.4 0.799 0.812 0.617 0.795 0.739 0.635 0.487 0.750 0.733 Ftc 0.2 0.782 0.777 0.556 0.727 0.696 0.489 0.359 0.280 0.671 Seq-SVD 0.6 0.784 0.809 0.607 0.786 0.728 0.666 0.483 0.635 0.730 Seq-SVD 0.4 0.768 0.762 0.536 0.728 0.714 0.565 0.380 0.545 0.679 Seq-SVD 0.2 0.582 0.380 0.253 0.370 0.530 0.134 0.233 0.000 0.375 enh-LeSTD 0.6 0.799 0.811 0.596 0.784 0.727 0.587 0.532 0.765 0.717 enh-LeSTD 0.4 0.786 0.784 0.595 0.751 0.734 0.441 0.469 0.510 0.682 enh-LeSTD 0.2 0.685 0.494 0.356 0.542 0.593 0.092 0.281 0.015 0.460 TLLM-gen 0.6 0.795 0.788 0.580 0.767 0.728 0.463 0.505 0.700 0.687 TLLM-gen 0.4 0.785 0.718 0.513 0.708 0.706 0.278 0.409 0.210 0.618 TLLM-gen 0.2 0.594 0.358 0.268 0.366 0.530 0.017 0.228 0.000 0.355 MPIFA 0.6 0.783 0.788 0.632 0.779 0.724 0.628 0.475 0.715 0.722 MPIFA 0.4 0.734 0.711 0.501 0.657 0.665 0.504 0.370 0.405 0.629 MPIFA 0.2 0.594 0.335 0.246 0.341 0.517 0.200 0.230 0.015 0.372 SVD-LLM 0.6 0.765 0.745 0.566 0.751 0.698 0.579 0.426 0.665 0.684 SVD-LLM 0.4 0.708 0.587 0.380 0.523 0.605 0.348 0.293 0.035 0.525 SVD-LLM 0.2 0.545 0.303 0.218 0.287 0.521 0.135 0.242 0.000 0.335 Basis Sharing 0.6 0.781 0.753 0.579 0.771 0.723 0.600 0.479 0.600 0.701 Basis Sharing 0.4 0.761 0.741 0.534 0.720 0.683 0.504 0.358 0.440 0.657 Basis Sharing 0.2 0.601 0.363 0.242 0.343 0.501 0.163 0.241 0.010 0.369 Qwen3.5-9B dense 1.0 0.794 0.776 0.565 0.770 0.734 0.696 0.553 0.885 0.723 Ftc 0.8 0.794 0.833 0.602 0.748 0.716 0.671 0.512 0.785 0.727 Ftc 0.6 0.755 0.822 0.550 0.681 0.705 0.625 0.421 0.580 0.690 A.4 Hyperparameter ablations Ablation protocol and additional design studies. Table 3 reports one-at-a-time ablations of the final recipe on all four GQA models at ρ∈0.6,0.2ρ∈\0.6,0.2\. The recipe uses fixed head-concat tensorization, budget-coupled allocation, a=0.5a=0.5, α=0.02α=0.02, two refits, 256 calibration samples, slice-wise normalization, no QKV teacher map, and a separate teacher-calibrated WOW_O target. Each variant changes one component while keeping the remaining configuration and compression budget fixed. “K/V repeat-expand” replaces head-concat by naive GQA→ replication, while “dense core” uses a dense core under the same parameter budget. Additional development-stage studies below examine design choices not isolated by the main grid. QKV teacher-map regularization. Table 11 shows that disabling the QKV teacher map consistently gives the lowest perplexity at ρ=0.2ρ=0.2. Among the mapped variants, μ=3μ=3 performs best on Mistral-7B, Llama-3-8B, and Qwen3-8B, while Qwen2.5-7B favors μ=10μ=10. Even at these best mapped settings, perplexity remains 2.12.1–11.3%11.3\% higher than the no-map configuration. This supports using the original QKV weights as the compression target in the final recipe, without an additional teacher-output map. Table 11: Sensitivity to QKV teacher-map regularization at ρ=0.2ρ=0.2. ρ=0.2ρ=0.2, wt2 ppl μ=0.001μ=0.001 0.030.03 0.30.3 11 33 1010 no QKV map (final) Mistral-7B 8.10 8.13 8.04 7.85 7.64 7.91 7.33 Llama-3-8B 12.40 12.35 11.87 11.32 11.18 11.37 10.44 Qwen2.5-7B 16.52 16.80 17.42 16.84 14.72 13.97 12.55 Qwen3-8B 12.26 12.09 11.38 11.07 10.93 11.61 10.70 Rank allocation. Replacing the budget-coupled allocator with fixed Tucker ranks increases perplexity by 3.73.7–6.2%6.2\% at ρ=0.2ρ=0.2. The allocator therefore provides a consistent but moderate gain, complementing the larger benefits from the activation-induced metric, joint tensorization, and sequential calibration. Output-projection metric. Reusing the QKV activation metric for WOW_O causes severe degradation, increasing perplexity by 5555–708%708\% at ρ=0.6ρ=0.6 and by more than 4000%4000\% at ρ=0.2ρ=0.2. This shows that WOW_O requires a metric constructed from its own attention-output representations rather than reusing the input-side QKV metric. Table 12: Perplexity ablations for rank allocation and the output-projection metric. Model ρ Variant variant full Δ Mistral-7B 0.2 fixed R1=640,R2=128R_1=640,R_2=128 7.786 7.33 +6.2%+6.2\% Llama-3-8B 0.2 fixed R1=640,R2=128R_1=640,R_2=128 10.975 10.44 +5.1%+5.1\% Qwen3-8B 0.2 fixed R1=640,R2=128R_1=640,R_2=128 11.100 10.70 +3.7%+3.7\% Mistral-7B 0.6 WOW_O with QKV metric 8.456 5.44 +55.4%+55.4\% Mistral-7B 0.2 WOW_O with QKV metric 303.397 7.33 +4039.1%+4039.1\% Llama-3-8B 0.6 WOW_O with QKV metric 53.677 6.64 +708.4%+708.4\% Llama-3-8B 0.2 WOW_O with QKV metric 606.211 10.44 +5706.6%+5706.6\% Qwen3-8B 0.6 WOW_O with QKV metric 12.865 7.36 +74.8%+74.8\% Qwen3-8B 0.2 WOW_O with QKV metric 454.125 10.70 +4144.2%+4144.2\% Table 13: Perplexity sensitivity to one versus two refit iterations. Model Variant Full Δ Llama-3-8B 6.655 6.613 6.607 Qwen3-8B 7.855 7.764 7.759 Qwen2.5-7B 7.322 7.312 7.308 Refit iterations. Table 13 shows that one refit captures nearly all of the improvement, while a second refit changes perplexity only marginally in the evaluated settings. We therefore retain two refits in the final recipe. Anchoring. The final recipe mixes the teacher Gram into the WOW_O statistics with a=0.5a=0.5, while leaving the QKV statistics unanchored. As shown in Table 3, anchoring has little effect at ρ=0.6ρ=0.6: removing it changes perplexity by only −0.6%-0.6\% to +0.8%+0.8\% across the four GQA models. Its contribution becomes more pronounced at ρ=0.2ρ=0.2, where removing the anchor increases perplexity by 1.5%1.5\% on Mistral-7B, 4.1%4.1\% on Llama-3-8B, 14.5%14.5\% on Qwen2.5-7B, and 6.1%6.1\% on Qwen3-8B. The effect is therefore mainly visible under aggressive compression, with Qwen2.5-7B showing the largest sensitivity. Slice-wise normalization. The effect of slice-wise normalization is strongly budget-dependent. As shown in Table 3, removing the slice-wise norm has only a limited and model-dependent effect at ρ=0.6ρ=0.6, changing perplexity by −0.3%-0.3\% to +5.5%+5.5\% across the four GQA models. At ρ=0.2ρ=0.2, however, the degradation becomes substantially larger: 27.2%27.2\% on Mistral-7B, 89.4%89.4\% on Llama-3-8B, 492.4%492.4\% on Qwen2.5-7B, and 11.8%11.8\% on Qwen3-8B. This indicates that slice-wise normalization becomes increasingly important under aggressive compression, where a tighter core budget makes the decomposition more sensitive to energy imbalance across head slices. A.5 Uniform recipe, calibration source, seeds, structured core The final recipe at every ratio. Table 14 reports the fixed recipe of Section 4 (allocator, a=0.5a=0.5, no QKV teacher map, WOW_O map with μ=1μ=1, α=0.02α=0.02, two refits, 256 calibration windows, and slice-wise norm) at all four compression ratios. The same recipe is used across models and ratios without model-specific tuning. Table 14: The final recipe on the six primary evaluation models at all four keep ratios. WikiText-2 perplexity, avg6, and GSM8K accuracy. GSM8K was not evaluated for GPT-J-6B and Llama-2-13B at ρ=0.2ρ=0.2. ρ=0.8ρ=0.8 ρ=0.6ρ=0.6 ρ=0.4ρ=0.4 ρ=0.2ρ=0.2 Model ppl avg6 GSM8K ppl avg6 GSM8K ppl avg6 GSM8K ppl avg6 GSM8K GPT-J-6B 9.04 0.613 0.010 9.30 0.598 0.005 10.03 0.564 0.025 13.60 0.448 n/a Llama-2-13B 4.98 0.715 0.190 5.11 0.703 0.115 5.52 0.660 0.030 7.52 0.497 n/a Mistral-7B 5.36 0.733 0.235 5.44 0.726 0.240 5.70 0.689 0.105 7.33 0.512 0.015 Llama-3-8B 6.36 0.729 0.395 6.64 0.716 0.290 7.31 0.668 0.100 10.44 0.488 0.020 Qwen2.5-7B 6.99 0.715 0.780 7.20 0.689 0.715 7.84 0.659 0.445 12.55 0.405 0.015 Qwen3-8B 7.12 0.728 0.765 7.36 0.707 0.735 8.48 0.624 0.105 10.70 0.500 0.005 Whitening metric and damping. We use Hϵ,l=Cs,l+ϵt¯lIH_ε,l=C_s,l+ε t_lI, where t¯=tr(Cs,l)dmodel t= tr(C_s,l)d_model., with ϵ=10−4ε=10^-4 throughout. Table 15 examines sensitivity to this damping parameter. At ρ=0.2ρ=0.2, increasing ϵε to 10−210^-2 raises perplexity by only 1.41.4–2.7%2.7\%, whereas the heavily damped setting ϵ=1ε=1 incurs 11.711.7–24.8%24.8\% higher perplexity across the three models. At ρ=0.6ρ=0.6, the effect is much smaller and model-dependent, with changes ranging from −1.5%-1.5\% to +3.0%+3.0\%. These results indicate that aggressive compression benefits from preserving the original functional geometry rather than strongly damping the whitening metric. We therefore fix ϵ=10−4ε=10^-4 across all models and compression ratios. Table 15: Sensitivity to whitening damping ϵε in Hϵ,l=Cs,l+ϵt¯lIH_ε,l=C_s,l+ε t_lI, where t¯=tr(Cs,l)dmodel t= tr(C_s,l)d_model. The recipe uses ϵ=10−4ε=10^-4. Model ρ ε=10−4 =10^-4 (recipe) ε=10−2 =10^-2 ε=1 =1 Llama-3-8B .6 6.64 6.63 (−0.1%-0.1\%) 6.54 (−1.5%-1.5\%) Llama-3-8B .2 10.44 10.59 (+1.4%+1.4\%) 12.21 (+17.0%+17.0\%) Mistral-7B .6 5.44 5.44 (−0.0%-0.0\%) 5.45 (+0.2%+0.2\%) Mistral-7B .2 7.33 7.43 (+1.4%+1.4\%) 8.19 (+11.7%+11.7\%) Qwen3-8B .6 7.36 7.39 (+0.4%+0.4\%) 7.58 (+3.0%+3.0\%) Qwen3-8B .2 10.70 10.99 (+2.7%+2.7\%) 13.36 (+24.8%+24.8\%) Whitening stability guard. The default whitening metric uses ϵ=10−4ε=10^-4. To ensure numerical stability, we increase ϵε by factors of ten whenever the de-whitened reconstruction is non-finite or its norm exceeds 10×10× that of the original weight. The same guard is applied uniformly to all configurations. It is never activated at ρ≥0.4ρ≥ 0.4, and no run falls back to the identity metric or leaves any layer uncompressed. At ρ=0.2ρ=0.2, the guard is triggered only for a subset of layers (Table 16). The most sensitive case is Qwen2.5-7B: under the final no-QKV-map recipe, 9 of 28 layers require increased damping, with 3 reaching ϵ=1ε=1. The escalation frequency also changes with the QKV target, indicating that different compression choices can alter the conditioning of the activation statistics seen by subsequent layers. Together with the damping sweep in Table 15, where larger ϵε generally degrades performance at this budget, these results show that the guard serves as a numerical safeguard rather than a mechanism for improving compression quality. Table 16: Layers whose whitening damping had to be escalated above ε=10−4 =10^-4 at ρ=0.2ρ=0.2, and how many of those ended at the ridge metric ε=1 =1. Parsed from the run logs. At ρ≥0.4ρ≥ 0.4 all counts are zero on all four models. Model QKV target layers escalated reaching ε=1 =1 Mistral-7B no QKV map (final) 4/32 0 Mistral-7B μ=1μ=1 map 5/32 1 Llama-3-8B no QKV map (final) 2/32 0 Llama-3-8B μ=1μ=1 map 2/32 0 Qwen2.5-7B no QKV map (final) 9/28 3 Qwen2.5-7B μ=1μ=1 map 5/28 0 Qwen3-8B no QKV map (final) 1/36 0 Qwen3-8B μ=1μ=1 map 7/36 1 Calibration-set size. The data-dependent baselines use 128 calibration sequences following their published configurations, whereas FTC uses 256 sequences in the main experiments. Table 17 evaluates the sensitivity of FTC to this difference. At ρ=0.6ρ=0.6, reducing the calibration set from 256 to 128 windows changes perplexity by only −0.1-0.1–+0.3%+0.3\% across all four models. At ρ=0.2ρ=0.2, the effect remains modest on Mistral-7B and Llama-3-8B (+2.3%+2.3\% and +3.7%+3.7\%, respectively), and is negligible on Qwen3-8B (−0.2%-0.2\%). Qwen2.5-7B is a notable exception: its perplexity increases from 12.5512.55 to 43.9043.90 when the calibration set is reduced to 128 windows. These results indicate that calibration size has little effect in most settings, but can become important for particular models under aggressive compression. Table 17: Sensitivity of Ftc to calibration-set size. We compare the same final recipe using 256 versus 128 WikiText-2 calibration windows of 2,048 tokens. Model ρ 256 windows 128 windows Δ Llama-3-8B 0.6 6.640 6.660 +0.3%+0.3\% Llama-3-8B 0.2 10.440 10.821 +3.7%+3.7\% Mistral-7B 0.6 5.440 5.447 +0.1%+0.1\% Mistral-7B 0.2 7.330 7.499 +2.3%+2.3\% Qwen2.5-7B 0.6 7.200 7.192 -0.1% Qwen2.5-7B 0.2 12.550 43.901 +249.8% Qwen3-8B 0.6 7.360 7.352 -0.1% Qwen3-8B 0.2 10.700 10.678 -0.2% Structured (2:4) core. Table 18 evaluates a hardware-aligned 2:4 core against the default unstructured sparse core under the corresponding realised budget. At moderate compression, the additional quality cost of the structured pattern is small: the perplexity increase is only 0.30.3–1.3%1.3\% at ρ=0.6ρ=0.6 and 0.80.8–2.0%2.0\% at ρ=0.4ρ=0.4 across the four models. The effect becomes substantially more model-dependent at ρ=0.2ρ=0.2: perplexity increases by 11.3%11.3\% on Qwen3-8B, 16.5%16.5\% on Llama-3-8B, 17.9%17.9\% on Mistral-7B, and 366.2%366.2\% on Qwen2.5-7B. In particular, Qwen2.5-7B increases from 12.5512.55 to 58.5058.50 perplexity under the 2:4 constraint, whereas its unstructured FTC checkpoint at the same nominal budget is 1.83×1.83× the dense perplexity (12.5512.55 vs. 6.856.85). Thus, the 2:4 structure is nearly cost-free at moderate compression but can impose a substantial and highly model-dependent penalty under the tightest budget. Table 18: Unstructured vs. 2:4 core at equal ρ. Both columns are the final recipe at a realised ρ, differing only in the core structure. Model ρ Unstructured core 2:4 core Δppl _ ppl Mistral-7B 0.6 5.44 5.47 +0.6%+0.6\% Mistral-7B 0.4 5.70 5.76 +1.1%+1.1\% Mistral-7B 0.2 7.33 8.64 +17.9%+17.9\% Llama-3-8B 0.6 6.64 6.72 +1.2%+1.2\% Llama-3-8B 0.4 7.31 7.46 +2.0%+2.0\% Llama-3-8B 0.2 10.44 12.16 +16.5%+16.5\% Qwen2.5-7B 0.6 7.20 7.22 +0.3%+0.3\% Qwen2.5-7B 0.4 7.84 7.90 +0.8%+0.8\% Qwen2.5-7B 0.2 12.55 58.50 +366.2%+366.2\% Qwen3-8B 0.6 7.36 7.46 +1.3%+1.3\% Qwen3-8B 0.4 8.48 8.56 +0.9%+0.9\% Qwen3-8B 0.2 10.70 11.91 +11.3%+11.3\% Calibration-source sensitivity. Table 19 compares WikiText-2 and C4 calibration for the data-dependent methods while keeping each method’s calibration budget fixed. C4 calibration consistently improves C4 perplexity, while WikiText-2 perplexity generally increases. The effect is mild at ρ=0.6ρ=0.6 but becomes substantially larger at ρ=0.2ρ=0.2, indicating greater sensitivity to the calibration distribution under aggressive compression. PTB shows a less uniform trend. Qwen2.5-7B is a notable exception at ρ=0.2ρ=0.2, where the calibration source also affects numerical stability. We therefore use WikiText-2 as the calibration source for all main data-dependent comparisons and analyze calibration-set size separately. Table 19: Data-dependent methods under WikiText-2 and C4 calibration. Model ρ Method Calibration wt2 PTB C4 Mistral-7B 0.6 SVD-LLM WikiText-2 5.92 48.17 10.76 Mistral-7B 0.6 SVD-LLM C4 6.59 46.76 9.74 Mistral-7B 0.6 MPIFA WikiText-2 5.63 41.47 9.78 Mistral-7B 0.6 MPIFA C4 6.01 41.09 9.17 Mistral-7B 0.6 Basis WikiText-2 5.75 46.79 10.11 Mistral-7B 0.6 Basis C4 6.21 46.32 9.38 Mistral-7B 0.6 Ftc WikiText-2 5.44 43.16 8.91 Mistral-7B 0.6 Ftc C4 5.58 39.23 8.77 Mistral-7B 0.2 SVD-LLM WikiText-2 25.40 479.05 82.38 Mistral-7B 0.2 SVD-LLM C4 81.34 564.41 55.14 Mistral-7B 0.2 MPIFA WikiText-2 15.92 299.66 51.15 Mistral-7B 0.2 MPIFA C4 45.94 274.50 30.73 Mistral-7B 0.2 Basis WikiText-2 16.18 350.76 54.43 Mistral-7B 0.2 Basis C4 48.04 285.01 33.74 Mistral-7B 0.2 Ftc WikiText-2 7.33 114.66 15.91 Mistral-7B 0.2 Ftc C4 12.41 99.54 12.19 Llama-3-8B 0.6 SVD-LLM WikiText-2 8.02 17.51 38.23 Llama-3-8B 0.6 SVD-LLM C4 10.48 17.73 13.50 Llama-3-8B 0.6 MPIFA WikiText-2 7.26 14.70 20.53 Llama-3-8B 0.6 MPIFA C4 8.16 14.38 11.37 Llama-3-8B 0.6 Basis WikiText-2 7.60 15.92 15.03 Llama-3-8B 0.6 Basis C4 9.25 15.83 12.49 Llama-3-8B 0.6 Ftc WikiText-2 6.64 12.44 11.01 Llama-3-8B 0.6 Ftc C4 6.93 12.25 10.31 Llama-3-8B 0.2 SVD-LLM WikiText-2 48.09 264.41 253.66 Llama-3-8B 0.2 SVD-LLM C4 212.86 305.31 117.94 Llama-3-8B 0.2 MPIFA WikiText-2 27.02 128.53 132.05 Llama-3-8B 0.2 MPIFA C4 92.45 107.70 51.89 Llama-3-8B 0.2 Basis WikiText-2 35.60 217.80 180.43 Llama-3-8B 0.2 Basis C4 176.36 239.78 106.65 Llama-3-8B 0.2 Ftc WikiText-2 10.44 50.71 29.18 Llama-3-8B 0.2 Ftc C4 23.77 43.20 19.42 Qwen2.5-7B 0.6 SVD-LLM WikiText-2 8.56 19.39 22.11 Qwen2.5-7B 0.6 SVD-LLM C4 12.40 17.56 15.38 Qwen2.5-7B 0.6 MPIFA WikiText-2 7.84 16.93 19.00 Qwen2.5-7B 0.6 MPIFA C4 9.63 15.70 13.70 Qwen2.5-7B 0.6 Basis WikiText-2 7.82 16.79 16.06 Qwen2.5-7B 0.6 Basis C4 8.89 15.77 13.54 Qwen2.5-7B 0.6 Ftc WikiText-2 7.20 14.26 12.50 Qwen2.5-7B 0.6 Ftc C4 7.55 14.00 12.23 Qwen2.5-7B 0.2 SVD-LLM WikiText-2 255.23 3251 1406 Qwen2.5-7B 0.2 SVD-LLM C4 1020 2547 450.36 Qwen2.5-7B 0.2 MPIFA WikiText-2 7791 55759 26390 Qwen2.5-7B 0.2 MPIFA C4 341.05 688.78 161.15 Qwen2.5-7B 0.2 Basis WikiText-2 140.65 1760 786.30 Qwen2.5-7B 0.2 Basis C4 854.58 1152 315.74 Qwen2.5-7B 0.2 Ftc WikiText-2 12.55 89.42 73.37 Qwen2.5-7B 0.2 Ftc C4 numerically unstable† Qwen3-8B 0.6 SVD-LLM WikiText-2 9.24 19.69 17.61 Qwen3-8B 0.6 SVD-LLM C4 10.45 20.15 16.39 Qwen3-8B 0.6 MPIFA WikiText-2 8.17 16.89 15.18 Qwen3-8B 0.6 MPIFA C4 8.72 16.15 13.83 Qwen3-8B 0.6 Basis WikiText-2 9.18 18.26 16.70 Qwen3-8B 0.6 Basis C4 9.83 17.78 15.35 Qwen3-8B 0.6 Ftc WikiText-2 7.36 13.58 12.72 Qwen3-8B 0.6 Ftc C4 7.58 13.42 12.46 Qwen3-8B 0.2 SVD-LLM WikiText-2 75.77 806.93 249.44 Qwen3-8B 0.2 SVD-LLM C4 185.24 497.86 145.08 Qwen3-8B 0.2 MPIFA WikiText-2 22.23 103.14 64.87 Qwen3-8B 0.2 MPIFA C4 52.09 83.33 43.36 Qwen3-8B 0.2 Basis WikiText-2 46.68 204.73 133.10 Qwen3-8B 0.2 Basis C4 117.24 171.98 104.44 Qwen3-8B 0.2 Ftc WikiText-2 10.70 33.02 23.73 Qwen3-8B 0.2 Ftc C4 15.72 26.82 18.87 A.6 Where drift accumulates along depth Figure 4: Layer-wise relative Frobenius drift of the residual stream between compressed and dense models on 16 sequences disjoint from the calibration set. We compare Ftc with five representative methods on four GQA models at ρ∈0.6,0.2ρ∈\0.6,0.2\. Lower is better. Ftc generally maintains smaller drift across depth, particularly under aggressive compression on Mistral-7B and Llama-3-8B. On the Qwen models, the metric becomes less discriminative through the middle layers at ρ=0.2ρ=0.2, with clearer separation near the output. A.7 Robustness and Sensitivity Table 20: Calibration-seed robustness of Ftc. Mean± over three calibration sets (seeds 42,1,2\42,1,2\). Model WT2 (ρ=.6ρ=.6) WT2 (ρ=.2ρ=.2) avg6 (ρ=.2ρ=.2) Mistral-7B 5.44±0.005.44± 0.00 7.30±0.027.30± 0.02 0.511±0.0050.511± 0.005 Llama-3-8B 6.65±0.006.65± 0.00 10.45±0.0310.45± 0.03 0.493±0.0030.493± 0.003 Qwen2.5-7B 7.20±0.017.20± 0.01 12.65±0.0712.65± 0.07 0.405±0.0020.405± 0.002 Qwen3-8B 7.39±0.037.39± 0.03 10.70±0.0210.70± 0.02 0.494±0.0080.494± 0.008 To assess the sensitivity of Ftc to calibration sampling, we repeat the compression procedure with three independently sampled calibration sets. Table 20 shows that perplexity is highly stable across all four GQA models: the standard deviation is at most 0.030.03 ppl in all displayed settings except Qwen2.5-7B at the most aggressive budget, where it remains 0.070.07 ppl. Downstream avg6 is similarly stable at ρ=0.2ρ=0.2, with standard deviations between 0.0020.002 and 0.0080.008. These results indicate limited sensitivity to calibration sampling across the tested seeds. Appendix B Implementation and Optimization Details B.1 Overall Compression Algorithm Figure 1 provides a conceptual overview of the proposed framework, while Algorithm 1 summarizes its complete layerwise implementation. The compression proceeds sequentially through the attention layers of the model. At each layer, activations from the current student model are used to construct the activation-induced metric for joint Q/K/V compression. The resulting Q/K/V projections are inserted into the student model before the attention output projection WO,lW_O,l is processed, so that its compression is based on the representations produced by the updated student attention branch. After replacing the corresponding projection weights with their compressed counterparts, the updated student model is used to generate the activations required for compressing the next layer. After the Q/K/V projections of layer l have been compressed, let ZlSZ_l^S denote the input to the attention output projection WO,lW_O,l produced by the updated student attention branch, and let ZlTZ_l^T denote the corresponding input produced by the pretrained teacher model. These representations are used only for the separate compression of WO,lW_O,l. The details of the activation-induced metric, shared Tucker representation, and budget-coupled rank–sparsity allocation are provided in Appendices C–E, respectively, while sparse-core refinement and output-projection compression are detailed below. Algorithm 1 Sequential Function-Driven Structured Compression 1: Pretrained teacher model fTf^T, calibration set calD_cal, layerwise Q/K/V budgets Bll=1L\B_l\_l=1^L, and candidate rank sets ℛll=1L\R_l\_l=1^L. 2: Compressed student model fSf^S 3: Initialize the student model as fS←fTf^S← f^T. 4: for l=1,…,Ll=1,…,L do 5: Collect the current student inputs XlSX_l^S to attention layer l using calD_cal. 6: Construct the activation-induced metric Hε,lH_ ,l from XlSX_l^S 7: Form the joint head-wise Q/K/V tensor lW_l from WQ,lW_Q,l, WK,lW_K,l, and WV,lW_V,l. 8: Apply metric whitening and frontal-slice normalization to obtain the tensor used for structured approximation. 9: Compute an HOSVD-based shared Tucker initialization for evaluating the candidate rank pairs. 10: for each l=(R1,l,R2,l)∈ℛlR_l=(R_1,l,R_2,l) _l do 11: Determine the admissible core cardinality Kl(l)K_l(R_l) under budget BlB_l. 12: Evaluate the corresponding projection core and allocation score Jl(l)J_l(R_l). 13: end for 14: Fix the Tucker factors at the selected rank (R1,l⋆,R2,l⋆)(R_1,l ,R_2,l ) and recompute the corresponding dense projection core. 15: Gradually prune the projection core and perform support-constrained HOOI-style refinement until the target cardinality Kl⋆K_l is reached. 16: Map the compressed Tucker representation back to the original weight coordinates and recover W^Q,l W_Q,l, W^K,l W_K,l, and W^V,l W_V,l 17: Run the calibration sequences through the updated student attention branch to collect ZlSZ_l^S, and collect the corresponding teacher representation ZlTZ_l^T from fTf^T. 18: Construct the output-projection activation-induced metric from ZlSZ_l^S and ZlTZ_l^T. 19: Construct the calibrated target from (ZlS,ZlT)(Z_l^S,Z_l^T) when enabled. 20: Compress WO,lW_O,l separately at the prescribed compression ratio to obtain W^O,l W_O,l. 21: Replace WO,lW_O,l by W^O,l W_O,l in fSf^S. 22: Propagate the updated student to provide the representations used at layer l+1l+1. 23: end for 24: return fSf^S B.2 Gradual Pruning and Sparse-Core Refinement Once the rank–sparsity allocator selects (R1,l⋆,R2,l⋆,Kl⋆)(R_1,l ,R_2,l ,K_l ), we further refine the corresponding Tucker representation before mapping it back to the original weight coordinates. The candidate Tucker subspaces used in the allocation stage are initialized by HOSVD, which provides a shared low-dimensional basis for evaluating different rank pairs. The one-step top-K projection used by the allocator serves only as a surrogate for estimating the sparsification cost of each candidate rank pair. The final sparse core is instead obtained progressively, allowing the surviving Tucker representation to adapt after each reduction in core cardinality. Let ¯l(0) G_l^(0) denote the dense projection core of the selected Tucker ranks in the balanced coordinates introduced in Section 3.4. We gradually reduce its number of active coefficients according to a decreasing sequence Kl(0)>Kl(1)>⋯>Kl(T)=Kl⋆.K_l^(0)>K_l^(1)>·s>K_l^(T)=K_l . Gradual support reduction. The cardinality sequence is controlled by a gradual core-pruning rate α∈(0,1]α∈(0,1]: Kl(t+1)=maxKl(t)−⌈αKl(t)⌉,Kl⋆.K_l^(t+1)= \K_l^(t)- α K_l^(t) ,\,K_l \. At stage t, we rank the currently active core coefficients by magnitude and retain the Kl(t+1)K_l^(t+1) largest ones. Denoting the resulting support by Ωl(t+1) _l^(t+1), we obtain the nested sequence Ωl(t+1)⊆Ωl(t),|Ωl(t)|=Kl(t). _l^(t+1) _l^(t), | _l^(t)|=K_l^(t). For 0<α<10<α<1, the number of pruning stages is approximately T≈log(Kl⋆/Kl(0))log(1−α).T≈ (K_l /K_l^(0)) (1-α). We use α=0.02α=0.02 in the main experiments, so each stage removes approximately 2%2\% of the currently surviving core coefficients. Setting α=1α=1 recovers one-shot magnitude pruning directly to Kl⋆K_l . Latent-core sparsity. The pruning is applied entry-wise to the Tucker core rather than to the reconstructed weight matrices. For the j-th frontal slice, the reconstructed projection can be written as W¯l,j=∑a=1R1,l⋆∑b=1R2,l⋆¯l(a,b,j)u1,l(a)(u2,l(b))⊤. W_l,j= _a=1^R_1,l _b=1^R_2,l G_l(a,b,j)\,u_1,l^(a) (u_2,l^(b) ) . Each core entry ¯l(a,b,j) G_l(a,b,j) serves as the coefficient of a latent rank-one component u1,l(a)(u2,l(b))⊤u_1,l^(a)(u_2,l^(b)) in the reconstruction of the j-th frontal slice. Setting this coefficient to zero deactivates the corresponding latent component. As more core coefficients are pruned, fewer latent components remain active, and the Tucker core becomes increasingly sparse. Importantly, this sparsity acts in the latent Tucker representation rather than directly in the original weight space. Since the Tucker factors are generally dense, sparsifying the core reduces the number of active latent components but does not necessarily introduce zeros into the reconstructed Q/K/V projection matrices. Support-constrained refinement. After each support reduction, we refine the Tucker representation while keeping both the selected ranks and the current support fixed. Let Ωl(t) _l^(t) denote the active support at stage t. Factor updates are allowed to change the shared Tucker subspaces, but coefficients outside Ωl(t) _l^(t) are constrained to remain zero throughout the refinement. For fixed factors, the dense projection core is ¯proj,l=¯l×1U~1,l⊤×2U2,l⊤. G_proj,l= W_l×_1 U_1,l ×_2U_2,l . We then restrict this core to the current support ¯l=Ωl(t)(¯proj,l), G_l=P_ _l^(t) ( G_proj,l ), where ΩP_ keeps entries indexed by Ω and sets all others to zero. We alternate the support-constrained core update with orthogonal factor-refitting steps. Each pruning stage uses two rounds of factor refitting, after which the core is again projected onto the same support. Thus, refinement may change the Tucker factors and the values of the surviving coefficients, but it does not increase the number of active core entries. The refined representation is then used to determine the magnitude ordering for the next support reduction. The resulting procedure therefore alternates support reduction⟶refit(1)⟶refit(2)⟶support reduction,support reduction\; \;refit^(1)\; \;refit^(2)\; \;support reduction, until ‖¯l‖0=Kl⋆. \| G_l \|_0=K_l . This gradual prune–refit schedule allows the shared Tucker subspaces and the surviving core coefficients to adapt as the sparsity constraint is progressively tightened, rather than fixing the final support from the initial dense core in a single pruning step. B.3 Calibration and Compression of the Output Projection The attention output projection WO,lW_O,l is treated separately from the Q/K/V projections not only because it receives a different input representation, but also because it occupies a fundamentally different position in the attention computation. The Q/K/V projections enter the coupled nonlinear map (Q,K,V)=softmax(QK⊤dh)V.A(Q,K,V)=softmax ( QK d_h )V. In contrast, after the attention output is formed, WO,lW_O,l is applied through a linear transformation to the resulting representation. At the output-projection stage, a candidate weight matrix W produces the student output ZlSW,Z_l^SW, whereas the corresponding teacher output is ZlTWO,l.Z_l^TW_O,l. Therefore, for fixed ZlSZ_l^S and ZlTZ_l^T, the quantity 1N‖ZlSW−ZlTWO,l‖F2 1N \|Z_l^SW-Z_l^TW_O,l \|_F^2 directly quantifies the local discrepancy between the student and teacher outputs at the output-projection stage. Teacher–student calibrated target. Because the upstream Q/K/V compression changes the representation entering the output projection from ZlTZ_l^T to ZlSZ_l^S, directly compressing the pretrained WO,lW_O,l ignores this representation shift. When output-projection calibration is enabled, we then construct a regularized weight target by solving minW1N‖ZlSW−ZlTWO,l‖F2+μO‖W−WO,l‖F2, _W\; 1N \|Z_l^SW-Z_l^TW_O,l \|_F^2+ _O \|W-W_O,l \|_F^2, (5) where μO>0 _O>0 controls the deviation of the calibrated solution from the pretrained output projection. The first term seeks a linear operator whose output on the current student representation matches the corresponding teacher branch output, while the second term regularizes this correction toward the original WO,lW_O,l. Thus, the calibration step compensates for the upstream representation shift without allowing an unconstrained departure from the pretrained projection. Define the empirical second moment and cross second moment Cs,lO=1N(ZlS)⊤ZlS,Cst,lO=1N(ZlS)⊤ZlT.C_s,l^O= 1N(Z_l^S) Z_l^S, C_st,l^O= 1N(Z_l^S) Z_l^T. The first-order optimality condition of (5) gives (Cs,lO+μOI)W=(Cst,lO+μOI)WO,l, (C_s,l^O+ _OI )W= (C_st,l^O+ _OI )W_O,l, and hence the calibrated target is WO,lcal=(Cs,lO+μOI)−1(Cst,lO+μOI)WO,l.W_O,l^cal= (C_s,l^O+ _OI )^-1 (C_st,l^O+ _OI )W_O,l. (6) In practice, (6) is evaluated through a linear solve rather than by explicitly forming the matrix inverse. To cover both calibrated and non-calibrated configurations, we write WO,ltgt=WO,lcal,if output-projection calibration is enabled,WO,l,otherwise.W_O,l^tgt= casesW_O,l^cal,&if output-projection calibration is enabled,\\[5.69054pt] W_O,l,&otherwise. cases Thus, the calibration step determines the weight matrix to be approximated, whereas the approximation geometry is constructed separately below. Output-specific activation-induced metric. The compression error of an approximation W^O,l W_O,l is measured under the input distribution of the current student branch. Let ΔWO,l=WO,ltgt−W^O,l. W_O,l=W_O,l^tgt- W_O,l. Then 1N‖ZlSΔWO,l‖F2 1N \|Z_l^S W_O,l \|_F^2 =tr(ΔWO,l⊤Cs,lOΔWO,l). =tr ( W_O,l C_s,l^O W_O,l ). This identity induces a specific metric from the current student representations. We denote by C¯lO C_l^O the empirical second-moment matrix used to construct the output-projection metric, with C¯lO=Cs,lO C_l^O=C_s,l^O when only the current student representations are used. Metric whitening and low-rank compression. Let Hε,lO=(LlO)⊤LlOH_ ,l^O=(L_l^O) L_l^O be a Cholesky factorization. The weighted approximation objective can then be rewritten as ‖WO,ltgt−W^O,l‖Hε,lO2 \|W_O,l^tgt- W_O,l \|_H_ ,l^O^2 =tr[(WO,ltgt−W^O,l)⊤Hε,lO(WO,ltgt−W^O,l)] =tr [ (W_O,l^tgt- W_O,l ) H_ ,l^O (W_O,l^tgt- W_O,l ) ] =‖LlO(WO,ltgt−W^O,l)‖F2. = \|L_l^O (W_O,l^tgt- W_O,l ) \|_F^2. Accordingly, we define the whitened target W~O,ltgt=LlOWO,ltgt. W_O,l^tgt=L_l^OW_O,l^tgt. We compute its singular value decomposition W~O,ltgt=UlOΣlO(VlO)⊤, W_O,l^tgt=U_l^O _l^O(V_l^O) , and retain the leading rO,lr_O,l singular components, where rO,lr_O,l is chosen according to the prescribed output-projection compression budget. This gives W~O,l(r)=Ul,rOΣl,rO(Vl,rO)⊤. W_O,l^(r)=U_l,r^O _l,r^O(V_l,r^O) . The compressed output projection is finally recovered in the original weight coordinates by solving LlOW^O,l=W~O,l(r).L_l^O W_O,l= W_O,l^(r). Thus, the output projection is compressed through an independent metric-whitened low-rank approximation whose metric is induced by its own attention-input representations. B.4 Optional Pattern-Aligned Refinement for 2:4 Sparsity For hard structured sparsity such as 2:42:4, the pruning result depends on the coordinate representation of the Tucker core. To improve the compatibility between the dense Tucker representation and the prescribed sparsity pattern, we optionally apply an orthogonal gauge transformation before structured pruning. Recall that the compressed Q/K/V tensor is represented as ^l=l×1U1,l×2U2,l. W_l=G_l×_1U_1,l×_2U_2,l. For any orthogonal matrices Q1,l⊤Q1,l=I,Q2,l⊤Q2,l=I,Q_1,l Q_1,l=I, Q_2,l Q_2,l=I, we may equivalently write ^l=l,Q×1U1,lQ×2U2,lQ, W_l=G_l,Q×_1U_1,l^Q×_2U_2,l^Q, where l,Q=l×1Q1,l⊤×2Q2,l⊤,U1,lQ=U1,lQ1,l,U2,lQ=U2,lQ2,l.G_l,Q=G_l×_1Q_1,l ×_2Q_2,l , U_1,l^Q=U_1,lQ_1,l, U_2,l^Q=U_2,lQ_2,l. Thus, the rotation leaves the dense Tucker reconstruction unchanged, while redistributing the coefficients within the latent core. Let 2:4P_2:4 denote the projection that enforces the prescribed 2:4 pattern on the core. We seek an orthogonal rotation that approximately reduces the structured projection error, minQ1,l,Q2,l‖l,Q−2:4(l,Q)‖F2,Q1,l⊤Q1,l=Q2,l⊤Q2,l=I. _Q_1,l,Q_2,l \|G_l,Q-P_2:4 (G_l,Q ) \|_F^2, Q_1,l Q_1,l=Q_2,l Q_2,l=I. (7) Since the rotations are orthogonal, ‖l,Q‖F=‖l‖F\|G_l,Q\|_F=\|G_l\|_F, and (7) can equivalently be viewed as concentrating more core energy on the coefficients that survive the 2:4 projection. The resulting rotation is used only when it provides a sufficient increase in the retained core energy. Specifically, letting η2:4()=‖2:4()‖F2‖F2, _2:4(G)= \|P_2:4(G) \|_F^2\|G\|_F^2, we apply the rotated representation only if η2:4(l,Q)−η2:4(l)>τ. _2:4(G_l,Q)- _2:4(G_l)>τ. (8) Otherwise, the original Tucker coordinates are retained. The accepted representation is then used as the initialization for the subsequent 2:4-constrained sparse-core refinement. This procedure does not alter the Tucker ranks or the dense represented subspace. Rather, it selects a latent coordinate system in which the prescribed 2:4 constraint removes less energy before structured refinement. Consequently, the rotation is used as a guarded pattern-alignment step rather than as an additional source of compression. B.5 Numerical Stability and Practical Details Since the activation-induced statistics are estimated from finite calibration data, the resulting Gram matrices may be singular or poorly conditioned. We therefore apply diagonal damping to both the Q/K/V and output-projection metrics before whitening. The damping parameter ε is used only for numerical stabilization and is distinct from the regularization parameter μO _O used to construct the calibrated target for WO,lW_O,l. Teacher-Gram anchoring for WOW_O. For the output projection, we additionally anchor the student attention statistics with the corresponding teacher statistics. Let Cs,lO=1N(ZlS)⊤ZlS,Ct,lO=1N(ZlT)⊤ZlT.C_s,l^O= 1N(Z_l^S) Z_l^S, C_t,l^O= 1N(Z_l^T) Z_l^T. We form the anchored Gram matrix as Canc,lO=(1−a)Cs,lO+aCt,lO,C_anc,l^O=(1-a)C_s,l^O+aC_t,l^O, with a=0.5a=0.5 in the final recipe. This anchoring is used only for WOW_O; the Q/K/V metric remains unanchored and is constructed solely from the current student representations. Both the Q/K/V and output-projection metrics are damped before whitening using the same damping coefficient ε . Stable metric transformations. For a damped metric Hε,l=Ll⊤Ll,H_ ,l=L_l L_l, (9) we perform whitening and inverse whitening through linear or triangular solves rather than explicitly forming Hε,l−1H_ ,l^-1 or Ll−1L_l^-1. The same procedure is used for the output-projection metric. Adaptive damping and fallback. The default damping is used whenever the resulting compressed representation passes the stability check. If numerical instability is detected, we increase the damping geometrically, εl(r+1)=10εl(r), _l^(r+1)=10\, _l^(r), (10) and recompute the compression. If the metric-based approximation remains unstable, we use the identity metric as a final fallback. All retries preserve the original compression target and parameter budget. In particular, the Tucker ranks and sparse-core budget for Q/K/V, as well as the low-rank budget for WO,lW_O,l, are kept unchanged. Therefore, the stability safeguards improve numerical robustness without relaxing the prescribed compression ratio or leaving unstable layers uncompressed. Appendix C Additional Details for Sequential Activation-Induced Tensor Approximation This appendix provides additional details for the sequential activation-induced tensor approximation introduced in Section 3.2. We first derive the local activation-induced metric, then describe the construction of the joint head-wise Q/K/V tensor and its metric whitening, and finally clarify how the compression proceeds sequentially across Transformer layers. Sequential student representations. The compression is performed layer by layer. At layer l, the calibration input XlSX_l^S is obtained from the current student model with all preceding compressed layers already in place. Consequently, the metric at layer l is constructed from the actual student representations and naturally incorporates the representation shift induced by earlier compression. For a projection matrix Wt,lW_t,l, the perturbation ΔWt,l=Wt,l−W^t,l W_t,l=W_t,l- W_t,l induces the following local functional distortion on the current student activations: 1N‖XlSΔWt,l‖F2. 1N \|X_l^S W_t,l \|_F^2. (11) This quantity measures the effect of the weight perturbation on the projection outputs over the calibration representations at layer l, and naturally induces a data-dependent metric for evaluating the approximation error. Activation-induced approximation metric. Define the empirical second-moment matrix Cs,l=1N(XlS)⊤XlS.C_s,l= 1N(X_l^S) X_l^S. Then the local projection error in (11) can be written as 1N‖XlSΔWt,l‖F2 1N \|X_l^S W_t,l \|_F^2 =1Ntr(ΔWt,l⊤(XlS)⊤XlSΔWt,l) = 1Ntr ( W_t,l (X_l^S) X_l^S W_t,l ) =tr(ΔWt,l⊤Cs,lΔWt,l). =tr ( W_t,l C_s,l W_t,l ). We therefore define ‖ΔW‖Cs,l2:=tr(ΔW⊤Cs,lΔW).\| W\|_C_s,l^2:=tr ( W C_s,l W ). Hence, the activation-induced metric is directly obtained from the projection-output error evaluated on the current student representations. Since the Q, K, and V projections share the same input XlSX_l^S, they also share the same input-side metric Cs,lC_s,l. Joint head-wise Q/K/V tensor construction. We reshape the Q/K/V projection matrices into their native head-wise blocks. Let WQ,l(i)∈ℝdmodel×dhead,i=1,…,hq,l,W_Q,l^(i) ^d_model× d_head, i=1,…,h_q,l, denote the query-head blocks, and similarly let WK,l(i),WV,l(i)∈ℝdmodel×dhead,i=1,…,hkv,l,W_K,l^(i),\,W_V,l^(i) ^d_model× d_head, i=1,…,h_kv,l, denote the key and value blocks. We stack these matrices as frontal slices of a joint tensor l∈ℝdmodel×dhead×ml,ml=hq,l+2hkv,l.W_l ^d_model× d_head× m_l, m_l=h_q,l+2h_kv,l. The query slices are placed first, followed by the key and value slices. The joint tensor retains the native head organization of the attention module. In GQA and MQA architectures, the shared key/value heads are kept in their original form rather than replicated to match the number of query heads. Accordingly, the third mode serves only to enumerate the native Q/K/V head slices and is not compressed in the Tucker decomposition. For a tensor ∈ℝdmodel×dhead×mlA ^d_model× d_head× m_l, we extend the activation-induced metric as ‖Cs,l2:=∑j=1mltr(Aj⊤Cs,lAj),\|A\|_C_s,l^2:= _j=1^m_ltr (A_j C_s,lA_j ), where AjA_j denotes the j-th frontal slice. Thus, the joint tensor objective retains the same activation-induced metric for every Q/K/V slice while allowing their structural information to be modeled jointly. Metric whitening. For numerical stability, we use the damped metric Hε,l=Cs,l+εlt¯lI,εl>0.H_ ,l=C_s,l+ _l t_lI, _l>0. Let Hε,l=Ll⊤LlH_ ,l=L_l L_l be its Cholesky factorization. Since the metric acts on the shared input dimension, it affects only mode-11 of the joint tensor. For any tensor A of the same size as lW_l, ‖Hε,l2 \|A\|_H_ ,l^2 =∑j=1mltr(Aj⊤Hε,lAj) = _j=1^m_ltr (A_j H_ ,lA_j ) =‖×1Ll‖F2. = \|A×_1L_l \|_F^2. Accordingly, we define the whitened Q/K/V tensor as ~l=l×1Ll. W_l=W_l×_1L_l. The activation-weighted tensor approximation can therefore be carried out as an ordinary Frobenius-norm approximation in the whitened coordinates. Sequential layer update. After obtaining the compressed Q/K/V projections W^Q,l,W^K,l,W^V,l, W_Q,l, W_K,l, W_V,l, we replace their original counterparts in the student model. Subsequent representations at layer l are evaluated using this updated student, so that the remaining compression operations are conditioned on the current compressed state of the model. After completing the compression of layer l, the updated student model is propagated forward to obtain the input representation Xl+1SX_l+1^S for the next layer. The resulting sequential procedure can be summarized as XlS⟶Cs,l⟶^l⟶update student⟶Xl+1S.X_l^S\; \;C_s,l\; \; W_l\; \;update student\; \;X_l+1^S. Accordingly, the activation-induced metric is recomputed at each layer from the current student representations, thereby accounting for the representation changes introduced by the preceding compression steps. Appendix D Proof of the Weighted Shared-Subspace Characterization We provide the proof of the weighted shared mode-11 subspace characterization stated in Section 3.3. We follow the notation introduced there. Proposition D.1 (Weighted shared mode-11 subspace). Consider min∑j=1mlU~l⊤U~l=IB1,l,…,Bml,lwj,l‖W~l,j−U~lBj,l‖F2, _ subarrayc U_l U_l=I\\ B_1,l,…,B_m_l,l subarray _j=1^m_lw_j,l \| W_l,j- U_lB_j,l \|_F^2, (12) where wj,l>0w_j,l>0 and U~l∈ℝdmodel×R1,l U_l ^d_model× R_1,l. For any fixed U~l U_l, the optimal coefficient matrices are Bj,l⋆=U~l⊤W~l,j.B_j,l = U_l W_l,j. Moreover, an optimal U~l U_l is given by the leading R1,lR_1,l left singular vectors of W~cat,l=[w1,lW~l,1⋯wml,lW~l,ml]. W_cat,l= bmatrix w_1,l W_l,1&·s& w_m_l,l W_l,m_l bmatrix. (13) Proof. For fixed U~l U_l, the orthogonality condition U~l⊤U~l=I U_l U_l=I gives ‖W~l,j−U~lBj,l‖F2 \| W_l,j- U_lB_j,l \|_F^2 =‖W~l,j‖F2−2⟨U~l⊤W~l,j,Bj,l⟩+‖Bj,l‖F2 =\| W_l,j\|_F^2-2 U_l W_l,j,B_j,l +\|B_j,l\|_F^2 =‖W~l,j‖F2−‖U~l⊤W~l,j‖F2+‖Bj,l−U~l⊤W~l,j‖F2. =\| W_l,j\|_F^2- \| U_l W_l,j \|_F^2+ \|B_j,l- U_l W_l,j \|_F^2. Therefore, Bj,l⋆=U~l⊤W~l,j.B_j,l = U_l W_l,j. Substituting these minimizers into (12) yields ∑j=1mlwj,l‖W~l,j−U~lBj,l⋆‖F2 _j=1^m_lw_j,l \| W_l,j- U_lB_j,l \|_F^2 =∑j=1mlwj,l‖W~l,j‖F2−∑j=1mlwj,l‖U~l⊤W~l,j‖F2. = _j=1^m_lw_j,l\| W_l,j\|_F^2- _j=1^m_lw_j,l \| U_l W_l,j \|_F^2. The first term is independent of U~l U_l. Therefore, the problem is equivalent to max∑j=1mlU~l⊤U~l=Iwj,l‖U~l⊤W~l,j‖F2. _ U_l U_l=I _j=1^m_lw_j,l \| U_l W_l,j \|_F^2. Using the weighted concatenation in (13), we have ∑j=1mlwj,l‖U~l⊤W~l,j‖F2=‖U~l⊤W~cat,l‖F2. _j=1^m_lw_j,l \| U_l W_l,j \|_F^2= \| U_l W_cat,l \|_F^2. Equivalently, ‖U~l⊤W~cat,l‖F2=tr(U~l⊤W~cat,lW~cat,l⊤U~l). \| U_l W_cat,l \|_F^2=tr ( U_l W_cat,l W_cat,l U_l ). Maximizing this trace over orthonormal U~l U_l selects the dominant R1,lR_1,l-dimensional eigenspace of W~cat,lW~cat,l⊤ W_cat,l W_cat,l , equivalently the leading left singular subspace of W~cat,l W_cat,l. ∎ This result formally justifies the weighted concatenation used to identify the shared mode-11 subspace. The full Tucker representation in Section 3.3 extends the same sharing principle to both compressed modes, while retaining the third mode in full. Appendix E Additional Details for Budget-Coupled Rank–Sparsity Allocation This section provides the supporting derivations for Proposition 3.1. We first establish the fixed-subspace error decomposition and the optimal top-KlK_l core projection, and then show how these results lead to the budget-coupled rank search. The gradual pruning and support-constrained refinement used to obtain the final sparse core are described separately in Appendix B.2. E.1 Fixed-Subspace Error Decomposition For a candidate rank pair l=(R1,l,R2,l)R_l=(R_1,l,R_2,l), let U~1,l U_1,l and U2,lU_2,l be the corresponding orthonormal factors, and recall the projection core ¯l⋆=¯l×1U~1,l⊤×2U2,l⊤. G_l = W_l×_1 U_1,l ×_2U_2,l . Define the associated projection residual by ℰl:=¯l−¯l⋆×1U~1,l×2U2,l.E_l:= W_l- G_l ×_1 U_1,l×_2U_2,l. (14) Proposition E.1 (Fixed-subspace error decomposition). For any ∈ℝR1,l×R2,l×mlG ^R_1,l× R_2,l× m_l, ‖¯l−×1U~1,l×2U2,l‖F2 \| W_l-G×_1 U_1,l×_2U_2,l \|_F^2 =‖ℰl‖F2+‖¯l⋆−‖F2. =\|E_l\|_F^2+ \| G_l -G \|_F^2. (15) Proof. Using (14), we have ¯l−×1U~1,l×2U2,l W_l-G×_1 U_1,l×_2U_2,l =ℰl+(¯l⋆−)×1U~1,l×2U2,l. =E_l+ ( G_l -G )×_1 U_1,l×_2U_2,l. (16) It observes that ℰlE_l is the residual of the orthogonal projection onto the Tucker subspace generated by U~1,l U_1,l and U2,lU_2,l. Hence, ⟨ℰl,×1U~1,l×2U2,l⟩=0 _l,\,A×_1 U_1,l×_2U_2,l =0 for any compatible tensor A. Therefore, the two terms in (16) are orthogonal. Since the two factors have orthonormal columns, ‖×1U~1,l×2U2,l‖F=∥F. \|A×_1 U_1,l×_2U_2,l \|_F=\|A\|_F. It follows that ‖¯l−×1U~1,l×2U2,l‖F2 \| W_l-G×_1 U_1,l×_2U_2,l \|_F^2 =‖ℰl‖F2+‖¯l⋆−‖F2. =\|E_l\|_F^2+ \| G_l -G \|_F^2. The quantity ‖ℰl‖F2\|E_l\|_F^2 represents the approximation error due to the selected Tucker subspace, which we denote by Esub(l)E_sub(R_l) in the following. ∎ Proposition E.1 shows that the total approximation error separates into the error of the selected Tucker subspace and the error incurred when approximating the corresponding projection core. In particular, the latter can be evaluated directly in the latent core space. E.2 Optimal Cardinality Projection of the Core We next specialize the previous result to the entry-wise cardinality constraint used for candidate scoring. Proposition E.2. For fixed orthonormal factors U~1,l U_1,l and U2,lU_2,l, the solution of min‖0≤Kl‖¯l−×1U~1,l×2U2,l‖F2 _\|G\|_0≤ K_l \| W_l-G×_1 U_1,l×_2U_2,l \|_F^2 (17) is Kl⋆=Kl(¯l⋆),G _K_l=P_K_l ( G_l ), (18) where KlP_K_l retains the KlK_l largest-magnitude entries of the projection core. Consequently, Estruct(l,Kl)=Esub(l)+Dsp(l,Kl).E_struct(R_l,K_l)=E_sub(R_l)+D_sp(R_l,K_l). (19) Proof. By Proposition E.1, problem (17) is equivalent to min‖0≤Kl‖¯l⋆−‖F2, _\|G\|_0≤ K_l \| G_l -G \|_F^2, since the subspace error does not depend on G. The best KlK_l-term approximation in the Frobenius norm is obtained by retaining the KlK_l entries of largest magnitude. Substituting (18) into (15) yields (19). ∎ Eq. (19) is exact for the selected Tucker subspace, with the factors and ranks fixed and ¯l⋆ G_l taken as the corresponding projection core. Subsequent refinement may further adjust the core values, support, and Tucker factors. E.3 Interpretation of Latent-Core Sparsity The sparsity constraint in FTC is imposed on the entries of the latent Tucker core rather than directly on the reconstructed Q/K/V weight matrices. To make this distinction explicit, write U~1,l=[u~1,l(1)⋯u~1,l(R1,l)],U2,l=[u2,l(1)⋯u2,l(R2,l)]. U_1,l= bmatrix u_1,l^(1)&·s& u_1,l^(R_1,l) bmatrix, U_2,l= bmatrixu_2,l^(1)&·s&u_2,l^(R_2,l) bmatrix. For the j-th frontal slice, the reconstructed weight matrix is W¯^l,j=U~1,lGl,jU2,l⊤, W_l,j= U_1,lG_l,jU_2,l , where Gl,j=(:,:,j)G_l,j=G(:,:,j). Then expanding the matrix product gives W¯^l,j=∑r1=1R1,l∑r2=1R2,lgr1,r2,ju~1,l(r1)(u2,l(r2))⊤, W_l,j= _r_1=1^R_1,l _r_2=1^R_2,lg_r_1,r_2,j\, u_1,l^(r_1) (u_2,l^(r_2) ) , (20) where gr1,r2,jg_r_1,r_2,j denotes the (r1,r2)(r_1,r_2)-th entry of the j-th core slice. Eq. (20) provides a structural interpretation of core sparsity. Each core coefficient controls one rank-one interaction between a mode-11 basis vector u~1,l(r1) u_1,l^(r_1) and a mode-22 basis vector u2,l(r2)u_2,l^(r_2). Therefore, setting a core coefficient to zero removes the corresponding rank-one component from the factorized representation. Let l,j=(r1,r2):gr1,r2,j≠0S_l,j= \(r_1,r_2):g_r_1,r_2,j≠ 0 \ denote the active support of the j-th core slice. Then W¯^l,j=∑(r1,r2)∈l,jgr1,r2,ju~1,l(r1)(u2,l(r2))⊤. W_l,j= _(r_1,r_2) _l,jg_r_1,r_2,j\, u_1,l^(r_1) (u_2,l^(r_2) ) . (21) Hence, the sparse core determines which rank-one interactions between the shared factor directions are active for each Q/K/V projection slice. The Tucker factors are shared across all Q/K/V slices, while each slice has its own core coefficients and may activate a different subset of the resulting rank-one components. This view also clarifies the difference between latent-core sparsity and entry-wise weight sparsity. Although only a subset of the rank-one components is active in (21), each rank-one component is generally dense because both factor vectors are dense. Consequently, W¯^l,j W_l,j is typically a dense matrix even when Gl,jG_l,j is highly sparse. The sparsity therefore lies in the low-dimensional interaction structure between the two shared factor spaces, rather than in the entries of the reconstructed weight matrix. Accordingly, the computational form of the compressed representation is not a materialized dense matrix but the sparse rank-one expansion in (21). Computational benefits from core sparsity therefore require an implementation that exploits this sparse latent interaction structure instead of first reconstructing the dense Q/K/V weight matrices. This interpretation is also consistent with the parameter accounting in Section 3.4. The shared factors contribute dmodelR1,l+dheadR2,ld_modelR_1,l+d_headR_2,l dense parameters, while the sparse core contributes the retained latent coefficients as well as the storage required to encode their support. Thus, KlK_l measures the number of active rank-one interactions across the joint Q/K/V representation, rather than the number of nonzero entries in the reconstructed projection matrices. E.4 Efficient Evaluation of Candidate Rank Pairs The rank–sparsity allocation requires comparing multiple feasible rank pairs, but running a complete Tucker optimization and sparse-core refinement for every candidate would introduce substantial overhead. We therefore use a common HOSVD-based initialization to evaluate all candidate configurations efficiently. The key observation is that the candidate Tucker subspaces are nested and can be extracted from a single pair of mode-wise singular subspaces. Let ¯l,(1) W_l,(1) and ¯l,(2) W_l,(2) denote the mode-11 and mode-22 unfoldings of the balanced whitened tensor ¯l W_l. We compute ¯l,(1)=U~1,lfullΣ1,lV1,l⊤,¯l,(2)=U2,lfullΣ2,lV2,l⊤. W_l,(1)= U_1,l^full _1,lV_1,l , W_l,(2)=U_2,l^full _2,lV_2,l . For a candidate l=(R1,l,R2,l)R_l=(R_1,l,R_2,l), the corresponding Tucker factors are obtained by truncating these common bases, U~1,l(l)=U~1,lfull(:,1:R1,l),U2,l(l)=U2,lfull(:,1:R2,l). U_1,l^(R_l)= U_1,l^full(:,1:R_1,l), U_2,l^(R_l)=U_2,l^full(:,1:R_2,l). Hence, the mode-wise singular subspaces are computed only once for each layer and reused across all candidate rank pairs. Let R1,lmaxR_1,l and R2,lmaxR_2,l denote the largest ranks appearing in the candidate grid. We further form the maximal projection core ¯lmax=¯l×1(U~1,lfull(:,1:R1,lmax))⊤×2(U2,lfull(:,1:R2,lmax))⊤, G_l = W_l×_1 ( U_1,l^full(:,1:R_1,l ) ) ×_2 (U_2,l^full(:,1:R_2,l ) ) , its projection core is ¯l⋆(l)=¯lmax(1:R1,l, 1:R2,l,:). G_l (R_l)= G_l (1:R_1,l,\,1:R_2,l,\,:). (22) Thus, the projection core can be reused across different candidate rank pairs. The subspace approximation error can also be evaluated without explicitly reconstructing the Tucker approximation. It can be written as Esub(l)=‖¯l‖F2−‖¯l⋆(l)‖F2.E_sub(R_l)=\| W_l\|_F^2- \| G_l (R_l) \|_F^2. (23) Therefore, candidate evaluation requires only the energy of the corresponding projection core rather than an explicit reconstruction of ¯l W_l. For the same candidate, the layerwise budget determines the admissible core cardinality Kl(l)K_l(R_l). Let |gl,(1)|≥|gl,(2)|≥⋯|g_l,(1)|≥|g_l,(2)|≥·s denote the magnitudes of the entries of ¯l⋆(l) G_l (R_l) arranged in descending order. The one-step sparsification error can then be evaluated directly from the discarded core energy: Dsp(l,Kl(l))=‖¯l⋆(l)‖F2−∑q=1Kl(l)|gl,(q)|2.D_sp (R_l,K_l(R_l) )= \| G_l (R_l) \|_F^2- _q=1^K_l(R_l)|g_l,(q)|^2. (24) Combining Eqs. (23) and (24), each feasible rank pair is evaluated using only quantities extracted from the common singular subspaces and the corresponding projection core. The candidate score is Jl(l)=Esub(l)+βDsp(l,Kl(l)).J_l(R_l)=E_sub(R_l)+β D_sp (R_l,K_l(R_l) ). In our implementation, the candidate set is a predefined 12×412× 4 grid over the mode-11 and mode-22 ranks. Rank pairs whose factor and support metadata costs already exceed the layerwise budget are discarded before scoring. The remaining candidates share the same mode-wise decompositions, and their projection cores are obtained from the nested subtensors in (22). Consequently, the allocation stage requires one pair of mode-wise decompositions and one maximal core projection per layer, followed by inexpensive energy and top-KlK_l evaluations over the candidate grid. The resulting top-KlK_l projection is used only as a proxy for comparing rank configurations. After selecting l⋆R_l , the corresponding Tucker representation is passed to the gradual sparse-core refinement procedure, where the core is progressively sparsified and refitted rather than being directly fixed to the one-step top-Kl⋆K_l projection.