Paper deep dive
Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression
Alessandro Beatini, Marco Maronese, Emanuele Rodolà
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/11/2026, 3:46:05 AM
Summary
The paper introduces Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that improves post-training tensor-network compression (Tensor Train and Tree Tensor Network) for Large Language Models. AWT preconditions weight matrices using diagonal activation-derived scales before decomposition, closing a significant portion of the perplexity gap compared to vanilla tensorization across models like Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B.
Entities (11)
Relation Signals (7)
AWT → appliedto → Ministral-8B
confidence 98% · Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization
AWT → appliedto → LLaMA-3.1-8B
confidence 98% · Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization
AWT → appliedto → Qwen2.5-7B
confidence 98% · Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization
AWT → improves → Tensor Train
confidence 95% · AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression
AWT → improves → Tree Tensor Network
confidence 95% · AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression
AWT → reduces → WikiText perplexity gap
confidence 95% · AWT closes 12-35% of the WikiText perplexity gap to the dense baseline
SmoothQuant → relatedto → AWQ
confidence 85% · Methods such as SmoothQuant and AWQ show that simple calibration statistics can identify activation channels
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (TT) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer's activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged TT/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression: under single-operator replacement, AWT closes 12-35% of the WikiText perplexity gap to the dense baseline across the three model families and 2-6 times compression settings; while under multi-operator Llama suffix replacement it closes 27-60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness-modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed TT/TTN compression pipelines without modifying the decomposition solver.
Tags
Links
- Source: https://arxiv.org/abs/2610.10085v1
- Canonical: https://arxiv.org/abs/2610.10085v1
Trouble viewing inline? Open PDF directly →
Full Text
84,172 characters extracted from source content.
Expand or collapse full text
Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression Alessandro Beatini Affiliation: Sapienza University of Rome Email: alessandro.beatini@uniroma1.it Marco Maronese Affiliation: Independent Researcher Email: marco.maronese95@gmail.com Affiliation: ORCID: 0000-0001-5548-5947 Emanuele Rodolà Affiliation: Sapienza University of Rome Email: rodola@di.uniroma1.it Abstract Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (T) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer’s activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged T/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla T/TTN tensorization at 2–6× compression: under single-operator replacement, AWT closes 12–35% of the WikiText perplexity gap to the dense baseline across the three model families and 2–6× compression settings; while under multi-operator Llama suffix replacement it closes 27–60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness–modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed T/TTN compression pipelines without modifying the decomposition solver. 1 Introduction Large language models (LLMs) are increasingly expensive to serve because dense linear operators dominate memory movement and compute during inference [2, 25, 3, 22]. This has motivated a broad set of post-training compression techniques, including quantization, pruning, distillation, low-rank factorization, and structured tensor decompositions [33]. Tensor-network compression offers a structured alternative to standard matrix factorizations. A dense matrix W∈ℝm×nW ^m× n can be reshaped into a higher-order tensor and represented by a network of low-rank cores: a Tensor Train (T) or matrix-product operator uses a chain of cores [21, 20], while a Tree Tensor Network (TTN) uses a loop-free hierarchy over tensor modes [23, 7]. Unlike truncated SVD, which imposes a single global rank on a matrix, tensor networks distribute rank budgets across tensor modes and internal edges of a chosen topology. This richer design space has made tensorization an increasingly relevant direction for LLM compression, with recent methods and systems such as CompactifAI, TensorLLM, and Minima demonstrating its use for compressing large neural operators and LLM components [24, 5, 1, 8, 15]. Despite this progress, post-training tensorization is typically activation-blind. Once a tensorization shape, topology, and rank budget are chosen, standard T/TTN backends approximate the weight matrix by minimizing an isotropic reconstruction objective in weight space. This objective is computationally convenient, but it does not directly capture the error induced by replacing a layer inside a pretrained model. This observation is well established in post-training quantization. Methods such as SmoothQuant and AWQ show that simple calibration statistics can identify activation channels whose perturbation disproportionately affects model behavior [29, 18]. However, the same principle has not been systematically exploited for post-training tensor-network decomposition. Tensor decompositions introduce a different constraint from quantization: the approximation is determined by a fixed tensorization layout, network topology, and internal rank budget rather than by a bit-width. This raises a natural question: can activation statistics be used to make existing tensor-network solvers more function-preserving, without changing the solver itself? We answer this question with Activation-aware Weight Tensorization (AWT). AWT computes a diagonal scale DαD_α from cached layer inputs, decomposes the reparameterized matrix W′=WDαW =WD_α using the same tensor-network backend, tensorization layout, topology, and rank budget as vanilla tensorization, and deploys the resulting approximation as y^=W~′(Dα−1x) y= W (D_α^-1x). Thus, activation information changes the matrix presented to the decomposition solver, but not the solver itself. The method is intentionally minimal: it transfers the equivalent reparameterization idea from activation-aware quantization to post-training tensor-network compression, requiring no retraining, no custom T/TTN solver, and no change to the tensorized operator apart from a lightweight input-side diagonal scaling. Empirically, we find that this simple preconditioning consistently improves the functional fidelity of tensorized LLM operators. The gains persist across model families, operator types, compression ratios, and tensor-network topologies, and become especially important when multiple operators are compressed jointly. In multi-operator suffix replacement, AWT closes 27–60% of the WikiText perplexity gap across both attention-group and all-seven-matrix settings (Table 2), indicating that activation-aware preconditioning helps control the accumulation of tensorization error inside the model. Contributions. Our main contributions are: (i) we introduce AWT, a diagonal activation-aware preconditioner for post-training T/TTN compression that preserves the underlying tensor-network solver and deployment path; (i) we show that AWT is equivalent to a column-weighted reconstruction proxy (Equation 5), connecting tensorization error to activation-conditioned functional distortion; (i) we demonstrate consistent gains over vanilla tensorization in single-operator and multi-operator LLM replacement experiments, with the largest absolute gains arising when compression errors accumulate across operators (Table 2); (iv) we evaluate AWT across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, three compression ratios, seven operator types, and T/TTN backends, including matched-compression SVD and downstream task checks; (v) we compare diagonal AWT with a dense full-covariance oracle and show that the diagonal restriction is a robustness–modularity tradeoff rather than a diagonal-covariance assumption. 2 Related Work Low-rank and tensorized neural compression. Low-rank factorization has long been used to compress neural networks, from matrix and filter decompositions for convolutional models to post-compression fine-tuning [4, 12, 16, 14]. Tensor-network parameterizations extend this idea by replacing dense operators with structured multilinear factors. Tensor Train (T) layers represent a matrix as a chain of low-order cores, building on the T format from numerical linear algebra [21, 20]; related formats such as Tensor Rings and Block-Term decompositions have also been used for neural compression [32, 17, 27]. These methods provide richer rank allocation than a single global matrix rank, but are typically optimized using weight-space reconstruction objectives. Tensorization for language models. Tensorized representations have been explored in NLP for large embedding tables and Transformer components, including T-parameterized embeddings, tensorized Transformer layers, and tensor-compressed models trained or distilled directly [10, 19, 31]. Recent work has moved toward post-training tensorization of pretrained LLMs: TensorGPT studies training-free T/MPS compression of token embeddings [30]; CompactifAI uses quantum-inspired tensor networks, including MPO/T-matrix-style decompositions, to compress attention and MLP operators with recovery [24, 5, 1]; TensorLLM tensorizes multi-head attention with Tucker-style structure [8]; and Minima presents a production-oriented pipeline combining sensitivity prediction, Tucker/T/TR decompositions, healing, and custom kernels [15]. These works establish tensorization as an active LLM-compression direction. AWT is orthogonal: rather than proposing a new topology or recovery procedure, it introduces a calibration-time activation-aware preconditioner before standard T/TTN decomposition. Function-aware compression objectives. A central limitation of weight-space compression is that Frobenius reconstruction error can be misaligned with the functional error induced inside a pretrained model. Fisher-weighted and distribution-aware decompositions address this by weighting the objective with curvature, Fisher information, or input covariance statistics [11, 13]. These methods more directly target output distortion, but often require modified optimization, dense preconditioners, or solver-specific updates. AWT instead uses activation statistics only to construct a diagonal equivalent reparameterization, leaving the tensor-network solver unchanged. Activation-aware post-training compression. Activation-aware scaling has been especially successful in post-training quantization. SmoothQuant applies an equivalent rescaling that transfers quantization difficulty from activation outliers into weights [29], while AWQ uses activation statistics to identify salient channels for low-bit weight-only quantization [18]. Related outlier-suppression methods also use equivalent transformations and clipping to reduce structured activation outliers [28]. AWT brings this activation-aware perspective to tensor-network decomposition, where the approximation budget is not a bit-width but a fixed tensorization shape, topology, and rank budget. 3 Background Tensorizing linear operators. We consider a linear layer y=Wxy=Wx with W∈ℝm×nW ^m× n. Tensor-network compression begins by factorizing the input and output dimensions as m=∏k=1Kmkm= _k=1^Km_k and n=∏k=1Knkn= _k=1^Kn_k, and reshaping W into an order-2K2K tensor ∈ℝm1×⋯×mK×n1×⋯×nKW ^m_1×·s× m_K× n_1×·s× n_K. The indices associated with mkm_k correspond to output modes, while those associated with nkn_k correspond to input modes. A tensor-network approximation then replaces W by a contraction of smaller core tensors whose internal bond dimensions control the parameter budget. Tensor Train and Tree Tensor Network decompositions. The Tensor Train (T) format, equivalently a Matrix Product Operator (MPO) for linear maps, represents W as a chain of cores (k)k=1K\G^(k)\_k=1^K, where (k)∈ℝrk−1×mk×nk×rkG^(k) ^r_k-1× m_k× n_k× r_k and r0=rK=1r_0=r_K=1 [21, 20]. Its entries are given by i1,…,iK,j1,…,jK=∑α1,…,αK−1∏k=1Kαk−1,ik,jk,αk(k).W_i_1,…,i_K,j_1,…,j_K= _ _1,…, _K-1 _k=1^KG^(k)_ _k-1,i_k,j_k, _k. (1) The T ranks rk\r_k\ are the internal bond dimensions and determine the compression–expressivity trade-off. In practice, T decompositions are commonly computed by successive truncated SVDs of tensor unfoldings. Tree Tensor Networks (TTNs) generalize this chain topology to a loop-free hierarchical graph over tensor modes [23, 7]. The leaves correspond to the physical modes of W, while internal nodes store latent cores that merge groups of modes. Each internal edge e carries a bond dimension χe _e, often bounded by a maximum rank χ. Compared with T, a TTN allocates rank across a hierarchy rather than along a single ordering, yielding a different inductive bias and parameter allocation for the same tensorized operator. Activation-conditioned reconstruction. Standard post-training T/TTN decompositions approximate W by minimizing a weight-space objective such as ‖W−W~‖F2\|W- W\|_F^2. However, replacing a linear layer inside a pretrained model induces output error that depends on the layer input distribution. For y=Wxy=Wx and approximation W W, ‖(W−W^)x‖22=‖(W−W^)Σx1/2‖F2,Σx=[xx⊤].E\|(W- W)x\|_2^2=\|(W- W) _x^1/2\|_F^2, _x=E[x ]. (2) Thus, directions that are frequently or strongly activated should contribute more to the reconstruction objective. Full covariance-weighted decompositions can directly target this metric, but they generally require modified solvers or dense preconditioners [11, 13]. AWT instead uses an equivalent diagonal reparameterization. For any invertible diagonal matrix D, Wx=(WD)(D−1x),Wx=(WD)(D^-1x), (3) so one can rescale the columns of W before decomposition and compensate by the inverse scaling at inference time. This is the same functional-preserving principle used by activation-aware quantization methods such as SmoothQuant and AWQ [29, 18], but applied here to tensor-network decomposition rather than bit-width allocation. 4 Activation-aware Weight Tensorization (AWT) AWT is a calibration-time preconditioning wrapper for tensor-network compression of linear layers. Given a linear map y=Wxy=Wx and a tensor-network backend (T/TTN) with a fixed tensorization layout, topology, and rank/parameter budget, we compute a diagonal importance scaling DαD_α from cached input activations and use it to bias the decomposition toward input channels that matter more under typical inputs. Concretely, AWT (i) right-preconditions the weight matrix by scaling its input-channel columns, (i) tensorizes and decomposes the scaled operator using a standard Frobenius-norm T/TTN routine, and (i) deploys the compressed layer without materializing the dense unscaled matrix: the inverse diagonal scaling is applied to the layer input, followed by the T/TTN operator (Equations 9 and 1). As discussed in the background, the expected functional distortion of a compressed linear layer induces a covariance-weighted metric ‖(W−W^)Σ1/2‖F\|(W- W) ^1/2\|_F. Directly optimizing this full-metric objective within T/TTN would require either a solver that handles a non-isotropic least-squares metric or an explicit dense preconditioner. The latter globally mixes input channels, can be poorly aligned with a fixed tensorization shape and rank budget, and would require a dense transform at deployment. AWT instead uses a diagonal preconditioner. This restriction is deliberate: it preserves the original channel layout seen by the tensorization, requires only single-pass gradient-free calibration statistics, and leaves the T/TTN solver and contraction path unchanged. Thus, AWT trades covariance optimality for modularity, statistical robustness, and compatibility with off-the-shelf tensor-network decomposition. Key identity: diagonal preconditioning yields a weighted Frobenius proxy. Let D≻0D 0 be diagonal, define the preconditioned matrix W′=WDW =WD, and let W~′ W be any approximation to W′W returned by an off-the-shelf Frobenius-norm T/TTN routine after tensorization. Define the effective deployed operator as W^=W~′D−1 W= W D^-1. Then (W−W^)D=WD−W~′,‖(W−W^)D‖F=‖WD−W~′‖F.(W- W)D=WD- W , 28.45274pt\|(W- W)D\|_F=\|WD- W \|_F. (4) Equivalently, AWT minimizes a column-weighted reconstruction proxy, ‖(W−W^)D‖F2 \|(W- W)D\|_F^2 =∑i=1ndi2∥W:i−W^:i∥22, = _i=1^nd_i^2\,\|W_:i- W_:i\|_2^2, (5) so columns associated with larger did_i are fit more accurately. This is the precise sense in which AWT injects activation information while reusing an off-the-shelf T/TTN Frobenius solver. Diagonal scales from activation importance. Given calibration activations (t)t=1T\x^(t)\_t=1^T for a layer, we compute per-channel importance scores impii=1n\imp_i\_i=1^n and normalize them by their mean imp¯=1n∑i=1nimpi. imp= 1n _i=1^nimp_i. For numerical stability, we clamp the mean as imp¯ε=max(imp¯,ε) imp_ = ( imp, ) and define scales sis_i via si=max(smin,(impiimp¯ε)α),smin=max(min_scale,ε).s_i= \! (s_ , ( imp_i imp_ )^α ), 28.45274pts_ = ( min\_scale, ). (6) where α>0α>0 controls the strength of reweighting, ε is a small numerical constant, and smins_ prevents large factors in Dα−1D_α^-1. We set Dα=diag(s1,…,sn)D_α=diag(s_1,…,s_n). In our experiments, we instantiate impiimp_i using either MeanAbs, impiMeanAbs=t[|xi(t)|]imp_i^MeanAbs=E_t[|x_i^(t)|], or Std, impiStd=Vart(xi(t))+εimp_i^Std= Var_t(x_i^(t))+ , with statistics taken over cached calibration activations. We use MeanAbs and Std because they are single-pass, gradient-free statistics that can be estimated from cached calibration activations, preserving the training-free character of AWT. Gradient- or Fisher-based saliency measures are possible, but would require backpropagation and would change the computational profile of the method. The diagonal restriction should not be interpreted as assuming diagonal activation covariance; rather, it preserves the original channel layout and keeps the preconditioner statistically and computationally lightweight. Precondition–tensorize–deploy (without materializing W W). For input-channel scaling (right preconditioning), we scale the columns of W and tensorize the scaled operator: W′=WDα,W =WD_α, (7) apply a tensor-network decomposition to obtain W~′≈W′ W ≈ W , and conceptually define W^=W~′Dα−1. W= W D_α^-1. (8) In practice, to avoid materializing W W (which would be dense), we implement the layer as the equivalent computation y^=W^x=(W~′Dα−1)x=W~′(Dα−1x), y= Wx=( W D_α^-1)x= W (D_α^-1x), (9) i.e., we apply the inverse diagonal scaling to the layer input and then apply the T/TTN-compressed operator W~′ W . Since DαD_α is diagonal, this adds only an elementwise per-channel multiply and can be fused with surrounding kernels. This preserves the T/TTN parameter budget while remaining functionally equivalent to Eq. (8). The algorithm is intentionally modular: any T/TTN decomposition implementation can be used in the decomposition step, and the only data-aware component is the diagonal scaling computed from a small calibration set. Algorithm 1 Activation-aware Weight Tensorization (AWT) 0: Weight matrix W∈ℝm×nW ^m× n, calibration inputs C, importance statistic imp(⋅)imp(·) (MeanAbs or Std), exponent α>0α>0, numerical constant ε>0 >0, minimum scale min_scale>0>0, tensorization shape, tensor-network routine Decomp(⋅) Decomp(·) (T/TTN) with target ranks/budget. 0: Compressed operator implemented as (W~′,Dα−1)( W ,D_α^-1). 1: Run the model on C and collect layer input activations (t)t=1T\x^(t)\_t=1^T. 2: Compute per-channel importances impi←imp(xi(t)t=1T)imp_i (\x_i^(t)\_t=1^T) for i=1,…,ni=1,…,n. 3: Compute mean importance imp¯←1n∑i=1nimpi imp← 1n _i=1^nimp_i and clamp imp¯ε←max(imp¯,ε) imp_ ← ( imp, ). 4: Set smin←max(min_scale,ε)s_ ← ( min\_scale, ) and form scales si←max(smin,(impiimp¯ε)α)s_i← \! (s_ , ( imp_i imp_ )^α ). 5: Form Dα←diag(s1,…,sn)D_α (s_1,…,s_n). 6: Precondition (right scaling): W′←WDαW ← WD_α. 7: Tensorize/reshape W′W into ′W using the specified shape. 8: Decompose: ~′←Decomp(′) W ← Decomp(W ) (subject to target ranks/budget). 9: Untensorize ~′ W into a (compressed) operator W~′ W . 10: Deploy without materializing W W: implement y^=W~′(Dα−1x) y= W (D_α^-1x). 11: return (W~′,Dα−1)( W ,D_α^-1). 5 Experimental Validation Models and scope. We evaluate AWT on Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, considering the seven dense Transformer operators Wq,Wk,Wv,Wo,Wup,Wgate,Wdown\W_q,W_k,W_v,W_o,W_up,W_gate,W_down\ at 2/4/6×2/4/6× compression. Unless stated otherwise, vanilla and AWT use the same tensorization layout, topology, rank budget, calibration set, and T/TTN backend; only the diagonal preconditioning differs. Our experiments are organized around four questions: (i) whether activation-aware preconditioning improves the local functional fidelity of a fixed T/TTN approximation; (i) whether these local improvements translate into model-level behavior under single-operator replacement; (i) how the gains behave when multiple compressed operators are composed; and (iv) how the diagonal AWT proxy compares with alternative baselines, including matched-compression SVD and dense full-covariance preconditioning. Calibration data and activation caches. AWT estimates per-channel activation statistics from a small unlabeled calibration set. We draw calibration prompts from a subset of lm-eval tasks (MMLU, HellaSwag, and BoolQ), run the original dense model, and record the input activations to each targeted linear operator using forward hooks. Only the input vectors immediately before multiplication by W are cached; labels and task losses are never used. Calibration examples are kept disjoint from all examples used for metric caches and downstream evaluation. For layer-level evaluation, we construct a separate held-out metric cache of input activations X. This cache is used only to measure local output error and to select among candidate tensorized configurations at a fixed parameter budget. Separating calibration and metric caches avoids selecting tensorized operators on the same activations used to construct the AWT scales. 5.1 Compared methods Tensorization backends. We compare Tensor Train (T) and two Tree Tensor Network variants (TTN4, TTN8), which differ in tree depth/fanout and therefore impose different rank-allocation biases. For a given matrix and compression budget, vanilla tensorization and AWT use the same tensorization shape, mode ordering, topology, and rank budget. This fixed-layout protocol isolates the effect of activation-aware preconditioning rather than jointly optimizing the tensorization search space. Activation-aware preconditioning. For each T/TTN backend, we evaluate Vanilla tensorization with no preconditioning, AWT-MeanAbs with channel scales from [|xi|]E[|x_i|], and AWT-Std with channel scales from the per-channel standard deviation. All AWT variants use right preconditioning (input-channel scaling) and deploy the compressed operator as W~′(Dα−1x) W (D_α^-1x). We sweep α∈0.5,1.0,1.5α∈\0.5,1.0,1.5\ and use α=1.0α=1.0 as the default unless otherwise specified. Baselines and oracles. We include matched-compression truncated SVD as a standard matrix low-rank baseline under the same single-operator protocol. We also compare diagonal AWT with a dense full-covariance preconditioner in an offline operator-level analysis. The full-covariance variant is used as an oracle-style comparison for the richer covariance-weighted objective, while diagonal AWT is the practical plug-in method used in deployment. Compression matching. To compare methods at equal compression, we sweep T/TTN rank configurations and select, for each target matrix, candidates satisfying the specified compression ratio. Unless otherwise stated, parameter counts include the tensor-network cores; the additional AWT diagonal scale is one input-channel vector and is negligible relative to the compressed operator, but we report it explicitly in the implementation details. Candidates are ranked using the held-out layer-level functional fidelity metric defined below. Layer-level metrics. Let X∈ℝB×nX ^B× n be a held-out activation cache for a layer with weight matrix W∈ℝm×nW ^m× n, and let Y=XW⊤Y=XW . For AWT, predictions are computed without materializing W W as Y^=(XDα−1)(W~′)⊤ Y=(XD_α^-1)( W ) , where W~′ W approximates W′=WDαW =WD_α. Our primary local metric is activation-conditioned relative output error, RelFrobout=‖Y^−Y‖F‖Y‖F+ε,ε=10−12.RelFrob_out= \| Y-Y\|_F\|Y\|_F+ , =10^-12. (10) For comparison, we also report weight-space reconstruction error RelFrobw=‖W−W^‖F/(‖W‖F+ε)RelFrob_w=\|W- W\|_F/(\|W\|_F+ ). For the full-covariance oracle analysis only, we use RelFrobΣ=‖(W−W^)ΣX1/2‖F/(‖WΣX1/2‖F+ε)RelFrob_ =\|(W- W) _X^1/2\|_F/(\|W _X^1/2\|_F+ ). 5.2 Evaluation protocols Local layer-wise sweep. For each targeted matrix instance, backend, compression ratio, and AWT setting, we sweep admissible T/TTN rank configurations and compute local metrics on the held-out metric cache. This sweep is performed on activations from the original dense model and is independent of end-to-end benchmark scores. At each parameter budget, we retain the candidate with the lowest RelFroboutRelFrob_out unless otherwise stated. This protocol gives a controlled comparison between vanilla tensorization and AWT under the same tensor-network hypothesis class. Single-operator replacement. To test whether local improvements translate into model-level behavior, we replace one dense operator at a time with its selected tensorized approximation and evaluate the resulting model. Results are aggregated over blocks, operator types, and backends at each compression ratio. This protocol isolates the effect of compressing a single matrix while preserving the surrounding model. Multi-operator suffix replacement. Single-operator replacement is a controlled diagnostic, but deployed compression typically composes errors from multiple approximated operators. We therefore also evaluate suffix replacement on Llama 3.1 8B: for the last k Transformer blocks, we jointly replace either the attention group Wq,Wk,Wv,Wo\W_q,W_k,W_v,W_o\ or all seven dense operators. This evaluates whether the AWT-vs-vanilla advantage persists when tensorization errors accumulate through multiple compressed operators. End-to-end metrics. We report WikiText perplexity for both single-operator and multi-operator replacement. We additionally evaluate downstream accuracy on HellaSwag and ARC-Challenge as external task-level checks. These downstream results are used as corroborating evidence rather than as the primary selection criterion; no candidate is selected using downstream task accuracy. Runtime and deployment overhead. AWT performs calibration and scale construction offline. At inference time, the only additional operation relative to the underlying tensorized layer is the input-side elementwise multiplication Dα−1xD_α^-1x before the T/TTN contraction. The tensorized operator and contraction path are unchanged, and the scaling can be fused into surrounding kernels. We therefore separate AWT’s preconditioning overhead from the latency properties of a particular optimized T/TTN backend. 6 Results Across local, single-operator, and multi-operator evaluations, AWT consistently improves vanilla T/TTN tensorization at matched parameter budgets. The local effect appears in activation-conditioned output error, transfers to WikiText perplexity across three model families, and becomes more pronounced when several operators are compressed jointly. We also find that truncated SVD remains a strong matrix low-rank baseline at matched parameter count, while dense full-covariance preconditioning is not a uniformly better practical substitute for the diagonal AWT proxy. Local fidelity and metric alignment AWT first improves the quantity it is designed to affect: held-out activation-conditioned output fidelity. At matched T/TTN parameter budgets, moderate activation reweighting reduces RelFroboutRelFrob_out relative to vanilla tensorization (Table 1). The optimum for functional fidelity differs from the optimum for weight-space reconstruction: increasing α continues to reduce RelFrobwRelFrob_w, but the best RelFroboutRelFrob_out occurs around α=1α=1. This separation supports the view that preserving the action of a compressed layer on real activations is not equivalent to minimizing plain Frobenius error in weight space. The same pattern appears at the model level. Across single-operator replacement trials, changes in activation-conditioned output error are more predictive of changes in perplexity than changes in weight-space error (Table 1, right). We therefore use held-out output fidelity as the candidate-selection metric for end-to-end replacement. Table 1: Local functional fidelity and metric alignment. Left: effect of the AWT exponent α at 4×4× compression on Llama 3.1 8B. AWT reduces activation-conditioned output error, while weight-space error follows a different trend. Right: Spearman correlation between changes in local metrics and changes in WikiText perplexity under single-operator replacement (α=1.0α=1.0), grouped by operator family. Groups: QK=Wq,Wk=\W_q,W_k\, VO=Wv,Wo=\W_v,W_o\, MLP=Wup,Wdown,Wgate=\W_up,W_down,W_gate\. Higher correlation means the local metric better predicts model-level degradation. AWT exponent ablation at 4×4× Out. Weight α Stat. Val. Δ Val. Δ 0.5 Std 0.606 -9.2 0.754 -1.7 0.5 MeanAbs 0.594 -10.9 0.751 -2.0 1.0 Std 0.583 -12.6 0.691 -9.9 1.0 MeanAbs 0.577 -13.5 0.665 -13.2 1.5 Std 0.589 -11.6 0.526 -31.4 1.5 MeanAbs 0.592 -11.3 0.455 -40.6 Δ values are percentages. Metric–PPL rank correlation Model Metric Overall QK VO MLP Llama 3.1 8B Δ 0.427 0.680 0.219 0.373 Δ 0.246 -0.207 -0.290 0.265 Ministral 8B Δ 0.395 0.794 0.245 0.235 Δ 0.042 -0.299 0.438 -0.072 Qwen2.5 7B Δ 0.327 0.401 0.458 0.165 Δ -0.049 -0.213 -0.260 0.164 QK =Wq,Wk=\W_q,W_k\, VO =Wv,Wo=\W_v,W_o\, MLP =Wup,Wdown,Wgate=\W_up,W_down,W_gate\. Δ denotes ΔRelFrobout _out and Δ denotes ΔRelFrobw _w. Single-operator replacement across model families We next replace one dense operator at a time with its selected tensorized approximation and evaluate WikiText perplexity. This protocol isolates the effect of compressing one operator while preserving the rest of the model. As shown in Table 2, AWT improves over vanilla tensorization at every tested compression ratio across all three model families. The absolute perplexity changes are small in this controlled setting, but the direction is consistent and the fraction of the dense-to-vanilla gap recovered remains substantial across architectures. Table 2: Single-operator and multi-operator WikiText evaluation. Left: single-operator replacement across model families; lower PPL is better. “Gap closed” is computed relative to the dense baseline of the corresponding model. Right: multi-operator suffix replacement on Llama 3.1 8B; entries are Δ == PPLAWT−_AWT-PPLvanilla, so negative is better. Single-operator replacement Model Comp. Dense Vanilla AWT Gap closed Llama 3.1 8B 2×2× 8.64 8.79 8.76 20.4% 4×4× 8.64 8.90 8.85 17.1% 6×6× 8.64 8.97 8.93 12.2% Ministral 8B 2×2× 8.66 8.74 8.71 35.3% 4×4× 8.66 8.81 8.76 31.3% 6×6× 8.66 8.86 8.80 28.6% Qwen2.5 7B 2×2× 9.59 9.76 9.70 33.5% 4×4× 9.59 9.85 9.79 25.8% 6×6× 9.59 9.91 9.84 22.2% Multi-operator suffix replacement Scope k 2×2× 4×4× 6×6× Attention 1 -0.41 -0.47 -0.46 2 -0.59 -0.72 -0.81 4 -0.76 -1.02 -1.11 8 -1.40 -2.48 -2.75 All seven 1 -1.86 -2.61 -3.62 2 -7.74 -9.21 -11.71 Composition: multi-operator suffix replacement Single-operator replacement tests isolated substitutions; practical compression composes several approximations. We therefore evaluate suffix replacement on Llama 3.1 8B, jointly replacing either the attention group Wq,Wk,Wv,Wo\W_q,W_k,W_v,W_o\ or all seven dense operators in the last k Transformer blocks. Table 2 reports ΔPPL=PPLAWT−PPLvanilla =PPL_AWT-PPL_vanilla, so negative values favor AWT. AWT improves over vanilla tensorization in every tested suffix setting. The absolute gain grows as more operators are replaced, indicating that activation-aware preconditioning helps control the accumulation of tensorization error through the model. 6.1 Baselines, downstream tasks, and deployment overhead Matched-compression SVD. We include truncated SVD as a strong matrix low-rank reference under the same single-operator protocol. as detailed in Appendix Table A.11., A matched-compression SVD baseline is stronger than the T/TTN pipeline under the single-operator protocol, with WikiText PPL 8.71/8.81/8.878.71/8.81/8.87 at 2/4/6×2/4/6×, compared to 8.76/8.85/8.938.76/8.85/8.93 for AWT and 8.79/8.90/8.978.79/8.90/8.97 for vanilla T/TTN. This clarifies the role of AWT: it improves a fixed tensor-network compression family rather than claiming that T/TTN universally dominates matrix low-rank compression. Downstream task validation. As an additional task-level check, we evaluate HellaSwag and ARC-Challenge under single-operator replacement on Llama 3.1 8B. AWT improves over vanilla tensorization at all tested compression ratios on both tasks. The absolute gains are small, as expected in the controlled single-operator setting: HellaSwag improves by 0.00030.0003–0.00080.0008 accuracy points and ARC-Challenge improves by 0.00090.0009–0.00170.0017. We report the full backend-specific downstream breakdowns in the appendix. Deployment overhead. AWT performs calibration, importance estimation, and scale construction offline. At inference time, the only extra operation relative to the underlying tensorized layer is the elementwise input scaling Dα−1xD_α^-1x before the T/TTN contraction. The tensorized operator and contraction path are unchanged, and the scaling can typically be fused into surrounding kernels. We therefore treat optimized T/TTN latency as a property of the tensorization backend, with AWT adding only a lightweight input-side rescaling. Diagonal-design analysis. The diagonal preconditioner should not be interpreted as a diagonal-covariance assumption. Activations are strongly non-diagonal: the diagonal covariance mass is only 0.032/0.142/0.2120.032/0.142/0.212 for Llama/Ministral/Qwen, while the off-diagonal Frobenius ratio is 0.984/0.926/0.8850.984/0.926/0.885. A dense full-covariance oracle wins its own covariance-weighted objective in 80/8180/81 cases, but diagonal AWT gives lower held-out output error in 53/8153/81 cases. This supports diagonal AWT as a robustness–modularity tradeoff; full details are reported in Table A.12. 7 Discussion and Conclusion Our results show that activation-aware preconditioning improves post-training tensor-network compression of LLM linear operators. Across local fidelity, single-operator replacement, and multi-operator suffix replacement, AWT consistently improves vanilla T/TTN tensorization at matched parameter budgets across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B. The gains become especially pronounced when multiple operators are replaced jointly, suggesting that activation-aware preconditioning helps control accumulated tensorization error. Functional fidelity versus weight reconstruction. Weight-space reconstruction error is not the right proxy for the behavior of a compressed layer inside an LLM. The activation-conditioned output metric is more aligned with perplexity changes than plain Frobenius weight error, especially for attention projections (Table 1). This supports the design of AWT: the preconditioner does not change the tensor-network hypothesis class, but changes how a fixed rank budget is allocated by the solver, biasing reconstruction toward input channels that matter under real activations. It also explains why the best α for functional fidelity need not minimize weight-space error. Topology- and operator-dependent tensorizability. Tensorizability is not uniform across operators or topologies. TTN backends often outperform T at the same parameter budget, with the clearest gap on attention projections, particularly WqW_q and WkW_k. A plausible explanation is structural alignment: multi-head attention induces head-wise and block-structured subspaces, and a hierarchical tree can merge correlated groups earlier than a one-dimensional T chain [26, 23, 7, 9]. This remains an empirical hypothesis, since performance depends on tensorization shape, mode ordering, rank allocation, and TTN topology. MLP matrices are generally harder to tensorize at matched compression, consistent with the view that feed-forward blocks implement richer feature mixing and content storage [6]. These differences suggest that practical tensorization pipelines should allocate ranks and choose topologies in an operator-aware manner. Relationship to SVD and other compression families. The matched-compression SVD baseline highlights that standard matrix low-rank approximation remains strong at matched parameter count. Tensor networks offer a different structured compression family, with multi-core representations, explicit internal rank budgets, and topology choices that may be useful for deployment and for modeling structured correlations after tensorization. AWT should therefore be viewed as improving this tensor-network family rather than replacing other compression primitives. It is also complementary to quantization, pruning, and matrix factorization: one could apply AWT before tensorizing weights and then further quantize the resulting tensor-network cores. Diagonal preconditioning as a robustness–modularity tradeoff. The diagonal restriction in AWT should not be interpreted as assuming diagonal activation covariance. Our covariance analysis shows the opposite: activations are strongly non-diagonal. Dense full-covariance preconditioning optimizes its intended covariance-weighted objective more directly, but does not reliably improve held-out local output fidelity. Diagonal AWT gives up optimality on that objective, but preserves the original channel layout, avoids dense deployment transforms, reduces the statistical estimation burden, and leaves the T/TTN solver and contraction path unchanged. Limitations and future work. Several limitations remain. First, tensorization quality depends on design choices that we keep fixed or only partially sweep, including dimension factorization, mode ordering, rank allocation, and TTN topology. Jointly optimizing these choices with activation-aware preconditioning is a natural next step. Second, our runtime analysis isolates the overhead of AWT itself, an input-side diagonal scaling, but end-to-end latency depends on optimized T/TTN kernels and hardware-specific implementations. Third, calibration statistics are estimated from a small input set and may not capture all downstream distributions. Finally, while our multi-operator suffix experiments move beyond isolated substitutions, full-model compression requires coordinated choices across layers and operators. In summary, AWT is a training-free, solver-agnostic mechanism for making post-training tensor-network compression more activation-aware. By inserting a diagonal equivalent reparameterization before standard T/TTN decomposition, it improves functional fidelity without retraining, custom solvers, or changes to the tensorized operator, especially when approximation errors accumulate across multiple compressed operators. References [1] A. Alvarez, A. Genuardi, N. Sinha, A. Tiene, M. Okyay, B. Ryskulov, D. Montero, S. Mugel, and R. Orús (2025) Scaling laws for energy efficiency of local llms. arXiv preprint arXiv:2512.16531. Cited by: §1, §2. [2] T. B. Brown et al. (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, Cited by: §1. [3] A. Chowdhery et al. (2023) PaLM: scaling language modeling with pathways. Journal of Machine Learning Research 24 (240), p. 1–113. Cited by: §1. [4] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus (2014) Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, Cited by: §2. [5] D. Fovet, S. Chamoli, S. Oury, and S. Singhal (2025) Accuracy and consumption analysis from a compressed model by compactifai from multiverse computing. arXiv preprint arXiv:2507.08836. Cited by: §1, §2. [6] M. Geva, R. Schuster, J. Berant, and O. Levy (2021) Transformer feed-forward layers are key-value memories. In Empirical Methods in Natural Language Processing, Cited by: §7. [7] L. Grasedyck (2010) Hierarchical singular value decomposition of tensors. SIAM Journal on Matrix Analysis and Applications. Cited by: §1, §3, §7. [8] Y. Gu, W. Zhou, G. Iacovides, and D. Mandic (2025) TensorLLM: tensorising multi-head attention for enhanced reasoning and compression in llms. In 2025 International Joint Conference on Neural Networks (IJCNN), p. 1–8. Cited by: §1, §2. [9] W. Hackbusch (1999) A sparse matrix arithmetic based on h-matrices. part i: introduction to h-matrices. Computing. Cited by: §7. [10] O. Hrinchuk, V. Khrulkov, L. Mirvakhabova, E. Orlova, and I. Oseledets (2019) Tensorized embedding layers for efficient model compression. In Findings of EMNLP, Cited by: §2. [11] C. Hsu et al. (2022) Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, Cited by: §2, §3. [12] M. Jaderberg, A. Vedaldi, and A. Zisserman (2014) Speeding up convolutional neural networks with low rank expansions. In British Machine Vision Conference, Cited by: §2. [13] C. Kalle et al. (2025) Distribution-aware tensor decomposition for neural network compression. arXiv preprint. Cited by: §2, §3. [14] Y. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin (2016) Compression of deep convolutional neural networks for fast and low power mobile applications. In International Conference on Learning Representations, Cited by: §2. [15] S. Kozyrev and D. Maiboroda (2026) A practical tensor-network compression pipeline for production-scale large language models. arXiv preprint arXiv:2602.01613. Cited by: §1, §2. [16] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky (2015) Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In International Conference on Learning Representations, Cited by: §2. [17] Y. Li et al. (2017) Block-term tensor neural networks. In arXiv preprint, Cited by: §2. [18] J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han (2024) AWQ: activation-aware weight quantization for llm compression and acceleration. In International Conference on Machine Learning, Cited by: §1, §2, §3. [19] X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, M. Zhou, and D. Song (2019) A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems, Cited by: §2. [20] A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov (2015) Tensorizing neural networks. In Advances in Neural Information Processing Systems, Cited by: §1, §2, §3. [21] I. V. Oseledets (2011) Tensor-train decomposition. SIAM Journal on Scientific Computing 33 (5), p. 2295–2317. Cited by: §1, §2, §3. [22] R. Pope et al. (2023) Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems, Cited by: §1. [23] Y.-Y. Shi, L.-M. Duan, and G. Vidal (2006) Classical simulation of infinite-size quantum lattice systems in the tree tensor network approach. Physical Review A 74 (2), p. 022320. Cited by: §1, §3, §7. [24] A. Tomut, S. S. Jahromi, A. Sarkar, U. Kurt, S. Singh, F. Ishtiaq, C. Muñoz, P. S. Bajaj, A. Elborady, G. del Bimbo, et al. (2024) Compactifai: extreme compression of large language models using quantum-inspired tensor networks. arXiv preprint arXiv:2401.14109, p. 12. Cited by: §1, §2. [25] H. Touvron et al. (2023) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. [26] A. Vaswani et al. (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Cited by: §7. [27] W. Wang, Y. Sun, B. Eriksson, W. Wang, and V. Aggarwal (2018) Wide compression: tensor ring nets. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §2. [28] X. Wei, R. Gong, Y. Li, X. Liu, and F. Yu (2022) Outlier suppression: pushing the limit of low-bit transformer language models. In Advances in Neural Information Processing Systems, Cited by: §2. [29] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han (2023) SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, Cited by: §1, §2, §3. [30] M. Xu et al. (2023) TensorGPT: efficient compression of gpt models via tensor-train decomposition. arXiv preprint. Cited by: §2. [31] Z. Yang et al. (2023) Quantization-aware training for tensor-compressed transformers. arXiv preprint. Cited by: §2. [32] Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki (2016) Tensor ring decomposition. In arXiv preprint arXiv:1606.05535, Cited by: §2. [33] X. Zhu et al. (2023) A survey on model compression for large language models. arXiv preprint. Cited by: §1. Appendix A Additional Experimental Results This appendix provides additional experimental details and breakdowns omitted from the main text for space: downstream task results, deployment overhead, covariance analysis, cross-model ablations, backend comparisons, and multi-operator replacement tables. A.1 Llama per-operator output-error breakdowns Table A.1: Per-operator activation-conditioned output error (relative Frobenius; lower is better) under TTN depth4 tensorization with α=1α=1. Entries are reported as mean μ and standard deviation σ over N=32N=32 single-operator trials per matrix type. 2× compression 4× compression 6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.180±0.0470.180± 0.047 0.147±0.0410.147± 0.041 0.147±0.0410.147± 0.041 0.245±0.0560.245± 0.056 0.205±0.0520.205± 0.052 0.205±0.0520.205± 0.052 0.279±0.0600.279± 0.060 0.236±0.0570.236± 0.057 0.236±0.0570.236± 0.057 32 WkW_k 0.192±0.0490.192± 0.049 0.147±0.0370.147± 0.037 0.147±0.0370.147± 0.037 0.304±0.0670.304± 0.067 0.230±0.0500.230± 0.050 0.230±0.0500.230± 0.050 0.358±0.0700.358± 0.070 0.274±0.0540.274± 0.054 0.274±0.0530.274± 0.053 32 WvW_v 0.595±0.0650.595± 0.065 0.470±0.0480.470± 0.048 0.467±0.0470.467± 0.047 0.749±0.0610.749± 0.061 0.623±0.0500.623± 0.050 0.621±0.0490.621± 0.049 0.811±0.0540.811± 0.054 0.692±0.0450.692± 0.045 0.689±0.0450.689± 0.045 32 WoW_o 0.526±0.0870.526± 0.087 0.387±0.0890.387± 0.089 0.389±0.0850.389± 0.085 0.665±0.0980.665± 0.098 0.525±0.1070.525± 0.107 0.526±0.1040.526± 0.104 0.722±0.1000.722± 0.100 0.591±0.1130.591± 0.113 0.591±0.1100.591± 0.110 32 WupW_up 0.597±0.0780.597± 0.078 0.559±0.0750.559± 0.075 0.561±0.0760.561± 0.076 0.646±0.0840.646± 0.084 0.611±0.0810.611± 0.081 0.613±0.0820.613± 0.082 0.698±0.0910.698± 0.091 0.666±0.0870.666± 0.087 0.668±0.0880.668± 0.088 32 WdownW_down 0.677±0.0940.677± 0.094 0.535±0.1570.535± 0.157 0.547±0.1580.547± 0.158 0.730±0.0980.730± 0.098 0.587±0.1710.587± 0.171 0.600±0.1720.600± 0.172 0.784±0.1010.784± 0.101 0.644±0.1840.644± 0.184 0.656±0.1860.656± 0.186 32 WgateW_gate 0.396±0.0610.396± 0.061 0.374±0.0580.374± 0.058 0.376±0.0590.376± 0.059 0.432±0.0670.432± 0.067 0.411±0.0630.411± 0.063 0.413±0.0640.413± 0.064 0.470±0.0730.470± 0.073 0.452±0.0690.452± 0.069 0.454±0.0700.454± 0.070 32 Table A.2: Per-operator activation-conditioned output error (relative Frobenius; lower is better) under TTN depth8 tensorization with α=1α=1. Entries are reported as mean μ and standard deviation σ over N=32N=32 single-operator trials per matrix type. 2× compression 4× compression 6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.182±0.0470.182± 0.047 0.148±0.0410.148± 0.041 0.148±0.0410.148± 0.041 0.247±0.0570.247± 0.057 0.207±0.0520.207± 0.052 0.207±0.0520.207± 0.052 0.282±0.0600.282± 0.060 0.239±0.0570.239± 0.057 0.239±0.0570.239± 0.057 32 WkW_k 0.197±0.0500.197± 0.050 0.151±0.0380.151± 0.038 0.151±0.0380.151± 0.038 0.305±0.0660.305± 0.066 0.233±0.0510.233± 0.051 0.232±0.0500.232± 0.050 0.357±0.0690.357± 0.069 0.275±0.0530.275± 0.053 0.275±0.0530.275± 0.053 32 WvW_v 0.603±0.0650.603± 0.065 0.478±0.0480.478± 0.048 0.475±0.0470.475± 0.047 0.770±0.0650.770± 0.065 0.627±0.0480.627± 0.048 0.624±0.0480.624± 0.048 0.833±0.0610.833± 0.061 0.691±0.0450.691± 0.045 0.688±0.0440.688± 0.044 32 WoW_o 0.530±0.0880.530± 0.088 0.390±0.0900.390± 0.090 0.393±0.0860.393± 0.086 0.670±0.0980.670± 0.098 0.530±0.1080.530± 0.108 0.531±0.1050.531± 0.105 0.727±0.1000.727± 0.100 0.597±0.1130.597± 0.113 0.597±0.1110.597± 0.111 32 WupW_up 0.597±0.0780.597± 0.078 0.559±0.0750.559± 0.075 0.561±0.0760.561± 0.076 0.649±0.0850.649± 0.085 0.613±0.0820.613± 0.082 0.615±0.0820.615± 0.082 0.700±0.0910.700± 0.091 0.668±0.0880.668± 0.088 0.670±0.0880.670± 0.088 32 WdownW_down 0.677±0.0940.677± 0.094 0.535±0.1570.535± 0.157 0.547±0.1570.547± 0.157 0.733±0.0990.733± 0.099 0.590±0.1710.590± 0.171 0.603±0.1720.603± 0.172 0.786±0.1020.786± 0.102 0.646±0.1850.646± 0.185 0.658±0.1850.658± 0.185 32 WgateW_gate 0.396±0.0610.396± 0.061 0.374±0.0580.374± 0.058 0.376±0.0590.376± 0.059 0.433±0.0670.433± 0.067 0.413±0.0630.413± 0.063 0.415±0.0640.415± 0.064 0.471±0.0730.471± 0.073 0.453±0.0690.453± 0.069 0.456±0.0700.456± 0.070 32 Table A.3: Per-operator activation-conditioned output error (relative Frobenius; lower is better) under T (cores4) tensorization with α=1α=1. Entries are reported as mean μ and standard deviation σ over N=32N=32 single-operator trials per matrix type. 2× compression 4× compression 6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.371±0.0200.371± 0.020 0.301±0.0300.301± 0.030 0.293±0.0320.293± 0.032 0.572±0.0230.572± 0.023 0.474±0.0410.474± 0.041 0.457±0.0420.457± 0.042 0.671±0.0240.671± 0.024 0.568±0.0450.568± 0.045 0.547±0.0470.547± 0.047 32 WkW_k 0.450±0.0240.450± 0.024 0.383±0.0370.383± 0.037 0.366±0.0390.366± 0.039 0.676±0.0200.676± 0.020 0.609±0.0450.609± 0.045 0.584±0.0470.584± 0.047 0.769±0.0160.769± 0.016 0.709±0.0460.709± 0.046 0.685±0.0470.685± 0.047 32 WvW_v 0.643±0.0750.643± 0.075 0.541±0.0240.541± 0.024 0.545±0.0250.545± 0.025 0.817±0.0660.817± 0.066 0.733±0.0260.733± 0.026 0.735±0.0270.735± 0.027 0.876±0.0560.876± 0.056 0.807±0.0250.807± 0.025 0.808±0.0250.808± 0.025 32 WoW_o 0.601±0.0650.601± 0.065 0.473±0.1010.473± 0.101 0.484±0.1090.484± 0.109 0.765±0.0540.765± 0.054 0.651±0.1030.651± 0.103 0.656±0.1190.656± 0.119 0.827±0.0470.827± 0.047 0.728±0.1010.728± 0.101 0.728±0.1180.728± 0.118 32 WupW_up 0.587±0.0370.587± 0.037 0.564±0.0320.564± 0.032 0.567±0.0350.567± 0.035 0.774±0.0280.774± 0.028 0.754±0.0240.754± 0.024 0.753±0.0260.753± 0.026 0.843±0.0210.843± 0.021 0.826±0.0190.826± 0.019 0.823±0.0210.823± 0.021 32 WdownW_down 0.594±0.0460.594± 0.046 0.528±0.1280.528± 0.128 0.542±0.1300.542± 0.130 0.773±0.0420.773± 0.042 0.696±0.1610.696± 0.161 0.705±0.1630.705± 0.163 0.839±0.0370.839± 0.037 0.758±0.1720.758± 0.172 0.765±0.1750.765± 0.175 32 WgateW_gate 0.449±0.0340.449± 0.034 0.429±0.0350.429± 0.035 0.428±0.0370.428± 0.037 0.620±0.0380.620± 0.038 0.595±0.0430.595± 0.043 0.589±0.0450.589± 0.045 0.688±0.0410.688± 0.041 0.663±0.0470.663± 0.047 0.654±0.0490.654± 0.049 32 A.2 Llama topology and alpha ablations Table A.4: T vs. TTN on WikiText at matched compression (parameter budget). Entries are within-backend medians over the same set of single-operator trials (all blocks and matrices), reported separately for each backend (T core4, TTN depth4, TTN depth8). This table is not pooled across backends; the main text reports pooled results. AWT value is the lower median across AWT methods. Comp. Variant PPL (Van.) Δ PPL (AWT) Δ 2× T (core4) 8.80 0.00 8.78 0.00 2× TTN (depth4) 8.73 -0.07 8.72 -0.06 2× TTN (depth8) 8.74 -0.06 8.72 -0.06 4× T (core4) 9.01 0.00 8.98 0.00 4× TTN (depth4) 8.82 -0.19 8.79 -0.19 4× TTN (depth8) 8.83 -0.18 8.79 -0.19 6× T (core4) 9.10 0.00 9.07 0.00 6× TTN (depth4) 8.87 -0.23 8.84 -0.23 6× TTN (depth8) 8.89 -0.21 8.84 -0.23 Table A.5: Effect of the reweighting exponent α at 2×2× compression. AWT consistently reduces activation-conditioned output error (our functional fidelity proxy), while weight-space error exhibits a different optimum (continuing to improve at larger α). Output error (median) Weight error (median) α Method Value Δ vs Van. Value Δ vs Van. 0.5 Std 0.4471 -15.7% 0.5985 -1.0% 0.5 MeanAbs 0.4430 -16.4% 0.5953 -1.6% 1.0 Std 0.4298 -18.9% 0.5500 -9.1% 1.0 MeanAbs 0.4339 -18.2% 0.5309 -12.2% 1.5 Std 0.4385 -17.3% 0.3935 -34.9% 1.5 MeanAbs 0.4466 -15.8% 0.3499 -42.1% Table A.6: Effect of the reweighting exponent α at 6×6× compression. AWT consistently reduces activation-conditioned output error (our functional fidelity proxy), while weight-space error exhibits a different optimum (continuing to improve at larger α). Output error (median) Weight error (median) α Method Value Δ vs Van. Value Δ vs Van. 0.5 Std 0.6734 -9.1% 0.8212 -1.4% 0.5 MeanAbs 0.6659 -10.2% 0.8180 -1.8% 1.0 Std 0.6518 -12.1% 0.7616 -8.5% 1.0 MeanAbs 0.6477 -12.6% 0.7374 -11.4% 1.5 Std 0.6606 -10.9% 0.5973 -28.3% 1.5 MeanAbs 0.6630 -10.5% 0.5169 -37.9% Table A.7: Per-matrix Spearman correlation between Δ _out and Δ (AWT−-Vanilla), α=1.0α=1.0. Matrix N Spearman ρ Wdown 440 0.441 Wgate 448 0.026 Wk 570 0.490 Wo 576 0.284 Wq 569 0.702 Wup 448 0.383 Wv 576 0.281 Table A.8: Per-matrix Spearman correlation between Δ Error and Δ (AWT−-Vanilla), α=1.0α=1.0. Matrix N Spearman ρ Wdown 440 0.486 Wgate 448 0.129 Wk 570 -0.178 Wo 576 0.034 Wq 569 -0.395 Wup 448 0.163 Wv 576 -0.056 A.3 Downstream evaluation on Llama 3.1 8B Table A.9: Downstream single-operator evaluation on Llama 3.1 8B. AWT improves over vanilla tensorization at every tested compression on both HellaSwag and ARC-Challenge. Higher is better. Task Comp. Dense Vanilla AWT Method Gap closed HellaSwag 2×2× 0.7931 0.7891 0.7894 Std 8.8% 4×4× 0.7931 0.7867 0.7873 Std 9.4% 6×6× 0.7931 0.7849 0.7857 Std 10.3% ARC-Challenge 2×2× 0.5512 0.5486 0.5495 Std 33.3% 4×4× 0.5512 0.5461 0.5469 Std 16.7% 6×6× 0.5512 0.5435 0.5452 Std 22.2% Table A.10: Backend-specific downstream results on Llama 3.1 8B under single-operator replacement. Entries are within-backend pooled medians over the canonical single-operator benchmark rows. Δ is measured against T core4 at the same compression when available. Higher is better. Task Comp. Backend Vanilla Δ AWT Δ HellaSwag 2×2× TTN depth4 0.7895 +0.0007 0.7901 +0.0013 HellaSwag 2×2× TTN depth8 0.7895 +0.0007 0.7900 +0.0012 HellaSwag 4×4× T core4 0.7839 +0.0000 0.7850 +0.0000 HellaSwag 4×4× TTN depth4 0.7876 +0.0038 0.7882 +0.0032 HellaSwag 4×4× TTN depth8 0.7875 +0.0037 0.7882 +0.0032 HellaSwag 6×6× T core4 0.7817 +0.0000 0.7826 +0.0000 HellaSwag 6×6× TTN depth4 0.7861 +0.0044 0.7872 +0.0046 HellaSwag 6×6× TTN depth8 0.7859 +0.0042 0.7869 +0.0043 ARC-Challenge 2×2× TTN depth4 0.5495 +0.0017 0.5495 +0.0009 ARC-Challenge 2×2× TTN depth8 0.5495 +0.0017 0.5495 +0.0009 ARC-Challenge 4×4× T core4 0.5418 +0.0000 0.5435 +0.0000 ARC-Challenge 4×4× TTN depth4 0.5478 +0.0060 0.5486 +0.0051 ARC-Challenge 4×4× TTN depth8 0.5478 +0.0060 0.5478 +0.0043 ARC-Challenge 6×6× T core4 0.5384 +0.0000 0.5414 +0.0000 ARC-Challenge 6×6× TTN depth4 0.5461 +0.0077 0.5469 +0.0055 ARC-Challenge 6×6× TTN depth8 0.5461 +0.0077 0.5469 +0.0055 A.4 SVD and dense full-covariance comparisons Table A.11: Additional baselines and diagonal-design analysis. Left: matched-compression SVD under the Llama 3.1 8B single-operator protocol. Right: diagonal AWT versus dense full-covariance preconditioning. SVD baseline Comp. T/TTN AWT SVD 2×2× 8.79 8.76 8.71 4×4× 8.90 8.85 8.81 6×6× 8.97 8.93 8.87 Diagonal vs. full covariance Comparison Result Diagonal covariance mass, L/M/Q 0.032 / 0.142 / 0.212 Off-diagonal Frobenius ratio, L/M/Q 0.984 / 0.926 / 0.885 Full-cov better on weighted objective 80 / 81 Diagonal better on held-out output error 53 / 81 Table A.12: Diagonal AWT versus dense full-covariance preconditioning. The activation covariance is strongly non-diagonal, and full covariance wins on its own weighted objective, but diagonal AWT more often gives lower held-out local output error. Statistic / comparison Result Diagonal covariance mass, Llama / Ministral / Qwen 0.0320 / 0.1415 / 0.2122 Off-diagonal Frobenius ratio, Llama / Ministral / Qwen 0.9838 / 0.9255 / 0.8850 Full-cov better on RelFrobΣRelFrob_ objective 80 / 81 cases Full-cov better on held-out RelFroboutRelFrob_out 28 / 81 cases Diagonal better on held-out RelFroboutRelFrob_out 53 / 81 cases Llama: full-cov vs diagonal wins on held-out output error 13 vs 14 Ministral: full-cov vs diagonal wins on held-out output error 10 vs 17 Qwen: full-cov vs diagonal wins on held-out output error 5 vs 22 Mean Δ output error, Llama / Ministral / Qwen -0.0284 / +0.6068 / +0.9093 Δ=RelFrobout(full-cov)−RelFrobout(diagonal) =RelFrob_out(full-cov)-RelFrob_out(diagonal), so positive values favor the diagonal method. A.5 Cross-model single-operator details Table A.13: Single-operator WikiText perplexity summaries for Ministral 8B and Qwen2.5 7B. These are the per-family counterparts to the pooled cross-model table in the main text. Lower PPL is better. Model Comp. Dense Vanilla AWT Method Gap closed Ministral 8B 2×2× 8.66 8.74 8.71 Std 35.3% 4×4× 8.66 8.81 8.76 Std 31.3% 6×6× 8.66 8.86 8.80 Std 28.6% Qwen2.5 7B 2×2× 9.59 9.76 9.70 Std 33.5% 4×4× 9.59 9.85 9.79 Std 25.8% 6×6× 9.59 9.91 9.84 Std 22.2% Table A.14: Spearman rank correlation between changes in layer-wise metrics and Δ on WikiText for Ministral 8B and Qwen2.5 7B (α=1.0α=1.0). Groups: QK=Wq,Wk=\W_q,W_k\, VO=Wv,Wo=\W_v,W_o\, MLP=Wup,Wdown,Wgate=\W_up,W_down,W_gate\. Model Metric Overall QK VO MLP Ministral 8B ΔRelFrobout _out 0.395 0.794 0.245 0.235 ΔRelFrobw _w 0.042 -0.299 0.438 -0.072 Qwen2.5 7B ΔRelFrobout _out 0.327 0.401 0.458 0.165 ΔRelFrobw _w -0.049 -0.213 -0.260 0.164 A.6 Cross-model alpha ablations Table A.15: Alpha ablations for Ministral 8B. Entries report pooled medians over single-operator sweep rows. Lower error is better. 2×2× 4×4× 6×6× α Method Out Δ Weight Δ Out Δ Weight Δ Out Δ Weight Δ 0.5 Std 0.398 -15.1 0.585 +0.1 0.518 -14.7 0.736 -2.7 0.584 -12.2 0.805 -2.4 0.5 MeanAbs 0.395 -15.8 0.582 -0.4 0.517 -14.9 0.734 -3.0 0.582 -12.5 0.802 -2.7 1.0 Std 0.376 -19.9 0.525 -10.2 0.500 -17.7 0.660 -12.8 0.562 -15.5 0.726 -12.0 1.0 MeanAbs 0.377 -19.7 0.502 -14.1 0.500 -17.8 0.635 -16.1 0.564 -15.2 0.699 -15.2 1.5 Std 0.380 -18.9 0.365 -37.5 0.505 -16.9 0.477 -36.9 0.570 -14.2 0.549 -33.5 1.5 MeanAbs 0.387 -17.5 0.285 -51.3 0.509 -16.2 0.379 -49.9 0.574 -13.6 0.430 -47.8 Vanilla pooled medians: output / weight error = 0.469/0.5850.469/0.585 at 2×2×, 0.608/0.7560.608/0.756 at 4×4×, and 0.665/0.8240.665/0.824 at 6×6×. Δ values are percentages. Table A.16: Alpha ablations for Qwen2.5 7B. Entries report pooled medians over single-operator sweep rows. Lower error is better. 2×2× 4×4× 6×6× α Method Out Δ Weight Δ Out Δ Weight Δ Out Δ Weight Δ 0.5 Std 0.377 -18.6 0.594 +0.8 0.522 -16.8 0.745 -2.6 0.590 -15.1 0.814 -2.2 0.5 MeanAbs 0.375 -19.1 0.588 -0.1 0.515 -17.9 0.740 -3.3 0.579 -16.7 0.808 -3.0 1.0 Std 0.366 -21.1 0.487 -17.3 0.501 -20.0 0.627 -18.1 0.568 -18.4 0.702 -15.7 1.0 MeanAbs 0.366 -21.0 0.445 -24.4 0.505 -19.4 0.578 -24.4 0.568 -18.4 0.654 -21.4 1.5 Std 0.395 -14.8 0.271 -54.0 0.520 -16.9 0.364 -52.5 0.582 -16.3 0.409 -50.9 1.5 MeanAbs 0.405 -12.6 0.196 -66.7 0.527 -16.0 0.286 -62.7 0.587 -15.6 0.332 -60.1 Vanilla pooled medians: output / weight error = 0.463/0.5890.463/0.589 at 2×2×, 0.627/0.7650.627/0.765 at 4×4×, and 0.695/0.8330.695/0.833 at 6×6×. Δ values are percentages. A.7 Cross-model backend comparisons Table A.17: Backend comparison on WikiText under single-operator replacement for Ministral 8B. Entries are within-backend pooled medians. Δ is measured against T core4 at the same compression. Lower PPL is better. Comp. Backend PPL (Van.) Δ PPL (AWT) Δ 2×2× T core4 8.75 +0.00 8.71 +0.00 2×2× TTN depth4 8.72 -0.03 8.70 -0.01 2×2× TTN depth8 8.72 -0.03 8.70 -0.01 4×4× T core4 8.86 +0.00 8.79 +0.00 4×4× TTN depth4 8.77 -0.09 8.75 -0.04 4×4× TTN depth8 8.78 -0.08 8.75 -0.04 6×6× T core4 8.92 +0.00 8.84 +0.00 6×6× TTN depth4 8.81 -0.11 8.78 -0.06 6×6× TTN depth8 8.81 -0.10 8.78 -0.06 Table A.18: Backend comparison on WikiText under single-operator replacement for Qwen2.5 7B. Entries are within-backend pooled medians. Δ is measured against T core4 at the same compression. Lower PPL is better. Comp. Backend PPL (Van.) Δ PPL (AWT) Δ 2×2× T core4 9.75 +0.00 9.72 +0.00 2×2× TTN depth4 9.76 +0.01 9.69 -0.02 2×2× TTN depth8 9.78 +0.02 9.70 -0.02 4×4× T core4 9.88 +0.00 9.82 +0.00 4×4× TTN depth4 9.84 -0.04 9.75 -0.07 4×4× TTN depth8 9.85 -0.03 9.77 -0.05 6×6× T core4 9.95 +0.00 9.89 +0.00 6×6× TTN depth4 9.88 -0.07 9.81 -0.08 6×6× TTN depth8 9.91 -0.04 9.81 -0.08 A.8 Multi-operator replacement details Table A.19: Multi-operator WikiText suffix replacement on Llama 3.1 8B. The attention group replaces Wq,Wk,Wv,Wo\W_q,W_k,W_v,W_o\ jointly in the last k blocks. Lower PPL is better; negative Δ favors AWT. k Comp. Vanilla PPL AWT PPL Δ Gap closed Finite backends 1 2×2× 9.33 8.92 -0.41 59.5% T core4, TTN depth4, TTN depth8 1 4×4× 9.58 9.12 -0.47 49.0% T core4, TTN depth4, TTN depth8 1 6×6× 9.71 9.33 -0.46 35.3% T core4, TTN depth4, TTN depth8 2 2×2× 9.78 9.19 -0.59 51.8% T core4, TTN depth4, TTN depth8 2 4×4× 10.28 9.60 -0.72 41.3% T core4, TTN depth4, TTN depth8 2 6×6× 10.56 9.91 -0.81 33.9% T core4, TTN depth4, TTN depth8 4 2×2× 10.56 9.80 -0.76 39.4% T core4, TTN depth4, TTN depth8 4 4×4× 11.93 10.92 -1.02 30.8% T core4, TTN depth4, TTN depth8 4 6×6× 12.68 11.57 -1.11 27.4% T core4, TTN depth4, TTN depth8 8 2×2× 11.85 10.45 -1.40 43.6% T core4, TTN depth4, TTN depth8 8 4×4× 15.55 13.08 -2.48 35.8% T core4, TTN depth4, TTN depth8 8 6×6× 17.55 14.81 -2.75 30.8% T core4, TTN depth4, TTN depth8 Table A.20: Multi-operator WikiText suffix replacement on Llama 3.1 8B when all seven dense matrices are replaced jointly in the last k blocks. Lower PPL is better; negative Δ favors AWT. k Comp. Vanilla PPL AWT PPL Δ Gap closed 1 2×2× 13.87 11.99 -1.86 36.0% 1 4×4× 15.54 12.76 -2.61 40.2% 1 6×6× 17.65 13.60 -3.62 45.0% 2 2×2× 21.53 13.79 -7.74 60.0% 2 4×4× 24.47 15.09 -9.21 59.3% 2 6×6× 30.18 17.28 -11.71 59.9% Table A.21: Multi-operator WikiText suffix replacement on Llama 3.1 8B when the MLP group Wup,Wgate,Wdown\W_up,W_gate,W_down\ is replaced jointly. Empty rows in the source runs correspond to non-finite PPL. Lower PPL is better; negative Δ favors AWT. k Comp. Vanilla PPL AWT PPL Δ Gap closed Finite backends 1 2×2× 13.54 12.27 -1.27 26.0% T core4, TTN depth4, TTN depth8 1 4×4× 14.19 12.73 -1.45 26.4% T core4, TTN depth4, TTN depth8 1 6×6× 14.78 13.25 -1.53 24.9% T core4, TTN depth4, TTN depth8 2 2×2× 17.65 14.21 -3.43 38.1% T core4, TTN depth4, TTN depth8 2 4×4× 18.52 14.67 -3.79 39.0% T core4, TTN depth4, TTN depth8 2 6×6× 19.78 15.30 -4.47 40.2% T core4, TTN depth4, TTN depth8 4 2×2× 30.38 17.41 -12.97 59.6% T core4, TTN depth4, TTN depth8 4 4×4× 34.02 18.33 -15.69 61.8% TTN depth4, TTN depth8 4 6×6× 39.31 19.61 -19.70 64.2% TTN depth4, TTN depth8 8 2×2× 36.21 38.31 +2.09 -7.6% T core4 A.9 Multi-operator BoolQ details Table A.22: Multi-operator BoolQ suffix replacement on Llama 3.1 8B. Attention group replaces Wq,Wk,Wv,Wo\W_q,W_k,W_v,W_o\ jointly in the last k blocks. Higher is better; positive Δ favors AWT. k Comp. Vanilla AWT Δ Error reduction vs vanilla Finite backends 1 2×2× 0.7900 0.8000 +0.0100 4.8% T core4, TTN depth4, TTN depth8 1 4×4× 0.8000 0.7900 -0.0100 -5.0% T core4, TTN depth4, TTN depth8 1 6×6× 0.8000 0.8000 +0.0000 0.0% T core4, TTN depth4, TTN depth8 2 2×2× 0.8000 0.8100 +0.0100 5.0% T core4, TTN depth4, TTN depth8 2 4×4× 0.8100 0.8300 +0.0200 10.5% T core4, TTN depth4, TTN depth8 2 6×6× 0.8100 0.8100 +0.0000 0.0% T core4, TTN depth4, TTN depth8 4 2×2× 0.8000 0.8300 +0.0200 15.0% T core4, TTN depth4, TTN depth8 4 4×4× 0.8200 0.8300 +0.0100 5.6% T core4, TTN depth4, TTN depth8 4 6×6× 0.8100 0.8300 +0.0200 10.5% T core4, TTN depth4, TTN depth8 8 2×2× 0.8100 0.8300 +0.0200 10.5% T core4, TTN depth4, TTN depth8 8 4×4× 0.8200 0.8300 +0.0100 5.6% T core4, TTN depth4, TTN depth8 8 6×6× 0.8300 0.8400 +0.0300 5.9% T core4, TTN depth4, TTN depth8 Table A.23: Multi-operator BoolQ suffix replacement on Llama 3.1 8B when all seven dense matrices are replaced jointly in the last k blocks. Higher is better. k Comp. Vanilla AWT Δ Error reduction vs vanilla 1 2×2× 0.8100 0.8300 +0.0200 10.5% 1 4×4× 0.7900 0.8200 +0.0300 14.3% 1 6×6× 0.7600 0.8200 +0.0600 25.0% 2 2×2× 0.7900 0.7800 -0.0100 -4.8% 2 4×4× 0.7800 0.7900 +0.0100 4.5% 2 6×6× 0.7900 0.7800 +0.0000 -4.8% Table A.24: Multi-operator BoolQ suffix replacement on Llama 3.1 8B when the MLP group Wup,Wgate,Wdown\W_up,W_gate,W_down\ is replaced jointly. Higher is better. k Comp. Vanilla AWT Δ Error reduction vs vanilla Finite backends 1 2×2× 0.8200 0.8200 +0.0000 0.0% T core4, TTN depth4, TTN depth8 1 4×4× 0.8100 0.8200 +0.0100 5.3% T core4, TTN depth4, TTN depth8 1 6×6× 0.8100 0.8300 +0.0200 10.5% T core4, TTN depth4, TTN depth8 2 2×2× 0.7800 0.7800 +0.0000 0.0% T core4, TTN depth4, TTN depth8 2 4×4× 0.7800 0.7800 +0.0000 0.0% T core4, TTN depth4, TTN depth8 2 6×6× 0.7900 0.7800 -0.0100 -4.8% T core4, TTN depth4, TTN depth8 4 2×2× 0.7900 0.7900 +0.0000 0.0% T core4, TTN depth4, TTN depth8 4 4×4× 0.7700 0.7800 +0.0100 4.3% T core4, TTN depth4, TTN depth8 4 6×6× 0.7600 0.7900 +0.0300 12.5% T core4, TTN depth4, TTN depth8 8 2×2× 0.7000 0.7100 +0.0100 3.3% T core4, TTN depth4, TTN depth8 8 4×4× 0.7000 0.7000 +0.0000 0.0% T core4, TTN depth4, TTN depth8 8 6×6× 0.7000 0.7200 +0.0200 6.7% T core4, TTN depth4, TTN depth8 A.10 Per-operator output-error breakdowns for Ministral and Qwen Table A.25: Per-operator activation-conditioned output error for Ministral 8B under TTN depth4 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.192±0.0440.192± 0.044 0.158±0.0380.158± 0.038 0.159±0.0380.159± 0.038 0.258±0.0520.258± 0.052 0.217±0.0460.217± 0.046 0.218±0.0460.218± 0.046 0.291±0.0540.291± 0.054 0.248±0.0490.248± 0.049 0.249±0.0490.249± 0.049 36 WkW_k 0.191±0.0370.191± 0.037 0.151±0.0320.151± 0.032 0.152±0.0320.152± 0.032 0.289±0.0440.289± 0.044 0.231±0.0390.231± 0.039 0.233±0.0390.233± 0.039 0.339±0.0440.339± 0.044 0.272±0.0410.272± 0.041 0.275±0.0410.275± 0.041 36 WvW_v 0.558±0.0800.558± 0.080 0.458±0.0610.458± 0.061 0.458±0.0600.458± 0.060 0.708±0.0810.708± 0.081 0.608±0.0680.608± 0.068 0.608±0.0680.608± 0.068 0.770±0.0790.770± 0.079 0.673±0.0670.673± 0.067 0.675±0.0670.675± 0.067 36 WoW_o 0.492±0.1110.492± 0.111 0.375±0.0990.375± 0.099 0.380±0.0970.380± 0.097 0.625±0.1350.625± 0.135 0.506±0.1240.506± 0.124 0.508±0.1220.508± 0.122 0.680±0.1440.680± 0.144 0.569±0.1350.569± 0.135 0.571±0.1330.571± 0.133 36 WupW_up 0.559±0.0780.559± 0.078 0.520±0.0790.520± 0.079 0.522±0.0800.522± 0.080 0.617±0.0870.617± 0.087 0.577±0.0880.577± 0.088 0.579±0.0880.579± 0.088 0.669±0.0940.669± 0.094 0.628±0.0950.628± 0.095 0.631±0.0950.631± 0.095 36 WdownW_down 0.655±0.0900.655± 0.090 0.507±0.1490.507± 0.149 0.523±0.1530.523± 0.153 0.716±0.0960.716± 0.096 0.568±0.1660.568± 0.166 0.585±0.1710.585± 0.171 0.769±0.1010.769± 0.101 0.625±0.1800.625± 0.180 0.642±0.1860.642± 0.186 36 WgateW_gate 0.348±0.0500.348± 0.050 0.328±0.0500.328± 0.050 0.329±0.0500.329± 0.050 0.387±0.0550.387± 0.055 0.366±0.0560.366± 0.056 0.368±0.0560.368± 0.056 0.423±0.0610.423± 0.061 0.403±0.0610.403± 0.061 0.404±0.0610.404± 0.061 36 Table A.26: Per-operator activation-conditioned output error for Ministral 8B under TTN depth8 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.193±0.0440.193± 0.044 0.160±0.0380.160± 0.038 0.160±0.0380.160± 0.038 0.260±0.0520.260± 0.052 0.220±0.0460.220± 0.046 0.220±0.0460.220± 0.046 0.294±0.0540.294± 0.054 0.251±0.0490.251± 0.049 0.252±0.0490.252± 0.049 36 WkW_k 0.195±0.0380.195± 0.038 0.155±0.0320.155± 0.032 0.156±0.0320.156± 0.032 0.293±0.0440.293± 0.044 0.235±0.0380.235± 0.038 0.236±0.0390.236± 0.039 0.338±0.0430.338± 0.043 0.274±0.0390.274± 0.039 0.276±0.0400.276± 0.040 36 WvW_v 0.566±0.0800.566± 0.080 0.466±0.0610.466± 0.061 0.465±0.0610.465± 0.061 0.718±0.0790.718± 0.079 0.612±0.0660.612± 0.066 0.613±0.0660.613± 0.066 0.778±0.0770.778± 0.077 0.672±0.0660.672± 0.066 0.673±0.0660.673± 0.066 36 WoW_o 0.495±0.1120.495± 0.112 0.378±0.1000.378± 0.100 0.383±0.0970.383± 0.097 0.629±0.1360.629± 0.136 0.510±0.1250.510± 0.125 0.513±0.1230.513± 0.123 0.685±0.1450.685± 0.145 0.575±0.1360.575± 0.136 0.577±0.1350.577± 0.135 36 WupW_up 0.559±0.0780.559± 0.078 0.521±0.0790.521± 0.079 0.522±0.0800.522± 0.080 0.618±0.0870.618± 0.087 0.578±0.0880.578± 0.088 0.580±0.0880.580± 0.088 0.670±0.0940.670± 0.094 0.630±0.0950.630± 0.095 0.632±0.0950.632± 0.095 36 WdownW_down 0.655±0.0900.655± 0.090 0.507±0.1490.507± 0.149 0.523±0.1530.523± 0.153 0.717±0.0960.717± 0.096 0.569±0.1660.569± 0.166 0.586±0.1710.586± 0.171 0.770±0.1010.770± 0.101 0.626±0.1800.626± 0.180 0.644±0.1860.644± 0.186 36 WgateW_gate 0.348±0.0500.348± 0.050 0.328±0.0500.328± 0.050 0.329±0.0500.329± 0.050 0.388±0.0550.388± 0.055 0.367±0.0560.367± 0.056 0.368±0.0560.368± 0.056 0.424±0.0610.424± 0.061 0.404±0.0610.404± 0.061 0.405±0.0610.405± 0.061 36 Table A.27: Per-operator activation-conditioned output error for Ministral 8B under T core4 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.333±0.0190.333± 0.019 0.228±0.0240.228± 0.024 0.227±0.0240.227± 0.024 0.489±0.0240.489± 0.024 0.331±0.0280.331± 0.028 0.331±0.0290.331± 0.029 0.565±0.0290.565± 0.029 0.382±0.0320.382± 0.032 0.381±0.0320.381± 0.032 36 WkW_k 0.321±0.0440.321± 0.044 0.219±0.0220.219± 0.022 0.219±0.0220.219± 0.022 0.500±0.0650.500± 0.065 0.326±0.0310.326± 0.031 0.325±0.0310.325± 0.031 0.581±0.0710.581± 0.071 0.378±0.0370.378± 0.037 0.377±0.0360.377± 0.036 36 WvW_v 0.616±0.0390.616± 0.039 0.448±0.0430.448± 0.043 0.448±0.0430.448± 0.043 0.780±0.0460.780± 0.046 0.611±0.0570.611± 0.057 0.611±0.0570.611± 0.057 0.841±0.0450.841± 0.045 0.677±0.0610.677± 0.061 0.677±0.0610.677± 0.061 36 WoW_o 0.551±0.1240.551± 0.124 0.449±0.1190.449± 0.119 0.466±0.1210.466± 0.121 0.698±0.1490.698± 0.149 0.609±0.1440.609± 0.144 0.620±0.1480.620± 0.148 0.753±0.1580.753± 0.158 0.677±0.1530.677± 0.153 0.684±0.1570.684± 0.157 36 WupW_up 0.575±0.0430.575± 0.043 0.493±0.0610.493± 0.061 0.493±0.0610.493± 0.061 0.748±0.0380.748± 0.038 0.652±0.0720.652± 0.072 0.651±0.0720.651± 0.072 0.814±0.0330.814± 0.033 0.713±0.0740.713± 0.074 0.712±0.0740.712± 0.074 36 WdownW_down 0.570±0.0800.570± 0.080 0.470±0.1390.470± 0.139 0.464±0.1370.464± 0.137 0.722±0.1240.722± 0.124 0.626±0.1770.626± 0.177 0.623±0.1750.623± 0.175 0.777±0.1420.777± 0.142 0.688±0.1890.688± 0.189 0.687±0.1870.687± 0.187 36 WgateW_gate 0.396±0.0400.396± 0.040 0.331±0.0530.331± 0.053 0.331±0.0530.331± 0.053 0.535±0.0480.535± 0.048 0.446±0.0710.446± 0.071 0.445±0.0710.445± 0.071 0.592±0.0510.592± 0.051 0.493±0.0770.493± 0.077 0.492±0.0770.492± 0.077 36 Table A.28: Per-operator activation-conditioned output error for Qwen2.5 7B under TTN depth4 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.331±0.0790.331± 0.079 0.222±0.0690.222± 0.069 0.223±0.0690.223± 0.069 0.446±0.0990.446± 0.099 0.313±0.0890.313± 0.089 0.313±0.0890.313± 0.089 0.505±0.1090.505± 0.109 0.363±0.0990.363± 0.099 0.363±0.0990.363± 0.099 28 WkW_k 0.354±0.0600.354± 0.060 0.207±0.0480.207± 0.048 0.208±0.0480.208± 0.048 0.566±0.0760.566± 0.076 0.346±0.0610.346± 0.061 0.347±0.0600.347± 0.060 0.652±0.0750.652± 0.075 0.419±0.0630.419± 0.063 0.419±0.0630.419± 0.063 28 WvW_v 0.665±0.0930.665± 0.093 0.469±0.1190.469± 0.119 0.468±0.1180.468± 0.118 0.823±0.0720.823± 0.072 0.647±0.0970.647± 0.097 0.644±0.1020.644± 0.102 0.876±0.0600.876± 0.060 0.728±0.0820.728± 0.082 0.726±0.0840.726± 0.084 28 WoW_o 0.473±0.0970.473± 0.097 0.375±0.0800.375± 0.080 0.377±0.0800.377± 0.080 0.619±0.1210.619± 0.121 0.510±0.1060.510± 0.106 0.510±0.1060.510± 0.106 0.684±0.1320.684± 0.132 0.574±0.1190.574± 0.119 0.573±0.1190.573± 0.119 28 WupW_up 0.602±0.0970.602± 0.097 0.469±0.1190.469± 0.119 0.470±0.1190.470± 0.119 0.659±0.0980.659± 0.098 0.527±0.1260.527± 0.126 0.528±0.1260.528± 0.126 0.710±0.1000.710± 0.100 0.580±0.1320.580± 0.132 0.581±0.1320.581± 0.132 28 WdownW_down 0.447±0.1840.447± 0.184 0.284±0.1840.284± 0.184 0.298±0.1890.298± 0.189 0.490±0.2000.490± 0.200 0.323±0.2070.323± 0.207 0.337±0.2120.337± 0.212 0.528±0.2130.528± 0.213 0.360±0.2280.360± 0.228 0.373±0.2330.373± 0.233 28 WgateW_gate 0.312±0.1300.312± 0.130 0.256±0.1150.256± 0.115 0.257±0.1160.257± 0.116 0.343±0.1410.343± 0.141 0.287±0.1270.287± 0.127 0.288±0.1280.288± 0.128 0.371±0.1500.371± 0.150 0.316±0.1380.316± 0.138 0.317±0.1380.317± 0.138 28 Table A.29: Per-operator activation-conditioned output error for Qwen2.5 7B under TTN depth8 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.333±0.0800.333± 0.080 0.224±0.0700.224± 0.070 0.224±0.0690.224± 0.069 0.450±0.1000.450± 0.100 0.316±0.0890.316± 0.089 0.317±0.0900.317± 0.090 0.511±0.1100.511± 0.110 0.368±0.1000.368± 0.100 0.368±0.1000.368± 0.100 28 WkW_k 0.369±0.0600.369± 0.060 0.216±0.0480.216± 0.048 0.216±0.0480.216± 0.048 0.554±0.0790.554± 0.079 0.338±0.0550.338± 0.055 0.338±0.0550.338± 0.055 0.637±0.0780.637± 0.078 0.399±0.0580.399± 0.058 0.399±0.0580.399± 0.058 28 WvW_v 0.680±0.0940.680± 0.094 0.476±0.1200.476± 0.120 0.475±0.1190.475± 0.119 0.829±0.0850.829± 0.085 0.607±0.1350.607± 0.135 0.606±0.1350.606± 0.135 0.882±0.0650.882± 0.065 0.681±0.1110.681± 0.111 0.677±0.1140.677± 0.114 28 WoW_o 0.476±0.0970.476± 0.097 0.378±0.0810.378± 0.081 0.380±0.0810.380± 0.081 0.623±0.1220.623± 0.122 0.515±0.1070.515± 0.107 0.515±0.1070.515± 0.107 0.690±0.1330.690± 0.133 0.580±0.1200.580± 0.120 0.579±0.1200.579± 0.120 28 WupW_up 0.638±0.0950.638± 0.095 0.503±0.1190.503± 0.119 0.503±0.1190.503± 0.119 0.697±0.0920.697± 0.092 0.561±0.1240.561± 0.124 0.560±0.1240.560± 0.124 0.762±0.0890.762± 0.089 0.623±0.1290.623± 0.129 0.621±0.1300.621± 0.130 28 WdownW_down 0.464±0.1900.464± 0.190 0.293±0.1900.293± 0.190 0.308±0.1950.308± 0.195 0.500±0.2020.500± 0.202 0.321±0.2070.321± 0.207 0.336±0.2120.336± 0.212 0.541±0.2140.541± 0.214 0.355±0.2260.355± 0.226 0.369±0.2310.369± 0.231 28 WgateW_gate 0.341±0.1340.341± 0.134 0.283±0.1130.283± 0.113 0.283±0.1160.283± 0.116 0.381±0.1450.381± 0.145 0.316±0.1260.316± 0.126 0.315±0.1290.315± 0.129 0.427±0.1570.427± 0.157 0.351±0.1400.351± 0.140 0.350±0.1430.350± 0.143 28 Table A.30: Per-operator activation-conditioned output error for Qwen2.5 7B under T core4 tensorization with α=1α=1. Entries report mean μ and standard deviation σ over per-block sweep values. Lower is better. 2×2× compression 4×4× compression 6×6× compression Matrix Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) Vanilla (μ±σμ\!±\!σ) Std (μ±σμ\!±\!σ) MeanAbs (μ±σμ\!±\!σ) N WqW_q 0.487±0.0660.487± 0.066 0.372±0.0700.372± 0.070 0.373±0.0720.373± 0.072 0.668±0.0670.668± 0.067 0.535±0.0880.535± 0.088 0.533±0.0900.533± 0.090 0.745±0.0670.745± 0.067 0.613±0.0940.613± 0.094 0.608±0.0950.608± 0.095 28 WkW_k 0.458±0.0340.458± 0.034 0.364±0.0300.364± 0.030 0.362±0.0300.362± 0.030 0.668±0.0370.668± 0.037 0.557±0.0400.557± 0.040 0.553±0.0390.553± 0.039 0.760±0.0320.760± 0.032 0.651±0.0420.651± 0.042 0.646±0.0390.646± 0.039 28 WvW_v 0.607±0.0400.607± 0.040 0.501±0.0480.501± 0.048 0.501±0.0500.501± 0.050 0.794±0.0310.794± 0.031 0.696±0.0580.696± 0.058 0.695±0.0600.695± 0.060 0.862±0.0250.862± 0.025 0.775±0.0630.775± 0.063 0.774±0.0640.774± 0.064 28 WoW_o 0.529±0.0900.529± 0.090 0.438±0.0920.438± 0.092 0.443±0.0960.443± 0.096 0.690±0.1100.690± 0.110 0.600±0.1190.600± 0.119 0.598±0.1240.598± 0.124 0.756±0.1180.756± 0.118 0.672±0.1310.672± 0.131 0.666±0.1350.666± 0.135 28 WupW_up 0.661±0.0950.661± 0.095 0.498±0.0790.498± 0.079 0.503±0.0810.503± 0.081 0.831±0.0720.831± 0.072 0.668±0.0900.668± 0.090 0.670±0.0950.670± 0.095 0.887±0.0550.887± 0.055 0.734±0.0920.734± 0.092 0.734±0.0980.734± 0.098 28 WdownW_down 0.429±0.1190.429± 0.119 0.320±0.1770.320± 0.177 0.321±0.1780.321± 0.178 0.608±0.1350.608± 0.135 0.444±0.2300.444± 0.230 0.443±0.2320.443± 0.232 0.685±0.1350.685± 0.135 0.495±0.2490.495± 0.249 0.493±0.2490.493± 0.249 28 WgateW_gate 0.375±0.1180.375± 0.118 0.289±0.1150.289± 0.115 0.292±0.1170.292± 0.117 0.499±0.1570.499± 0.157 0.391±0.1510.391± 0.151 0.393±0.1520.393± 0.152 0.546±0.1720.546± 0.172 0.433±0.1650.433± 0.165 0.435±0.1650.435± 0.165 28