Paper deep dive

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability

Matvei Tarasov, Salman Ahmadi-Asl, Andre L. F. de Almeida, Andrzej Cichocki

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 219

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/15/2026, 8:45:53 AM

Summary

This survey paper reviews tensor methods for Large Language Models (LLMs), arguing that the multilinear structure of LLM components (tokens, weights, caches, activations) is underexploited by conventional matrix-centric views. It organizes tensor decompositions and networks across a seven-stage lifecycle (tokenization to interpretability) and a component view (embeddings, attention, FFN). The paper introduces a unified notation, compares methods, and defines a metric called rho_gap to measure the discrepancy between theoretical memory reduction and actual system-level speedup.

Entities (12)

Large Language Models · model-class · 99%Tensor Methods · methodology · 98%Transformer · architecture · 97%Tensor Decompositions · technique · 95%Tensor Networks · technique · 95%rho_gap · metric · 93%KV cache · component · 92%CP Decomposition · algorithm · 90%TT Decomposition · algorithm · 90%Tucker Decomposition · algorithm · 90%Llama-3-8B · model-instance · 88%Innopolis University · organization · 85%

Relation Signals (8)

Tensor Methods → appliesto → Large Language Models

confidence 95% · This survey organizes tensor methods for LLMs

CP Decomposition → istypeof → Tensor Decompositions

confidence 95% · We cover the CP decomposition... This section reviews the main tensor decomposition formats

Tucker Decomposition → istypeof → Tensor Decompositions

confidence 95% · the Tucker decomposition... reviews the main tensor decomposition formats

TT Decomposition → istypeof → Tensor Decompositions

confidence 95% · the TT decomposition... reviews the main tensor decomposition formats

rho_gap → measures → compression-realization gap

confidence 92% · introduce rho_gap, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup

Tensor Methods → analyzes → Transformer

confidence 90% · analyze tensorization strategies for individual Transformer components

Tensor Methods → optimizes → KV cache

confidence 85% · tensor methods... compression... KV cache can become the dominant memory bottleneck

Llama-3-8B → usedascasestudyfor → Tensor Methods

confidence 85% · using LLaMA-3-8B as a running case study

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce $\rho_{\rm gap}$, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at \href{this https URL}{this https URL}.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

215,422 characters extracted from source content.

Expand or collapse full text

Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Compression, Inference, and Interpretability Matvei Tarasov Affiliation: Innopolis University, Innopolis 420500, Russia m.tarasov@innopolis.university Affiliation: Innopolis University, Innopolis 420500, Russia s.ahmadiasl@innopolis.ru Salman Ahmadi-Asl Affiliation: Innopolis University, Innopolis 420500, Russia m.tarasov@innopolis.university Affiliation: Innopolis University, Innopolis 420500, Russia s.ahmadiasl@innopolis.ru André L. F. de Almeida Affiliation: Department of Teleinformatics Engineering, Federal University of Ceará, Fortaleza 60455-760, Brazil andre@gtel.ufc.br Andrzej Cichocki Affiliation: Systems Research Institute of Polish Academy of Science and Warsaw University of Technology, Poland cichockiand@gmail.com Abstract Large language models (LLMs) are built from structured high-dimensional objects such as token representations, weights, adaptation updates, caches, and activations, whose multilinear structure is underexploited by the conventional matrix-centric view. Tensor decompositions and tensor networks provide a principled algebraic language for this structure, yet the literature often treats them as isolated compression mechanisms. This survey organizes tensor methods for LLMs through two complementary views: a seven-stage lifecycle taxonomy covering tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability, and a component view covering embeddings, attention, and feed-forward networks. We provide unified notation and theoretical foundations, analyze tensorization strategies for individual Transformer components, and compare methods at each lifecycle stage while making differences in evaluation protocols and model scales explicit. We further connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. Finally, we synthesize open challenges and introduce ρgap _ gap, a metric for the compression-realization gap between theoretical memory reduction and measured system-level speedup. By treating tensorization as a common structural principle, the survey provides a structured entry point to tensorized language models and clarifies when parameter savings can plausibly translate into memory efficiency, computational efficiency, or interpretability. The GitHub page dedicated to this paper is accessible at this https URL. 1 Introduction Large language models (LLMs) based on the Transformer architecture [168] have reshaped deep learning and artificial intelligence by demonstrating broad capabilities and performance gains across successive model generations [17, 121, 163]. Their success stems from three factors: a highly parallelizable architecture that scales efficiently with model size and compute [168], self-supervised pre-training on internet-scale text corpora [139, 140], and the predictable improvements that arise from scaling this combination [85, 73]. Today’s production-scale LLMs, such as Qwen [138], DeepSeek [32], and Kimi [87], contain tens to hundreds of billions of parameters and are deployed at scale, demanding substantial computational resources for both training and inference. In parallel, the past several decades have seen active development of tensor decompositions and tensor networks within the numerical linear algebra community. Emerging independently in psychometrics [165, 18, 63] and in quantum many-body physics [122, 169, 170], these tools have gradually migrated into general-purpose computational algebra. Along the way, researchers reformulated the existing formats in the language of numerical analysis and developed stable algorithms for computing them [27, 125, 190], alongside a broader theoretical effort to systematize and unify the resulting landscape [92, 21, 22]. This growing activity has, in turn, fueled interest in applying tensor methods to deep learning [175], reflecting a wider recognition that numerical linear algebra sits at the core of modern neural architectures [9]. Motivation. Despite their impressive capabilities, modern LLMs face three persistent challenges. Training a frontier model requires substantial accelerator time [113]. Inference at scale demands high memory bandwidth, and the KV cache can become the dominant memory bottleneck in long-context applications [100]. Meanwhile, internal representations, including activations and weights, remain largely opaque, and the mechanisms by which these models reason and store knowledge are still poorly understood, motivating a growing body of interpretability research [39, 38]. These challenges are usually treated separately, but a tensor perspective provides a common language for all three. LLMs are conventionally described and implemented in matrix terms: Transformer computations reduce largely to batched matrix multiplication, which is convenient and highly optimized but conceptually incomplete. Many Transformer objects, including stacked multi-head attention projections, layer collections, activation streams, and KV caches, are naturally higher-order, and a purely matrix-centric view leaves their multilinear structure underexploited. Table 1: Literature clusters surrounding tensor methods and language models. Cluster Main objects or methods Specific gap for this survey Classical tensor decompositions and tensor networks [92], [21], [22], [122] Tensors, tensor-algebraic operations; decompositions, ranks; computation algorithms Discusses tensors abstractly; does not provide LLM applications of tensors Tensor methods for machine learning and neural networks [151, 83, 128, 126, 175, 68] Tensorized machine learning; tensorization and decomposition of generic neural modules, layers Often predates decoder-only LLMs; omits LLM-specific objects such as the KV cache LLM efficiency techniques [172, 174, 194, 158, 86, 177, 193, 183, 101, 114, 11] PEFT; compression: quantization, pruning, knowledge distillation; efficient attention; efficient inference; KV-cache compression Usually omits tensor counterparts; does not emphasize tensor methods as a separate class LLM mechanistic interpretability [153, 142, 14] Features, activations, logits, neurons, heads, circuits; activation patching, ablations, SAEs Overlooks multilinear structure, both as a descriptive language and as an analysis tool Current state of the literature. The literature on tensors and LLMs is fragmented across several partially overlapping clusters, summarized in Table 1. We cite representative surveys and reviews, so that the reader can turn to a fuller treatment of each area. Each row identifies a community with its own objects, methods, and terminology; the remainder of this section discusses each cluster in turn, together with the specific gap it leaves for a survey connecting tensor methods to LLMs. The first cluster is the classical tensor decomposition literature [92, 21, 22, 122]. Works from this cluster are mathematically deep and essential for notation, algorithms, and theoretical concepts such as tensor rank. While this literature provides the general algebraic and numerical foundation for tensor computations, it does not map these concepts onto LLM-specific objects. A second line of work applies tensor decompositions and tensor networks to classical machine learning [151, 83] and neural networks (NNs) [128, 126, 175, 68]. This literature establishes neural-network tensorization as a general design principle. Much of it, however, predates modern decoder-only LLMs or focuses on generic layers, convolutional neural networks (CNNs), and recurrent neural networks (RNNs), leaving LLM-specific objects and constraints untreated. A third body of work covers the broader landscape of LLM efficiency techniques. It has expanded rapidly over the past few years, spanning many directions [172]. Some of this work targets parameter-efficient fine-tuning (PEFT) [174] and post-training compression techniques such as quantization, pruning, and knowledge distillation (KD) [194, 158, 177]; some covers PEFT and compression together [86]. Other work targets inference-time efficiency [193, 183, 101] and serving [114], whereas [11] cuts across all these directions by structure rather than application, reviewing low-rank matrix factorizations in weights, adapters, and gradients. Within each of these sub-clusters, tensor decompositions surface only occasionally, mentioned as one technique among many. Yet tensor methods across all these sub-clusters rest on the same underlying algebraic theory, which suggests considering them as a standalone group. As a fourth cluster, mechanistic interpretability has become an active and rapidly growing research direction, aiming to reverse-engineer language models’ internal computations in terms of features, activations, neurons, heads, and circuits, using tools such as activation patching, ablations, and sparse autoencoders (SAEs) [153, 142, 14]. Within this broader field, a smaller community studies Transformer circuits specifically through the lens of tensor networks, using graphical tensor notation [160]. Beyond notation, a handful of very recent works make multilinear structure the object of the analysis itself, and no existing interpretability survey covers them. This survey treats both the notation and these methods as a distinct direction (Section 5.7). Scope. The same multilinear structure recurs across all four clusters, but the clusters use it differently. The classical tensor literature develops a general theory, whereas tensorized neural modules, LLM efficiency techniques, and mechanistic interpretability typically use that theory locally for a specific task. This survey connects these lines of work through a unified tensor-theoretic lens grounded in LLM-specific objects. Its distinguishing feature is the combination of two complementary organizations: a component view that asks what inside a Transformer is tensorized, and a lifecycle view that asks when and why tensorization is introduced. To our knowledge, existing surveys do not combine this two-view organization with unified notation, protocol-aware comparisons, and a systems-level analysis of whether theoretical compression is realized in practice. Contributions This survey makes four contributions: 1. We frame tensorization as a common structural principle acting on token representations, weights, adaptation updates, caches, and activations. We organize this literature with a seven-stage lifecycle taxonomy: tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability. 2. We complement the lifecycle taxonomy with a component view of embeddings, attention, and feed-forward networks. Together, the two views distinguish the structural compatibility of a decomposition with a model object from the training or deployment objective for which that decomposition is used. 3. We provide unified notation and theoretical foundations for tensor operations, decompositions, and networks in Transformer-like language models, and we compare methods while explicitly recording differences in model scale, baselines, evaluation protocols, and reported metrics. We also connect tensor methods to neighboring efficiency techniques and probabilistic tensor networks. 4. We synthesize open challenges and formulate the compression-realization gap through ρgap _ gap, which separates algorithmic overhead from hardware realization and makes explicit why parameter reduction alone does not imply end-to-end speedup. Structure of the survey. The remainder of this paper is organized as follows. Section 2 presents the mathematical preliminaries, including notation, tensor-network diagrams, tensor decompositions, and a brief review of the Transformer architecture. Section 3 introduces the lifecycle taxonomy and illustrates tensorization at each populated stage using LLaMA-3-8B as a running case study. Section 4 develops the component view for individual Transformer modules, whereas Section 5 develops the lifecycle view through formal problem definitions, literature reviews, and comparison tables where coverage permits. Section 6 situates tensor methods within the broader landscape of LLM efficiency techniques, and Section 7 clarifies their connections to probabilistic tensor networks. Section 8 surveys software and presents case studies. Section 9 defines the compression-realization gap, identifies open challenges, and outlines future research directions. Finally, Section 10 concludes the survey. 2 Preliminaries Notation Meaning α scalar a vector A matrix A tensor i1,…,iNA_i_1,…,i_N (i1,…,iN)(i_1,…,i_N)-th entry i1,…,in−1,:,in+1,…,iNA_i_1,…,i_n-1,:,i_n+1,…,i_N n-th mode slice ⊗A Kronecker product ∘a outer product ×nA×_nB mode-n product i1,…,iK×j1,…,jKℬA 22.66655pt ×_ [0pt][r]i_1,…,i_K 22.66655pt ×_ [0pt][r]i_1,…,i_K 16.86935pt ×_ [0pt][r]i_1,…,i_K 16.86935pt ×_ [0pt][r]i_1,…,i_K×^j_1,…,j_KB contraction aI (a) vector AIIJJ (b) matrix AI1I_1INI_N⋯·s (c) tensor I1I2I_1I_2J1J2J_1J_2ABI1I_1J1J_1I2I_2J2J_2 (e) Kronecker product abIIJJ (d) outer product ABIIKKJJ (f) contraction Figure 1: Notation reference. Left: symbols used throughout this survey for scalars, vectors, matrices, tensors, and the main tensor-algebraic operations. Right: tensor-network diagram representations of the same objects, with examples of the operations. 2.1 Notation Objects Calligraphic uppercase letters ∈ℝI1×I2×⋯×INA ^I_1× I_2×·s× I_N denote an NNth-order tensor; matrices and vectors are denoted by bold uppercase A and lowercase letters a. The symbols in Figure 1 (left) collect the notation used throughout the survey; the tensor-network diagrams [133] in Figure 1 (right) give the corresponding graphical illustration for each object and operation. Outer product Given N vectors (1)∈ℝI1,…,(N)∈ℝINa^(1) ^I_1,…,a^(N) ^I_N, their outer product is the NNth-order tensor (1)∘⋯∘(N)∈ℝI1×⋯×INa^(1) ·s ^(N) ^I_1×·s× I_N with entries: ((1)∘⋯∘(N))i1…iN=i1(1)i2(2)⋯iN(N). (a^(1) ·s ^(N) )_i_1… i_N=a^(1)_i_1a^(2)_i_2·sa^(N)_i_N. (1) This tensor is called a canonical rank-one tensor. For N=2N=2, this reduces to the familiar rank-one matrix ∘∈ℝI×Ja ^I× J with entries (∘)ij=aibj(a )_ij=a_ib_j. In tensor-network diagrams, the outer product is represented by placing the tensors next to each other, as shown in Figure 1(d). Kronecker product and decomposition The Kronecker product of two matrices ∈ℝI1×J1A ^I_1× J_1 and ∈ℝI2×J2B ^I_2× J_2 is the block matrix obtained by replacing each entry of A with a copy of B scaled by that entry, ⊗=[A11⋯A1J1⋱AI11⋯AI1J1]∈ℝI1I2×J1J2.A = bmatrixA_11B&·s&A_1J_1B\\ & & \\ A_I_11B&·s&A_I_1J_1B bmatrix ^I_1I_2× J_1J_2. (2) The corresponding tensor-network diagram is shown in Figure 1(e). This diagram has a strong connection with the outer product; see [182] for more details. For a fixed R∈ℕR , one may look for the best approximation of a matrix by a sum of Kronecker products minr,r⁡‖−∑r=1Rr⊗r‖F, _U_r,V_r\ \|A- _r=1^RU_r _r\|_F, (3) where ∈ℝI1I2×J1J2A ^I_1I_2× J_1J_2, r∈ℝI1×J1U_r ^I_1× J_1, and r∈ℝI2×J2V_r ^I_2× J_2. This problem admits a closed-form solution; rearranging the I2×J2I_2× J_2 blocks of A into the rows of a matrix ℛ⁡()∈ℝI1J1×I2J2R(A) ^I_1J_1× I_2J_2 turns each Kronecker term into a rank-one matrix, so Eq. (3) reduces to a low-rank approximation of ℛ⁡()R(A) and is solved by its rank-R truncated SVD, whose singular vectors are then reshaped back into rU_r and rV_r; see [50] for the precise construction. This is known as the Kronecker product SVD, and we refer to it as the Kronecker decomposition. The smallest R for which the approximation is exact is called the Kronecker rank of A. The decomposition is not intrinsic to A alone, since the factor shapes I1I2=I_1I_2=I and J1J2=J_1J_2=J can be chosen in several ways, and each choice defines a different problem (3) with its own Kronecker rank. We consider only the matrix case, although the construction generalizes to tensors [13]. Contractions Contraction can be viewed as summing with respect to one or more paired modes of two tensors, while leaving other modes free. Let ∈ℝI1×I2×⋯×INA ^I_1× I_2×·s× I_N be an NNth-order tensor and ℬ∈ℝJ1×J2×⋯×JMB ^J_1× J_2×·s× J_M an MMth-order tensor, and suppose there are K mode pairs (i1,j1),…,(iK,jK)(i_1,j_1),…,(i_K,j_K) satisfying 1≤ik≤N,1≤jk≤M,k∈1,2,…,Kik≠iℓ,jk≠jℓ,∀k≠ℓIik=Jjk,k∈1,2,…,K. \ array[]l1≤ i_k≤ N, 1≤ j_k≤ M,&k∈\1,2,…,K\\\ i_k≠ i_ , j_k≠ j_ ,&∀ k≠ \\ I_i_k=J_j_k,&k∈\1,2,…,K\. array . (4) Then the contraction of A and ℬB over these K mode pairs, denoted as i1,…,iK×j1,…,jKℬ,A 28.624pt ×_ [0pt][r]i_1,…,i_K 28.624pt ×_ [0pt][r]i_1,…,i_K 27.80115pt ×_ [0pt][r]i_1,…,i_K 27.80115pt ×_ [0pt][r]i_1,…,i_K×^j_1,…,j_KB, (5) is an (N+M−2K)(N+M-2K)th-order tensor, obtained by summing over the K matched indices and keeping every remaining mode of A and ℬB free. In tensor-network diagram notation, contractions are represented by connecting the legs of the contracted modes, while the free modes remain unconnected. This abstract operation corresponds to the einsum function in modern array programming libraries such as NumPy [62], PyTorch [131], and JAX [16]. For example, given ∈ℝI×K,∈ℝK×JA ^I× K,B ^K× J, the contraction over the mode pair (2,1)(2,1) yields the standard matrix product: ==2×1∈ℝI×J,ij=∑k=1Kikj.C=AB=A 4.48613pt ×_ [0pt][r]2 4.48613pt ×_ [0pt][r]2 3.90283pt ×_ [0pt][r]2 3.90283pt ×_ [0pt][r]2×^1B ^I× J, _ij= _k=1^KA_ikB_kj. (6) This example is depicted in Figure 1(f), where modes of size K are contracted and modes of size J and I remain free. Expressed using pseudocode einsum notation, the same operation is: =einsum("ik,kj→ij",,).C= einsum("ik,\ kj→ ij",A,B). (7) Mode-n product The mode-n product contracts an NNth-order tensor ∈ℝI1×I2×⋯×INA ^I_1× I_2×·s× I_N with a matrix ∈ℝJ×InB ^J× I_n over the mode pair (n1,m1)=(n,2)(n_1,m_1)=(n,2): ×n=n×2∈ℝI1×⋯×In−1×J×In+1×⋯×IN,(×n)i1…in−1jin+1…iN=∑in=1Ini1…iNj,in. gatheredA×_nB=A\ 5.44333pt ×_ [0pt][r]n 5.44333pt ×_ [0pt][r]n 4.90399pt ×_ [0pt][r]n 4.90399pt ×_ [0pt][r]n×^2\ B ^I_1×·s× I_n-1× J× I_n+1×·s× I_N,\\ (A×_nB)_i_1… i_n-1\,j\,i_n+1… i_N= _i_n=1^I_nA_i_1… i_NB_j,i_n. gathered (8) In einsum notation, this operation can be expressed as: =einsum("i1…in…iN,jin→i1…j…iN",,).C= einsum("i_1… i_n… i_N,\ ji_n→ i_1… j… i_N",A,B). (9) Intuitively, this operation multiplies every fiber (i.e., every 1D line) running along the n-th dimension of the tensor by the matrix B. If one unfolds the tensor (i.e., reshapes the tensor into a matrix) so that the n-th dimension becomes the rows of a 2D matrix, then ×n×_n reduces to standard matrix multiplication on that unfolded matrix, after which the result is folded back into a tensor of the new shape. 2.2 Tensor Decompositions This section reviews the main tensor decomposition formats relevant to this work. We cover the CP decomposition (sum of canonical rank-one terms), the Tucker decomposition (core tensor with mode-specific factors and its Tucker-1/2/3 variants), the T decomposition and its matrix variant (TTM), and the block term decomposition, which generalizes both CP and Tucker. For each model, we present the mathematical formulation, highlight key properties, and discuss uniqueness and trade-offs. Throughout this section, ∈ℝI1×I2×⋯×INX ^I_1× I_2×·s× I_N stands for an NNth-order tensor. 2.2.1 Canonical Polyadic Decomposition The canonical polyadic (CP) decomposition [72], [18], [63] approximates a tensor by a sum of rank-one outer products ≈∑r=1Rλr(1)r∘(2)r∘⋯∘(N)r,X≈ _r=1^R _ra^(1)_r ^(2)_r ·s ^(N)_r, (10) or equivalently in scalar form i1,i2,…,iN≈∑r=1Rλr(1)i1,r(2)i2,r⋯(N)iN,r,X_i_1,i_2,…,i_N≈ _r=1^R _rA^(1)_i_1,rA^(2)_i_2,r·sA^(N)_i_N,r, (11) where (n)=[1(n),2(n),…,R(n)]∈ℝIn×RA^(n)=[a^(n)_1,a^(n)_2,…,a^(n)_R] ^I_n× R is the factor matrix for mode n, and λr _r are scaling coefficients. When equality holds in Eqs. (10) and (11), the decomposition is exact and we refer to it as a CP representation of X. The minimum admissible value of R in such a representation is the tensor rank, or CP rank, of X. An exact representation is unique up to trivial scaling and column permutations under the Kruskal condition ∑n=1Nk(n)≥2R+(N−1) _n=1^Nk_A^(n)≥ 2R+(N-1), which is sufficient but not necessary, where k_A is the k-rank of the factor matrix [97], [150], [92]. CP decomposition is compact and often interpretable, but poorly behaved as an approximation problem owing to the NP-hardness of computing CP rank [64], [70]. Consequently, R has to be selected heuristically, and at a fixed R≥2R≥ 2 a best rank-R approximation may fail to exist on a set of tensors of positive volume [29]. Fitting algorithms inherit these difficulties and numerical instability at high orders [92], [21]. Additional constraints such as non-negativity, orthogonality, or sparsity are therefore often imposed to improve stability and accuracy, and to relax the uniqueness conditions [21]. 2.2.2 Tucker Decomposition The Tucker decomposition [165] admits the following model: ≈×1(1)×2(2)×3⋯×N(N),X ×_1A^(1)×_2A^(2)×_3·s×_NA^(N), (12) where ∈ℝR1×R2×⋯×RNG ^R_1× R_2×·s× R_N is an NNth-order core tensor, and each (n)∈ℝIn×RnA^(n) ^I_n× R_n is the mode-n factor matrix. In scalar form, the decomposition expresses each element of X as: i1,i2,…,iN≈∑r1=1,…,rN=1R1,…,RNr1,r2,…,rN(1)i1,r1(2)i2,r2⋯(N)iN,rN.X_i_1,i_2,…,i_N≈ _r_1=1,…,r_N=1^R_1,…,R_NG_r_1,r_2,…,r_NA^(1)_i_1,r_1A^(2)_i_2,r_2·sA^(N)_i_N,r_N. (13) If exact equality in Eqs. (12) and (13) holds, we refer to it as a Tucker representation of X, and the element-wise minimal N-tuple (R1,R2,…,RN)(R_1,R_2,…,R_N) for which such a representation exists is called the Tucker, or multilinear, rank of X (such a tuple always exists, with each RnR_n equal to the rank of the mode-n matricization of X [92]). The Tucker decomposition is non-unique since, for any invertible n∈ℝRn×RnS_n ^R_n× R_n, replacing the factor matrix (n)A^(n) with (n)nA^(n)S_n and compensating in the core with n−1S_n^-1 leaves the reconstructed tensor unchanged. Orthogonality restricts, but does not in general remove, this gauge freedom. Stronger uniqueness statements require additional assumptions, such as non-degenerate mode singular subspaces and a specified canonical convention. Common algorithms such as HOSVD [27] therefore fix a particular choice, producing orthogonal factor matrices ((n))⊤(n)=Rn(A^(n)) A^(n)=I_R_n and a core with additional constraints (see [27] for more details). As a result of this non-uniqueness, unlike in the CP decomposition, the vectors given by the Tucker factors and the values in the core are not directly interpretable. For the third-order case, there is additional terminology, indexed by the number of compressed modes. Tucker-3 denotes the standard Tucker decomposition applied to a 3rd-order tensor, where all three modes are compressed. If ∈ℝI×J×KX ^I× J× K has only two modes of large size (say, I and J), and a dimensionality reduction is required only along those modes, the corresponding decomposition is called the Tucker-2 model, ≈×1×2X ×_1A×_2B, where ∈ℝR1×R2×KG ^R_1× R_2× K, ∈ℝI×R1A ^I× R_1 and ∈ℝJ×R2B ^J× R_2; R1<IR_1<I, R2<JR_2<J are the reduced ranks; and the third mode (K) remains uncompressed. Similarly, if the tensor has only one mode of large size (say, I), and reduction is needed only in that mode, the decomposition is called the Tucker-1 model, ≈×1X ×_1A, with ∈ℝR1×J×KG ^R_1× J× K, ∈ℝI×R1A ^I× R_1, R1<IR_1<I; this reduces to a truncated SVD along the first mode. In both cases, the core tensor retains the remaining uncompressed modes in full, making these models efficient when only a subset of modes exhibits high dimensionality. 2.2.3 Tensor Train Decomposition The tensor train (T) decomposition [125] expresses a higher-order tensor ∈ℝI1×I2×⋯×INX ^I_1× I_2×·s× I_N in chain-like form as follows: ≈(1)−1×1(2)−1×1⋯−1×1(N),X ^(1) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ G^(2) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ ·s 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ G^(N), (14) where the −1-1 in the left contraction index denotes the last mode of the left tensor, following Python-style indexing. In scalar form, this is equivalent to i1,i2,…,iN≈∑α0,α1,…,αN=1R0,R1,…,RN(1)α0,i1,α1(2)α1,i2,α2⋯(N)αN−1,iN,αN,X_i_1,i_2,…,i_N≈ _ _0, _1,…, _N=1^R_0,R_1,…,R_NG^(1)_ _0,i_1, _1G^(2)_ _1,i_2, _2·sG^(N)_ _N-1,i_N, _N, (15) where each core (n)∈ℝRn−1×In×RnG^(n) ^R_n-1× I_n× R_n is 3rd-order, and the (N+1)(N+1)-tuple (R0,R1,…,RN)(R_0,R_1,…,R_N) is called the T-rank, with R0=RN=1R_0=R_N=1 always holding. This decomposition can also be expressed in terms of the core slices: i1,i2,…,iN≈(1)i1(2)i2⋯(N)iN,X_i_1,i_2,…,i_N ^(1)_i_1G^(2)_i_2·sG^(N)_i_N, (16) where (n)in=(n):,in,:∈ℝRn−1×RnG^(n)_i_n=G^(n)_:,i_n,: ^R_n-1× R_n. This form of the decomposition motivates the alternative name Matrix Product State (MPS). If equality holds in the above equations, we refer to it as a T representation of X. The T decomposition is unique only up to gauge transformations. Specifically, for any sequence of invertible matrices nS_n inserted between adjacent cores, the cores can be transformed via in(n)↦n−1in(n)n−1G^(n)_i_n _n-1G^(n)_i_nS_n^-1 without changing the reconstructed tensor. This non-uniqueness is a fundamental property of the T decomposition, although orthogonality conditions on the cores restrict this gauge freedom and yield a canonical form [21]. T decomposition is attractive for high-order tensors due to its favorable scaling with order. Specifically, the chain-like structure ensures that the total number of parameters grows only as ⁡(NIR2)O(NIR^2) (assuming uniform dimensions and ranks), which is linear in the order N, in contrast to the exponential growth of the full tensor. The Tensor Train Matrix (TTM) format [124] applies the T decomposition to matrices. More precisely, a given matrix ∈ℝI×JT ^I× J with I=∏n=1NInI= _n=1^NI_n and J=∏n=1NJnJ= _n=1^NJ_n is reshaped into a higher-order tensor ∈ℝI1×J1×⋯×IN×JNT ^I_1× J_1×…× I_N× J_N, which is then approximated via the T-like structure ≈(1)−1×1(2)−1×1⋯−1×1(N),T ^(1) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ G^(2) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ ·s 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1\ G^(N), (17) where, unlike the T decomposition, each core (n)∈ℝRn−1×In×Jn×RnG^(n) ^R_n-1× I_n× J_n× R_n is 4th-order. In scalar form, this can be expressed as i1,j1,…,iN,jN≈∑α0,α1,…,αN=1R0,R1,…,RN(1)α0,i1,j1,α1(2)α1,i2,j2,α2⋯(N)αN−1,iN,jN,αN,T_i_1,j_1,…,i_N,j_N≈ _ _0, _1,…, _N=1^R_0,R_1,…,R_NG^(1)_ _0,i_1,j_1, _1G^(2)_ _1,i_2,j_2, _2·sG^(N)_ _N-1,i_N,j_N, _N, (18) where (R0,R1,…,RN)(R_0,R_1,…,R_N) is the TTM-rank, with R0=RN=1R_0=R_N=1. As in the MPS form of T, TTM can also be expressed in terms of the matrix slices of the cores i1,j1,…,iN,jN≈(1)i1,j1(2)i2,j2⋯(N)iN,jN,T_i_1,j_1,…,i_N,j_N ^(1)_i_1,j_1G^(2)_i_2,j_2·sG^(N)_i_N,j_N, (19) where (n)in,jn=(n):,in,jn,:∈ℝRn−1×RnG^(n)_i_n,j_n=G^(n)_:,i_n,j_n,: ^R_n-1× R_n. In the TTM decomposition, each (i1,j1,…,iN,jN)(i_1,j_1,…,i_N,j_N)-th entry of T maps one-to-one to the (i1,…,iN¯,j1,…,jN¯)( i_1,…,i_N, j_1,…,j_N)-th entry of T, where i1,…,iN¯ i_1,…,i_N denotes the multi-index and equals i1,…,iN¯=i1+(i2−1)I1+⋯+(iN−1)I1…IN−1. i_1,…,i_N=i_1+(i_2-1)I_1+·s+(i_N-1)I_1… I_N-1. Since this mapping is a bijection, if equality holds in Eqs. (17), (18), and (19), we refer to it as a TTM representation of the matrix T itself. The advantage of TTM is that it natively handles matrices, representing them as a multilinear operator and thereby preserving the operator structure, in contrast to T. That is why, in quantum physics, TTM is often referred to as a Matrix Product Operator (MPO) [145]. We will refer to TTM and MPO interchangeably, preferring TTM, while using MPO when it is the terminology adopted by the paper under discussion. 2.2.4 Block Term Decomposition The block term (BT) decomposition [28] writes X as a sum of K Tucker terms ≈∑k=1K[k×1k(1)×2k(2)⋯×Nk(N)],X≈ _k=1^K [G_k×_1A_k^(1)×_2A_k^(2)·s×_NA_k^(N) ], (20) where each k∈ℝRk1×Rk2×⋯×RkNG_k ^R_k1× R_k2×·s× R_kN is a small core tensor and each k(n)∈ℝIn×RknA_k^(n) ^I_n× R_kn is a factor matrix associated with term k; the ranks (Rk1,Rk2,…,RkN)(R_k1,R_k2,…,R_kN) need not be shared across terms, so each term can have its own core size, in addition to its own core and factor matrices. In scalar form: i1,…,iN≈∑k=1K[∑l1=1,…,lN=1Rk1,…,RkN(k)l1,…,lN(k(1))i1,l1⋯(k(N))iN,lN].X_i_1,…,i_N≈ _k=1^K [ _l_1=1,…,l_N=1^R_k1,…,R_kN(G_k)_l_1,…,l_N\,(A_k^(1))_i_1,l_1·s(A_k^(N))_i_N,l_N ]. (21) When equality holds in Eqs. (20) and (21), we refer to it as a BT representation of X. BT decomposition interpolates between CP and Tucker. Setting K=1K=1 removes the outer sum and recovers the Tucker decomposition with core 1G_1 and ranks (R11,R12,…,R1N)(R_11,R_12,…,R_1N), whereas setting Rkn=1R_kn=1 for every term k and mode n instead collapses every core kG_k to a scalar λk _k, recovering the CP decomposition as a sum of K rank-one terms. Its uniqueness is more nuanced than that of the standard Tucker decomposition, as a BT decomposition carries the within-term ambiguity of Tucker in every block, and on top of that the K blocks may be permuted arbitrarily. Uniqueness beyond these indeterminacies requires additional conditions on the factor matrices, which are known for third-order tensors [28]. Its main feature is that, by choosing K and the per-term ranks (Rk1,Rk2,…,RkN)(R_k1,R_k2,…,R_kN) jointly, BT decomposition can trade off CP’s compactness against Tucker’s per-mode flexibility, at the cost of more hyperparameters and a more involved fitting procedure than either decomposition alone [28]. 2.3 Attention Mechanism Attention mechanism is a parametric neural component that enables a model to selectively concentrate on salient features of an input sequence while processing each element of an output sequence. It overcomes the bottleneck of fixed-dimensional context vectors in RNNs by allowing direct access to all previous hidden states at each decoding step [10]. Attention computes a context vector as a convex combination of values, where the coefficients are derived from a compatibility function between queries and keys. 2.3.1 Single-Head Attention The most widely adopted formulation, introduced in the Transformer architecture [168], is scaled dot-product attention. Given a query matrix ∈ℝN×dkQ ^N× d_k, a key matrix ∈ℝM×dkK ^M× d_k, and a value matrix ∈ℝM×dvV ^M× d_v, the attention output is computed as Attention(,,)=Softmax(⊤dk),Attention(Q,K,V)=Softmax ( QK d_k )V, (22) where N and M denote the sequence lengths of queries and keys/values, respectively, and dkd_k, dvd_v are the dimensionalities of keys and values. The scaling factor 1dk 1 d_k normalizes the variance of the softmax input, making the training process more stable. Eq. 22 is the general formulation, in which queries and keys may come from different sequences and differ in length. As our survey focuses on decoder-only LLMs, two restrictions apply throughout. First, all three matrices are derived from the same sequence of T tokens, so that N=M=TN=M=T; attention of this kind is called self-attention. Second, a token must not attend to its successors, which is enforced by adding a mask matrix ∈ℝT×TM ^T× T with zeros on and below the diagonal and −∞-∞ entries above it: MaskedAttention(,,)=Softmax(⊤dk+),MaskedAttention(Q,K,V)=Softmax ( QK d_k+M )V, (23) where now ,∈ℝT×dkQ,K ^T× d_k and ∈ℝT×dvV ^T× d_v. The −∞-∞ entries vanish under the softmax, so the i-th output depends only on the tokens 1,2,…,i1,2,…,i. This masked self-attention is what makes autoregressive decoding possible, and it is the variant we assume from here on. 2.3.2 Interpretation of the Attention Weights The Softmax operation is applied row-wise to produce a probability distribution over the keys for each query αij=exp⁡(i⊤j/dk)∑l=1Mexp⁡(i⊤l/dk), _ij= (q_i k_j/ d_k ) _l=1^M (q_i k_l/ d_k ), (24) where αij _ij represents the attention score assigned by the i-th query to the j-th key. The output for query i is then: i=∑j=1Mαijj.o_i= _j=1^M _ijv_j. (25) In the masked case the same expressions hold with the summations restricted to j≤ij≤ i, since the mask sets the remaining weights to zero. Each term can be understood with the following interpretation: • Queries (Q): Represent the current position for which we seek relevant context. In self-attention, queries are derived from the same input sequence as keys and values. • Keys (K): Act as labels or identifiers for each element in the input sequence. The dot product ⊤QK measures the pairwise compatibility or similarity between each query and all keys. • Values (V): Contain the actual information to be aggregated. The final output is a weighted sum of values, where the weighting is determined by the attention scores. This formulation allows the model to dynamically weigh the importance of each input element, effectively creating a context-aware representation for each position in the output sequence. 2.3.3 Multi-Head Attention To capture different types of relationships and dependencies simultaneously, the Transformer runs several attention operations in parallel, each with its own learned projection matrices. At this point the input is no longer Q, K, and V themselves, but the hidden states from which they are projected: given Q∈ℝN×dX_Q ^N× d and K,V∈ℝM×dX_K,X_V ^M× d, where d is the model dimension MultiHeadAttention(Q,K,V)=Concat(head1,…,headH)O,MultiHeadAttention(X_Q,X_K,X_V)=Concat(head_1,…,head_H)W^O, (26) where each head is computed as headi=Attention(QiQ,KiK,ViV).head_i=Attention(X_QW_i^Q,X_KW_i^K,X_VW_i^V). (27) Here, iQ∈ℝd×dkW_i^Q ^d× d_k, iK∈ℝd×dkW_i^K ^d× d_k, iV∈ℝd×dvW_i^V ^d× d_v are projection matrices for the i-th head, and O∈ℝHdv×dW^O ^Hd_v× d projects the concatenated outputs back to the model dimension. Typically, dk=dv=dHd_k=d_v= dH, where H is the number of heads. We will denote dh=dHd_h= dH as the head dimension, so dh=dk=dvd_h=d_k=d_v in standard implementations. In the decoder-only setting, the two restrictions introduced above carry over to every head. A single hidden-state matrix ∈ℝT×dX ^T× d supplies all three projections, and each head applies masked self-attention, MultiHeadAttention()=Concat(head1,…,headH)O,headi=MaskedAttention(iQ,iK,iV). gatheredMultiHeadAttention(X)=Concat(head_1,…,head_H)W^O,\\ head_i=MaskedAttention(XW_i^Q,XW_i^K,XW_i^V). gathered (28) This combination, called multi-head masked self-attention, is the standard attention mechanism of decoder-only LLMs, and we refer to it as MHA. In standard implementations dk=dv=d/Hd_k=d_v=d/H, and we write dh=d/Hd_h=d/H for this common head dimension. Some architectures let several query heads share one key-value projection. From here on, HQH_ Q denotes the number of query heads and HKVH_ KV the number of distinct key-value heads; for MHA the two coincide, HQ=HKV=H_ Q=H_ KV=H. 2.3.4 KV-cache Let us consider autoregressive decoding and observe that, within a head, the attention output at time step t is Eq. (25) restricted to j≤tj≤ t: t=∑j=1tαtjj.o_t= _j=1^t _tjv_j. (29) Of the queries, only tq_t enters this expression, through αtj _tj in Eq. (24), whereas the keys and values of all tokens 1,…,t1,…,t are required. The keys and values of the first t−1t-1 tokens have already been computed at the preceding steps, so they can be stored and reused instead of being recomputed. Before appending the current token’s KV pair, an implementation holds exactly these t−1t-1 cached entries. This technique is known as KV-cache and is widely used in LLMs to speed up inference by avoiding redundant computations, at the cost of an increased memory footprint. Reducing this footprint is the goal of multi-query attention (MQA) [148], which sets HKV=1H_ KV=1, and grouped-query attention (GQA) [3], which takes 1<HKV<HQ1<H_ KV<H_ Q. Multi-head latent attention (MLA) [30] instead caches one low-rank latent vector per token, from which the per-head keys and values are reconstructed. 2.3.5 Computational and Memory Complexity Three regimes have to be distinguished. During training the whole sequence is processed at once. At inference, the model first runs a single forward pass over the input prompt, which is called prefill, and then generates tokens one at a time, each generated token requiring one decode step. The cost of an attention layer splits into two contributions that scale differently across these regimes. The projection cost covers the computation of Q, K, V and of the output projection OW^O, while the attention cost covers the formation of ⊤QK and the aggregation of V. Which of the two dominates depends on whether the sequence is longer than the model is wide. As for memory, training and prefill have to deal with the attention matrix, which occupies ⁡(T2)O(T^2) memory if materialized fully. During a decode step, thanks to the KV-cache, only the last row of this matrix is needed, that is ⁡(T)O(T) memory. The cache itself, however, occupies MKV=2LBTHKVdhbM_ KV=2LBTH_ KVd_hb bytes, where L is the number of layers, B the batch size, and b the number of bytes per scalar. For full MHA one has HKVdh=dH_ KVd_h=d, while MQA and GQA take HKV<HQH_ KV<H_ Q. This expression does not cover MLA, which caches a latent vector of dimension dcd_c per token and layer instead of HKVH_ KV key-value heads. Table 2 collects the computational cost and the corresponding memory of one attention layer in the three regimes. Table 2: Per-sequence, per-layer complexity of one MHA block. T denotes the sequence length, and in the decode column the number of already cached tokens; d is the model dimension, HKVH_KV the number of key-value heads, and dhd_h the head dimension. Training Inference Prefill Decode (per step) Projection cost ⁡(Td2)O(Td^2) ⁡(Td2)O(Td^2) ⁡(d2)O(d^2) Attention cost ⁡(T2d)O(T^2d) ⁡(T2d)O(T^2d) ⁡(Td)O(Td) Attention-matrix memory ⁡(T2)O(T^2) ⁡(T2)O(T^2) ⁡(T)O(T) KV-cache memory – ⁡(THKVdh)O(TH_ KVd_h) ⁡(THKVdh)O(TH_ KVd_h) Training and prefill share the same asymptotics, since both process the whole sequence at once; the nature of attention allows for efficient parallel training on GPUs [168], as all pairwise interactions and heads can be computed simultaneously. A decoding step, in contrast, processes a single token against the cache, which removes one factor of T from both cost rows. Consequently, training and prefill are commonly compute-bound, while decoding is commonly memory-bound [183], given that a decoding step performs only ⁡(Td+d2)O(Td+d^2) operations but has to read the whole cache from High Bandwidth Memory (HBM). This is a typical regime, as the actual behavior depends on the batch size, the context length, the kernel implementation, the numerical precision, and the hardware. 2.4 Feed-Forward Network The feed-forward network (FFN) is a critical component of the Transformer architecture, typically accounting for approximately two-thirds of the model’s parameters [47]. In modern decoder-only LLMs, the FFN follows a gated architecture [149]: FFN⁡()=[σ⁡(⊤gate)⊙⊤up]down,FFN(x)= [σ(x W^gate) W^up ]W^down, (30) where gate,up∈ℝd×dffW^gate,W^up ^d× d_f are the gate and up projections, down∈ℝdff×dW^down ^d_f× d is the down projection, σ⁡(⋅)σ(·) denotes a non-linear activation function (commonly SiLU/Swish1Swish_1 [143] or GELU [69]), and ⊙ represents element-wise multiplication. The expansion factor dffd d_fd typically ranges from 22 to 44 in practice. The FFN is applied independently and identically at each sequence position; that is, the same weights are reused for every token, so, unlike attention, it does not mix information across positions. Two properties make this component distinctive. First, it holds a large part of the weights, yet consists of nothing but dense matrix multiplications. Since general matrix multiplication (GEMM) [52] is thoroughly optimized for modern accelerator architectures such as GPUs, an FFN is evaluated very fast despite its size, and any structured replacement has to compete with that baseline. Second, FFNs have been argued to act as key-value memories [47] and to mediate some factual associations [111], which makes their factorization relevant to interpretability and not only to efficiency. 2.5 Overview of Tensor Decompositions for LLMs Table 3 compares the five formats along the properties discussed above. CP is the most compact of them and the only one that comes with a uniqueness guarantee, which makes it the natural choice when the factors themselves are meant to carry meaning. Tucker assigns a separate rank to every mode and therefore suits tensors whose modes have distinct semantics, such as layers and heads, at the price of a core that grows with the order. T and TTM keep the parameter count linear in the order and are the usual choice for large matrices, including embedding layers, projection matrices, and adapters; TTM additionally preserves the operator structure of the matrix it replaces. BT sits in between: richer than CP, but more structured than a single large Tucker core. Table 3: Overview of the tensor decomposition formats used in language models: parameter count, main strengths and weaknesses, and representative use cases. The notation follows Section 2.2. Decomposition Number of parameters Strengths Weaknesses Natural use cases in LLMs CP [72], [18], [63] R∑nInR _nI_n Most compact; unique under mild condition, hence interpretable factors NP-hard CP-rank; a best approximation may not exist; unstable fitting Compact embedding tables [192]; factored KV-cache [189] Tucker [165] ∏nRn+∑nInRn _nR_n+ _nI_nR_n Per-mode ranks; flexible compression Core grows exponentially in the order N; gauge freedom leaves factors not directly interpretable Cross-head factor sharing [57], [103]; cross-head redundancy in the KV-cache [88] T [125] ∑nRn−1InRn _nR_n-1I_nR_n Parameters linear in the order N Gauge freedom; T-rank depends on the chosen reshaping Tensorized embedding layers [178], projection matrices [181], adapters [5] TTM [124] ∑nRn−1InJnRn _nR_n-1I_nJ_nR_n Preserves the operator structure Same gauge freedom as T Tensorized embedding layers [76], projection matrices [19], adapters [78] BT [28] ∑k[ΠnRkn+∑nInRkn] _k [ _nR_kn+ _nI_nR_kn ] Interpolates between CP compactness and Tucker flexibility More hyperparameters (K and per-term ranks) Attention rewritten as a BT representation [110] 3 Lifecycle Overview This section introduces the lifecycle taxonomy that organizes the survey. Section 3.1 defines the stages, their boundaries, and the rule used to assign methods that affect more than one stage. Section 3.2 then grounds the taxonomy with concrete examples based on LLaMA-3-8B. The lifecycle view is complemented by the component view in Section 4, which asks which Transformer objects are tensorized, and by Section 5, which formalizes the objective and reviews the literature at each stage. 3.1 The Lifecycle Taxonomy We organize tensor methods according to the lifecycle of a language model: tokenization, embeddings, pre-training, adaptation, compression, inference, and interpretability. Adaptation is represented primarily by parameter-efficient fine-tuning (PEFT), although the stage also includes other procedures that modify a pre-trained model for a downstream use. The term lifecycle identifies the point at which tensor structure is introduced, exploited, or analyzed; it does not imply that every model follows one irreversible linear sequence. For example, adaptation and compression may be repeated or applied in either order, and interpretability can be performed before, during, or after deployment. We distinguish the stages using four attributes: (i) the intervention time in the model lifecycle, (i) the primary tensor object, (i) the objective optimized by the method, and (iv) the evaluation criterion. Tokenization acts on a discrete vocabulary and segmentation rule; embeddings act on continuous token representations; pre-training learns the base model parameters; adaptation learns task- or domain-specific updates to a pre-trained model; compression transforms an already trained model to reduce its resource footprint; inference operates on runtime objects such as attention factors and the KV cache; and interpretability analyzes, reformulates, or deliberately imposes multilinear structure to expose model mechanisms. These boundaries matter, since two methods may use the same decomposition while solving different optimization problems and requiring different evidence of success. The lifecycle view and the component view are therefore orthogonal taxonomies. A method can be located by both when tensorization is introduced and what it acts on. For example, a TTM representation of an FFN weight may be learned during pre-training, fitted post hoc for compression, or used only for an adaptation update. For methods that span several stages, we assign the primary lifecycle label according to where the tensor constraint is introduced and which objective is optimized; downstream effects are recorded as cross-stage consequences. A complete method description should consequently specify at least the lifecycle stage, component or tensor object, tensorization scheme, decomposition family and ranks, training regime, and evaluation metrics. Figure 2 presents the taxonomy with representative works, while the corresponding subsections of Section 5 give formal problem definitions and broader literature coverage. TokenizationEmbeddingsPre-trainingAdaptationCompressionInferenceInterpretability §5.1 no published works §5.2 T-embeddings [76], TensorGPT [178], TN-gram [192], MorphTE [45] §5.3 Multi-Linear Attention [110], GPT-TTM [19], Hypoformer [102], Shapeshifter [129], CoMERA [181] §5.4 KronA [36], DoTA [78], LoRETTA [180], LoRTA [75], LoTR [15], T-LoRA [5], MetaTT [107], QuanTA [20], Quantum-PEFT [91], TeRA [58] §5.5 KnGPT [37], TQCompressor [2], TensorLLM [57], LeSTD [103], CompactifAI [162], Saten [152], LatentLLM [89] §5.6 TPA [189], Tucker Attention [88], DecoQuant [104], EinSort [90] §5.7 Bilinear MLPs [132], PolySAE [93], TensorLens [7] Figure 2: The seven-stage lifecycle taxonomy, with representative tensor methods at each stage. The arrows show a common model-development and deployment flow: stages may be revisited, reordered, or analyzed retrospectively. Each stage corresponds to a subsection of Section 5. Each lifecycle stage exposes a different structured object and a different notion of success. Table 4 summarizes the primary tensorization target, expected benefit, and characteristic risk at each stage. The listed risk is stage-specific; cross-cutting issues, including rank selection, tensorization-shape sensitivity, optimization stability, kernel support, and the compression-realization gap, are discussed in Section 9.2. Table 4: Lifecycle view of tensor methods: the primary tensorization target, expected benefit, and characteristic risk at each stage. Stage Tensorization target Expected benefit Key risk or cost Tokenization Vocabulary, text segmentation Vocabulary structured by token relatedness (no published work) A discrete, non-differentiable map offers no object to factorize Embeddings Token representations Compact vocabulary-scale embedding table Less expressive token representations Pre-training Weights, optimizer states Lower memory for weights and optimizer states; structural priors Optimization stability; unknown scaling behavior Adaptation (primarily PEFT) Adaptation updates Ultra-parameter-efficient fine-tuning Unmerged adapters add contractions to every forward pass Compression Pre-trained weights Memory reduction without full retraining Quality loss and increased latency Inference KV-cache, attention factors Per-token cache memory savings at long context Compression that does not translate into decoding speedup Interpretability Activations, weights Explicit multilinear structure; circuits as diagrams Imposing multilinearity requires pre-training from scratch The taxonomy also clarifies why decomposition names alone are insufficient for comparison. T and TTM, for example, occur in embeddings, pre-training, adaptation, and compression, but their roles differ: they may define a trainable base-model parameterization, constrain an update, or approximate a fixed pre-trained weight. Conversely, one lifecycle stage may admit several decomposition families with different rank notions and contraction costs. Cross-stage dependencies further complicate evaluation: ranks selected during pre-training may constrain later adaptation capacity; post-training compression changes the kernels and memory traffic observed during inference; and interpretability can study either the original or tensorized model. Comparisons should therefore be made within a shared lifecycle objective and should report both stage-specific quality and system-level consequences. 3.2 Illustrative Examples for LLaMA-3-8B To ground the abstract lifecycle stages in concrete terms, this section walks through one representative method per stage, whereas a formal description and comprehensive comparison of methods at each stage appear in Section 5. Tokenization has no example since, although it remains part of the taxonomy, no published work tensorizes this stage (Section 5.1); consequently, the walkthrough starts with embeddings. Throughout, LLaMA-3-8B [113] serves as a running example, with configuration |V|=128,000|V|=128,000, d=4096d=4096, L=32L=32, HQ=32H_ Q=32, HKV=8H_ KV=8, dff=14336d_ f=14336, dh=128d_h=128, ≈8.03≈8.03B parameters total. Not every stage has a published method targeting this exact model, which is why, where necessary, notation is adapted accordingly. 1e_12e_2⋮ _Vd1(1)G^(1)_12(1)G^(1)_23(1)G^(1)_34(1)G^(1)_4R1R_1R2R_2R3R_3d1d_1d2d_2d3d_3d4d_41(2)G^(2)_12(2)G^(2)_23(2)G^(2)_34(2)G^(2)_4R1R_1R2R_2R3R_3d1d_1d2d_2d3d_3d4d_4⋮ 1(V)G^(V)_12(V)G^(V)_23(V)G^(V)_34(V)G^(V)_4R1R_1R2R_2R3R_3d1d_1d2d_2d3d_3d4d_4xWIIJJX(1)W^(1)(2)W^(2)(3)W^(3)(4)W^(4)R1R_1R2R_2R3R_3J1J_1J2J_2J3J_3J4J_4I1I_1I2I_2I3I_3I4I_4Per-Row T Compression of the Embedding TableReshape++ T-SVDTTM Parameterization of an FFN ProjectionTTM Figure 3: Left: each embedding row ie_i is reshaped into a tensor ℰ(i)E^(i), which is then approximated by a T decomposition with cores n(i)∈ℝRn−1×dn×RnG^(i)_n ^R_n-1× d_n× R_n via T-SVD [125]. Right: a dense matrix ∈ℝI×JW ^I× J is represented by a chain of TTM cores (n)∈ℝRn−1×In×Jn×RnW^(n) ^R_n-1× I_n× J_n× R_n. The reshaped input ∈ℝI1×I2×I3×I4X ^I_1× I_2× I_3× I_4 is contracted directly with the core chain, producing a factorized output ∈ℝJ1×J2×J3×J4Y ^J_1× J_2× J_3× J_4. Embeddings. One way to tensorize an embedding table is per-row factorization: each row is reshaped into a higher-order tensor and decomposed independently. TensorGPT [178] compresses each token embedding individually. This preserves row independence, which suits the look-up table nature of the embedding layer and keeps the scheme valid under a changing vocabulary. In LLaMA-3-8B, the embedding table ∈ℝ128000×4096E ^128000× 4096 holds roughly 525M parameters, and since the model does not tie input and output embeddings [134], [81], the table alone accounts for about 6.5%6.5\% of all parameters. The feature axis factors cleanly as d=4096=84d=4096=8^4, so each row becomes a fourth-order tensor and admits a T decomposition with T-rank (1,R1,R2,R3,1)(1,R_1,R_2,R_3,1), as illustrated in Figure 3 (left). The technique is training-free, as T-SVD [125] is applied directly to the pre-trained embeddings. With R1=R2=R3=R_1=R_2=R_3=R, the whole table costs 128,000×16R(1+R)≈2R(1+R)×106128,000× 16R(1+R)≈ 2R(1+R)× 10^6 parameters. The rank governs the trade-off: taking R=1R=1 compresses the table by 128×128×, while R=4R=4 gives 12.8×12.8× and retains more of the original embeddings, since the T-SVD truncation error decreases as the ranks grow. The empirical quality cost of such compression is reported in [178]. Pre-training. Training a language model with tensor-structured weight matrices from scratch is the most direct way to use tensor structure as an inductive bias. One natural approach is to parameterize each large dense weight matrix with a TTM representation before training [19]. The dense matrix is then never materialized. Instead, the input x is reshaped into a tensor and contracted directly with the TTM cores, so the weights and their optimizer states scale with the cores. As a concrete illustration, consider applying the TTM representation to the down-projection ∈ℝ14336×4096W ^14336× 4096 of one FFN block in LLaMA-3-8B, whose input and output dimensions factor as 14336=8×14×16×814336=8× 14× 16× 8 and 4096=844096=8^4, respectively. As shown in Figure 3 (right), this yields a TTM chain of four cores 1∈ℝ1×8×8×R1,2∈ℝR1×14×8×R2,3∈ℝR2×16×8×R3,4∈ℝR3×8×8×1.W_1 ^1× 8× 8× R_1, _2 ^R_1× 14× 8× R_2, _3 ^R_2× 16× 8× R_3, _4 ^R_3× 8× 8× 1. If we take R1=R2=R3=R_1=R_2=R_3=R, the total parameter count is 128R+240R2128R+240R^2, compared with 14336×4096≈58.7M14336× 4096≈ 58.7M for the dense matrix. At rank R=64R=64, for instance, the TTM stores approximately 1M parameters, corresponding to a compression ratio of roughly 60×60× for this single projection. Chekalina et al. [19] demonstrate that training GPT-2 with such TTM-parametrized layers (at a sufficiently high rank) incurs only a small perplexity increase, suggesting that the dense parameterization is not necessary to reach this quality level WI1I2I_1I_2J1J2J_1J_2==I1I2I_1I_2J1J2J_1J_2ABI1I_1I2I_2J1J_1J2J_2Wdddhd_hH44Tucker++ sparsityGH(1)U^(1)(2)U^(2)(3)U^(3)R1R_1R2R_2R3R_3dddhd_h44Kronecker Parameterization of the Adapter UpdateTucker Compression with a Sparse Core Figure 4: Left: the weight update is parameterized as a Kronecker product of two factor matrices, ∈ℝI1×J1A ^I_1× J_1 and ∈ℝI2×J2B ^I_2× J_2, so that Δ∈ℝI1I2×J1J2 ^I_1I_2× J_1J_2. Right: the attention projections of one layer are stacked into a fourth-order tensor ∈ℝd×dh×4×HW ^d× d_h× 4× H, whose modes index, in order, the model dimension d, the head dimension dhd_h, the projection type (Q,K,V,OW^Q,W^K,W^V,W^O), and the H attention heads. The tensor is approximated by a Tucker decomposition with core ∈ℝR1×R2×R3×HG ^R_1× R_2× R_3× H and factor matrices (1)∈ℝd×R1U^(1) ^d× R_1, (2)∈ℝdh×R2U^(2) ^d_h× R_2, (3)∈ℝ4×R3U^(3) ^4× R_3, leaving the head mode uncompressed. The core is then sparsified, as indicated by the dashed diagonal. Adaptation. At the adaptation stage, the incremental weight update Δ is either parameterized directly as a tensor network or structured through tensor-algebraic operations such as the Kronecker product. Instead of adding a low-rank matrix update ⊤∈ℝI×JUV ^I× J [77] with ∈ℝI×RU ^I× R and ∈ℝJ×RV ^J× R, KronA [36] applies a Kronecker product increment 0←0+α⁡(⊗)W_0 _0+α(A ), where ∈ℝI1×J1A ^I_1× J_1, ∈ℝI2×J2B ^I_2× J_2, with scaling α, and I1I2=I_1I_2=I, J1J2=J_1J_2=J. Figure 4 (left) shows the corresponding tensor network diagram. By the standard identity rank⁡(⊗)=rank⁡()⋅rank⁡()rank(A )=rank(A)·rank(B) [50], a single Kronecker term reaches matrix rank up to min⁡(I1,J1)⋅min⁡(I2,J2) (I_1,J_1)· (I_2,J_2) at a cost of I1J1+I2J2I_1J_1+I_2J_2 parameters. A LoRA update of rank R, by contrast, has matrix rank up to R and costs R⁡(I+J)R(I+J) parameters. Thus, its rank grows linearly with the parameter budget, whereas the Kronecker form multiplies the ranks of its factors while only adding their sizes. The gap is concrete for LLaMA-3-8B. For the per-head query projection Q∈ℝ4096×128W_Q ^4096× 128, the factorization (I1,J1)=(64,16)(I_1,J_1)=(64,16), (I2,J2)=(64,8)(I_2,J_2)=(64,8) gives a single-term KronA adapter of 1,5361,536 parameters whose rank can reach 128128, the full column rank. LoRA spends 4,2244,224 parameters per unit of rank here, so the same rank costs 540540K, and even R=1R=1 costs nearly three times the whole Kronecker adapter. The comparison concerns attainable rank rather than attainable updates, since Kronecker terms span only a structured subset of the matrices of that rank, while the low-rank factorization is unconstrained. Compression. At the compression stage, tensor decompositions are applied post-hoc to pretrained weights to reduce their memory footprint, exploiting the low-rank structure that emerges in trained networks. LeSTD [103] targets the MHA block as a whole. For each head, the four projections, namely query, key, value, and output, are stacked into a third-order tensor, and the per-head tensors are stacked along a fourth mode indexing the heads. Compression is data-free and proceeds in two stages. In the first stage, a Tucker decomposition over the first three modes is computed , so all heads share the factor matrices (1),(2),(3)U^(1),U^(2),U^(3) while head-specific information stays in the corresponding slice of the core ∈ℝR1×R2×R3×HG ^R_1× R_2× R_3× H; the head mode is left uncompressed. In the second stage, the core is sparsified by an importance score. Figure 4 (right) illustrates both transformations jointly. In LLaMA-3-8B, GQA prevents a single head mode, given that HQ=32H_ Q=32 whereas HKV=8H_KV=8, so the construction could be split into QO∈ℝ4096×128×2×32W_ QO ^4096× 128× 2× 32 and KV∈ℝ4096×128×2×8W_ KV ^4096× 128× 2× 8, holding 33.633.6M and 8.48.4M parameters. The smaller K/VK/V tensor uses the lower mode-1 rank since its mode-1 factor would otherwise dominate. With 70%70\% core sparsity and Tucker ranks (1024,64,2,HQ)(1024,64,2,H_ Q) and (256,64,2,HKV)(256,64,2,H_ KV), the two shrink to 5.55.5M and 1.11.1M, a factor of 6.4×6.4× for the block. What remains is dominated by the shared factors (1)U^(1) (5.25.2M of 6.66.6M), so the benefit grows with the number of heads owing to the amortization of these factors across heads, while only the core scales with H. GQA works against this, since the K/VK/V tensor offers only eight heads to amortize over. KTTHHdhd_h==1A_11B_1RKR_ KHHdhd_h⋮ _TTB_TRKR_ KHHdhd_htK_tHHdhd_h==tA_ttB_tRKR_ KHHdhd_hKey Tensor in TPA Figure 5: Left: the full key tensor ∈ℝT×H×dhK ^T× H× d_h decomposes as stacked tK_t matrices [189]. Right: (dotted box): for a single token t, the key matrix t∈ℝH×dhK_t ^H× d_h is expressed as the contraction t⊤tA_t B_t over the rank mode RKR_ K, with t∈ℝRK×HA_t ^R_ K× H and t∈ℝRK×dhB_t ^R_ K× d_h. Scaling by the factor of 1RK 1R_ K is omitted for this illustration. Inference. The natural target for tensorization at the inference stage is the KV-cache, whose size grows linearly with sequence length. Tensor Product Attention (TPA) [189] factorizes the query, key, and value matrices of each token, so that the cached keys and values are never materialized. For token t, the key matrix t∈ℝH×dhK_t ^H× d_h is written as 1RKt⊤t 1R_ KA_t B_t with t∈ℝRK×HA_t ^R_K× H and t∈ℝRK×dhB_t ^R_K× d_h, and analogously for tQ_t and tV_t. Figure 5 shows this two-factor case; variants with more factors are also possible. Caching the factors costs (RK+RV)(H+dh)(R_K+R_V)(H+d_h) values per token instead of 2Hdh2Hd_h: for a LLaMA-3-8B-sized layer with H=32H=32 and dh=128d_h=128, that is 640640 against 8,1928,192, assuming RK=RV=2R_ K=R_ V=2; a reduction of 12.8×12.8×. The baseline here is multi-head attention since TPA subsumes both: MHA, MQA, and GQA are its non-contextual special cases. For comparison, the grouped-query attention of LLaMA-3-8B shares keys and values across HKV=8H_ KV=8 groups and therefore stores 2HKVdh=2,0482H_ KVd_h=2,048 values per token; the factored representation is 3.2×3.2× smaller still. Interpretability. Bilinear MLPs [132] drop the element-wise nonlinearity from the gated FFN, which turns the layer into an exact contraction with a third-order tensor ℬ∈ℝd×d×dB ^d× d× d (Section 4.4, Eq. (39)). For LLaMA-3-8B this means replacing all 3232 gated FFN blocks by bilinear ones. The parameter count stays the same, but SwiGLU disappears from the architecture, so the model has to be pre-trained from scratch. The resulting tensor admits three modes of analysis, differing in how much is already known about the features. Given dictionaries for both the input in∈ℝf×4096F in ^f× 4096 and the output out∈ℝf×4096F out ^f× 4096 features of a layer, obtained for example from sparse autoencoders, contracting them into all three modes rewrites ℬ∈ℝ4096×4096×4096B ^4096× 4096× 4096 in the feature basis ℬ~=ℬ×1out×2in×3in∈ℝf×f×f B=B×_1F out×_2F in×_3F in ^f× f× f, so that each entry of ℬ~ B is the interaction of a pair of input features for one output feature. Given only an output feature ∈ℝ4096u ^4096, such as an unembedding row, contracting the output mode leaves the symmetric interaction matrix =ℬ×1⊤∈ℝ4096×4096Q=B×_1u ^4096× 4096, whose eigenvectors are the input directions driving u and whose eigenvalues weight their quadratic contributions. With no features at all, the tensor could be analyzed directly, for example through a HOSVD. The three routes therefore differ in cost as well as in prerequisites, since contracting features away keeps the analysis at matrix scale, whereas the last route touches the full tensor, where d3≈69d^3≈ 69B entries per layer. 4 Tensorizing the Transformer The current literature tensorizes the Transformer module primarily in a component-wise fashion. Concretely, existing works mostly focus on a single Transformer submodule, such as the embedding layer [178], the attention mechanism [110], or the FFN [19]. This section follows the same component-wise organization, which we call the component view of the survey, given the substantial differences in structural constraints across the three submodules. First, Section 4.1 introduces a two-way classification of tensorization strategies. Then, Sections 4.2, 4.3 and 4.4 discuss each submodule in turn, covering where the tensor object arises, which tensorization strategies are structurally compatible with it, and what the key challenges are. Finally, Section 4.5 gives a structural overview. 4.1 Tensorization Strategies Tensorization strategies can be classified by whether the modes of a tensor object carry semantic meaning. If the modes have known semantics, such as position in the sequence or the attention-head index, we refer to this as mode-specific tensorization. In this case, knowledge about the semantic meaning allows us to exploit the structure of the tensor. For instance, decomposition factors can be shared across structurally similar modes, reducing memory consumption. Imposed tensorization arises when modes are assigned arbitrarily, for example by reshaping a weight matrix into a compatible shape. In this case nothing in principle restricts the choice of tensor network; instead, it may be motivated by the inductive bias one wishes to impose on the parameterization [60]. This distinction organizes the component analysis in the following sections. 4.2 Embedding Layer Three structurally distinct lines of work on embedding tensorization can be identified, differing in how the tensor object arises and which decompositions are structurally compatible. Classical embedding table. Let ∈ℝ|V|×dE ^|V|× d be the standard token embedding matrix, where each row is an independent token representation. Since the rows are independent, the matrix carries no higher-order organization. Tensorization is therefore imposed in one of two ways. Either each row is reshaped into a tensor independently, or the full matrix is treated as a tensor. While the row-wise approach preserves row independence, tensorizing the full matrix can hurt it. In both cases, however, decomposed embeddings require a chain of contractions to recover the full vector at inference time, unlike a direct table lookup. This creates a tradeoff between compression ratio and inference latency. N-gram embedding table. A structurally different tensor arises when the classical embedding layer is replaced by an n-gram embedding ngram∈ℝ|V|n×dE_ ngram ^|V|^n× d [79], where each entry represents a sequence of n tokens. After reshaping the vocabulary axis, each mode of the resulting tensor corresponds to a distinct position within the n-gram. Therefore, tensorization here is mode-specific. For n≥2n≥ 2 and any realistic vocabulary size, the full table is prohibitively large to store in memory [192]. Tensor decomposition thus serves not as a compression choice applied post-hoc, but as a structural necessity for operating with this object at all. Parameterization in the form of a tensor decomposition for the n-gram table introduces a rank hyperparameter. This rank directly controls the expressiveness of the learned representations, with higher ranks yielding lower final loss [192]. This demonstrates the key tradeoff involved in selecting the optimal rank subject to memory constraints. Morpheme-based embedding table. Instead of storing a representation for each token, one can maintain a smaller table over morphemes, i.e., the sub-token units that compose tokens. This morpheme-based matrix morphemeE_ morpheme has size |M|×d|M|× d, where |M|≪|V||M| |V|, making it substantially more compact than the standard embedding layer. Token vectors are then constructed from morpheme embeddings via tensor-algebraic operations [45], making this a mode-specific tensorization, as each mode corresponds to a morpheme position within the word. However, this introduces additional hyperparameters and may yield less expressive word representations than direct lookup. 4.3 Attention Mechanisms The attention mechanism exposes several structurally distinct tensor objects, each admitting a different tensorization rationale. We organize them into three cases. Q, K, V tensors. For hidden states ∈ℝB×T×dX ^B× T× d, MHA produces query, key, and value tensors ∈ℝB×HQ×T×dh,,∈ℝB×HKV×T×dh,Q ^B× H_ Q× T× d_h, ,V ^B× H_ KV× T× d_h, (31) where the four modes are batch, head, sequence position, and head dimension. These tensors are activations rather than static weights, so we also use the notation ⁡(x),⁡(x),⁡(x)Q(x),K(x),V(x). Since each mode carries a distinct semantic role, tensorization here is mode-specific. The primary object of interest is the KV-cache, namely the ,K,V tensors. Exploiting the structure of these tensors can significantly reduce the KV-cache memory footprint and the volume of inference-time data transfer. However, exploiting this structure remains non-trivial for the following reasons. First, tensor methods must either be compatible with efficient Attention implementations such as FlashAttention [26], or achieve a comparable level of GPU utilization. Second, tensor methods must match or exceed non-tensor counterparts such as GQA and MLA in terms of both compression ratio and model quality. Third, RoPE [155] compatibility is an additional constraint, given that RoPE applies position-dependent rotations to Q and K at inference time; consequently, any factored or compressed KV-cache representation must support these rotations. The crucial design challenges for a tensorized KV cache are therefore efficient GPU implementation, competitiveness with non-tensor schemes, and RoPE compatibility. Projection matrices. We distinguish two approaches to tensorizing the projection matrices. The first one treats each matrix Q,K,V,OW_Q,W_K,W_V,W_O independently via imposed reshaping, which is structurally identical to FFN tensorization and is discussed in Section 4.4. The second one stacks projections across layers, heads, and projection types to form a joint tensor, introducing semantic modes. The projection matrices Q,h(ℓ),K,h(ℓ),V,h(ℓ)∈ℝd×dhW_Q,h^( ),W_K,h^( ),W_V,h^( ) ^d× d_h (per head h and layer ℓ ) and O(ℓ)∈ℝHdh×dW_O^( ) ^Hd_h× d differ in shape: O(ℓ)W_O^( ) maps from the concatenated outputs of all H heads and is therefore H times wider along one dimension, which must be accounted for when forming a joint tensor. If O(ℓ)W_O^( ) is excluded, the query, key, and value projections assemble as: QKV=stackℓ,p,h⁡(p,h(ℓ))∈ℝL×3×H×d×dh,p∈Q,K,V,W_QKV=stack_ ,p,h\! (W_p,h^( ) ) ^L× 3× H× d× d_h, p∈\Q,K,V\, (32) where ℓ and h are the layer and head indices, respectively. To include O(ℓ)W_O^( ), we divide (O(ℓ))⊤(W_O^( )) into H blocks O,h(ℓ)∈ℝd×dhW_O,h^( ) ^d× d_h, each corresponding to a single head. Then: QKVO=stackℓ,p,h⁡(p,h(ℓ))∈ℝL×4×H×d×dh,p∈Q,K,V,O.W_QKVO=stack_ ,p,h\! (W_p,h^( ) ) ^L× 4× H× d× d_h, p∈\Q,K,V,O\. (33) For QKVW_QKV and QKVOW_QKVO, the layer, projection type, and head modes carry distinct semantic meaning. This mode-specific structure enables cross-layer and cross-head factor sharing, which has been widely exploited by existing methods [103]. The key tradeoff is between the compression gains from sharing and the increased coupling between modes that sharing introduces. 4.4 Feed-Forward Network The FFN matrices gateW_ gate, upW_ up, and downW_ down can be treated independently as linear maps or organized into a joint tensor. The following two paragraphs discuss these approaches, which turn out to fall on opposite sides of the mode-specific/imposed distinction. T and TTM. Under imposed tensorization, the projection matrix W is reshaped into a higher-order tensor W, which is then parametrized or decomposed using a tensor network. Both the tensorization scheme, namely the shape and order of W, and the topology of the tensor network are free design choices. The T family, represented by T and TTM, prevails in the current literature. Let I=∏n=1NInI= _n=1^NI_n and J=∏m=1MJmJ= _m=1^MJ_m. Under T tensorization, the projection ∈ℝI×JW ^I× J is reshaped into ∈ℝI1×⋯×IN×J1×⋯×JMW ^I_1×·s× I_N× J_1×·s× J_M and the input ∈ℝIx ^I into ∈ℝI1×⋯×INX ^I_1×·s× I_N. Representing W in the T format with third-order cores (1),…,(N+M)G^(1),…,G^(N+M), the output ⊤=⊤y =x W is computed entrywise as j1,…,jM=∑i1=1,…,iN=1I1,…,INi1,…,iN(1):,i1,:⋯(N):,iN,:⋅(N+1):,j1,:⋯(N+M):,jM,:,Y_j_1,…,j_M= _i_1=1,…,i_N=1^I_1,…,I_NX_i_1,…,i_NG^(1)_:,i_1,:·sG^(N)_:,i_N,:·G^(N+1)_:,j_1,:·sG^(N+M)_:,j_M,:, (34) and ∈ℝJ1×⋯×JMY ^J_1×·s× J_M is reshaped back into y. TTM tensorization instead requires the two factorizations to have the same length, so that I=∏n=1NInI= _n=1^NI_n and J=∏n=1NJnJ= _n=1^NJ_n. The projection is reshaped into ∈ℝI1×J1×⋯×IN×JNW ^I_1× J_1×·s× I_N× J_N, pairing each input mode with the corresponding output mode, and the contraction becomes j1,…,jN=∑i1=1,…,iN=1I1,…,INi1,…,iN(1):,i1,j1,:⋯(N):,iN,jN,:,Y_j_1,…,j_N= _i_1=1,…,i_N=1^I_1,…,I_NX_i_1,…,i_NG^(1)_:,i_1,j_1,:·sG^(N)_:,i_N,j_N,:, (35) where (n)∈ℝRn−1×In×Jn×RnG^(n) ^R_n-1× I_n× J_n× R_n are the TTM cores. The two formats differ in how the modes are ordered: T chains all N+MN+M modes in sequence, whereas TTM couples the n-th input mode with the n-th output mode inside a single core, halving the length of the chain. In T tensorization, the cut between cores N and N+1N+1 separates all input modes from all output modes, so the rank of the reconstructed matrix W is bounded by RNR_N [125]. For TTM tensorization, Theorem 1 of [76] states that almost every TTM whose ranks are bounded by a fixed set RR reconstructs to a matrix of full rank min⁡(I,J) (I,J). An arbitrary learned TTM projection can therefore represent a full-rank matrix in principle. In practice, however, Li et al. [102] report that TTM parameterizations with low ranks struggle to retain expressivity. The key tradeoff here is between compression and latency. A tensorized layer replaces a single GEMM with a sequential chain of small contractions, whereas modern GPUs are well optimized for large matrix computations but handle such small-size contractions poorly, so a tensorized model can cut parameters and FLOPs while still running slower than its dense counterpart; realizing an actual speedup requires kernel-level work, such as optimizing the contractions themselves and removing backend overhead [181]. The general solution to this challenge remains an open direction (Section 9). Bilinear FFN A gated FFN becomes bilinear when the element-wise nonlinearity is dropped [149]: ⊤=(⊤gate)⊙(⊤up),⊤=⊤down.z =(x W gate) (x W up), =z W down. (36) This case differs from those above: the layer is multilinear in x, so it admits an exact tensor representation. Fix a hidden coordinate α. Since (⊤)α=⊤:,α(x W)_α=x W_:,α, the two branches combine into a quadratic form: α=(⊤gate)α⋅(⊤up)α=(⊤:,αgate)(⊤:,αup)=⊤:,αgate(:,αup)⊤=⊤α.z_α=(x W gate)_α·(x W up)_α= (x W gate_:,α ) (x W up_:,α )=x W gate_:,α (W up_:,α ) x=x B_αx. (37) Note that only the symmetric part of αB_α contributes to the quadratic form, so αB_α may be replaced by 12(α+α⊤) 12(B_α+B_α ) without loss of generality. Propagating through the down-projection and summing over α gives: β=∑α=1dffα,βdown=∑α=1dff(⊤α)α,βdown=⊤(∑α=1dffα,βdownα)=⊤ℬβ,:,:,y_β= _α=1^d_ fz_αW down_α,β= _α=1^d_ f(x B_αx)W down_α,β=x ( _α=1^d_ fW down_α,βB_α )x=x B_β,:,:x, (38) so that the whole layer collapses into a single contraction ⊤=ℬ×2⊤×3⊤,y =B×_2x ×_3x , (39) where ℬ∈ℝd×d×dB ^d× d× d and each slice ℬβ,:,:B_β,:,: is symmetric. The modes of ℬB carry semantic meaning, with two input coordinates and one output coordinate, so this is the place in the FFN where tensorization is mode-specific. Pearce et al. [132] use this directly as a mechanistic interpretability tool, discovering circuits from ℬB. The practical obstacle is that gated variants remain the default in deployed LLMs [149], so obtaining a bilinear FFN requires training from scratch. With a nonlinear activation no such representation exists. The three matrices can still be stacked into: =[gate,up,(down)⊤]∈ℝ3×d×dff,S= [W gate,W up,(W down) ] ^3× d× d_ f, (40) where the modes index the projection type, the input dimension, and the hidden dimension; as in Eq. (33), these stacks can be further concatenated across layers, or across experts in MoE models, yielding tensors of higher order. Unlike ℬB in Eq. (39), however, S carries no multilinear operator meaning: its first mode does not participate in the forward pass at all. 4.5 Components Overview Table 5 provides an overview of the component-wise analysis. The Tensorization Type column follows the mode-specific/imposed classification of Section 4.1, and the Key tradeoff column names the dominant constraint. Table 5: Structural overview of tensorized Transformer components. The tensorization strategy follows the classification of Section 4.1. Component Tensor object Tensorization strategy Key tradeoff Embedding layer ∈ℝ|V|×dE ^|V|× d imposed Compression vs. lookup latency ngram∈ℝ|V|n×dE_ ngram ^|V|^n× d mode-specific Memory tractability vs. rank selection difficulty morpheme∈ℝ|M|×dE_ morpheme ^|M|× d mode-specific Memory reduction vs. expressiveness of representations Attention mechanism ∈ℝB×HQ×T×dh;,∈ℝB×HKV×T×dhQ ^B× H_ Q× T× d_h; K,V ^B× H_ KV× T× d_h mode-specific KV-cache compression vs. kernel efficiency and quality parity QKV∈ℝL×3×H×d×dhW_QKV ^L× 3× H× d× d_h, QKVO∈ℝL×4×H×d×dhW_QKVO ^L× 4× H× d× d_h mode-specific Parameter sharing vs. reduced independence FFN up,gate∈ℝd×dff,W_ up,W_ gate ^d× d_ f, down∈ℝdff×dW_ down ^d_ f× d imposed Compression vs. latency (contractions vs. GEMM) ℬ∈ℝd×d×dB ^d× d× d (bilinear) mode-specific Exact multilinear structure vs. training from scratch ∈ℝ3×d×dffS ^3× d× d_ f (stacked) mode-specific Joint representation of a layer vs. no operator semantics 5 Tensorization Across the Lifecycle This section develops the lifecycle view: it organizes applications of tensor methods for LLMs by stage, following the lifecycle taxonomy introduced in Section 3. Each subsection opens with a formal problem definition, followed by a literature overview and a summary table where coverage is sufficient. Stages with sparse literature acknowledge gaps directly. 5.1 Tokenization The tokenization stage maps raw text to a sequence of subword units drawn from a fixed vocabulary VV, typically produced by byte-pair encoding [146] or the unigram language model [99], commonly applied through SentencePiece [98]. The vocabulary is a set of discrete symbols and the tokenizer is a non-differentiable map, which makes this stage an unusual target for tensor methods, as no continuous multilinear object exists until the tokens are embedded. To our knowledge, no published work applies tensor methods to the tokenization procedure itself. The closest tensor work starts one step later, at the representation of tokens: MorphTE [45] injects morphological structure into the embedding table, and TN-gram [192] tensorizes n-gram embeddings (Section 5.2). Both take the vocabulary as given. The tokenizer has been made stochastic [99], [135] and even learnable [159], but not tensorized. We outline what tensorization could offer at this stage in Section 9. 5.2 Embeddings Problem definition. Tensorizing the embedding layer can pursue different optimization objectives mirroring the two objectives for pre-training (see Section 5.3) and compression (see Section 5.5), namely training the embedding matrix from scratch or applying a training-free compression scheme to a pretrained embedding matrix. Both cases affect only the embedding layer while leaving the rest of the model unchanged, unlike the methods discussed in Section 5.3 and Section 5.5. This two-way classification is orthogonal to the table-type classification from Section 4.2, since any table type may appear under either objective. Literature overview. Before LLMs, tensor methods in NLP were primarily used for representation learning of linguistic units. Early works constructed representations for subject-verb-object triplets via Tucker-inspired factorization [167] and tensor-algebraic operations on word vectors [115]. A second line of work emerged from reformulating the embedding-learning problem as matrix factorization of co-occurrence matrices. This motivated tensor extensions that obtain richer representations by aggregating additional information, such as sentiment [141] or frame-semantic role annotations [41], into a higher-order co-occurrence tensor, which was then factorized to produce word representations. However, with the development of LLMs, the objective shifted from tensor-based enrichment to compression and parameterization of learned embedding table. In modern LLMs, the dominant approach for tensorizing the classical embedding layer (see Section 4.2) relies on T-family decompositions. Hrinchuk et al. [76] replace the embedding layer with a TTM representation and trains the model from scratch. This parameterization achieves a substantial compression for Transformer-XL [25] on the language modeling task [76]. However, this approach requires training from zero in this parameterization, and, as discussed in Section 4.2, row independence is violated: token ordering may affect representation quality [76]. TensorGPT [178] treats each row of the embedding layer independently; specifically, each row is reshaped and compressed via T-SVD [125] applied to an already pretrained model (see Figure 3, left). This eliminates the need for retraining but may yield less expressive representations and add inference latency. Beyond the standard embedding table, tensor methods have been applied to two structurally different settings: n-gram embedding table and morpheme-based embedding table (see Section 4.2). In the first case, TN-gram [192] parameterizes the n-gram embedding table as a CP representation, reducing the parameter count from exponential ⁡(|V|nd)O(|V|^nd) in n to linear ⁡(Rn|V|+Rd)O(Rn|V|+Rd). At the same time, each CP factor corresponds to a specific position within the n-gram, making the decomposition interpretable. However, as with T-Embedding, TN-gram must be trained from scratch since the CP parameterization is intrinsic to the n-gram table. In the second case, MorphTE [45] takes a different direction: instead of compressing a word-level table, it maintains a small morpheme embedding table and constructs word representations as Kronecker products (referred to as tensor products in the original paper) of morpheme vectors. This drastically reduces embedding table size but, again, requires training from scratch to adapt the model to such embeddings. Comparison. Table 6 compares the embedding methods discussed above along structural lines. Each method targets a structurally different embedding table, each serving a different representational purpose and application area. Therefore, these methods are not fairly comparable, which is why no compression-ratio or perplexity columns are included here. The table identifies the decomposition applied, the embedding table type, and whether a training stage is required (training-free vs. from-scratch training). Table 6: Structural comparison of tensor methods for embeddings stage. Method Decomposition Embedding table Training T-embeddings [76] TTM Classical from scratch TensorGPT [178] T row-wise Classical training-free TN-gram [192] CP N-gram from-scratch MorphTE [45] Kronecker product of morpheme embeddings Morpheme-based from scratch 5.3 Pre-training Problem definition. The optimization goal for tensorized pre-training can be stated as the following constrained optimization problem: min∈(R)⁡ℒLM(,LM), _W∈ TN(R)L_ LM(W;\ D_ LM), (41) s.t.m⁡()≤M,c⁡()≤C s.t.~m(W)≤ M,\ c(W)≤ C Here, W denotes the tensorized weights of the language model, (R) TN(R) is the parametric family of tensor networks given a set of ranks RR, and ℒLML_ LM is the language-modelling loss on a corpus LMD_ LM. The budget uses two functions to reflect the different effects of tensorization: m⁡(⋅)m(·) measures memory, typically the stored parameters together with the peak footprint reached during training, whereas c⁡(⋅)c(·) measures computation, either the contraction FLOPs or the latency they induce on the target hardware. The pre-training objective, in contrast to the compression objective (see Section 5.5), has no reference dense W, so the tensorized weights W are trained from scratch. And, unlike the embedding stage (see Section 5.2), tensorization here can target any part of the Transformer. Literature overview. Besides Transformers, tensor methods were already used in NLP. Probabilistic language models were built directly with tensor decompositions: Tensor Space Language Model (TSLM) [185] represents sentences as a tensor product of word vectors and then expresses the conditional probability of the next token as the inner product of two tensors. Tensor Train Language Model (TTLM) [156] continues this line and parameterizes the language model via T decomposition. Several works reformulate the attention mechanism itself as a tensor network in order to mitigate the complexity disadvantages of the Transformer architecture. Ma et al. [110] rewrite it entirely in BT form (Eq. (20)): queries, keys and values act as the factor matrices (1)=A^(1)=Q, (2)=A^(2)=K, (3)=A^(3)=V of a third-order tensor, shared across all K blocks, which differ only in their diagonal cores k∈ℝR×R×RG_k ^R× R× R; each block term then plays the role of one head in classical MHA, with the head count replaced by K and the head dimension by the rank R. The construction buys factor sharing and low-rank cores at the price of non-trivial causal masking, an issue that the authors leave largely unexplained and that should be borne in mind when reading the perplexities in Table 7. TensorCoder [187] pays a similar price: its dimension-wise attention lowers the complexity from ⁡(T2d)O(T^2d) to ⁡(Td2)O(Td^2), but only in the encoder, as the causal mask restores ⁡(T2d2)O(T^2d^2) in the decoder. Overall, tensor networks can change attention structurally, with the autoregressive mask as the binding constraint. A larger group of methods leaves the attention mechanism intact and tensorizes the weight projections separately instead, following earlier work on tensorizing neural networks [120]. GPT-TTM [19] applies this idea to an LLM, replacing every projection matrix in the dense layers by a TTM parameterization (Section 4.4). The difficulty is the one described there: a low TTM-rank is the regime in which the format is worth using, and also the regime in which its full-rank expressivity is hardest to retain. Hypoformer [102] addresses this with a hybrid layer =[denseTTM]W= bmatrixW_ dense\\ W_ TTM bmatrix with dense∈ℝαI×JW_ dense ^α I× J and TTM∈ℝ(1−α)I×JW_ TTM ^(1-α)I× J, in which a low-rank TTM branch propagates in parallel with a dense one and the input vector is divided between the two in a controlled proportion (α / 1−α1-α), combining the compression power of the tensor network with the full-rank expressivity of the dense matrix. Both methods, however, fix the ranks before training, so meeting a given budget requires a costly search. CoMERA [181], instead, makes the ranks themselves part of the optimization: it interleaves the T parameterization with diagonal factors =(1)−1×1(1)−1×1(2)−1×1…−1×1(N−1)−1×1(N)W=G^(1) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1~D^(1) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1~G^(2) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1~… 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1~D^(N-1) 7.18059pt ×_ [0pt][r]-1 7.18059pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1 6.26398pt ×_ [0pt][r]-1×^1~G^(N) so that, within a multi-objective optimization problem, the matrices (n)∈ℝRn×RnD^(n) ^R_n× R_n allow to adjust the T-rank dynamically during training. The authors additionally propose implementations of their framework optimized via CUDA Graph. The T family is not the only choice; Shapeshifter [129] parameterizes all matrices as a sum of Kronecker products instead, although there the number of terms is again fixed in advance. Comparison. Table 7 compares tensorized pre-training methods restricted to those that report perplexity on a language-modeling benchmark. Only multi-linear attention [110] and GPT-TTM [19] do this, with both appearing at two validated scales. The two are not equally solid, however. Multi-linear attention [110] is compared against a Transformer-XL baseline of different depth and width, so its compression may reflect the change of configuration. Shapeshifter [129], Hypoformer [102], and CoMERA [181] are omitted, as they report no language-modeling validation at all, making their compression claims incomparable with the LM pre-training-scale evidence. This distinction matters here, given that scaling behavior is a central concern of the pre-training stage in terms of Scaling Laws [85]; it is therefore necessary to know how far the empirical evidence for a method extends before drawing conclusions. Table 7: Comparison of tensorized pre-training methods. Column definitions are as follows: Decomposition denotes the underlying decomposition and its rank R; Baseline denotes the model against which Δ PPL is calculated; Params gives the number of parameters in the tensorized model; and CR denotes the compression ratio, CR=# parametersbaseline# parameterstensorized CR= \# parameters_ baseline\# parameters_ tensorized. We calculate ΔPPL=perplexitytensorized−perplexitybaselineperplexitybaseline×100 PPL= perplexity_ tensorized-perplexity_ baselineperplexity_ baseline× 100, and ↓ indicates that lower values are better. Method Decomposition Baseline Train / Test dataset #\#Params CR Δ PPL (↓ ) Multi-linear attention (core 2) [110] BT Transformer-XL Large (257M) [25] WikiText-103 / WikiText-103 85M 3.01* 3.28 Transformer-XL Large (0.8B) One-Billion / One-Billion 160M 5.0* -10.55 GPT-TTM [19] TTM (R=64) GPT-2smallGPT-2_ small [139] WikiText-103 / WikiText-103 84M 1.49 3.02 TTM (R=72) GPT-2mediumGPT-2_ medium [139] OpenWebText / WikiText-103 218M 1.63 50.05 * The ratio is computed against a baseline of different depth and width; see the discussion in the text. 5.4 Adaptation Problem definition. Tensorized PEFT generalizes the idea of LoRA [77] by parametrizing the update as a low-rank tensor network. The general optimization problem for tensorized adaptation is minΔ∈(R)⁡ℒtask(0+Δ,task), _ ∈ TN(R)L_ task(W_0+ ;\ D_ task), (42) s.t.m⁡(Δ)≤M,c⁡(Δ)≤C s.t.~m( )≤ M,\ c( )≤ C where 0W_0 is a frozen pretrained backbone, Δ is constrained to a tensor family (R) TN(R) for a given set of ranks RR, and taskD_ task is a task-specific dataset. The memory budget m⁡(⋅)m(·) is the same as in Section 5.3, only applied to the update. The compute budget c⁡(⋅)c(·) depends on how the adapter is deployed. If it is merged into 0W_0 after training, inference costs no more than the backbone alone. If it cannot be merged, as in multi-task serving where many adapters share one backbone, its contractions run in every forward pass and have to be counted. The tensorization of Δ can take one of two forms. In the local approach, each update is reshaped and parameterized as a tensor separately. In the global approach, the updates are aggregated across all matrices into a single tensor, similar to the tensors in (32) and (40), and then parameterized jointly. Literature overview. The first line of work follows the local tensorization approach, meaning that each update matrix is tensorized independently. KronA [36] replaces low-rank matrix factorization with a Kronecker rank-1 factorization. As discussed in Section 3.2, the resulting update matrix can have full rank. DoTA [78] represents each update matrix using a MPO initialized from the MPO decomposition of the pretrained weight 0∈ℝI×JW_0 ^I× J, while keeping the residual matrix res=0−MPO⁡(0)W_ res=W_0-MPO(W_0) frozen. Moreover, analogously to QLoRA [34], QDoTA [78] was proposed: it trains updates in bFloat16 while casting the frozen parts to NormalFloat4 [34]. LoRETTA [180] provides two variants using T parameterization of matrices: LoRETTAadpLoRETTA_adp incorporates a T-parameterized Houlsby-inspired adapter [74] after each attention and FFN block, whereas LoRETTArepLoRETTA_rep makes a LoRA-like low-rank factorized incremental update Δ= =BA, where the matrices ∈ℝI×RB ^I× R and ∈ℝR×JA ^R× J are T-parameterized. T-LoRA [5] continues this line of T parameterization, but instead of parametrizing B and A separately, it parameterizes Δ directly in T form. Overall, all of the above methods adapt each weight matrix independently. There exists a small cluster of works inspired by the quantum tensor network literature. QuanTA [20] reshapes the update matrix and then parameterizes it as a quantum-circuit tensor network constructed from a set of small ”gates,” i.e., tensors that connect only two axes. The main advantage is that QuanTA’s tensor-network representation is not restricted to low rank, since such an update can have high rank [20]. A related but structurally different method, Quantum-PEFT [91], instead follows the AdaLoRA [186] SVD-like update parameterization: Δ=⊤ =U where ∈ℝI×RU ^I× R, ∈ℝR×R ^R× R, and ∈ℝJ×RV ^J× R. However, unlike AdaLoRA, where orthogonality is imposed inexactly through a regularizer, Quantum-PEFT makes U and V orthogonal by construction through the Pauli parameterization. Their parameter count then grows logarithmically with the matrix dimension rather than linearly as in LoRA, while the diagonal factor contributes R parameters. This yields comparable downstream performance at a substantially smaller parameter budget. The second line of work follows the global approach, where all update weights are aggregated into a single tensor, which is then parameterized using some decomposition. LoRTA [75] suggests a CP parameterization for all weights, composed into a 5th-order tensor Δ∈ℝd×dh×H×L×M ^d× d_h× H× L× M, where the modes represent the model and head dimensions, the number of heads, the layer index, and the projection type (query, key, value, output). LoTR [15] instead proposes to compose a third-order tensor Δ∈ℝd×d×N ^d× d× N by stacking all update matrices one by one along the third mode. Then LoTR applies a Tucker-2 parameterization to this tensor: Δ=×1×2 =G×_1A×_2B, where the factors A and B are shared across all stacked projection updates, which helps make fine-tuning more parameter-efficient. TeRA [58] reaches the same global principle by a different path, starting by reshaping and Tucker-decomposing each update matrix as an N-th order tensor on its own and then constraining the resulting core and factors to be shared across all update matrices. The shared core and factors are randomly initialized and kept frozen, while only diagonal per-update additional factors are trained. Despite this per-matrix starting point, the frozen shared core and factors are what place TeRA in this cluster rather than the local one. MetaTT [107] constructs a 4-th order update tensor Δ∈ℝd×L×M×dh ^d× L× M× d_h, where d and dhd_h are the model and head dimensions, L is the number of layers, and M is the projection type. MetaTT then utilizes a T parameterization for this tensor. Moreover, this method proposes an adaptive rank optimization strategy [107]. Comparison. Table 8 compares the tensorized PEFT methods discussed above. For each method, the table first reports the underlying decomposition and whether it follows the local or global tensorization principle introduced earlier. Then, the table reports parameter and accuracy results as published in the original papers, for one or several models. Although they give a general idea, these numbers are still not fully comparable across methods: the models are roughly similar but not always identical, and the relative-accuracy reported is itself an average computed by each paper over its own benchmark suite, which sometimes differs from method to method. Table 8: Comparison of tensorized PEFT methods, %Params=parametersmethodparametersbaseline×100\% Params= parameters_ methodparameters_ baseline× 100, Rel. Perf. = scoremethodscorebaseline×100 score_ methodscore_ baseline× 100, where baseline is the full fine-tuning. scoremethodscore_ method and scorebaselinescore_ baseline are the average metric across benchmarks reported in the corresponding paper. R denotes the rank of the decomposition, if several ranks are reported. ↑ indicates that higher values are better. Method Decomposition Type Model %\% Params Rel. Perf. (↑ ) KronA [36] Kronecker Local T5 [140] 0.07 100.57 DoTA [78] MPO Local LLaMA-2-7B [164] 0.15 100.49 LLaMA-3-8B [113] 0.06 99.43 LoRETTArepLoRETTA_ rep [180] T Local LLaMA-2-7B § [164] 0.0076 92.31 T-LoRA [5] T Local LLaMA-2-7B [164] 0.0015 106.87 LLaMA3-8B [113] 0.0025 107.56* QuanTA [20] Quantum circuit Local LLaMA-2-7B [164] 0.041 100.49 LLaMA-3-8B [164] 0.035 106.19* DeBERTaV3baseDeBERTaV3_ base† [67] 0.051 99.56 Quantum-PEFT [91] SVD-like Local DeBERTaV3baseDeBERTaV3_ base [67] 0.007 100.57 LoRTA(R=8) [75] CP Global LLaMA-2-7B [164] 0.00043 99.22 LoRTA(R=16) [75] RoBERTabaseRoBERTa_ base [105] 0.012 102.15* LoTR [15] Tucker Global RoBERTabaseRoBERTa_ base [105] 0.26 91.44 RoBERTalargeRoBERTa_ large [105] 0.092 93.36 TeRA [58] Tucker Global LLaMA-2-7B [164] 0.0039 94.13 LLaMA-3-8B [113] 0.0033 98.46 MetaTT(R=16) [107] T Global LLaMA-2-7B [164] 0.0021 96.86* MetaTT(R=64) [107] LLaMA-2-7B [164] 0.098 98.86* MetaTT(R=64) [107] RoBERTabaseRoBERTa_ base [105] 0.125 93.35 § These results are taken from [5] * Calculated with respect to the best LoRA score instead of full FT † These results are taken from [91] 5.5 Compression Problem definition. Post-training compression can be viewed as a two-stage process. The first stage minimizes the reconstruction error between the original and the factorized weights ∗=arg⁡min∈(R)⁡‖−‖F2,Z^*= _Z∈ TN(R)\|W-Z\|_F^2, (43) where W collects the pretrained weights targeted for compression, Z is their tensorized approximation, and (R) TN(R) is the set of tensors admitting a given decomposition with rank set RR; the minimizer ∗Z^* is the compressed model. It is followed by an optional second stage, hereafter called healing, that retrains the factorized model on a healing dataset, initialized with the factorized weights ∗Z^* obtained from Eq. (43) min∈(R)⁡ℒheal(,heal),(0)=∗. _Z∈ TN(R)L_ heal(Z;\ D_ heal), ^(0)=Z^*. (44) Healing can range from short retraining on some dataset [162] to a lightweight knowledge distillation approach [157], so the healing loss ℒhealL_ heal is not restricted to the pre-training language modeling loss. The healing stage is not a cosmetic correction, as Zagitov et al. [184] give theoretical and empirical evidence that the reconstruction objective of Eq. (43) is misaligned with functional preservation. Consequently, a Frobenius-optimal ∗Z^* need not be a good solution in the operator sense. In their experiments, even a lightweight LoRA-style repair recovers only part of the lost quality, which suggests that healing should be a full post-compression training stage. Alternatively, the first stage can minimize the error on activations over a calibration dataset [89] ∗=arg⁡min⁡∑i∈I∈(R)⁡λix∼cal‖(i−i)i‖22.Z^*= _Z∈ TN(R) _i _i\,E_x _ cal \|(W_i-Z_i)\,x_i \|_2^2. (45) Here II indexes the components being compressed, including individual projections or projection groups that a method factorizes jointly, and iW_i, iZ_i are the corresponding slices of W and Z. The activation ix_i is the input induced at component i by a calibration sample x∼calx _ cal in the uncompressed model, and calD_ cal is the calibration dataset. The weights λi>0 _i>0 make the terms proportional, as components could differ in output dimension and in activation scale, so an unweighted sum would let the largest of them dominate. Literature overview. One line of work uses Kronecker decomposition, which yields a high compression ratio while keeping the factorized matrices high-rank. Initially, Tahaei et al. [157] applied a Kronecker rank-1 decomposition of each projection to the BERT model [35], followed by a KD healing stage. Continuing this line, KnGPT [37] uses the same approach to compress the GPT-2 model [139], achieving the same compression ratio as DistilGPT2 [144] with better perplexity and substantially lower training time. TQCompressor [2] then adds column and row permutations to the Kronecker parameterization, which improves the compression ratio while matching quality on benchmarks. Several works explore Tucker decomposition for model compression. TRAWL [109] was among the first to apply a tensor decomposition to a tensor aggregated from projection matrices, motivated not only by compression but also by noise reduction. TRAWL explores CP and Tucker decomposition applied to a third-order tensor formed by stacking projection matrices one by one. Following this line, TensorLLM [57] proposes operating on a fourth-order tensor that separates the head dimension. Tucker is then applied to this tensor, leaving the head mode uncompressed; it can be viewed as a per-head full Tucker decomposition with shared factors and distinct third-order cores. This yields a high compression ratio with better accuracy than the uncompressed model, for GPT-J [173] and LLaMA2 [164]. However, an accurate decomposition requires a high Tucker rank, so the dense core becomes a new memory bottleneck. To mitigate this issue, LeSTD [103] proposes a two-stage scheme: (1) finding the cores and shared factors as in TensorLLM, and (2) sparsifying the fourth-order core by pruning unimportant values. The authors also propose an algorithm for inference without reconstruction. It yields better accuracy and matches or exceeds the throughput of TensorLLM and of non-tensor counterparts. Other decompositions were explored as well. CompactifAI [162] applies MPO decomposition to the pretrained weights of LLaMA2-7B [164], then heals on several datasets in less than one epoch. The authors systematically compare this approach against, and in combination with, quantization techniques. Saten [152] also uses sparsity, but in a different way from LeSTD [103]. Saten first decomposes the weight matrices with error-based T-SVD [125] and then constructs an additive sparse correction to the residual −TW-W_ T. LatentLLM [89] follows the activation-aware optimization objective in Eq. (45), building the compression from a joint preconditioned factorization of three projection groups: query with key, value with output, and up with down. Validation across all scales of OPT [188] shows that LatentLLM outperforms activation-aware non-tensor methods. Comparison. Table 9 compares and summarizes the compression methods discussed above. The Model column identifies the model used for validation in the original paper; where a method reports results on multiple models, we select the one closest to the 6-7B scale, and where no such model is reported, we use whatever model the paper provides. If the table contains two rows for the same method, one corresponds to the more aggressive compression (small) and the other to the softest compression (large). The table is meant to give a general picture, taking into account different benchmark sets, some training-free, some with a healing stage, some fine-tuned per task. Consequently, conclusions drawn from this table carry real limitations and should be read with these differences in mind. Table 9: Comparison of tensorized compression methods. Model - compression target, Healing - dataset used for healing or calibration (”-” indicates no healing), CR - compression ratio, CR=# parametersuncompressed# parameterscompressed CR= \# parameters_ uncompressed\# parameters_ compressed. For a single dataset/benchmark: ΔPPL=perplexitycompressed−perplexityuncompressedperplexityuncompressed×100 PPL= perplexity_ compressed-perplexity_ uncompressedperplexity_ uncompressed× 100, Rel. Perf.=scorecompressedscoreuncompressed×100Rel. Perf.= score_ compressedscore_ uncompressed× 100. Where several datasets/benchmarks are listed, the resulting metrics are computed as an average. ↓ , ↑ indicate that lower or higher values are better respectively. Method Model Healing PPL dataset / Benchmark CR Δ PPL (↓ ) / Rel. Perf. (↑ ) KnGPT [37] GPT-2smallGPT-2_ small [139] OpenWebText WikiText-103† / GLUE 1.51.5 9.04 / 99.82 TQ- -Compressor [2] GPT-2smallGPT-2_ small [139] OpenWebText WikiText-103, WikiText-2, Lambada / - 1.51.5 37.31 / - TensorLLM [57] LLaMA2-7B [164] - - / HotPotQA, FEVER, Bios Profession, BigBenchWikidataQA 3.543.54 - - 5.815.81* - / 106.92 LeSTD [103] (large) GPT-J [173] - WikiText-2 / MathQA, GSM8K, TruthfulQA 1.25 0.68 / 88.30 LeSTD [103] (small) WikiText-2 / MathQA, GSM8K, TruthfulQA 2.5 459.37 / 43.78 CompactifAI [162] LLaMA2-7B (float-16) [164] Ultrachat, Alpaca, OpenHermess - / MMLU, HellaSwag, BoolQ, TriviaQ, GSM8K 3.34§ - / 97.25 Saten(2:4) [152] LLaMA3.2-1B [112] Fine-tuning on each task - / BoolQ, CB, WSC, COPA 1.59 - / 101.33 Saten(u) [152] - / BoolQ, CB, WSC, COPA 3.26 - / 88.34 LatentLLM [89] (large) OPT-6.7B [188] C4 for calibration WikiText-2, PTB, C4 / - 1.11 0.81 / - LatentLLM [89] (small) WikiText-2, PTB, C4 / - 1.67 54.39 / - † Reported perplexity is computed on the WikiText-103 test set, but the model was fine-tuned on the WikiText-103 training set [37]. * Compression ratio is reported only for the MHA parameters, not for the entire model [57]. § Note that the compression ratio accounts only parameter counts, not the memory footprint, which is also affected by quantization [162]. 5.6 Inference Problem definition. Given the KV-cache tensor produced by MHA for an input x: ⁡(x)=(⁡(x),⁡(x))∈ℝ2×L×B×HKV×T×dhC(x)= (K(x),V(x) ) ^2× L× B× H_ KV× T× d_h, the objective for the inference stage can, roughly speaking, be stated as follows: minΠ⁡x∼[μ⁡(⁡(x),Π⁡(x))] _ E_x [μ (C(x), (x) ) ] (46) s.t.memory/latency budget, s.t. /latency budget, where Π⁡(⋅) (·) is such that, for every x drawn from the input distribution D, Π⁡(x)∈(R) (x)∈ TN(R) is a tensor-network approximation of the KV-cache, with (R) TN(R) denoting the tensor network family given a set of ranks RR; μ⁡(⋅,⋅)μ(·,·) is a measure of error, such as reconstruction error on the cache tensor or an activation-aware loss, analogous to Section 5.5. We do not attempt a fully rigorous formalization here, as no work poses or solves this joint constrained problem explicitly; the common implicit goal across these methods is instead to find an efficient tensor parameterization or decomposition of the KV-cache that substantially reduces memory footprint while preserving quality and maintaining or improving inference throughput. Literature overview. Several works explore different tensor-based approaches to compress the KV-cache. TPA [189] parameterizes per-token slices t,t,t∈ℝH×dhQ_t,K_t,V_t ^H× d_h as a sum of context-aware tensor products t=1RK∑r=1RKrK(xt)∘rK(xt),K_t= 1R_ K _r=1^R_ Ka_r K(x_t) _r K(x_t), (47) where rK(xt)∈ℝH,rK(xt)∈ℝdha_r K(x_t) ^H,b_r K(x_t) ^d_h are context-dependent transformations. This is equivalent to 1RKtK(xt)⊤tK(xt) 1R_KA K_t(x_t) B K_t(x_t) with t(xt)=[1K(xt),…,RKK(xt)]∈ℝRK×HA_t(x_t)=[a K_1(x_t),…,a K_R_ K(x_t)] ^R_ K× H and t(xt)=[1K(xt),…,RKK(xt)]∈ℝRK×dhB_t(x_t)=[b_1 K(x_t),…,b_R_ K K(x_t)] ^R_ K× d_h, which is a low-rank factorization of the key matrix, as discussed in Section 3.2 for inference stage. The same is done for query and values. This results in (RK+RV)(H+dh)(R_K+R_V)(H+d_h) memory per token instead of 2Hdh2Hd_h for MHA. Tucker Attention [88] starts from the observation that cross-head redundancy is left unexploited by non-tensor KV-cache methods (MQA, GQA, MLA). Inspired by this, the attention computation is first rewritten in tensor form, and the weight tensors are then parameterized via Tucker decomposition, letting the method exploit that cross-head redundancy. The per-token memory footprint is reduced to 2R, where R is a rank hyperparameter. DecoQuant [104] observes that MPO-decomposing a weight matrix into a large and a small core yields a narrow weight distribution in the large core and a wider one in the small core. This motivates a mixed-precision quantization scheme, where the large core is quantized using low precision and the small core is quantized using high precision. EinSort [90] explores the idea of permutations, motivated by the finding that sorting the elements of a matrix yields a substantially better low-rank approximation. Building on this, the framework proposes using an invertible permutation π⁡[⋅]π[·] before compression and, after compression, restoring the initial order via π−1π^-1. This compression framework is applied to the KV-cache (note that such an approach could also be applied in the weight-compression scenario). This poses a new challenge, since the permutation order needs to be stored for each token, which can be costly. Comparison. Table 10 compares the tensorized KV-cache compression methods discussed above along structural dimensions. PPL, benchmark score, and latency are not included, given the limited comparability of the reported numbers across methods; for example, TPA conducts experiments matching the baseline’s parameter count, while Tucker Attention actually reduces the parameter count. The table instead gives a structural comparison along two axes: the underlying decomposition family and whether the method is applied post-hoc without additional training (training-free) or requires training the entire model from scratch. Table 10: Structural comparison of tensorized inference methods. Method Decomposition Training TPA [189] CP from scratch Tucker Attention [88] Tucker from scratch DecoQuant [104] MPO + quantization training-free EinSort [90] Arbitrary training-free 5.7 Interpretability Problem definition. Interpretability operates under a structurally different goal than the rest of the survey. The preceding stages follow the implicit paradigm of making the model as capable as possible while minimizing memory, training, and inference costs. Interpretability instead aims to understand and explain existing models. Due to the vagueness of this goal we do not state a formal objective here. Literature overview. After tokenization, interpretability is the least developed stage in this survey, and nearly all of the work in it is recent. The contribution of tensor theory here is not efficiency but description, providing a way to write down what a model computes so that its structure becomes visible. In its weakest form, this contribution is purely notational; Taylor [160] rewrites the mathematical framework of [39] in graphical tensor notation, up to the construction of the induction head, so that circuits become diagrams. The three substantive works on interpretability differ in how they treat multilinearity. Bilinear MLPs [132] impose it. A Gated Linear Unit without its element-wise nonlinearity is described exactly by a third-order weight tensor, so it can be analyzed from the weights alone. We discussed this bilinear construction in Section 4.4 and provide it as the example in Section 3.2. PolySAE [93] instead adds multilinearity where it was missing, since standard sparse autoencoders [24] reconstruct activations as linear combinations of dictionary atoms and therefore cannot separate a compound concept from the co-occurrence of its parts. PolySAE keeps the linear encoder but extends the decoder with quadratic and cubic terms, made tractable by a low-rank tensor factorization on a shared projection subspace, at roughly 3%3\% extra parameters on GPT-2 small. TensorLens [7] reformulates the model as a whole: the entire Transformer stack, including attention, FFNs, normalization, and residual connections, becomes a single input-dependent operator represented as a fourth-order tensor ∈ℝT×d×T×dT ^T× d× T× d. This tensor acts as a generalized attention matrix by replacing both the individual attention maps and their heuristic aggregations across heads and layers, thereby providing a global description of the model. Comparison. The described interpretability works analyze different objects, so we do not provide a comparison table for them. 6 Relationship with Neighboring Efficiency Methods Table 11: Comparison of tensor methods with neighboring LLM-efficiency techniques. The table emphasizes interaction and complementarity. Technique Target Typical advantages Typical limitations How tensors interact Low-rank Adapters [77, 186] Individual matrices or updates Simple, mature, easy to implement Limited to two-dimensional structure Tensorized PEFT generalizes low-rank structure across layers, modules and heads (Section 5.4) Quantization [44, 33, 106] Numerical precision Strong memory reduction, hardware support Quality degrades at low bit-widths; post-training methods often need calibration data Tensor factors can be quantized to different precisions [104] Pruning / sparsity [43, 6] Weights: neurons, blocks, heads Removes weights and the corresponding compute Irregular sparsity may be slow Tensor networks can be combined with structured sparsity [152, 103] KD [71] Model behavior Recovers performance in smaller models Requires data and training Tensorized student models can be distilled from dense teachers [37] Efficient attention [26, 176, 31] Attention compute and memory traffic Addresses long-context bottleneck Often does not reduce FFN or embedding parameters Tensorized attention may complement IO-aware and linear/sparse methods [110] KV-cache [148, 3, 30] Cached keys and values at inference Reduces cache memory and bandwidth in long-context decoding May degrade model quality Non-tensor methods such as MQA and GQA are special cases of more general tensor parameterizations [189, 88] MoE routing [147, 40] Parameters activated per token Scales capacity without activating all parameters Routing and load-balancing complexity The expert index becomes an additional mode, so a single factorization can share structure across experts [179] Tensor methods interact with other LLM-efficiency techniques. Table 11 describes, for each neighboring technique, what it targets, its typical advantages and limitations, and exactly how it interacts with tensor methods. A key observation is that tensor methods do not need to compete with these techniques as mutually exclusive alternatives. In realistic deployments, tensor factors can be quantized, tensorized adapters can be trained on a quantized backbone model, and tensor compression can be followed by knowledge distillation. How to read the interactions. The last column of Table 11 contains three conceptually different relationships. First, techniques can be compositional: tensorization changes the structural parameter count or operator graph, whereas quantization changes precision, pruning changes support, and knowledge distillation changes the training signal. These methods act on different design variables and can therefore be optimized jointly. Second, a tensor method can generalize a neighboring method. Matrix low-rank adaptation is recovered when an update has only two modes, while tensorized PEFT extends sharing across layers, heads, or modules; similarly, MQA and GQA can be viewed as restricted cases of more general tensor factorizations of attention projections or cached states. Third, methods can overlap, i.e., efficient-attention kernels, linear or sparse attention, and tensorized attention may all target the same compute or memory traffic. In this case, the gain of the combination is the marginal gain after the first method has already changed the bottleneck, not the product of two independently reported speedups. For example, in the case of quantization, a compact memory model is Mbits=∑k=1Knkbk+Mmeta,M_ bits= _k=1^Kn_kb_k+M_ meta, (48) where the tensorized model is stored as K factors/cores, nkn_k is the number of scalars retained in part k, bkb_k its precision in bits, and MmetaM_ meta collects what the format needs besides the values themselves such as quantization scales and zero-points. Written this way, the two techniques act on different arguments of one expression: tensorization determine nkn_k, quantization determines bkb_k, and both are chosen per factor. A sequential pipeline such as decompose and then quantize, or decompose an already quantized model, fixes one argument before the other is examined and therefore searches only a slice of this space, with no reason for the result to be optimal. Schemes that assign different precisions to different factors are a first step out of that slice [104], and the argument is not specific to quantization. The last term MmetaM_ meta decides whether structural savings survive into the stored format. For instance, regular structure is almost free to describe, whereas irregular structure, which group carries which scale, which core entries survived pruning, has to be enumerated. Since high compression ratios are often obtained by introducing such irregularity, a ratio computed from the values alone can overstate the memory actually occupied. Order of composition and error interaction. Even when two techniques target different variables, their order can matter. Decomposition changes the distributions and sensitivities of the resulting factors, which changes how aggressively each factor can be quantized. Pruning before decomposition may destroy a low-rank pattern that the factorization could otherwise exploit, whereas pruning the factors after decomposition introduces a different structured approximation. Distillation can be used either to train a tensorized student directly or as a healing stage after compression. Consequently, the errors of two techniques need not add linearly: they may amplify one another, partially cancel, or act as an implicit regularizer. A combined method should therefore be optimized and evaluated as a joint pipeline instead of being assembled from hyperparameters tuned for each technique in isolation. Bottleneck-aware combinations. The appropriate combination depends on which resource dominates the operating regime. If weight bandwidth is the bottleneck, tensorization combined with quantization or structured sparsity can reduce the bytes fetched per token. In long-context decoding, attention and KV-cache methods are more likely to dominate, so tensor cache factorization should be evaluated together with MQA/GQA and IO-aware kernels. For adaptation, a tensorized update can reduce trainable state while a quantized backbone controls the frozen-model footprint. Knowledge distillation is different again since it can recover quality or transfer behavior, but its benefit must be weighed against the additional training cost. This stage-specific view prevents a method that improves an inactive resource from being credited with an end-to-end gain it cannot deliver. The limitation can be stated through an Amdahl-style bound. Let p be the fraction of baseline runtime affected by a technique and let StargetS_ target be the speedup of that fraction. Then the end-to-end speedup satisfies Se2e≤1(1−p)+p/Starget.S_ e2e≤ 1(1-p)+p/S_ target. (49) Thus, even an arbitrarily fast tensorized module gives only a modest gain when untensorized FFNs, attention, communication, or framework overhead dominate the remaining runtime. Methods that target disjoint bottlenecks can be complementary, whereas methods that target the same fraction exhibit diminishing returns. This also explains why the same combination may help prefill, decoding, fine-tuning, and batch serving by very different amounts. Evaluation of combined methods. A convincing comparison should include four systems: the dense baseline, the tensor-only method, the neighboring technique alone, and their combination, all at matched quality and under the same hardware and precision. Reporting only the combined result makes it difficult to attribute the gain to tensorization. Moreover, memory accounting should include tensor metadata, quantization scales, sparsity indices, temporary contraction buffers, and adapter state. Latency should be separated by prefill and decoding and reported both per module and end to end. Finally, the order of operations and the tuning budget for every baseline should be stated explicitly, and ρgap _ gap from Equation 55 should be reported for the tensorized part and for the full pipeline. 7 Relationship to Probabilistic Tensor Networks The connection between tensor decompositions and probabilistic graphical models (PGMs) has been recognized for over a decade, primarily in the context of unsupervised learning and density estimation [59, 61, 116, 171, 49]. This relationship is often invoked to justify tensor-network architectures for sequential data, but the correspondence is subtle and requires careful qualification. In this section, we clarify the precise conditions under which a tensor decomposition corresponds to a PGM, discuss the limitations of these correspondences for real-valued tensor networks, and outline the implications for language modeling. 7.1 Tensor Decompositions as Parameterizations of Graphical Models A joint probability mass function over N discrete random variables X1,X2,…,XNX_1,X_2,…,X_N can be represented as a tensor ∈ℝI1×I2×⋯×INP ^I_1× I_2×…× I_N with non-negative entries summing to one. Many common PGMs correspond to low-rank or structured tensor factorizations of P, with the tensor decomposition directly encoding the conditional independence structure of the model. CP decomposition and naïve Bayes. The CP decomposition expresses an N-th-order tensor as i1,…,iN=∑r=1Rλr(1)i1,r(2)i2,r⋯(N)iN,r.P_i_1,…,i_N= _r=1^R _rA^(1)_i_1,rA^(2)_i_2,r·sA^(N)_i_N,r. (50) If P is a joint probability tensor and the entries λr,in,r(n) _r,A^(n)_i_n,r are non-negative and suitably normalized, then the CP representation corresponds exactly to a naïve Bayes model with a latent class variable Z∈1,2,…,RZ∈\1,2,…,R\. Specifically, each observed variable XnX_n is conditionally independent given Z, with conditional distributions Pr⁡(Xn=in∣Z=r)∝in,r(n) (X_n=i_n Z=r) ^(n)_i_n,r, and prior Pr⁡(Z=r)∝λr (Z=r) _r (up to normalization constants). This correspondence is well-known in the tensor decomposition literature and has been exploited for topic modeling [4] and non-negative tensor factorization [23]. T decomposition and hidden Markov models. The tensor train decomposition [125] represents a tensor as: i1,…,iN=(1)i1(2)i2⋯(N)iN,P_i_1,…,i_N=G^(1)_i_1G^(2)_i_2·sG^(N)_i_N, (51) where each in(n)∈ℝRn−1×RnG^(n)_i_n ^R_n-1× R_n is a matrix slice of the n-th core, and R0=RN=1R_0=R_N=1. If the entries are non-negative and the cores satisfy certain column-stochastic normalization conditions, the T format is equivalent to a hidden Markov model (HMM) [49]. Specifically, the bond dimension RnR_n corresponds to the number of hidden states at position n, and the core in(n)G^(n)_i_n encodes the transition probabilities from hidden state Rn−1R_n-1 to RnR_n together with the emission probability of Xn=inX_n=i_n. This connection has been used in tensor-network-based density estimation [61], generative modeling [48], and sequence modeling [154]. Once again, the correspondence is conditional, since an unconstrained real-valued T representation of a weight matrix does not correspond to an HMM. The cores are not stochastic matrices, the bond dimensions do not have a probabilistic interpretation, and there is no underlying latent process. The T format is simply a structured low-rank parameterization that happens to share the same algebraic form as an HMM’s parameterization but without the probabilistic semantics. Block-term decomposition and more general PGMs. The BT decomposition represents a tensor as a sum of Tucker blocks: =∑k=1Kk×1(1)k×2(2)k⋯×N(N)k.P= _k=1^KG_k×_1A^(1)_k×_2A^(2)_k·s×_NA^(N)_k. (52) When the factors and core are non-negative and normalized, BT decomposition can represent mixtures of latent-variable models, such as mixture of naïve Bayes (where each block is a naïve Bayes component) or more general hierarchical models. Conversely, an unconstrained BT decomposition has no PGM interpretation and should be viewed purely as a flexible tensor parameterization. 7.2 Tensor Networks as Matrix-Valued Stochastic Processes The term “matrix-valued Markov chain” is often used informally to describe the TTM or MPO parameterization of a sequence of matrices. This terminology is evocative but not formally precise. An MPO is a tensor network of the form: i1,j1,…,iN,jN=(1)i1,j1(2)i2,j2⋯(N)iN,jN,W_i_1,j_1,…,i_N,j_N=G^(1)_i_1,j_1G^(2)_i_2,j_2·sG^(N)_i_N,j_N, (53) where each in,jn(n)∈ℝRn−1×RnG^(n)_i_n,j_n ^R_n-1× R_n is a matrix slice indexed by an input-output pair (in,jn)(i_n,j_n). This representation is not a Markov chain in the probabilistic sense. It has no transition probabilities, no state space over which probability mass is conserved, and no stochastic interpretation. The term “matrix-valued” refers to the fact that each slice is a matrix, and “chain” refers to the sequential contraction of these matrices. It is a useful mnemonic, but it should not be interpreted as implying a probabilistic Markov process unless the cores are explicitly constrained to be column-stochastic and the appropriate normalization conditions are imposed 7.3 Implications for Language Models The correspondence between tensor decompositions and PGMs has several implications for tensor methods in language modeling: 1. Interpretability: When a tensor decomposition is applied to an embedding table, attention tensor, or activation tensor, the factors do not automatically correspond to interpretable latent variables such as topics, syntactic roles, or semantic features. For factors to be interpretable in a probabilistic sense, the decomposition must be non-negative and properly normalized. This is rarely the case in practice; most LLM tensorization uses real-valued decompositions optimized for reconstruction or task loss. 2. Generative modeling: If the goal is to build a generative language model directly from a tensor network, the decomposition must be constrained to represent a valid probability distribution. This requires non-negativity and normalization, which complicate optimization and may reduce expressivity compared to unconstrained real-valued factorizations. 3. Sequential structure: The T and MPO formats are natural for sequential data, as they impose a chain-like structure that mirrors the temporal or positional ordering of tokens. However, this structural similarity does not imply that the model captures temporal dependencies in a probabilistic sense; it simply means the parameterization is aligned with the sequential nature of the data. 4. Unified view: Treating tensor decompositions as parameterizations of PGMs provides a unified perspective on tensor methods for language models. It suggests that tensorization can be viewed as imposing a particular conditional independence structure on the model’s representations, which may serve as a useful inductive bias. This perspective is particularly relevant for understanding the expressivity and generalization properties of tensorized architectures. 8 Software overview 8.1 Packages and Frameworks. Python tensor libraries. This paragraph lists the most relevant Python packages that are either compatible with or specifically designed for deep learning scenarios. TensorLy [95] is a general-purpose tensor toolbox that runs on many backends, including NumPy [62], JAX [16], and PyTorch [131], whereas tntorch [166] targets PyTorch directly. Packages for tensorized training build upon mature deep learning frameworks: TensorLy-Torch [94] for PyTorch and T3F [119] for TensorFlow [1]. Other packages, including TensorKrowch [130] and tn4ml [136], address machine-learning applications more broadly. Finally, quimb [55] supports tensor networks of arbitrary topology across several backends, and cotengra [54] specializes in optimizing their contraction. Hardware co-design. A separate line of work co-designs software and hardware for tensorized layers. ETTE [51] and Huang et al. [80] optimize forward pass for tensorized linear layers to accelerate inference, whereas FETTA [108] and the FPGA design of Tian et al. [161] cover both training and inference. Others. The remaining tools fall into three groups. Some target languages other than Python, including Julia, C++, and MATLAB; examples include ITensor [42], TenDeC++ [84], and Tensor Toolbox [8]. Others implement a single decomposition family without specializing in deep learning, as in Scikit-T [46]. A third group has not been updated for five years or more: TTAX [118], TorchMPS [117], and TedNet [127]. 8.2 Case studies We now examine two recent studies that illustrate how these tools are used in practice at different scales, with one operating at research scale and the other at production scale. Javanmard et al. [82] apply MPO parameterization to every linear layer of PicoGPT [123], a small, GPT-2-style character-level model. Their work provides an open-source MPOLinear module, whose cores can be initialized from a dense matrix or trained from scratch with standard PyTorch autograd. They validate the tensorized model against a dense baseline on the Tiny Shakespeare dataset. Kozyrev et al. [96] propose Minima, a deployment-ready compression pipeline for production-scale LLMs. The pipeline consists of sensitivity analysis, Tucker/T/Tensor Ring compression, a short healing stage, custom Triton/CUDA kernels, and speculative decoding. The authors validate it on the Qwen3-32B [138] model. 9 Discussion In this section, we discuss the challenges, gaps, and future research directions regarding tensor methods for LLMs. 9.1 Compression-Realization Gap A recurrent problem with tensorized models is that a substantial reduction in parameter count rarely converts into a proportional speedup, either in training or at inference. We do not report new measurements here; the aim is to name the phenomenon, to separate its two sources, and to fix a protocol under which it becomes comparable across methods. Intuitively, the quantity of interest is ρgap=theoretical compressionpractical speedup, _ gap= theoretical compressionpractical speedup, whose numerator is what a paper usually claims and whose denominator is what a user obtains. Let B denote the total number of bytes, F the FLOPs count, and T the wall-clock time, with the subscripts “dense” and “TN” referring to the baseline dense model and to its tensorized counterpart. We define byte compression, FLOPs compression, and speedup as CB=BdenseBTN,CF=FdenseFTN,ST=TdenseTTN.C_B= B_ denseB_ TN, C_F= F_ denseF_ TN, S_T= T_ denseT_ TN. (54) These quantities are comparable only under a common setup, which we require to satisfy: (i) the same device and numerical precision, so that the peak throughput is shared; (i) an optimized dense baseline; (i) matched quality for the dense and the TN model; (iv) the same scope for all three quantities, either per module or end-to-end; and (v) a declared operating regime, with prefill and decoding reported separately, as they sit on opposite sides of the compute-bound/memory-bound boundary. We define the compression-realization gap as ρgap=CBST=CBCF⏟algorithmic×CFST⏟realization, _ gap= C_BS_T= C_BC_F_algorithmic× C_FS_T_realization, (55) which splits the gap into two complementary and independently actionable parts. The algorithmic term depends only on the decomposition and the contraction scheme, not on the hardware; it exceeds one when FLOPs increase more rapidly than memory as the decomposition-specific ranks grow, or when a suboptimal contraction order is used. The realization term admits an exact reading in terms of hardware utilization. Model FLOPs utilization is defined with respect to the device peak throughput as MFU=F/(T⋅Fpeak)MFU=F/(T· F_ peak), so that T=F/(MFU⋅Fpeak)T=F/(MFU· F_ peak). Substituting this into STS_T and using the shared FpeakF_ peak granted by condition (i), we obtain CFST=MFUdenseMFUTN. C_FS_T= MFU_ denseMFU_ TN. (56) This identity is informative only in the compute-bound regime, where FLOPs govern the runtime. In the memory-bound regime the runtime is set by the bytes moved, and the analogous quantity is the achieved memory-bandwidth utilization. A value ρgap=1 _ gap=1 means that the system realizes all of the idealized byte-compression factor as speedup. The typical case is ρgap>1 _ gap>1, and the two factors attribute it either to a decomposition that buys memory at the price of arithmetic, or to a model that utilizes the device worse than dense GEMMs. A value ρgap<1 _ gap<1 is also possible in the memory-bound regime, where shrinking the weights or the KV-cache reduces exactly the resource that bounds the runtime, so the speedup may exceed the byte compression. Since the metric only contrasts nominal compression with realized speedup, it applies equally to quantization [44, 33, 106], sparsification [43, 6], and IO-aware attention kernels [26], providing common ground for the comparison tensor and non-tensor methods; we return to it in Section 9.2. 9.2 Gaps, Challenges, and Possible Solutions This subsection collects the open problems that the reviewed works themselves report. We took them from the limitations and future-work sections of the surveyed papers and grouped them by theme. For each gap we describe the challenges that arise there and the directions that could resolve them. Table 12 presents this in compact form, and the paragraphs that follow discuss each gap in more detail. Table 12: Gaps and challenges overview across the tensor methods literature. Gap Challenges Possible solutions Hardware compatibility Small, sequential, and irregular contractions underutilize GPU parallelism, so fewer parameters need not mean faster computations Open-source Triton/CUDA libraries with fused and structure-aware kernels, evaluated via measured ρgap _ gap Rank selection Optimal rank differs across layers and component types; matrix adaptive schemes do not easily extend to tensor ranks Adaptive ranks allocation algorithms, benchmarked against fixed-rank heuristics Tensorization scheme Schemes with equal parameter count differ in quality and contraction cost; scheme selection is a complex search whose objective is expensive to evaluate Construct an admissible set of schemes and search it efficiently, scoring candidates by validation loss, memory, and contraction cost New tensor networks Richer topologies introduce more complex ranks/scheme choices; raise contraction cost and latency Evaluate underexplored topologies per component/stage, reporting quality, memory, and latency jointly Training/optimization dynamics Training can be unstable since the tensor parameterization changes the loss landscape, yet optimizers are tuned for dense weight matrices Diagnostics of gradient scale and initialization across cores; optimization routines that account for the tensor structure Scaling Training tensorized models from scratch at large scales is beyond most academic compute, which affects the pre-training stage most Measure how quality and compression ratio change with scale, giving an analogue of scaling laws for tensorized models Compatibility with neighboring efficiency methods Orthogonality to other efficiency methods holds in principle, but whether gains add up and how errors compose is unknown Ablations separating the contribution of tensorization from that of other methods Benchmarking and evaluation Methods use different baselines, datasets, and per-paper tuning, so reported gains are not directly comparable A per-stage benchmark on a shared baseline, reporting the compression-realization gap next to quality Hardware compatibility. Tensorization of all components and stages shares one common challenge: hardware compatibility. Modern GPUs are built for dense GEMMs, which are large, regular, and highly parallelizable, whereas tensor contractions are small, sequential, and irregular, so a model with fewer parameters need not run faster. Closing this compression-realization gap is probably a kernel-level problem. Two kinds of GPU kernels could address it. Fused kernels combine a chain of contractions into a single launch, removing per-launch overhead and keeping intermediate factors in fast memory instead of writing them back to global memory. Structure-aware kernels exploit the structure that a decomposition imposes on the factors, such as sparse cores, instead of treating them as dense arrays. One practical solution is open-source Triton or CUDA libraries implementing such kernels, with their benefit reported as measured ρgap _ gap (Eq. 55). Rank selection. Ranks are the main control knob of a tensorized model. Uniform rank selection is a convenient heuristic, but it is not optimal, since the optimal ranks may differ between layers and component types. Algorithms that allocate ranks adaptively should, in principle, dominate uniform assignment at an equal parameter budget. At the same time, generalizing matrix adaptive schemes, such as AdaLoRA [186], to tensor ranks is not straightforward. Several reviewed studies [181], [107], as well as work beyond our scope [66], [65], [56], propose adaptive schemes for automatic rank allocation. However, a universal and generally applicable method for adaptive rank allocation is still missing. Tensorization scheme. A second design choice concerns the tensorization scheme itself, namely how a model parameter becomes a tensor object, and which of the admissible mappings works best in practice. For example, in the imposed setting (Section 4.1), turning a weight matrix W into a tensor W requires choosing the number of modes and their sizes, and these choices are not obvious. Different choices may give the same parameter count but differ in approximation quality and contraction cost. This area remains under-theorized. New tensor networks. Applying new tensor networks is a natural and promising direction. In the imposed tensorization setting, for instance, nothing in principle restricts the applicable tensor network to TTM; richer topologies are equally valid. Deep tensor networks are potentially more expressive and memory-efficient than shallow chain-like topologies such as TTM, and they have shown promising results in domains beyond LLMs and NLP [137], which suggests they may transfer to the LLM setting as well. The deeper and more complicated the tensor network, however, the higher the computational latency, since the contraction cost grows. Tensor Ring [190], tree-structured [53], fully-connected [191], PEPS [169], MERA-inspired [170], and Kronecker Tensor Decomposition [13] networks remain underexplored in LLM tensorization. Training and optimization dynamics. Training a tensorized model with standard techniques and optimizers can be unstable [12]. One reason is that the tensor parameterization substantially changes the loss landscape, so optimizers designed and tuned for dense weight matrices, such as Adam, may not be optimal for tensorized models [181]. What is still missing is an understanding of how tensorized training behaves in this landscape, including how gradient magnitudes are distributed across cores, how core initialization affects convergence, and how these quantities evolve during training. Such diagnostics would also indicate what an optimizer actually has to correct. Studying optimization routines that account for the tensor structure could then make training both faster and more stable; some methods, such as MetaTT [107], point in this direction. Scaling. Training a tensorized language model from scratch is expensive, and training one at several scales, or at a large scale at all, is beyond the compute of most academic groups [19]. This affects the pre-training stage most. The open question is how the benefits of tensorization change as the model grows, specifically how compression ratio and quality behave at each scale, thereby providing an analogue of scaling laws [85] for tensorized language models. Current evidence is far from this (Table 7). Scaling is also a practical question at the other stages, since how much a model can be compressed depends on how it was trained. Recent models are often distilled and probably harder to compress [152]. A compression ratio measured on one model therefore need not translate to a newer or larger one. Compatibility with neighboring efficiency methods. Section 6 discusses how tensor methods relate to quantization, pruning, distillation, and other efficiency techniques. Most of them are orthogonal to tensorization in principle, but their practical interaction, including whether the gains add up and how the errors compose, remains an open question. Benchmarking and evaluation. Numbers reported across papers often are not directly comparable; methods are evaluated on different baselines and datasets, and each is tuned separately, so a difference in reported quality may come from the setup, not only from the method. Claiming that one method is better than another requires a comparison on a shared baseline, with hyperparameters tuned for every method under the same approach. A benchmark would provide this, and it is probably better to have one per stage, since the stages optimize for different things. Whatever the stage, such a benchmark has to report the compression-realization gap (Eq. 55) next to quality. Otherwise a method that saves parameters on paper but not time on a GPU is indistinguishable from one that saves both. 9.3 Future Directions The preceding subsection focuses on obstacles that must be removed for existing tensor methods to become reliable and efficient. Here we highlight longer-horizon directions that change where tensorization enters the model lifecycle, which objects it acts on, or what capabilities it is intended to provide. These directions extend the stage-specific literature toward tensor-native language-model design. Tokenization. No tensor method reaches the tokenizer (Section 5.1), although relatedness between tokens is lost as soon as the vocabulary is used, as the vocabulary built by BPE and its variants is a set of subword strings, but each string is used only through its index. Consequently, two tokens sharing a morpheme may receive unrelated embeddings. Tensor methods are a natural way to impose structure that reflects this relatedness. MorphTE [45] does this for embeddings, but takes the segmentation from an external analyzer and leaves the vocabulary unchanged. More broadly, the vocabulary and the embedding table are still built in sequence; forming them jointly is an open problem. The obstacle is that the tokenizer is a discrete, non-differentiable map, so there is no continuous object to factorize. Interpretability. Interpretability is underexplored for the opposite reason. The stage is not empty but new and, in our view, the most promising. Apart from the notational work, every method surveyed in Section 5.7 has appeared within the last year. Together they show that multilinear structure can support understanding in three ways. It can be added to the analysis tool, as in PolySAE [93]; reformulated from a model that is multilinear without having been written that way, as in TensorLens [7]; or imposed at training time as an inductive bias, as in bilinear layers [132]. The last route is the most demanding, since it requires pre-training from scratch, and the most far-reaching, as it turns interpretability into a design constraint on the architecture. A separate opportunity is graphical tensor notation [160]. Its immediate benefit is expository, as a diagram states which modes a method contracts, shares, or truncates; accordingly, we encourage authors of tensor interpretability methods to draw their constructions this way. Beyond that, the notation is a reasoning tool in its own right, since asking which contraction orders, factorizations, and shared factors a network admits can reveal new mechanisms. Lifecycle-level co-design. Most existing methods optimize tensorization at one lifecycle stage while treating the surrounding stages as fixed. A tensor-native model could instead co-design the embedding factorization, base-model ranks, adaptation subspaces, post-training compression, and inference contraction schedule. Such coupling matters, since a rank pattern that minimizes pre-training loss may be unsuitable for later PEFT, and a decomposition that is compact algebraically may be difficult to execute efficiently during decoding. A useful formulation is therefore multi-objective: select tensorization schemes and ranks jointly to optimize quality, training memory, adaptation capacity, serving latency, and energy under explicit resource budgets. Results should be reported as Pareto frontiers. Tensorized training state and distributed learning. The literature focuses primarily on model weights, adapters, and KV caches, but large-scale training also stores activations, gradients, optimizer moments, and communication buffers. These objects have natural modes associated with layers, microbatches, sequence positions, heads or experts, and devices. Exploiting low-rank structure across such modes could reduce both accelerator memory and inter-device communication, extending tensorization from model representation to the full training state. The central difficulty is that these tensors are transient and non-stationary: their useful ranks may change over the course of training, and approximation errors can accumulate through optimization. Future work should therefore evaluate convergence, numerical stability, communicated bytes, synchronization cost, and end-to-end training throughput together. Conditional and elastic tensorization. Nearly all surveyed methods use ranks and contraction graphs that remain fixed for every input. A more flexible model could allocate tensor capacity conditionally by layer, token, context length, task, or confidence, activating additional cores or rank components only when they improve the prediction sufficiently. This would turn rank from a static architecture hyperparameter into a runtime compute budget and could provide controllable quality–latency trade-offs. The systems challenge is substantial: variable contraction shapes interfere with batching, compilation, and fused kernels. Evaluation must consequently include worst-case memory and tail latency, not only average FLOPs or average rank. Multimodal and modular architectures. Multimodal language models and mixture-of-experts architectures expose semantic modes that are absent from a single dense text model, including modality, expert, routing group, spatial position, and temporal position. Tucker, block-term, tree-structured, or shared-core decompositions could separate globally shared linguistic factors from modality- or expert-specific factors. This may offer a principled alternative to duplicating full projection and adapter matrices across modules. The open questions are how to preserve modality-specific capacity, routing balance, and rare-expert behavior while avoiding negative transfer through an overly restrictive shared core. This direction would also test whether semantically meaningful tensor modes are more robust than imposed reshaping at scale. 10 Conclusion This survey has examined tensor methods for language models through a unified tensor-theoretic lens, organizing the literature across the seven stages of the LLM lifecycle. Two findings emerge from this synthesis. First, the tensorization strategies available to a given component depend on whether its modes carry semantic meaning. Attention tensors and stacked projection tensors admit mode-specific decompositions, while feed-forward networks and embedding tables typically rely on imposed reshaping and chain-like tensor networks such as T and TTM. Second, the same decomposition families recur across stages; for example, T and TTM appear in embeddings, pre-training, PEFT, and compression. Nevertheless, the objectives differ, with pre-training optimizing a language-modeling loss under a parameter budget and compression optimizing reconstruction error or activation-aware distortion. This distinction has practical consequences for method design and evaluation. The lifecycle view also makes the uneven coverage of the field visible. Tokenization is untouched by tensor methods, interpretability is the newest stage and, in our view, the most promising, and pre-training lacks the multi-scale evidence needed to say how tensorization interacts with scaling laws. Cutting across the stages, the central obstacle is the compression-realization gap, since parameter savings rarely convert into proportional speedups. We formalize it as the metric ρgap _ gap, which separates an algorithmic term, set by the decomposition and the contraction scheme, from a realization term, set by how well the model uses the device. The directions we outline address both sides of this problem. Fused and structure-aware kernels target the realization side, whereas adaptive rank allocation, better-understood tensorization schemes, and richer tensor-network topologies target the algorithmic side. These directions should be accompanied by per-stage benchmarks that report ρgap _ gap alongside quality. Overall, this survey treats tensor methods as a coherent subfield within LLM research. We hope it serves both as a reference for the existing work and as an entry point for the next generation of tensorized language models. References [1] M. Abadi et al. (2016) TensorFlow: a system for large-scale machine learning. External Links: 1605.08695, Link Cited by: §8.1. [2] V. Abronin, A. Naumov, D. Mazur, D. Bystrov, K. Tsarova, A. Melnikov, S. Dolgov, R. Brasher, and M. Perelshein (2024) TQCompressor: improving tensor decomposition methods in neural networks via permutations. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), Vol. , p. 503–506. External Links: Document Cited by: Figure 2, §5.5, Table 9. [3] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai (2023) GQA: training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. Cited by: §2.3.4, Table 11. [4] A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky (2014) Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research 15 (1), p. 2773–2832. Cited by: §7.1. [5] A. Anjum, M. E. Eren, I. Boureima, B. Alexandrov, and M. Bhattarai (2024) Tensor train low-rank approximation (t-lora): democratizing ai with accelerated llms. In 2024 International Conference on Machine Learning and Applications (ICMLA), Vol. , p. 583–590. External Links: Document Cited by: Table 3, Figure 2, §5.4, Table 8, Table 8. [6] S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman (2024) SliceGPT: compress large language models by deleting rows and columns. External Links: 2401.15024, Link Cited by: Table 11, §9.1. [7] I. A. Atad, I. Zimerman, S. Katz, and L. Wolf (2026) TensorLens: end-to-end transformer analysis via high-order attention tensors. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 3452–3468. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: Figure 2, §5.7, §9.3. [8] B. W. Bader and T. G. Kolda (2026) Tensor Toolbox for MATLAB, version 3.8. Note: https://w.tensortoolbox.org Cited by: §8.1. [9] A. Baggag and Y. Saad (2025) Deep learning, transformers and graph neural networks: a linear algebra perspective. Numerical Algorithms 100 (4), p. 2095–2134. External Links: Document, Link Cited by: §1. [10] D. Bahdanau, K. Cho, and Y. Bengio (2016) Neural machine translation by jointly learning to align and translate. External Links: 1409.0473, Link Cited by: §2.3. [11] L. Balzano, T. Ding, B. D. Haeffele, S. M. Kwon, Q. Qu, P. Wang, Z. Wang, and C. Yaras (2026) An overview of low-rank structures in the training and adaptation of large models: from implicit low-dimensionality to efficient training and fine-tuning [special issue on the mathematics of deep learning]. IEEE Signal Processing Magazine 43 (3), p. 52–70. External Links: Document Cited by: §1, Table 1. [12] F. Barratt, J. Dborin, and L. Wright (2022) Improvements to gradient descent methods for quantum tensor network machine learning. External Links: 2203.03366, Link Cited by: §9.2. [13] K. Batselier and N. Wong (2017) A constructive arbitrary-degree kronecker product decomposition of tensors. Numerical Linear Algebra with Applications 24 (5), p. e2097. Note: e2097 nla.2097 External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/nla.2097 Cited by: §2.1, §9.2. [14] L. Bereska and E. Gavves (2024) Mechanistic interpretability for ai safety – a review. External Links: 2404.14082, Link Cited by: §1, Table 1. [15] D. Bershatsky, D. Cherniuk, T. Daulbaev, A. Mikhalev, and I. Oseledets (2024) LoTR: low tensor rank weight adaptation. External Links: 2402.01376, Link Cited by: Figure 2, §5.4, Table 8. [16] JAX: composable transformations of Python+NumPy programs External Links: Link Cited by: §2.1, §8.1. [17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33, p. 1877–1901. External Links: Link Cited by: §1. [18] J. D. Carroll and J. Chang (1970) Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 35, p. 283–319. External Links: Document Cited by: §1, §2.2.1, Table 3. [19] V. Chekalina, G. Novikov, J. Gusak, A. Panchenko, and I. Oseledets (2023) Efficient GPT model pre-training using tensor train matrix representation. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, Hong Kong, China, p. 600–608. External Links: Link Cited by: Table 3, Figure 2, §3.2, §3.2, §4, §5.3, §5.3, Table 7, §9.2. [20] Z. Chen, R. Dangovski, O. Dugan, C. Loh, D. Luo, and M. Soljačić (2024) QuanTA: efficient high-rank fine-tuning of llms with quantum-informed tensor adaptation. In Advances in Neural Information Processing Systems 37, NeurIPS 2024, p. 92210–92245. External Links: Link, Document Cited by: Figure 2, §5.4, Table 8. [21] A. Cichocki, N. Lee, I. Oseledets, A. Phan, Q. Zhao, and D. P. Mandic (2016) Tensor networks for dimensionality reduction and large-scale optimization: part 1 low-rank tensor decompositions. Foundations and Trends® in Machine Learning 9 (4-5), p. 249–429. External Links: ISSN 1935-8245, Link, Document Cited by: §1, Table 1, §1, §2.2.1, §2.2.3. [22] A. Cichocki, A. Phan, Q. Zhao, N. Lee, I. Oseledets, M. Sugiyama, and D. Mandic (2017) Tensor networks for dimensionality reduction and large-scale optimizations part 2 applications and future perspectives. Foundations and Trends® in Machine Learning 9 (6), p. 431–673. External Links: ISSN 1935-8245, Link, Document Cited by: §1, Table 1, §1. [23] A. Cichocki, R. Zdunek, A. H. Phan, and S. Amari (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. John Wiley & Sons, United Kingdom. External Links: ISBN 9780470746660 Cited by: §7.1. [24] H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey (2023) Sparse autoencoders find highly interpretable features in language models. External Links: 2309.08600, Link Cited by: §5.7. [25] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov (2019) Transformer-xl: attentive language models beyond a fixed-length context. External Links: 1901.02860, Link Cited by: §5.2, Table 7. [26] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, Vol. 35, p. 16344–16359. Cited by: §4.3, Table 11, §9.1. [27] L. De Lathauwer, B. De Moor, and J. Vandewalle (2000) A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications 21 (4), p. 1253–1278. External Links: Document, Link, https://doi.org/10.1137/S0895479896305696 Cited by: §1, §2.2.2. [28] L. De Lathauwer (2008) Decompositions of a higher-order tensor in block terms—part i: definitions and uniqueness. SIAM Journal on Matrix Analysis and Applications 30 (3), p. 1033–1066. External Links: Document, Link, https://doi.org/10.1137/070690729 Cited by: §2.2.4, §2.2.4, Table 3. [29] V. de Silva and L. Lim (2008) Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications 30 (3), p. 1084–1127. External Links: Document, Link, https://doi.org/10.1137/06066518X Cited by: §2.2.1. [30] DeepSeek-AI (2024) DeepSeek-v2: a strong, economical, and efficient mixture-of-experts language model. External Links: 2405.04434, Link Cited by: §2.3.4, Table 11. [31] DeepSeek-AI (2025) DeepSeek-v3.2: pushing the frontier of open large language models. External Links: 2512.02556, Link Cited by: Table 11. [32] DeepSeek-AI (2026) DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §1. [33] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022) LLM.int8(): 8-bit matrix multiplication for transformers at scale. External Links: 2208.07339, Link Cited by: Table 11, §9.1. [34] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023) QLoRA: efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, Cited by: §5.4. [35] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, p. 4171–4186. External Links: Document Cited by: §5.5. [36] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh (2022) KronA: parameter efficient tuning with kronecker adapter. External Links: 2212.10650, Link Cited by: Figure 2, §3.2, §5.4, Table 8. [37] A. Edalati, M. Tahaei, A. Rashid, V. Nia, J. Clark, and M. Rezagholizadeh (2022) Kronecker decomposition for GPT compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland, p. 219–226. External Links: Link, Document Cited by: Figure 2, §5.5, Table 9, Table 9, Table 11. [38] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah (2022) Toy models of superposition. External Links: 2209.10652, Link Cited by: §1. [39] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. (2021) A mathematical framework for transformer circuits. Transformer Circuits Thread. Cited by: §1, §5.7. [40] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. In Journal of Machine Learning Research, Vol. 23, p. 1–39. Cited by: Table 11. [41] F. Ferraro, A. Poliak, R. Cotterell, and B. Van Durme (2017) Frame-based continuous lexical semantics through exponential family tensor factorization and semantic proto-roles. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), N. Ide, A. Herbelot, and L. Màrquez (Eds.), Vancouver, Canada, p. 97–103. External Links: Link, Document Cited by: §5.2. [42] M. Fishman, S. R. White, and E. M. Stoudenmire (2022) The ITensor Software Library for Tensor Network Calculations. SciPost Phys. Codebases, p. 4. External Links: Document, Link Cited by: §8.1. [43] E. Frantar and D. Alistarh (2023) SparseGPT: massive language models can be accurately pruned in one-shot. External Links: 2301.00774, Link Cited by: Table 11, §9.1. [44] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2023) GPTQ: accurate post-training quantization for generative pre-trained transformers. External Links: 2210.17323, Link Cited by: Table 11, §9.1. [45] G. Gan, P. Zhang, S. Li, X. Lu, and B. Wang (2022) MorphTE: injecting morphology in tensorized embeddings. External Links: 2210.15379, Link Cited by: Figure 2, §4.2, §5.1, §5.2, Table 6, §9.3. [46] P. Gelß, S. Klus, M. Scherer, F. Nüske, and M. Lücke (2019) Scikit-T: tensor-train computations in Python. Note: https://github.com/PGelss/scikit_t Cited by: §8.1. [47] M. Geva, R. Schuster, J. Berant, and O. Levy (2021) Transformer feed-forward layers are key-value memories. External Links: 2012.14913, Link Cited by: §2.4, §2.4. [48] I. Glasser, N. Pancotti, and J. I. Cirac (2020) From probabilistic graphical models to generalized tensor networks for supervised learning. IEEE Access 8, p. 68169–68182. Cited by: §7.1. [49] I. Glasser, R. Sweke, N. Pancotti, J. Eisert, and I. Cirac (2019) Expressive power of tensor-network factorizations for probabilistic modeling. Advances in neural information processing systems 32. Cited by: §7.1, §7. [50] G. H. Golub and C. F. Van Loan (2013) Matrix computations. Fourth edition, JHU Press. External Links: ISBN 1421407949 9781421407944, Link Cited by: §2.1, §3.2. [51] Y. Gong, M. Yin, L. Huang, J. Xiao, Y. Sui, C. Deng, and B. Yuan (2023) ETTE: efficient tensor-train-based computing engine for deep neural networks. In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA. External Links: ISBN 9798400700958, Link, Document Cited by: §8.1. [52] K. Goto and R. A. van de Geijn (2008) Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software 34 (3), p. 1–25. External Links: Document Cited by: §2.4. [53] L. Grasedyck (2010) Hierarchical singular value decomposition of tensors. SIAM Journal on Matrix Analysis and Applications 31 (4), p. 2029–2054. External Links: Document, Link, https://doi.org/10.1137/090764189 Cited by: §9.2. [54] J. Gray and S. Kourtis (2021) Hyper-optimized tensor network contraction. Quantum 5, p. 410. External Links: Document, Link, ISSN 2521-327X Cited by: §8.1. [55] J. Gray (2018) Quimb: a python package for quantum information and many-body calculations. Journal of Open Source Software 3 (29), p. 819. External Links: Document, Link Cited by: §8.1. [56] J. Gu, B. Keller, J. Kossaifi, A. Anandkumar, B. Khailany, and D. Z. Pan (2022) HEAT: hardware-efficient automatic tensor decomposition for transformer compression. External Links: 2211.16749, Link Cited by: §9.2. [57] Y. Gu, W. Zhou, G. Iacovides, and D. Mandic (2025) TensorLLM: tensorising multi-head attention for enhanced reasoning and compression in llms. In 2025 International Joint Conference on Neural Networks (IJCNN), Vol. , p. 1–8. External Links: Document Cited by: Table 3, Figure 2, §5.5, Table 9, Table 9. [58] Y. Gu, W. Zhou, G. Iacovides, and D. Mandic (2026) TeRA: vector-based random tensor network for high-rank adaptation of large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, p. 2314–2329. External Links: Link, Document, ISBN 979-8-89176-390-6 Cited by: Figure 2, §5.4, Table 8. [59] M. G. A. Hameed and G. Rabusseau (2025) Efficient probabilistic tensor networks. arXiv preprint arXiv:2510.00382. Cited by: §7. [60] S. Hamreras et al. (2025) Tensorization is a powerful but underexplored tool for neural network compression. arXiv preprint arXiv:2505.20132. Cited by: §4.1. [61] Z. Han, J. Wang, H. Fan, L. Wang, and P. Zhang (2018) Unsupervised generative modeling using matrix product states. Physical Review X 8 (3), p. 031012. Cited by: §7.1, §7. [62] C. R. Harris et al. (2020) Array programming with NumPy. Nature 585 (7825), p. 357–362. External Links: Document, Link Cited by: §2.1, §8.1. [63] R. A. Harshman (1970) Foundations of the parafac procedure: models and conditions for an explanatory multimodal factor analysis. UCLA Working Papers in Phonetics 16, p. 1–84. Cited by: §1, §2.2.1, Table 3. [64] J. Håstad (1990) Tensor rank is np-complete. Journal of Algorithms 11 (4), p. 644–654. External Links: ISSN 0196-6774, Document, Link Cited by: §2.2.1. [65] C. Hawkins, X. Liu, and Z. Zhang (2021) Towards compact neural networks via end-to-end training: a bayesian tensor approach with automatic rank determination. External Links: 2010.08689, Link Cited by: §9.2. [66] C. Hawkins and Z. Zhang (2019) Bayesian tensorized neural networks with automatic rank selection. External Links: 1905.10478, Link Cited by: §9.2. [67] P. He, J. Gao, and W. Chen (2023) DeBERTav3: improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Table 8, Table 8. [68] Y. He, H. Wu, W. Liu, and X. Luo (2026) A survey of latent factorization of tensor-based model compression: algorithms, toolboxes and future directions. Neurocomputing 682, p. 133455. External Links: ISSN 0925-2312, Document, Link Cited by: §1, Table 1. [69] D. Hendrycks and K. Gimpel (2023) Gaussian error linear units (gelus). External Links: 1606.08415, Link Cited by: §2.4. [70] C. J. Hillar and L. Lim (2013) Most tensor problems are np-hard. J. ACM 60 (6). External Links: ISSN 0004-5411, Link, Document Cited by: §2.2.1. [71] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: Table 11. [72] F. L. Hitchcock (1927) The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics 6 (1-4), p. 164–189. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/sapm192761164 Cited by: §2.2.1, Table 3. [73] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre (2022) Training compute-optimal large language models. External Links: 2203.15556, Link Cited by: §1. [74] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for nlp. External Links: 1902.00751, Link Cited by: §5.4. [75] I. Hounie, C. Kanatsoulis, A. Tandon, and A. Ribeiro (2025) LoRTA: low rank tensor adaptation of large language models. External Links: 2410.04060, Link Cited by: Figure 2, §5.4, Table 8, Table 8. [76] O. Hrinchuk, V. Khrulkov, L. Mirvakhabova, E. Orlova, and I. Oseledets (2020) Tensorized embedding layers. In Findings of the Association for Computational Linguistics: EMNLP 2020, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 4847–4860. External Links: Link, Document Cited by: Table 3, Figure 2, §4.4, §5.2, Table 6. [77] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §3.2, §5.4, Table 11. [78] X. Hu, X. Cheng, P. Liu, W. Liu, J. Luan, B. Wang, and Y. Liu (2025) DoTA: weight-decomposed tensor adaptation for large language models. In Advances in Knowledge Discovery and Data Mining, Singapore, p. 16–27. External Links: ISBN 978-981-96-8186-0 Cited by: Table 3, Figure 2, §5.4, Table 8. [79] H. Huang, D. Zhu, B. Wu, Y. Zeng, Y. Wang, Q. Min, and X. Zhou (2025) Over-tokenized transformer: vocabulary is generally worth scaling. External Links: 2501.16975, Link Cited by: §4.2. [80] S. Huang, T. Wang, A. Li, A. Shen, K. Li, K. Jiang, M. Huang, and H. Yu (2025) A tensor-train decomposition based compression of llms on group vector systolic accelerator. External Links: 2501.19135, Link Cited by: §8.1. [81] H. Inan, K. Khosravi, and R. Socher (2017) Tying word vectors and word classifiers: a loss framework for language modeling. External Links: 1611.01462, Link Cited by: §3.2. [82] Y. Javanmard, T. Pandit, and M. Mardani (2026) Compressing transformer language models via matrix product operator decomposition: a case study on picogpt. External Links: 2603.28534, Link Cited by: §8.2. [83] Y. Ji, Q. Wang, X. Li, and J. Liu (2019) A survey on tensor techniques and applications in machine learning. IEEE Access 7 (), p. 162950–162990. External Links: Document Cited by: §1, Table 1. [84] H. Jiapeng et al. (2019) A C++ library for tensor decomposition. In 2019 IEEE 38th International Performance Computing and Communications Conference (IPCCC), p. 1–2. External Links: Document Cited by: §8.1. [85] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020) Scaling laws for neural language models. External Links: 2001.08361, Link Cited by: §1, §5.3, §9.2. [86] G. I. Kim, S. Hwang, and B. Jang (2025) Efficient compressing and tuning methods for large language models: a systematic literature review. ACM Comput. Surv. 57 (10). External Links: ISSN 0360-0300, Link, Document Cited by: §1, Table 1. [87] Kimi Team (2026) Kimi k3: open frontier intelligence. External Links: 2607.24653, Link Cited by: §1. [88] T. Klein, J. Kusch, S. Sager, S. Schnake, and S. Schotthöfer (2026) Tucker attention: a generalization of approximate attention mechanisms. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Table 3, Figure 2, §5.6, Table 10, Table 11. [89] T. Koike-Akino, X. Chen, J. Liu, Y. Wang, Pu, Wang, and M. Brand (2025) LatentLLM: attention-aware joint tensor compression. External Links: 2505.18413, Link Cited by: Figure 2, §5.5, §5.5, Table 9, Table 9. [90] T. Koike-Akino, J. Liu, and Y. Wang (2026) EinSort: sorting is all we need for tensorizing LLM. In ICML’26 workshop on CoLoRAI - The 2nd Workshop on Connecting Low-rank Representations in AI, External Links: Link Cited by: Figure 2, §5.6, Table 10. [91] T. Koike-Akino, F. Tonin, Y. Wu, F. Z. Wu, L. N. Candogan, and V. Cevher (2025) Quantum-PEFT: ultra parameter-efficient fine-tuning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Figure 2, §5.4, Table 8, Table 8. [92] T. G. Kolda and B. W. Bader (2009) Tensor decompositions and applications. SIAM Review 51 (3), p. 455–500. External Links: Document Cited by: §1, Table 1, §1, §2.2.1, §2.2.2. [93] P. Koromilas, A. D. Demou, J. Oldfield, Y. Panagakis, and M. Nicolaou (2026) PolySAE: modeling feature interactions in sparse autoencoders via polynomial decoding. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Figure 2, §5.7, §9.3. [94] J. Kossaifi et al. (2024) TensorLy-Torch: deep tensorized learning. Note: https://tensorly.org/torch/dev/ Cited by: §8.1. [95] J. Kossaifi, Y. Panagakis, A. Anandkumar, and M. Pantic (2019) TensorLy: tensor learning in python. Journal of Machine Learning Research 20 (26), p. 1–6. External Links: Link Cited by: §8.1. [96] S. Kozyrev and D. Maiboroda (2026) Minima: a practical tensor-network compression pipeline for production-scale large language models. External Links: 2602.01613, Link Cited by: §8.2. [97] J. B. Kruskal (1977) Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra and its Applications 18 (2), p. 95–138. External Links: ISSN 0024-3795, Document, Link Cited by: §2.2.1. [98] T. Kudo and J. Richardson (2018) SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. External Links: 1808.06226, Link Cited by: §5.1. [99] T. Kudo (2018) Subword regularization: improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y. Miyao (Eds.), Melbourne, Australia, p. 66–75. External Links: Link, Document Cited by: §5.1, §5.1. [100] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. External Links: 2309.06180, Link Cited by: §1. [101] H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen (2025) A survey on large language model acceleration based on kv cache management. External Links: 2412.19442, Link Cited by: §1, Table 1. [102] S. Li, P. Zhang, G. Gan, X. Lv, B. Wang, J. Wei, and X. Jiang (2022) Hypoformer: hybrid decomposition transformer for edge-friendly neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, p. 7056–7068. External Links: Link, Document Cited by: Figure 2, §4.4, §5.3, §5.3. [103] Y. Li, Z. Guo, M. Yin, and B. Li (2026) LeSTD: LLM compression via learning-based sparse tensor decomposition. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Table 3, Figure 2, §3.2, §4.3, §5.5, §5.5, Table 9, Table 9, Table 11. [104] P. Liu, Z. Gao, X. Zhao, Y. Ma, T. Wang, and J. Wen (2024) Unlocking data-free low-bit quantization with matrix decomposition for KV cache compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 2430–2440. External Links: Link, Document Cited by: Figure 2, §5.6, Table 10, §6, Table 11. [105] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized bert pretraining approach. External Links: 1907.11692, Link Cited by: Table 8, Table 8, Table 8, Table 8. [106] Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra (2023) LLM-qat: data-free quantization aware training for large language models. External Links: 2305.17888, Link Cited by: Table 11, §9.1. [107] J. Lopez-Piqueres, P. Deshpande, A. Ray, M. J. Villani, M. Pistoia, and N. Kumar (2026) MetaTT: a global tensor-train adapter for parameter-efficient fine-tuning. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: Figure 2, §5.4, Table 8, Table 8, Table 8, §9.2, §9.2. [108] J. Lu, J. Tian, H. Li, Ian. A. Young, and Z. Zhang (2026) FETTA: flexible and efficient hardware accelerator for tensorized neural network training. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 45 (9), p. 4422–4435. External Links: Document Cited by: §8.1. [109] Y. Luo, H. Patel, Y. Fu, D. Ahn, J. Chen, Y. Dong, and E. E. Papalexakis (2025) TRAWL: tensor reduced and approximated weights for large language models. In Data Science: Foundations and Applications, Singapore, p. 402–413. External Links: ISBN 978-981-96-8298-0 Cited by: §5.5. [110] X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou (2019) A tensorized transformer for language modeling. External Links: 1906.09777, Link Cited by: Table 3, Figure 2, §4, §5.3, §5.3, Table 7, Table 11. [111] K. Meng, D. Bau, A. Andonian, and Y. Belinkov (2023) Locating and editing factual associations in gpt. External Links: 2202.05262, Link Cited by: §2.4. [112] Meta AI (2024) Llama 3.2-1b. Note: https://huggingface.co/meta-llama/Llama-3.2-1BModel card, Hugging Face. Release date: September 25, 2024 Cited by: Table 9. [113] Meta AI (2024) The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §1, §3.2, Table 8, Table 8, Table 8. [114] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia (2025) Towards efficient generative large language model serving: a survey from algorithms to systems. ACM Comput. Surv. 58 (1). External Links: ISSN 0360-0300, Link, Document Cited by: §1, Table 1. [115] D. Milajevs, D. Kartsaklis, M. Sadrzadeh, and M. Purver (2014) Evaluating neural word representations in tensor-based compositional settings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans (Eds.), Doha, Qatar, p. 708–719. External Links: Link, Document Cited by: §5.2. [116] J. Miller, G. Rabusseau, and J. Terilla (2021) Tensor networks for probabilistic sequence modeling. In International Conference on Artificial Intelligence and Statistics, p. 3079–3087. Cited by: §7. [117] J. Miller (2019) TorchMPS. GitHub. Note: https://github.com/jemisjoky/torchmps Cited by: §8.1. [118] A. Novikov and D. Belousov (2021) TTAX: tensor-train toolbox on Jax. Note: https://github.com/fasghq/ttax Cited by: §8.1. [119] A. Novikov, P. Izmailov, V. Khrulkov, M. Figurnov, and I. Oseledets (2020) Tensor train decomposition on tensorflow (t3f). Journal of Machine Learning Research 21 (30), p. 1–7. External Links: Link Cited by: §8.1. [120] A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov (2015) Tensorizing neural networks. In Advances in Neural Information Processing Systems, Vol. 28. Cited by: §5.3. [121] OpenAI (2024) GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1. [122] R. Orús (2014) A practical introduction to tensor networks: matrix product states and projected entangled pair states. Annals of Physics 349, p. 117–158. External Links: ISSN 0003-4916, Document, Link Cited by: §1, Table 1, §1. [123] T. J. Osborne (2026) PicoGPT.jl: from-scratch gpt in pure julia. Note: https://github.com/tobiasosborne/PicoGPT.jlGitHub repository Cited by: §8.2. [124] I. V. Oseledets (2010) Approximation of 2d×2d2^d× 2^d matrices using tensor decomposition. SIAM Journal on Matrix Analysis and Applications 31 (4), p. 2130–2145. External Links: Document, Link, https://doi.org/10.1137/090757861 Cited by: §2.2.3, Table 3. [125] I. V. Oseledets (2011) Tensor-train decomposition. SIAM Journal on Scientific Computing 33 (5), p. 2295–2317. External Links: Document Cited by: §1, §2.2.3, Table 3, Figure 3, §3.2, §4.4, §5.2, §5.5, §7.1. [126] X. Ou, Z. Chen, C. Zhu, and Y. Liu (2024) Low rank optimization for efficient deep learning: making a balance between compact architecture and fast training. Journal of Systems Engineering and Electronics 35 (3), p. 509–531. External Links: Document Cited by: §1, Table 1. [127] Y. Pan, M. Wang, and Z. Xu (2022) TedNet: A pytorch toolkit for tensor decomposition networks. Neurocomputing 469, p. 234–238. Cited by: §8.1. [128] Y. Panagakis, J. Kossaifi, G. G. Chrysos, J. Oldfield, M. A. Nicolaou, A. Anandkumar, and S. Zafeiriou (2021) Tensor methods in computer vision and deep learning. Proceedings of the IEEE 109 (5), p. 863–890. External Links: ISSN 1558-2256, Link, Document Cited by: §1, Table 1. [129] A. Panahi, S. Saeedi, and T. Arodz (2021) Shapeshifter: a parameter-efficient transformer using factorized reshaped matrices. In Advances in Neural Information Processing Systems, Vol. 34, p. 1337–1350. External Links: Link Cited by: Figure 2, §5.3, §5.3. [130] J. R. Pareja Monturiol, D. Pérez-García, and A. Pozas-Kerstjens (2024) TensorKrowch: Smooth integration of tensor networks in machine learning. Quantum 8, p. 1364. External Links: Document, 2306.08595 Cited by: §8.1. [131] A. Paszke et al. (2019) PyTorch: an imperative style, high-performance deep learning library. External Links: 1912.01703, Link Cited by: §2.1, §8.1. [132] M. T. Pearce, T. Dooms, A. Rigg, J. Oramas, and L. Sharkey (2025) Bilinear MLPs enable weight-based mechanistic interpretability. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Figure 2, §3.2, §4.4, §5.7, §9.3. [133] R. Penrose (1971) Applications of negative dimensional tensors. In Combinatorial Mathematics and Its Applications, p. 221–244. Cited by: §2.1. [134] O. Press and L. Wolf (2017) Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, M. Lapata, P. Blunsom, and A. Koller (Eds.), Valencia, Spain, p. 157–163. External Links: Link Cited by: §3.2. [135] I. Provilkov, D. Emelianenko, and E. Voita (2020) BPE-dropout: simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 1882–1892. External Links: Link, Document Cited by: §5.1. [136] E. Puljak, S. Sanchez-Ramirez, S. Masot-Llima, J. Vallès-Muns, A. Garcia-Saez, and M. Pierini (2025) Tn4ml: tensor network training and customization for machine learning. External Links: 2502.13090, Link Cited by: §8.1. [137] Y. Qing, K. Li, P. Zhou, and S. Ran (2025) Compressing neural networks using tensor networks with exponentially fewer variational parameters. Intelligent Computing 4 (), p. 0123. External Links: Document, Link, https://spj.science.org/doi/pdf/10.34133/icomputing.0123 Cited by: §9.2. [138] Qwen Team (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §8.2. [139] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. In OpenAI Technical Report, Cited by: §1, §5.5, Table 7, Table 7, Table 9, Table 9. [140] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2023) Exploring the limits of transfer learning with a unified text-to-text transformer. External Links: 1910.10683, Link Cited by: §1, Table 8. [141] Z. Rahimi and M. M. Homayounpour (2021) TensSent: a tensor based sentimental word embedding method. Applied Intelligence 51 (8), p. 6056–6071. External Links: ISSN 0924-669X, Link, Document Cited by: §5.2. [142] D. Rai, Y. Zhou, S. Feng, A. Saparov, and Z. Yao (2025) A practical review of mechanistic interpretability for transformer-based language models. External Links: 2407.02646, Link Cited by: §1, Table 1. [143] P. Ramachandran, B. Zoph, and Q. V. Le (2017) Searching for activation functions. External Links: 1710.05941, Link Cited by: §2.4. [144] V. Sanh, L. Debut, J. Chaumond, and T. Wolf (2019) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In NeurIPS EMC2 Workshop, Cited by: §5.5. [145] U. Schollwöck (2011) The density-matrix renormalization group in the age of matrix product states. Annals of Physics 326 (1), p. 96–192. External Links: ISSN 0003-4916, Link, Document Cited by: §2.2.3. [146] R. Sennrich, B. Haddow, and A. Birch (2016) Neural machine translation of rare words with subword units. External Links: 1508.07909, Link Cited by: §5.1. [147] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. External Links: 1701.06538, Link Cited by: Table 11. [148] N. Shazeer (2019) Fast transformer decoding: one write-head is all you need. External Links: 1911.02150, Link Cited by: §2.3.4, Table 11. [149] N. Shazeer (2020) GLU variants improve transformer. External Links: 2002.05202, Link Cited by: §2.4, §4.4, §4.4. [150] N. D. Sidiropoulos and R. Bro (2000) On the uniqueness of multilinear decomposition of n-way arrays. Journal of Chemometrics 14 (3), p. 229–239. External Links: Document, Link, https://analyticalsciencejournals.onlinelibrary.wiley.com/doi/pdf/10.1002/1099-128X%28200005/06%2914%3A3%3C229%3A%3AAID-CEM587%3E3.0.CO%3B2-N Cited by: §2.2.1. [151] N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalexakis, and C. Faloutsos (2017) Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing 65 (13), p. 3551–3582. External Links: ISSN 1941-0476, Link, Document Cited by: §1, Table 1. [152] R. Solgi, K. Zhen, R. V. Swaminathan, N. Susanj, A. Mouchtaris, S. Kunzmann, and Z. Zhang (2025) Saten: sparse augmented tensor networks for post-training compression of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 23674–23683. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: Figure 2, §5.5, Table 9, Table 9, Table 11, §9.2. [153] S. Somvanshi, M. M. Islam, A. Rafe, A. G. Tusti, A. Chakraborty, A. Baitullah, T. I. Chowdhury, N. Alnawmasi, A. Dutta, and S. Das (2026) Bridging the black box: a survey on mechanistic interpretability in ai. ACM Comput. Surv. 58 (8). External Links: ISSN 0360-0300, Link, Document Cited by: §1, Table 1. [154] E. M. Stoudenmire and D. J. Schwab (2016) Supervised learning with tensor networks. In Advances in Neural Information Processing Systems, Cited by: §7.1. [155] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu (2023) RoFormer: enhanced transformer with rotary position embedding. External Links: 2104.09864, Link Cited by: §4.3. [156] Z. Su, Y. Zhou, F. Mo, and J. G. Simonsen (2024) Language modeling using tensor trains. External Links: 2405.04590, Link Cited by: §5.3. [157] M. S. Tahaei, E. Charlaix, V. P. Nia, A. Ghodsi, and M. Rezagholizadeh (2021) KroneckerBERT: learning kronecker decomposition for pre-trained language models via knowledge distillation. External Links: 2109.06243, Link Cited by: §5.5, §5.5. [158] Y. Tang, Y. Wang, J. Guo, Z. Tu, K. Han, H. Hu, and D. Tao (2024) A survey on transformer compression. External Links: 2402.05964, Link Cited by: §1, Table 1. [159] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler (2022) Charformer: fast character transformers via gradient-based subword tokenization. External Links: 2106.12672, Link Cited by: §5.1. [160] J. K. Taylor (2024) An introduction to graphical tensor notation for mechanistic interpretability. External Links: 2402.01790, Link Cited by: §1, §5.7, §9.3. [161] J. Tian, J. Lu, H. Li, X. Wang, C. C. Hao, I. Young, and Z. Zhang (2026) Ultra memory-efficient on-fpga training of transformers via tensor-compressed optimization. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 45 (3), p. 1352–1365. External Links: Document Cited by: §8.1. [162] A. Tomut et al. (2024) CompactifAI: extreme compression of large language models using quantum-inspired tensor networks. External Links: 2401.14109, Link Cited by: Figure 2, §5.5, §5.5, Table 9, Table 9. [163] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample (2023) LLaMA: open and efficient foundation language models. External Links: 2302.13971, Link Cited by: §1. [164] Touvron et al. (2023) Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Link Cited by: §5.5, §5.5, Table 8, Table 8, Table 8, Table 8, Table 8, Table 8, Table 8, Table 8, Table 8, Table 9, Table 9. [165] L. R. Tucker (1966) Some mathematical notes on three-mode factor analysis. Psychometrika 31 (3), p. 279–311. External Links: Link, Document Cited by: §1, §2.2.2, Table 3. [166] M. Usvyatsov, R. Ballester-Ripoll, and K. Schindler (2022) Tntorch: tensor network learning with PyTorch. Journal of Machine Learning Research 23 (208), p. 1–6. External Links: Link Cited by: §8.1. [167] T. Van de Cruys, T. Poibeau, and A. Korhonen (2013) A tensor-based factorization model of semantic compositionality. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, L. Vanderwende, H. Daumé I, and K. Kirchhoff (Eds.), Atlanta, Georgia, p. 1142–1151. External Links: Link Cited by: §5.2. [168] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . External Links: Link Cited by: §1, §2.3.1, §2.3.5. [169] F. Verstraete, V. Murg, and J.I. Cirac (2008) Matrix product states, projected entangled pair states, and variational renormalization group methods for quantum spin systems. Advances in Physics 57 (2), p. 143–224. External Links: ISSN 1460-6976, Link, Document Cited by: §1, §9.2. [170] G. Vidal (2008) Class of quantum many-body states that can be efficiently simulated. Physical Review Letters 101 (11). External Links: ISSN 1079-7114, Link, Document Cited by: §1, §9.2. [171] T. Vieijra, L. Vanderstraeten, and F. Verstraete (2022) Generative modeling with projected entangled-pair states. arXiv preprint arXiv:2202.08177. Cited by: §7. [172] Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury, and M. Zhang (2024) Efficient large language models: a survey. External Links: 2312.03863, Link Cited by: §1, Table 1. [173] B. Wang and A. Komatsuzaki (2021) GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. Note: https://github.com/kingoflolz/mesh-transformer-jax Cited by: §5.5, Table 9. [174] L. Wang, S. Chen, L. Jiang, S. Pan, R. Cai, S. Yang, and F. Yang (2025) Parameter-efficient fine-tuning in large models: a survey of methodologies. External Links: 2410.19878, Link Cited by: §1, Table 1. [175] M. Wang, Y. Pan, Z. Xu, G. Li, X. Yang, D. Mandic, and A. Cichocki (2025) Tensor networks meet neural networks: a survey and future perspectives. External Links: 2302.09019, Link Cited by: §1, Table 1, §1. [176] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma (2020) Linformer: self-attention with linear complexity. In arXiv preprint arXiv:2006.04768, Cited by: Table 11. [177] W. Wang, W. Chen, Y. Luo, Y. Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He (2024) Model compression and efficient inference for large language models: a survey. External Links: 2402.09748, Link Cited by: §1, Table 1. [178] M. Xu, Y. L. Xu, and D. P. Mandic (2024) TensorGPT: efficient compression of large language models based on tensor-train decomposition. External Links: 2307.00526, Link Cited by: Table 3, Figure 2, §3.2, §4, §5.2, Table 6. [179] Y. Xu, Y. Wang, X. Peng, H. Zang, C. Minghao, P. Xia, and Z. Wen (2026) TD-moe: tensor decomposition for moe models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Table 11. [180] Y. Yang, J. Zhou, N. Wong, and Z. Zhang (2024) LoRETTA: low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, p. 3161–3176. External Links: Link, Document Cited by: Figure 2, §5.4, Table 8. [181] Z. Yang, Z. Liu, S. Choudhary, X. Xie, C. Gao, S. Kunzmann, and Z. Zhang (2024) CoMERA: computing- and memory-efficient training via rank-adaptive tensor optimization. External Links: 2405.14377, Link Cited by: Table 3, Figure 2, §4.4, §5.3, §5.3, §9.2, §9.2. [182] T. Yokota (2024) Very basics of tensors with graphical notations: unfolding, calculations, and decompositions. External Links: 2411.16094, Link Cited by: §2.1. [183] Z. Yuan, Y. Shang, Y. Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y. J. Lee, Y. Yan, B. Chen, G. Sun, and K. Keutzer (2024) LLM inference unveiled: survey and roofline model insights. External Links: 2402.16363, Link Cited by: §1, Table 1, §2.3.5. [184] A. Zagitov, A. Miasnikov, M. Krutikov, V. Aletov, G. Molodtsov, N. Bashirov, A. Tsedenov, and A. Beznosikov (2026) Rethinking the role of tensor decompositions in post-training llm compression. External Links: 2606.03465, Link Cited by: §5.5. [185] L. Zhang, P. Zhang, X. Ma, S. Gu, Z. Su, and D. Song (2019) A generalized language model in tensor space. External Links: 1901.11167, Link Cited by: §5.3. [186] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao (2023) AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. External Links: 2303.10512, Link Cited by: §5.4, Table 11, §9.2. [187] S. Zhang, P. Zhang, X. Ma, J. Wei, N. Wang, and Q. Liu (2020) TensorCoder: dimension-wise attention via tensor representation for natural language modeling. External Links: 2008.01547, Link Cited by: §5.3. [188] S. Zhang et al. (2022) OPT: open pre-trained transformer language models. External Links: 2205.01068, Link Cited by: §5.5, Table 9. [189] Y. Zhang, Y. Liu, H. Yuan, Z. Qin, Y. Yuan, Q. Gu, and A. C. Yao (2025) Tensor product attention is all you need. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 3, Figure 2, Figure 5, §3.2, §5.6, Table 10, Table 11. [190] Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki (2016) Tensor ring decomposition. External Links: 1606.05535, Link Cited by: §1, §9.2. [191] Y. Zheng, T. Huang, X. Zhao, Q. Zhao, and T. Jiang (2021) Fully-connected tensor network decomposition and its application to higher-order tensor completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 11071–11078. External Links: Document Cited by: §9.2. [192] W. Zhou, Y. Gu, G. Iacovides, Y. Qiu, Q. Zhao, and D. Mandic (2026) Tensorizing engram: sharing latents across n-gram embeddings is beneficial in llms. External Links: 2606.08347, Link Cited by: Table 3, Figure 2, §4.2, §5.1, §5.2, Table 6. [193] Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, S. Yan, G. Dai, X. Zhang, Y. Dong, and Y. Wang (2024) A survey on efficient inference for large language models. External Links: 2404.14294, Link Cited by: §1, Table 1. [194] X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang (2024) A survey on model compression for large language models. External Links: 2308.07633, Link Cited by: §1, Table 1.

Instant research discovery