Paper deep dive
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Gerard Conangla Planes
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/29/2026, 3:51:34 AM
Summary
This paper provides a theoretical framework for determining the required rank for Low-Rank Adaptation (LoRA) in Transformer attention mechanisms. It establishes bounds on the Kullback-Leibler (KL) divergence between target and candidate attention distributions as a function of the LoRA rank. Key contributions include deriving lower and upper bounds on approximation error based on score differences and downstream-weighted spectral tail energy, analyzing the impact of softmax saturation, and extending these bounds to fused multi-head and joint query/key LoRA updates.
Entities (6)
Relation Signals (4)
LoRA → appliesto → Transformer Attention
confidence 95% · In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention.
Rank → determines → KL Error
confidence 93% · bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-r query LoRA update.
LoRA → extendsto → Multi-head LoRA
confidence 92% · We then extend the analysis to fused multi-head LoRA and joint query/key updates
Softmax → affects → Rank Requirement
confidence 90% · softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $\psi(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $\psi(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $\psi(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.
Tags
Links
- Source: https://arxiv.org/abs/2608.26052v1
- Canonical: https://arxiv.org/abs/2608.26052v1
Trouble viewing inline? Open PDF directly →
Full Text
89,887 characters extracted from source content.
Expand or collapse full text
How Much Rank Does LoRA Need? Rank–Error Bounds for Transformer Attention Gerard Conangla Planes †thanks: Correspondence: gerardpc@gmail.com Affiliation: Aily Labs Abstract Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback–Leibler (KL) error achievable by a rank-r query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to ψ(‖d‖2)ψ(\|d\|_2), where d is the difference between candidate and target attention scores and ψ(t)=mint2,tψ(t)= \t^2,t\. We also prove an unconditional upper bound min‖d‖22/4,2‖d‖2 \\|d\|_2^2/4, 2\|d\|_2\. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-r error between an explicit multiple of ψ(Tr)ψ( T_r) and minTr/4,2Tr \T_r/4, 2T_r\, where TrT_r is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints. 1 Introduction LoRA [11] makes it cheaper to fine-tune a pretrained Transformer [22] by replacing a dense weight update with a low-rank one. Such a low rank is both an advantage and a restriction, reducing the number of trainable parameters and computation, but also limiting the representational power of the adapter. In practice, rank is commonly chosen using rules of thumb or by training several adapters of different rank and comparing their downstream performance [11, 27, 21]. Such a sweep reveals which rank worked in a particular run, but it doesn’t tell us whether a smaller adapter lacked capacity or was simply harder to train. To isolate the representation question, suppose a dense or high-rank target adapter has already been obtained. We then ask: for each rank budget r, how closely can an adapter reproduce the target on inputs from the downstream task? The answer cannot be read from the singular values of the target weight update ΔW W alone. For instance, a large singular direction may have no effect if inputs from the task never activate it, while a smaller direction may matter on nearly every input because it repeatedly changes which tokens receive attention. The softmax function in the attention mechanism introduces two further effects. First, adding the same constant to every score leaves the attention probabilities unchanged. Second, softmax saturates, so a score error with large magnitude need not produce a proportionally large attention error. The rank needed to reproduce the target function can therefore differ from the rank suggested by the raw singular values of its weight update. In this paper, we begin with a LoRA update to the query projection of a single attention head. That update changes the scores assigned to the available keys, which softmax converts into attention probabilities. We compare these probabilities with those of a dense or high-rank target adapter, ask for the smallest expected KL error achievable at rank r, and obtain bounds on the error, some unconditional and some conditional on explicit assumptions. Attention KL therefore measures how much the query update changes the attention distribution itself, before later layers can modify that change. We stress that our results bound the attention KL described above, but we do not provide bounds on the head output or final task loss. To do so would require additional assumptions about the architecture, including the value and output projections, residual connections, feed-forward network, and later layers. These components determine how a difference in one head is propagated and may amplify, suppress, or cancel it. Without such architecture-specific assumptions, attention KL alone does not determine output or task error. Figure 1 shows the scope of our analysis within a Transformer block in more detail, including extensions to fused multi-head and joint query/key LoRA. Transformer attention with LoRA: computation and theoretical scope X(u)∈ℝn(u)×dX(u) ^n(u)× d input token representations Adapted query projection Q~h=Q0+Ah Q_h=Q_0+A_h Qh(u)=X(u)Q~hQ_h(u)=X(u) Q_h T Adapted key projection K~h=K0+Bh K_h=K_0+B_h Kh(u)=X(u)K~hK_h(u)=X(u) K_h T Fixed value projection Vh(u)=X(u)V0V_h(u)=X(u)V_0 T Sh(u)=βQh(u)Kh(u)+maskS_h(u)=β Q_h(u)K_h(u) T+mask pairwise query–key scores Ph(u)=σrow(Sh(u))P_h(u)= _ row(S_h(u)) attention probabilities for head h Oh(u)=Ph(u)Vh(u)O_h(u)=P_h(u)V_h(u) value-weighted output of head h Single-head question. For rank r, how closely can Pr,h(u)P_r,h(u) reproduce target attention P∗,h(u)P_*,h(u) on the downstream task? Error is expected KL. Repeat the displayed attention computation for h=1,…,Hh=1,…,H to obtain O1(u),…,OH(u)O_1(u),…,O_H(u) Multi-head extension. For one fused rank-r adapter, we bound ∑hKL(P∗,h∥Pr,h) _hKL(P_*,h\|P_r,h) before the head outputs are mixed by WOW_O. Concat(O1(u),…,OH(u))WOConcat(O_1(u),…,O_H(u))W_O T concatenate heads and apply the output projection residual connection, normalization, feed-forward network, and later layers Legend: input activations projection changed by LoRA standard attention operation later Transformer computation not bounded here theory annotation, not an operation Figure 1: Transformer attention with LoRA and the scope of our analysis. Read from top to bottom, the diagram traces an input through the query, key, and value projections, attention scores, row-wise softmax, and multi-head aggregation. Our central question is how closely a rank-r LoRA update can reproduce a target attention function on inputs from the downstream task. The main theorem treats query-only adaptation in one head; later results cover a fused update shared across heads and simultaneous query/key updates. For an input u, X(u)X(u) contains n(u)n(u) token representations of width d. For each head index h∈1,…,Hh∈\1,…,H\, query/key width is p and value width is pvp_v, with Qh(u),Kh(u)∈ℝn(u)×pQ_h(u),K_h(u) ^n(u)× p, Vh(u)∈ℝn(u)×pvV_h(u) ^n(u)× p_v, and Sh(u),Ph(u)∈ℝn(u)×n(u)S_h(u),P_h(u) ^n(u)× n(u). Here σrow _ row applies softmax row-wise and β is typically 1/p1/ p. The pretrained projections Q0,K0,V0Q_0,K_0,V_0 are fixed, while AhA_h and BhB_h are rank-r LoRA updates. The multi-head result bounds the sum of headwise attention KL errors before WOW_O; the remaining Transformer computation and the final model output are outside this guarantee. Overview of the results. For each rank budget r, let ℰrE_r denote the smallest expected attention KL over all rank-r candidates; Section 2 defines this objective formally. Our analysis first connects attention KL to score approximation, then connects score approximation to LoRA rank. Theorem 3.1 gives the first connection. If d is the difference between candidate and target scores after removing their common shift, define ψ(t)=mint2,t,ψup(t)=mint2/4,2t.ψ(t)= \t^2,t\, _ up(t)= \t^2/4, 2\,t\. (1) The theorem proves that the pointwise attention KL is comparable to ψ(‖d‖2)ψ(\|d\|_2) when the target probabilities are bounded away from zero, and is at most ψup(‖d‖2) _ up(\|d\|_2) without this assumption. Thus the error is quadratic for small score differences and linear for large ones. Corollary 3.2 lifts this pointwise statement to the expected, rank-constrained objective ℰrE_r. Theorem 4.1 supplies the second connection. Under its realizability, geometry, probability-floor, and moment assumptions, it proves cloψ(Tr)≤ℰr≤ψup(Tr).c_ lo\,ψ( T_r) _r≤ _ up( T_r). (2) Here TrT_r is the residual tail energy after the target update has been weighted by the queries and keys occurring on the downstream task. Directions that the task never uses therefore do not contribute, and cloc_ lo is the explicit assumption-dependent constant in Theorem 4.1. The upper bound constructs a candidate whose error does not exceed its curve; the lower bound applies to every candidate. Together they bracket the best achievable error at each rank. Section 9 explains how to estimate this rank–error curve from downstream data and compare it with a desired error tolerance. When the probability floor in the main theorem is too small to give a useful lower bound, Section 5 provides two alternatives with different assumptions (Figure 2 shows a decision diagram to clarify the choices, and Table 2 gives an overview of the three laws of approximation). Theorem 5.1 gives target-Fisher upper and lower bounds for candidates whose scores remain within a fixed range of the target scores. Because that candidate class is restricted, its lower bound does not apply to the unrestricted ℰrE_r. Theorem 5.2 instead gives a lower bound on the unrestricted ℰrE_r by focusing on a subset of tokens carrying most of the target probability mass. The preceding results concern approximation at finite scores. Theorem 6.1 treats a different effect: after softmax saturates, a lower-rank sequence of logits can approach an attention function that requires higher rank to realize with finite logits. It constructs an explicit family with a constant-factor separation between these two ranks. Finally, Theorem 7.1 extends the rank–KL bounds to a fused update shared across attention heads, and Theorem 8.1 extends them to simultaneous query/key LoRA, where the score update must also satisfy a query/key factorization constraint. Contributions. 1. Under explicit assumptions, we characterize the best task-dependent attention approximation available at each LoRA rank. Global softmax bounds convert centered score error into KL, and a downstream-weighted spectral theorem converts the remaining score error into an explicit function of rank. 2. We give two alternatives when the main lower-bound constant is weak: target-Fisher bounds for a score-restricted candidate class, and a high-mass lower bound for the original unrestricted class. 3. We prove an exact separation for an explicit Walsh family: matching its finite logits requires rank k, but after softmax saturation the same limiting attention can be recovered to arbitrary accuracy at rank k−⌊k/3⌋k- k/3 within the score space reachable by query adaptation. 4. We extend the rank–KL analysis to a fused update shared across heads and to joint query/key LoRA, where a separate factorization gap measures whether the best effective score update can be realized by the two factors. Positioning relative to prior work. Prior work studies adaptive rank allocation during training [27, 18], activation-aware approximation [24, 4, 23], finite-sample LoRA rank selection [12], and the transformations that LoRA can express [26]. Our contribution is different: for a fixed pretrained head and a known target attention function, we bound the best attention KL attainable at every rank, averaged over u∼Pu P, the downstream task distribution (e.g. text-to-SQL examples). The underlying weighted approximation theorem is classical [7, 16]; the new step is to connect that task-weighted approximation problem to global upper and lower bounds on attention KL. Section 10 gives a more detailed comparison with these and the related attention-rank and softmax-saturation literature. 2 Problem formulation We begin with one query position in one attention head, so that every matrix and function can be defined explicitly. Sections 7 and 8 extend the formulation. Let P denote the downstream input distribution and draw u∼Pu P, with n(u)n(u) available token positions. Fix pretrained logits z0(u)∈ℝn(u)z_0(u) ^n(u), a pretrained key matrix K(u)∈ℝn(u)×dkK(u) ^n(u)× d_k, a query activation h(u)∈ℝdhh(u) ^d_h, and the attention scale β>0β>0. A query LoRA update is a matrix M∈ℝdk×dhM ^d_k× d_h with rank(M)≤rrank(M)≤ r. It produces the score vector and attention distribution zM(u)=z0(u)+βK(u)Mh(u),pM(⋅∣u)=softmax(zM(u)).z_M(u)=z_0(u)+β K(u)Mh(u), p_M(· u)=softmax(z_M(u)). (3) Here h(u)h(u) is the query representation entering the adapted projection, and the rows of K(u)K(u) are the key vectors against which that query is scored. M is one stored matrix; the attention distribution pM(⋅∣u)p_M(· u) changes with the input u. Let z∗(u)z_*(u) be fixed target logits and p∗(⋅∣u)=softmax(z∗(u))p_*(· u)=softmax(z_*(u)) the target attention function. The target may be arbitrary for the score-space results in Section 3. The spectral specialization in Theorem 4.1 requires it to be generated exactly by a dense query update Δ∗ _*. A generic fully fine-tuned Transformer can also change keys, values, earlier layers, and the activations entering this head, so it need not satisfy that assumption. For such a misspecified target, Equation (4) remains well defined, but the query-only spectral formula would need an extra irreducible approximation term. For rank budget r, define ℰr=infrank(M)≤ru∼PKL(p∗(⋅∣u)∥pM(⋅∣u)). _r= _rank(M)≤ rE_u PKL\! (p_*(· u)\,\|\,p_M(· u) ). (4) Thus ℰrE_r is the best error attainable by the entire rank-r candidate class. An upper bound exhibits a candidate with small error. A lower bound applies to every candidate and certifies unavoidable error. Inverting the bounds for a tolerance ϵε then yields sufficient and necessary rank conditions. Section 9 turns this interpretation into a calibration procedure. Softmax is invariant to adding a constant to all scores. We therefore center score differences with Πn=In−1n,dM(u)=Πn(u)(zM(u)−z∗(u)). _n=I_n- 1n11 T, d_M(u)= _n(u) (z_M(u)-z_*(u) ). (5) The associated approximation objective, robust to both small and large score errors, is Ψr=infrank(M)≤ru∼Pψ(‖dM(u)‖2). _r= _rank(M)≤ rE_u Pψ\! (\|d_M(u)\|_2 ). (6) For the sharper upper bound, also define Φr=infrank(M)≤ru∼Pψup(‖dM(u)‖2). _r= _rank(M)≤ rE_u P _ up\! (\|d_M(u)\|_2 ). (7) Object Meaning What changes? u One input from the downstream task Varies across examples z0(u)z_0(u) Attention scores produced by the pretrained head Depends on u p∗(⋅∣u)p_*(· u) Target attention probabilities to be reproduced Depends on u; the target model is fixed M Candidate query update with rank(M)≤rrank(M)≤ r One fixed matrix for all inputs pM(⋅∣u)p_M(· u) Attention probabilities produced with update M Depends on u and on the chosen M dM(u)d_M(u) Candidate–target score difference after removing a common shift Depends on u and on the chosen M ℰrE_r Smallest average target-to-candidate KL achievable at rank r One number for each rank budget Table 1: Objects in the target–candidate comparison. For each downstream input u, the pretrained head and fixed target determine the behavior to be matched; M ranges over candidate updates of rank at most r. Example 2 grounds this objective in a concrete downstream task. Example 1: Interpreting attention KL in text-to-SQL An input may contain the user’s question, the database schema, instructions, and previously generated SQL tokens. The target and candidate heads each assign attention probabilities to those token positions. Our objective asks how much rank is needed for the candidate to reproduce the target’s attention on such inputs. This gives a head-level approximation measure that can be studied alongside end-to-end measures such as SQL execution accuracy. 3 From score error to attention KL The first result is independent of LoRA: it relates centered score error to probability error. Theorem 3.1 (Global robust softmax bounds). Let d∈ℝnd ^n satisfy d=01 Td=0, let p∗=softmax(z∗)p_*=softmax(z_*), and put a=minip∗,i>0a= _ip_*,i>0. Then a2e2ψ(∥d∥2)≤KL(p∗∥softmax(z∗+d))≤ψup(∥d∥2). a2e^2ψ(\|d\|_2) \! (p_*\,\|\,softmax(z_*+d) )≤ _ up(\|d\|_2). (8) The upper bound does not require a positive lower bound on a. The quadratic part of ψ comes from local curvature of log-sum-exp. The linear part is unavoidable globally: a rare input can contain a very large score error while contributing only linear or bounded KL. A purely quadratic global lower bound therefore cannot hold; Proposition B.1 gives an explicit fixed-probability-floor construction. Corollary 3.2 (Global functional bounds). If minip∗,i(u)≥a>0 _ip_*,i(u)≥ a>0 almost surely, then a2e2Ψr≤ℰr≤Φr. a2e^2 _r _r≤ _r. (9) The proof of Theorem 3.1 is short and appears in Appendix A. The rare-context obstruction to a quadratic law is given in Appendix B. 4 Task-weighted spectral rank–KL bounds Assume in this section that the target is exactly realizable by a dense query update Δ∗ _*. For A=M−Δ∗A=M- _*, define G(u)=β2K(u)Πn(u)K(u),Σ=[h(u)h(u)].G(u)=β^2K(u) T _n(u)K(u), =E[h(u)h(u) T]. (10) G(u)G(u) records which update directions alter centered attention scores in context u; Σ records which query directions appear on the task. In the separable case, the random key Gram matrix G(u)G(u) is independent of the query activation h(u)h(u). Then ‖βΠn(u)K(u)Ah(u)‖22=‖G1/2AΣ1/2‖F2,G=G(u).E\|β _n(u)K(u)Ah(u)\|_2^2=\|G^1/2A ^1/2\|_F^2, G=EG(u). (11) The matrix whose spectrum matters is therefore D∗=G1/2Δ∗Σ1/2,Tr=∑j>rσj(D∗)2.D_*=G^1/2 _* ^1/2, T_r= _j>r _j(D_*)^2. (12) TrT_r is the residual squared score error after the best rank-r update. Directions annihilated by the keys or never activated by h(u)h(u) disappear automatically, a phenomenon that is illustrated in Example 4. Example 2: A large update direction that the task never uses Suppose the target update has two singular directions. The first has the larger singular value, but every query activation from the downstream task is orthogonal to it, so it never changes an attention score. The second is smaller but active on nearly every input. A raw truncated SVD keeps the first direction; the weighted spectrum above removes it and keeps the direction that actually changes attention. Theorem 4.1 turns this distinction into upper and lower error bounds. To turn this quadratic tail into a global robust lower bound, assume that, for every deterministic compatible matrix C, ∥Ch(u)∥24≤κh(∥Ch(u)∥22)2,G(u)⪯ΛGalmost surely.E\|Ch(u)\|_2^4≤ _h (E\|Ch(u)\|_2^2 )^2, G(u) G surely. (13) These uniform moment conditions prevent the quadratic error from being carried entirely by extremely rare contexts. Theorem 4.1 (Downstream spectral rank–KL bounds). Suppose the target attention function is generated by a dense query update Δ∗ _*, minip∗,i(u)≥a>0 _ip_*,i(u)≥ a>0 almost surely, G(u)G(u) and h(u)h(u) are independent, and the two conditions in Equation (13) hold. Then a2e2(1+Λκh)ψ(Tr)≤ℰr≤ψup(Tr). a2e^2(1+ _h)ψ( T_r) _r≤ _ up( T_r). (14) The upper bound is achieved by truncated SVD of D∗D_*, which already weights the target update by the task queries and keys. The lower bound applies to every rank-r candidate. Both sides are quadratic in the score scale near zero and linear at large scale, but their constants can be far apart when the target probability floor a is small. For a desired error tolerance ϵε, the upper bound gives the sufficient rank rsuff(ϵ)=minr:ψup(Tr)≤ϵ.r_ suff(ε)= \r: _ up( T_r)≤ε\. (15) Conversely, any rank attaining error at most ϵε must satisfy ψ(Tr)≤2e2(1+Λκh)aϵ.ψ( T_r)≤ 2e^2(1+ _h)aε. (16) Inverting the error bounds in this way yields sufficient and necessary rank conditions. Remark 4.2 (Size of the bracket). The worst-case ratio between the upper and lower constants is at most 22e2(1+Λκh)a. 2 2e^2(1+ _h)a. (17) For example, it is about 2.0×1042.0× 10^4 when a=10−2a=10^-2, Λ=5 =5, and κh=3 _h=3. This is a sensitivity calculation from the theorem, not an empirical estimate. For long, peaked attention vectors, the probability floor can make the global lower bound numerically weak; Section 5 gives alternatives. The proof is in Appendix C. In ordinary self-attention, G(u)G(u) and h(u)h(u) are computed from the same input and need not be independent. The following result replaces independence by direct comparison between the true mean squared score error and the reference quadratic form used to define TrT_r. For A=M−Δ∗A=M- _*, write Q(A)=‖βΠn(u)K(u)Ah(u)‖22,S(A)=‖G1/2AΣ1/2‖F2,Q(A)=E\|β _n(u)K(u)Ah(u)\|_2^2, S(A)=\|G^1/2A ^1/2\|_F^2, (18) and let ArsvdA_r svd be the weighted-SVD candidate used in the upper bound. Theorem 4.3 (Dependence-allowing spectral bounds). Suppose, for every feasible A=M−Δ∗A=M- _*, Q(A)≥crS(A),‖βΠn(u)K(u)Ah(u)‖24≤κrQ(A)2,Q(A)≥ c_rS(A), \|β _n(u)K(u)Ah(u)\|_2^4≤ _rQ(A)^2, (19) and suppose Q(Arsvd)≤Cr+TrQ(A_r svd)≤ C_r^+T_r. If the target probability floor is at least a>0a>0, then a2e2(1+κr)ψ(crTr)≤ℰr≤ψup(Cr+Tr). a2e^2(1+ _r)ψ( c_rT_r) _r≤ _ up( C_r^+T_r). (20) This theorem allows arbitrary dependence between keys and queries; its price is that the comparison and moment constants must hold on the candidate class. For example, if λG⪯G(u)⪯ΛGλ G G(u) G almost surely and the activation moment condition in Equation (13) holds, it applies with cr=λc_r=λ, Cr+=ΛC_r^+= , and κr=κh(Λ/λ)2 _r= _h( /λ)^2. The proof is in Appendix D. 5 Which rank–KL bound should be used? Because the minimum target probability can be very small in long, peaked attention vectors, we provide two refinements that address different needs. The target-Fisher theorem gives upper and lower bounds when score differences stay inside a fixed range. The high-mass theorem gives a lower bound for the original unrestricted class, using a set of tokens, chosen from the target, that carries most probability mass. Figure 2 and Table 2 make the distinction explicit: Need a lowerbound as well?All targetprobabilities boundedaway from zero?Score differencesin a fixed range?A few tokens carrymost target mass? Global upper bound No probability floor. Score-space Thm. 3.1. Global two-sided bounds Unrestricted rank-r. Thm. 4.1. Target-Fisher bounds Justified score range. Thm. 5.1, App. E. Combine two results Upper: Cor. 3.2. Lower: Thm. 5.2, App. F. Upper bound only No unrestricted lower bound. Thm. 3.1. yesnonoyesnoyesyesno Figure 2: Which rank–KL theorem to use. Start at the left and stop at the first result whose assumptions you can justify. An upper bound shows that some rank-r update achieves at most the stated error; a lower bound shows that every rank-r update incurs at least the stated error. The Fisher bounds apply only to candidates whose scores remain within a fixed range of the target scores. Consequently, their lower bound does not apply to the unrestricted optimum ℰrE_r. Route Upper bound Lower bound Candidate class Main price Global ✓ ✓ unrestricted rank-r minimum target probability and moment assumptions on keys/queries Target Fisher ✓ ✓ rank-r with a justified range of score differences span-dependent constants and the target Fisher inner product High mass × ✓ unrestricted rank-r tokens carrying most target mass, a conditional probability floor, and the key/query Gram on that set Table 2: The three rank–KL routes and their logical scope. 5.1 Target-Fisher bounds under a controlled score span Let H∗=diag(p∗)−p∗p∗H_*=diag(p_*)-p_*p_* T and let R(d)=maxidi−minidiR(d)= _id_i- _id_i. Define c−(R)=R−1+e−R2,c+(R)=eR−1−R2,c_-(R)= R-1+e^-RR^2, c_+(R)= e^R-1-RR^2, (21) with both values set to 1/21/2 at R=0R=0. Theorem 5.1 (Target-Fisher score bounds). For every score displacement d, c−(R(d))dH∗d≤KL(p∗∥softmax(z∗+d))≤c+(R(d))dH∗d.c_-(R(d))d TH_*d (p_*\|softmax(z_*+d))≤ c_+(R(d))d TH_*d. (22) Consequently, on the candidate class whose score differences from the target have range at most R0R_0, the optimal KL is bounded above and below by c+(R0)c_+(R_0) and c−(R0)c_-(R_0) times the corresponding target-Fisher quadratic optimum. There is no explicit minimum-probability constant; small target probabilities enter through the Fisher matrix H∗H_*. More importantly, the optimized class is restricted by the span condition. Because that class is a subset of the unrestricted rank-r class, its lower bound cannot be transferred to ℰrE_r. The spectral statements, including the extra span condition needed for a matching upper bound, and the proofs are in Appendix E. 5.2 An unrestricted high-mass lower bound For each context, let S(u)S(u) be a measurable token set chosen from the target before optimizing the candidate. Suppose it carries at least 1−δ1-δ of the target mass and the conditional target distribution on S(u)S(u) has minimum probability at least acorea_core. Theorem 5.2 (High-mass lower bound). Under the preceding assumptions, ℰr≥(1−δ)acore2e2infrank(M)≤rψ(‖ΠS(u)(zM(u)−z∗(u))S(u)‖2).E_r≥ (1-δ)a_core2e^2 _rank(M)≤ rEψ\! ( \| _S(u)(z_M(u)-z_*(u))_S(u) \|_2 ). (23) This is a lower bound on the original unrestricted ℰrE_r. Under additional control of the key/query Gram and moments on S(u)S(u), the right-hand side reduces to the same spectral tail, but only on those tokens. It does not give a full-KL upper bound: matching conditional scores inside S(u)S(u) need not match the total mass assigned to S(u)S(u). The proof and spectral forms are in Appendix F. 6 Saturation changes the relevant notion of rank The main theorem describes approximation of finite logits, but softmax also has a boundary regime in which logits can diverge while their distributions converge. Example 6 shows the basic saturation mechanism before the rank separation is stated formally. Example 3: Different logits, the same saturated attention For two tokens, softmax(T,0)⟶(1,0),softmax(2T,0)⟶(1,0)as T→∞.softmax(T,0) (1,0), (2T,0) (1,0) T→∞. The difference between the two logit vectors grows with T, but their attention distributions converge to the same limit. The results below prove a stronger effect: for explicit families of targets, saturation reduces the rank needed to approximate the entire attention function. For a boundary target P∞P_∞ and an allowed centered score space L, define its L-relative softmax closure rank rclL(P∞)r_ cl^L(P_∞) as the smallest r for which there is a sequence of score matrices of rank at most r, with every score column in L, whose columnwise softmax distributions converge to P∞P_∞. For query adaptation, L is the score space reachable through the fixed key matrix, so the restriction is part of the LoRA model. This rank can be smaller than the rank required to match every finite target logit exactly. Theorem 6.1 (Exact closure rank of an isolated-triple Walsh family). For every k≥3k≥ 3, there is a query-only Walsh-attention family with k contexts and fewer than 4k2+84k^2+8 token positions such that every exact realization with finite logits has update rank k. For the limiting target P∞(k)P_∞^(k) of this family and its Walsh score space LkL_k, the exact relative closure rank is rclLk(P∞(k))=k−⌊k/3⌋.r_cl^L_k(P_∞^(k))=k- k/3 . (24) This equality is for the isolated-triple family constructed in Appendix G; other target families can have smaller closure rank, as the construction below demonstrates. The construction groups Walsh characters into isolated triples. A rank-two path per triple gives the upper bound. For the lower bound, restricted Fourier identities and columnwise normalization force every limiting triple block to retain rank at least two. Saturation therefore yields a genuine constant-factor reduction, but in this family it cannot collapse the rank of the attention function to o(k)o(k). A separate linear-token family gives a stronger achieved ratio: with fewer than 8(k+1)8(k+1) token positions, rank at most k−max⌊k/3⌋,3⌊k/7⌋k- \ k/3 ,3 k/7 \ (25) attains vanishing KL, reaching ratio 4/74/7 on complete seven-context blocks. That construction concerns a different target family and does not identify its minimum closure rank. The two results and their proofs are kept separate in Appendix G. 7 Fused multi-head LoRA Standard query LoRA for multi-head attention often constrains one fused projection matrix, vertically divided into head blocks. This is not the same as assigning an independent rank to every head: one fused rank direction can serve several heads. For j=1,…,Hj=1,…,H, let Kj(u)∈ℝnj(u)×dk,jK_j(u) ^n_j(u)× d_k,j and let all heads share the query activation h(u)∈ℝdhh(u) ^d_h. Stack the head updates as M=(M1,…,MH)M=(M_1 T,…,M_H T) T and impose the single fused constraint rank(M)≤rrank(M)≤ r. Assume the target is generated by the similarly stacked dense update Δ∗MH _*^MH. Define dM,j(u) d_M,j(u) =βjΠnj(u)Kj(u)(Mj−Δ∗,j)h(u), = _j _n_j(u)K_j(u)(M_j- _*,j)h(u), (26) ℰrMH _r^MH =infrank(M)≤r∑j=1HKL(p∗,j(⋅∣u)∥pM,j(⋅∣u)), = _rank(M)≤ rE _j=1^HKL(p_*,j(· u)\|p_M,j(· u)), (27) GMH(u) G_MH(u) =blockdiag(βj2Kj(u)Πnj(u)Kj(u):j=1,…,H),GMH=GMH(u), =blockdiag\! ( _j^2K_j(u) T _n_j(u)K_j(u):j=1,…,H ), G_MH=EG_MH(u), (28) TrMH T_r^MH =∑j>rσj(GMH1/2Δ∗MHΣ1/2)2,Σ=[h(u)h(u)]. = _j>r _j\! (G_MH^1/2 _*^MH ^1/2 )^2, =E[h(u)h(u) T]. (29) Theorem 7.1 (Fused multi-head rank–KL bounds). Suppose every target probability is at least amin>0a_ >0 almost surely, GMH(u)G_MH(u) is independent of h(u)h(u), GMH(u)⪯ΛMHGMHG_MH(u) _MHG_MH almost surely, and ‖Ch(u)‖24≤κh(‖Ch(u)‖22)2for every compatible deterministic C.E\|Ch(u)\|_2^4≤ _h (E\|Ch(u)\|_2^2 )^2 every compatible deterministic C. (30) Then amin2e2(1+ΛMHκh)ψ(TrMH)≤ℰrMH≤minTrMH/4,2HTrMH. a_ 2e^2(1+ _MH _h)ψ( T_r^MH) _r^MH≤ \T_r^MH/4, 2HT_r^MH\. (31) The upper/lower bracket can widen by a factor of order H H because the objective sums KL over heads; the aggregation inequality producing this factor is tight without further assumptions on how error is distributed across heads. If each head instead has its own adapter and integer rank rhr_h, the feasible set separates after the ranks are fixed; allocating a total budget R is the discrete problem min∑h∑hrh≤Rℰh,rh _ _hr_h≤ R _hE_h,r_h. The proof and the underlying pointwise aggregation inequality are in Appendix H. 8 Joint query/key LoRA Practical LoRA may update both queries and keys. Let Q0,K0∈ℝp×dQ_0,K_0 ^p× d be pretrained factors and let A,B∈ℝp×dA,B ^p× d be query and key updates with ranks at most rQ,rKr_Q,r_K. On ordinary dot-product scores they change the scores by the effective update C(A,B)=(K0+B)(Q0+A)−K0Q0=K0A+B(Q0+A).C(A,B)=(K_0+B) T(Q_0+A)-K_0 TQ_0=K_0 TA+B T(Q_0+A). (32) The bound rank(C(A,B))≤rQ+rKrank(C(A,B))≤ r_Q+r_K is known [26]: grouping the bilinear term gives the sharp containment rank(C(A,B))≤rQ+rK.rank(C(A,B))≤ r_Q+r_K. (33) Let a context provide token representations X(u)∈ℝn(u)×dX(u) ^n(u)× d and a query representation h(u)∈ℝdh(u) ^d. For target factors (A∗,B∗)(A_*,B_*), put C∗=C(A∗,B∗)C_*=C(A_*,B_*) and define the centered score error and actual factor-class objective dA,B(u) d_A,B(u) =βΠn(u)X(u)[C(A,B)−C∗]h(u), =β _n(u)X(u)[C(A,B)-C_*]h(u), (34) ℰrQ,rKQK ^QK_r_Q,r_K =infrank(A)≤rQrank(B)≤rKKL(p∗∥pA,B). = _ subarraycrank(A)≤ r_Q\\ rank(B)≤ r_K subarrayEKL(p_*\|p_A,B). (35) Let GX(u)=β2X(u)Πn(u)X(u),GX=GX(u),Σ=[h(u)h(u)].G_X(u)=β^2X(u) T _n(u)X(u), G_X=EG_X(u), =E[h(u)h(u) T]. (36) If GX(u)G_X(u) and h(u)h(u) are independent, the mean squared score error equals ‖GX1/2[C(A,B)−C∗]Σ1/2‖F2\|G_X^1/2[C(A,B)-C_*] ^1/2\|_F^2. Define D∗eff=GX1/2C∗Σ1/2,Tseff=∑j>sσj(D∗eff)2,s=rQ+rK.D_*^eff=G_X^1/2C_* ^1/2, T_s^eff= _j>s _j(D_*^eff)^2, s=r_Q+r_K. (37) Let ssvdS_s svd be the set of rank-s matrices attaining the weighted approximation error TseffT_s^eff. An element of this set need not be writable as query and key updates of size p×dp× d with the separate rank budgets. Define the extra error from that factorization gap by ρrQ,rK=infCs∈ssvdinfrank(A)≤rQrank(B)≤rK‖GX1/2[C(A,B)−Cs]Σ1/2‖F. _r_Q,r_K= _C_s _s svd _ subarraycrank(A)≤ r_Q\\ rank(B)≤ r_K subarray\|G_X^1/2[C(A,B)-C_s] ^1/2\|_F. (38) Theorem 8.1 (Joint Q/K rank–KL bounds). Suppose minip∗,i(u)≥a>0 _ip_*,i(u)≥ a>0 almost surely, GX(u)G_X(u) is independent of h(u)h(u) so that the mean-squared identity above holds, and, for every feasible pair (A,B)(A,B), ‖dA,B(u)‖24≤κ(‖dA,B(u)‖22)2.E\|d_A,B(u)\|_2^4≤κ (E\|d_A,B(u)\|_2^2 )^2. (39) Then a2e2(1+κ)ψ(Tseff)≤ℰrQ,rKQK≤ψup(Tseff+ρrQ,rK). a2e^2(1+ κ)ψ( T_s^eff) ^QK_r_Q,r_K≤ _ up\! ( T_s^eff+ _r_Q,r_K ). (40) The lower bound follows from containment in the effective rank-s class. The reverse containment generally fails: every updated width-p head also satisfies rank(K0Q0+C)≤prank(K_0 TQ_0+C)≤ p. Exact one-sided support conditions or a sequential two-sided factorization make ρrQ,rK=0 _r_Q,r_K=0; otherwise that extra error can be large. The realization lemmas, a constructive target-factor upper bound, and a fused multi-head lower bound are proved in Appendix I. The two sides form a useful bracket only when ρrQ,rK _r_Q,r_K is comparable to, or smaller than, Tseff T_s^eff. In general, computing ρrQ,rK _r_Q,r_K is a nonconvex factorization problem over the weighted rank-s optimizers; the theorem does not provide a tractable procedure or a universal upper bound for it. The cases with ρrQ,rK=0 _r_Q,r_K=0 therefore identify the cleanest regime for this extension. RoPE inserts a relative rotation between the query and key factors. Every relative-position slice still has rank at most rQ+rKr_Q+r_K, so the global robust bound remains valid for the assembled scores. The candidate is then a coupled family of effective matrices, however, and the single-matrix spectral tail in Theorem 8.1 does not apply automatically. This RoPE-specific obstruction is not universal. Architectures that omit explicit positional embeddings and use NoPE, such as Kimi K3 [13], keep token order only through causal computation. For those heads, the ordinary dot-product formulation is the more direct starting point, subject to the model’s other attention mechanisms. 9 Estimating LoRA rank from downstream data The theory can be used once a dense or high-rank target update and a calibration set from the downstream task are available. Its output is a pair of error curves over rank: an upper curve achieved by a constructed candidate and a lower curve that no candidate in the stated class can beat. These curves may identify one rank, or they may leave an interval that the theory does not resolve. For query-only adaptation, the procedure is: 1. Check the target class. Verify that the target attention is generated by a query update Δ∗ _*. If both queries and keys change, use Section 8; a target produced by unrestricted fine-tuning may also contain an irreducible query-only approximation error. 2. Collect calibration inputs. Sample complete inputs from the downstream task and run the fixed pretrained and target heads. Store the query activations h(u)h(u), centered key Gram matrices G(u)G(u), and target attention probabilities. Complete inputs, rather than individual token positions treated as independent observations, are the sampling units. 3. Choose the applicable bound. Use Figure 2. If the separability assumptions of Theorem 4.1 are not credible, use the dependence-allowing geometry in Appendix D. If the target probability floor is too small, evaluate the target-Fisher or high-mass route instead. 4. Estimate the spectral tail. For the main route, estimate G=G(u)G=EG(u) and Σ=[h(u)h(u)] =E[h(u)h(u) T], form D^∗=G^1/2Δ∗Σ^1/2 D_*= G^1/2 _* ^1/2, and compute T^r=∑j>rσj(D^∗)2 T_r= _j>r _j( D_*)^2 for every rank of interest. The truncated SVD of D^∗ D_* gives the corresponding upper-bound candidate in the weighted coordinates. On the supported subspaces, the corresponding query update is M^r=G^†/2(D^∗)rΣ^†/2 M_r= G /2( D_*)_r /2, where (D^∗)r( D_*)_r is the rank-r truncated SVD. 5. Supply and stress-test the constants. The probability floor, almost-sure geometry bound, and uniform moment constant are assumptions of the population theorem, not quantities certified by a finite calibration set. Use analytic bounds when available, or report LrL_r over a range of assumed values. Sample estimates are diagnostics, not finite-sample certificates. The upper curve UrU_r comes from the constructed candidate and does not require these lower-bound constants. 6. Compare with the desired tolerance. If Ur≤ϵU_r≤ε, the upper bound exhibits a rank-r candidate within tolerance. If Lr>ϵL_r>ε, no rank-r candidate in the relevant class can meet the tolerance. The smallest rank certified by the upper curve is sufficient; ranks rejected by the lower curve are impossible under the theorem’s assumptions. Any gap between the two remains unresolved. Example 4: Estimating rank on a given calibration set Suppose a high-rank query adapter has already been trained, for instance for a text-to-SQL task as in Example 2. A calibration set contains complete prompts with the question, schema, instructions, and SQL prefix available at each attention call. The upper and conditional lower curves divide the tested ranks into three sets: ranks whose constructed candidate meets the KL tolerance, ranks excluded by the conditional lower bound, and ranks for which the two bounds do not decide. The width of the unresolved set depends on the gap between the upper and lower constants and should be reported rather than replaced by a single selected rank. The population theorems do not provide confidence intervals for these plug-in estimates. A held-out calibration split can test whether the predicted rank–KL curve tracks measured target-to-candidate KL, but it does not turn the current bounds into a finite-sample guarantee. The relevant comparison is against the raw spectrum of Δ∗ _* and activation-only rank criteria: the question is whether task-weighted tails predict attainable attention KL more accurately, not whether a particular optimizer produces a U-shaped validation curve. 10 Related work LoRA and expressivity. LoRA introduced low-rank updates as a parameter-efficient adaptation mechanism [11]. Zeng and Lee [26] analyze LoRA expressivity, including products of adapted matrices and exact Transformer score matching. We use their bound rank(C)≤rQ+rKrank(C)≤ r_Q+r_K in the joint Q/K analysis and then quantify the additional error caused when the best effective score update cannot be realized by factors with the separate rank budgets. Duranthon et al. [6] give a statistical theory of rank-one LoRA fine-tuning in a solvable attention model; their object is learning-curve asymptotics rather than error as a function of rank for a prescribed target. Arunan [12] studies finite-sample estimation and rank selection for empirical risk minimization over rank-constrained LoRA. In a well-specified locally quadratic model, that work derives matching Θ~(rd/n) (rd/n) estimation rates and an under-ranking bias governed by the raw singular values of the target update. That work studies statistical estimation from finite training data. We hold the target fixed and study the approximation error remaining at each rank, with a spectrum weighted by the task queries and keys and a global error scale determined by softmax. Weighted approximation. The weighted low-rank approximation theorem is a classic result. [7, 16]. Methods such as DRONE and SVD-LLM use related data-aware decompositions for model compression [4, 23]. CorDA orients a weight decomposition by downstream activation covariance, while EVA initializes and reallocates LoRA rank using activation variance [24, 18]. These methods motivate task-aware approximation directions. Our analysis places the weighted approximation inside the target-to-candidate attention KL and derives both upper and lower bounds, including the global scale ψ needed for small and large score errors. Task-intrinsic attention rank and low-rank logits. Yoon [25] defines attention-native intrinsic rank as the minimum query–key kernel rank realizing a task and studies controlled deficiency and recovery phenomena. We study the narrower class of LoRA updates of a pretrained head, and a prescribed attention target under KL. Golowich et al. [8] study low-rank approximation of extended language-model logit matrices under average KL, without pretrained LoRA attention or separate Q/K update constraints. Attention rank and saturation. Attention head dimension and score-matrix rank have been studied as expressivity constraints [3]. Sparse-target cross-entropy and diverging-margin geometry are adjacent to our saturation branch [28]; low pre-softmax rank can also produce high post-softmax rank [15]. Boundary supports of fixed discrete exponential families are classical [5, 20]. Sign rank and rounding rank concern finite, exact realization of prescribed sign or threshold patterns [19, 17], whereas our closure rank allows a sequence of scores to diverge while its softmax converges. Our lower bound uses restricted Fourier identities, columnwise normalization, and the closedness of a bounded-rank matrix set. We do not know a direct implication in either direction between the cited Walsh sign-rank bounds and the L-relative closure-rank formula in Theorem 6.1. Low-rank softmax models have also been studied through argmaxability [9, 10]. Basri and Jacobs [2] show that low-dimensional softmax embeddings can preserve ratios among selected top-token probabilities while off-support mass vanishes. These results establish the broad support-separation and diverging-margin mechanisms. Our saturation theorem makes a different comparison within one explicit multi-context family: the rank required to match finite logits versus the exact softmax closure rank. Softmax curvature. The Hessian comparison behind our target-Fisher bounds is related to generalized self-concordant analyses of logistic and softmax losses [1, 14]. We specialize this curvature principle through a finite-dimensional Bregman calculation for KL from the target attention to the candidate. 11 Limitations The theory starts from an available target update. It can assess compression of that target or the rank needed to reproduce it, but it does not prescribe a rank before any target behavior is known. The bounds are population statements. From a calibration set, one can plug in estimates of the task geometry (G, Σ ) and the corresponding spectral tails, although we do not provide finite-sample confidence intervals for the resulting rank–KL curves. Other inputs to the lower bounds—the probability floor, the almost-sure geometry constant, and the uniform moment constant—enter as theorem assumptions rather than as quantities certified by a finite sample, so a practical lower curve must rely on analytic bounds or report sensitivity to those values. The spectral specialization likewise assumes that the target attention is realizable within the stated query-only or joint query/key adapter class; unrestricted fine-tuning may leave an additional approximation error outside that class. The results describe the best candidate in a rank-constrained class, not the path taken by a training algorithm. They therefore separate representational capacity from optimization but do not guarantee that SGD or another optimizer finds the candidate attaining the upper bound. The constants can also be loose for long, sharply peaked attention vectors. The target-Fisher and high-mass results offer alternatives in that regime, but their own assumptions must still be checked. We have not established that any of these constants are numerically tight on trained, real-model attention heads (for more information on computational checks, see Appendix Computational checks.). Finally, the guarantees concern attention probabilities. Converting them into bounds on layer outputs or task loss requires assumptions about values, output projections, residual paths, feed-forward networks, and later layers. The saturation families isolate a genuine nonlinear effect but are explicit constructions rather than models of typical language data. For joint query/key LoRA, RoPE couples the factors across relative positions, so the single-matrix spectral specialization does not apply directly, although it is worth noting that this particular obstruction is absent in NoPE architectures such as Kimi K3 [13]. 12 Conclusion We studied how much LoRA rank is needed to reproduce a known attention function on inputs from a downstream task. The main message is that this question has a task-dependent answer: it is not settled by the raw rank or singular values of the target weight update alone, but by the part of that update that the task’s queries and keys actually activate. Under explicit assumptions, we connect that activated component to the best expected attention KL attainable at each rank. A global softmax comparison turns centered score error into KL, and a downstream-weighted spectral theorem turns the remaining score error into an explicit function of rank through the scale ψ(t)=mint2,tψ(t)= \t^2,t\. Together, these steps replace rank heuristics and post-hoc sweeps with a two-sided, rank-indexed bracket on representational error at the attention layer itself. The same framework extends when the standard constants are weak or the adapter is richer than query-only LoRA. Target-Fisher and high-mass lower bounds provide alternative routes when the global probability-floor assumption is uninformative, and an explicit saturation family shows that softmax can separate the rank needed to match finite logits from the rank needed to match the limiting attention function. Fused multi-head analysis accounts for a shared rank budget across heads, and joint query/key LoRA isolates the extra error from factorizing an effective score update into separate low-rank factors. Given a target adapter and a calibration set drawn from the task, the theory therefore yields upper and lower error curves over rank and makes explicit where the available assumptions certify a rank, exclude it, or leave the decision unresolved. References [1] F. Bach (2010) Self-concordant analysis for logistic regression. Electronic Journal of Statistics 4, p. 384–414. External Links: Document Cited by: §10. [2] R. Basri and D. W. Jacobs (2026) The softmax bottleneck does not limit the probabilities of the most likely tokens. In International Conference on Learning Representations, External Links: Link Cited by: §10. [3] S. Bhojanapalli, C. Yun, A. S. Rawat, S. J. Reddi, and S. Kumar (2020) Low-rank bottleneck in multi-head attention models. In International Conference on Machine Learning, External Links: Link Cited by: §10. [4] P. Chen, H. Yu, I. S. Dhillon, and C. Hsieh (2021) DRONE: data-aware low-rank compression for large NLP models. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §10. [5] I. Csiszár and F. Matúš (2005) Closures of exponential families. The Annals of Probability. External Links: Link Cited by: §10. [6] O. Duranthon, F. Boncoraglio, and L. Zdeborová (2026) High-dimensional theory of LoRA fine-tuning in a solvable attention model. arXiv preprint arXiv:2606.05899. External Links: Link Cited by: §10. [7] C. Eckart and G. Young (1936) The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. External Links: Document Cited by: §1, §10. [8] N. Golowich, A. Liu, and A. Shetty (2026) Sequences of logits reveal the low rank structure of language models. In International Conference on Learning Representations, External Links: Link Cited by: §10. [9] A. Grivas, N. Bogoychev, and A. Lopez (2022) Low-rank softmax can have unargmaxable classes in theory but rarely in practice. arXiv preprint arXiv:2203.06462. External Links: Link Cited by: §10. [10] A. Grivas, A. Vergari, and A. Lopez (2023) Taming the sigmoid bottleneck: provably argmaxable sparse multi-label classification. arXiv preprint arXiv:2310.10443. External Links: Link Cited by: §10. [11] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1, §10. [12] A. J (2026) Tight sample complexity for low-rank adaptation: matching bounds and rank selection. arXiv preprint arXiv:2607.27680. External Links: Link Cited by: §1, §10. [13] Kimi Team (2026) Kimi K3: open frontier intelligence. arXiv preprint arXiv:2607.24653. External Links: Link Cited by: §11, §8. [14] U. Marteau-Ferey, F. Bach, and A. Rudi (2019) Globally convergent newton methods for ill-conditioned generalized self-concordant losses. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §10. [15] W. Masarczyk, M. Ostaszewski, T. S. Cheng, T. Trzciński, A. Lucchi, and R. Pascanu (2025) Unpacking softmax: how temperature drives representation collapse, compression, and generalization. arXiv preprint arXiv:2506.01562. External Links: Link Cited by: §10. [16] L. Mirsky (1960) Symmetric gauge functions and unitarily invariant norms. The Quarterly Journal of Mathematics 11 (1), p. 50–59. External Links: Document Cited by: §1, §10. [17] S. Neumann, R. Gemulla, and P. Miettinen (2016) What you will gain by rounding: theory and algorithms for rounding rank. arXiv preprint arXiv:1609.05034. External Links: Link Cited by: §10. [18] F. Paischer, L. Hauzenberger, T. Schmied, B. Alkin, M. P. Deisenroth, and S. Hochreiter (2025) Parameter efficient fine-tuning via explained variance adaptation. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §10. [19] R. Paturi and J. Simon (1986) Probabilistic communication complexity. Journal of Computer and System Sciences 33 (1), p. 106–123. External Links: Document Cited by: §10. [20] J. Rauh, T. Kahle, and N. Ay (2011) Support sets in exponential families and oriented matroid theory. International Journal of Approximate Reasoning 52 (5), p. 613–626. External Links: Document Cited by: §10. [21] J. Schulman and Thinking Machines Lab (2025) LoRA without regret. Note: Thinking Machines Lab blogthinkingmachines.ai/blog/lora; accessed 2026-08-18 Cited by: §1. [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Cited by: §1. [23] X. Wang, Y. Zheng, Z. Wan, and M. Zhang (2024) SVD-LLM: truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378. External Links: Link Cited by: §1, §10. [24] Y. Yang, X. Li, Z. Zhou, S. L. Song, J. Wu, L. Nie, and B. Ghanem (2024) CorDA: context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tuning. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, §10. [25] B. H. Yoon (2026) The entropic bound for transformers: why static rank fails and attention-native rank recovers. arXiv preprint arXiv:2607.23050. External Links: Link Cited by: §10. [26] Y. Zeng and K. Lee (2024) The expressive power of low-rank adaptation. In International Conference on Learning Representations, External Links: Link Cited by: §1, §10, §8. [27] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao (2023) AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations, External Links: Link Cited by: §1, §1. [28] Z. Zhao, T. Behnia, A. Vakilian, and C. Thrampoulidis (2024) Implicit geometry of next-token prediction: from language sparsity patterns to model representations. arXiv preprint arXiv:2408.15417. External Links: Link Cited by: §10. Guide to the appendix Appendices A–D prove the global softmax, moment, spectral, and dependence-allowing bounds. Appendices E and F contain the two alternative lower-bound routes. Appendix G gives the saturation constructions and closure-rank lower bound, and Appendices H–I prove the multi-head and joint query/key extensions. Computational checks. The accompanying code is available at https://github.com/gerardpc/lora-rank-project and verifies the displayed finite constructions, rank calculations, and selected inequalities on small instances. These checks are provided for reproducibility and debugging and are not used in the proofs. Appendix contents A. Proof of the global softmax bounds B. Why a purely quadratic global lower bound is impossible C. Moment and spectral lemmas D. Dependence-allowing rank–KL bounds E. Target-Fisher bounds F. High-mass lower bounds G. Saturation constructions and lower bound H. Fused multi-head proofs I. Joint query/key proofs Appendix A Proof of the global softmax bounds For z∈ℝnz ^n, write lse(z)=log∑iezilse(z)= _ie^z_i and H(z)=diag(p(z))−p(z)p(z)H(z)=diag(p(z))-p(z)p(z) T, where p(z)=softmax(z)p(z)=softmax(z). Proof of Theorem 3.1. The KL divergence is the log-sum-exp Bregman divergence F(d)=lse(z∗+d)−lse(z∗)−⟨p∗,d⟩.F(d)=lse(z_*+d)-lse(z_*)- p_*,d . (41) Taylor’s formula with integral remainder gives F(d)=∫01(1−t)dH(z∗+td)dt.F(d)= _0^1(1-t)d TH(z_*+td)d\,dt. (42) Since ‖H(z)‖op≤1/2\|H(z)\|_op≤ 1/2, F(d)≤‖d‖22/4F(d)≤\|d\|_2^2/4. Also both lse(z∗+d)−lse(z∗)lse(z_*+d)-lse(z_*) and ⟨p∗,d⟩ p_*,d lie in [minidi,maxidi][ _id_i, _id_i]. Therefore F(d)≤maxidi−minidi≤2‖d‖2.F(d)≤ _id_i- _id_i≤ 2\|d\|_2. (43) Taking the smaller of the quadratic and linear bounds proves F(d)≤ψup(‖d‖2)F(d)≤ _ up(\|d\|_2). For the lower bound, first suppose ‖d‖2≤1\|d\|_2≤ 1. For every t∈[0,1]t∈[0,1], pi(z∗+td)=p∗,ietdi∑jp∗,jetdj≥ae−2.p_i(z_*+td)= p_*,ie^td_i _jp_*,je^td_j≥ ae^-2. (44) For any probability vector q satisfying miniqi≥α _iq_i≥α and any centered vector v, v(diag(q)−qq)v v T(diag(q)-q T)v =∑iqi(vi−μq)2 = _iq_i(v_i- _q)^2 (45) ≥α∑i(vi−μq)2≥α‖v‖22, ≥α _i(v_i- _q)^2≥α\|v\|_2^2, (46) where μq=∑iqivi _q= _iq_iv_i and the last inequality uses that zero is the Euclidean-optimal centering constant for a centered v. Inserting α=ae−2α=ae^-2 into (42) yields F(d)≥ae−22‖d‖22,‖d‖2≤1.F(d)≥ ae^-22\|d\|_2^2, \|d\|_2≤ 1. (47) If ‖d‖2=s≥1\|d\|_2=s≥ 1, put v=d/sv=d/s and g(t)=F(tv)g(t)=F(tv). Convexity and g(0)=0g(0)=0 imply that g(t)/tg(t)/t is nondecreasing. Hence F(d)=g(s)≥sg(1)≥ae−22s.F(d)=g(s)≥ sg(1)≥ ae^-22s. (48) Combining this with (47) proves the lower bound. ∎ Proof of Corollary 3.2. Apply Theorem 3.1 pointwise to the centered displacement dM(u)d_M(u), take expectations, and then take the infimum over the same rank-constrained candidate class on both sides. ∎ Appendix B Why a purely quadratic global lower bound is impossible Proposition B.1 (Fixed-floor rare-context obstruction). There is a family with target probability floor 1/31/3 and mean key Gram G=IG=I for which the best rank-one quadratic error tends to one while the best rank-one expected attention KL tends to zero. Proof. Let d=2d=2, n=3n=3, and choose K∈ℝ3×2K ^3× 2 with orthonormal columns in ⟂1 . Draw h=e1with probability 1−p,h=Te2with probability p,T=(1−p)/p.h=e_1 probability 1-p, h=Te_2 probability p, T= (1-p)/p. (49) Take the pretrained query map to be −I2-I_2 and the dense target update to be I2I_2. The target query map is therefore zero, so the target attention is uniform and its floor is 1/31/3. The activation covariance is Σ=(1−p)e1e1+pT2e2e2=(1−p)I2. =(1-p)e_1e_1 T+pT^2e_2e_2 T=(1-p)I_2. (50) The best quadratic rank-one approximation to I2I_2 has error 1−p→11-p→ 1. Choose the rank-one candidate M=e1e1M=e_1e_1 T. It is exact when h=e1h=e_1. When h=Te2h=Te_2, its centered scores are −Tq2-Tq_2, where q2q_2 is a centered unit column of K. Thus KL(unif3∥softmax(−Tq2))=lse(−Tq2)−log3≤T.KL(unif_3\|softmax(-Tq_2))=lse(-Tq_2)- 3≤ T. (51) The population KL is at most pT=p(1−p)→0pT= p(1-p)→ 0. ∎ Appendix C Moment and spectral lemmas For a deterministic error matrix A, define XA(u)=‖βΠn(u)K(u)Ah(u)‖2,Q(A)=XA(u)2.X_A(u)=\|β _n(u)K(u)Ah(u)\|_2, Q(A)=EX_A(u)^2. (52) Lemma C.1 (Fourth-moment bridge). If a nonnegative random variable X satisfies X4≤κ(X2)2EX^4≤κ(EX^2)^2, then, with μ=X2μ=EX^2, ψ(X)≥ψ(μ)1+κ.Eψ(X)≥ ψ( μ)1+ κ. (53) Proof. For x≥0x≥ 0, ψ(x)≥x2/(1+x)ψ(x)≥ x^2/(1+x). Cauchy–Schwarz gives μ2≤[X21+X][X2(1+X)].μ^2 \! [ X^21+X ]E[X^2(1+X)]. (54) Moreover, X3≤X2X4≤κμ3/2EX^3≤ EX^2EX^4≤ κμ^3/2. Therefore ψ(X)≥μ1+κμ≥ψ(μ)1+κ.Eψ(X)≥ μ1+ κμ≥ ψ( μ)1+ κ. (55) For the last inequality: if μ≤1μ≤ 1, divide by μ and use μ≤1 μ≤ 1; if μ≥1μ≥ 1, divide by μ μ and use 1/μ≤11/ μ≤ 1. ∎ Lemma C.2 (Uniform moment condition). Assume G(u)G(u) and h(u)h(u) are independent, that for every deterministic compatible matrix C, ‖Ch‖24≤κh(‖Ch‖22)2,E\|Ch\|_2^4≤ _h(E\|Ch\|_2^2)^2, (56) and that G(u)⪯ΛG(u) G almost surely, where G=G(u)G=EG(u). Then every deterministic A satisfies XA4≤κhΛ2(XA2)2.EX_A^4≤ _h ^2(EX_A^2)^2. (57) Proof. Condition on G(u)=GuG(u)=G_u. Independence and (56), applied to C=Gu1/2AC=G_u^1/2A, give h[XA4∣Gu]≤κhtr(GuAΣA)2.E_h[X_A^4 G_u]≤ _h\tr(G_uA A T)\^2. (58) Because AΣA⪰0A A T 0 and Gu⪯ΛG_u G, this is at most κhΛ2tr(GAΣA)2 _h ^2\tr(GA A T)\^2. Independence also gives XA2=tr(GAΣA)EX_A^2=tr(GA A T). Combine the two identities. ∎ Lemma C.3 (Singular weighted spectral optimizer). Let G,Σ⪰0G, 0, let Δ∗ _* be any compatible matrix, and put D∗=G1/2Δ∗Σ1/2D_*=G^1/2 _* ^1/2. Then infrank(M)≤r‖G1/2(M−Δ∗)Σ1/2‖F2=∑j>rσj(D∗)2=Tr. _rank(M)≤ r\|G^1/2(M- _*) ^1/2\|_F^2= _j>r _j(D_*)^2=T_r. (59) This remains true when G or Σ is singular. Proof. For every feasible M, the matrix G1/2MΣ1/2G^1/2M ^1/2 has rank at most r, so Eckart–Young–Mirsky gives the lower bound. Let (D∗)r(D_*)_r be a rank-r truncated SVD and define Mr=G†/2(D∗)rΣ†/2.M_r=G /2(D_*)_r /2. (60) The left and right singular vectors of (D∗)r(D_*)_r lie in the supports of G and Σ , respectively. Hence G1/2MrΣ1/2=(D∗)rG^1/2M_r ^1/2=(D_*)_r and rank(Mr)≤rrank(M_r)≤ r. This candidate attains the tail. ∎ Proof of Theorem 4.1. Exact dense-target realizability gives dM(u)=βΠn(u)K(u)(M−Δ∗)h(u)d_M(u)=β _n(u)K(u)(M- _*)h(u). Independence yields Q(A)=‖G1/2AΣ1/2‖F2.Q(A)=\|G^1/2A ^1/2\|_F^2. (61) By Lemma C.3, every feasible A=M−Δ∗A=M- _* satisfies Q(A)≥TrQ(A)≥ T_r. Lemma C.2 and Lemma C.1 therefore imply, uniformly over the complete candidate class, ψ(XA)≥ψ(Q(A))1+Λκh≥ψ(Tr)1+Λκh.Eψ(X_A)≥ ψ( Q(A))1+ _h≥ ψ( T_r)1+ _h. (62) Taking the infimum gives the robust-objective lower bound. For any nonnegative X, ψup(X)≤minX2/4,2X≤ψup(X2).E _ up(X)≤ \EX^2/4, 2\,EX\≤ _ up( EX^2). (63) Apply this to the explicit weighted-SVD candidate MrM_r from Lemma C.3, for which Q(Mr−Δ∗)=TrQ(M_r- _*)=T_r. Thus Φr≤ψup(Tr) _r≤ _ up( T_r). Corollary 3.2 converts the two robust-objective bounds into (14). ∎ Appendix D Dependence-allowing rank–KL bounds Without assuming independence, retain the definitions Q(A)=XA2,S(A)=‖G1/2AΣ1/2‖F2,G=G(u).Q(A)=EX_A^2, S(A)=\|G^1/2A ^1/2\|_F^2, G=EG(u). (64) Proof of Theorem 4.3. Every feasible A satisfies S(A)≥TrS(A)≥ T_r. Applying Lemma C.1 and (19) gives ψ(XA)≥ψ(Q(A))1+κr≥ψ(crTr)1+κr.Eψ(X_A)≥ ψ( Q(A))1+ _r≥ ψ( c_rT_r)1+ _r. (65) This holds for the entire candidate class and survives the infimum. For the upper bound, insert ArsvdA_r svd into (63). Apply Corollary 3.2. ∎ Appendix E Target-Fisher bounds Proof of Theorem 5.1. Let qt=softmax(z∗+td)q_t=softmax(z_*+td). Coordinatewise, e−tR(d)p∗,i≤qt,i≤etR(d)p∗,i.e^-tR(d)p_*,i≤ q_t,i≤ e^tR(d)p_*,i. (66) For any probability vector q, vH(q)v=min∑icqi(vi−c)2.v TH(q)v= _c _iq_i(v_i-c)^2. (67) Consequently, if αpi≤qi≤γpiα p_i≤ q_i≤γ p_i, then αH(p)⪯H(q)⪯γH(p)α H(p) H(q) γ H(p). The lower inequality follows by applying the coordinatewise lower bound before minimizing over c; for the upper inequality, evaluate the q-weighted variance at the p-optimal centering constant. Insert (66) into the Bregman integral KL(p∗∥softmax(z∗+d))=∫01(1−t)dH(qt)ddt.KL(p_*\|softmax(z_*+d))= _0^1(1-t)d TH(q_t)d\,dt. (68) The scalar integrals are ∫01(1−t)e−tRt=R−1+e−R2,∫01(1−t)etRt=eR−1−R2. _0^1(1-t)e^-tRdt= R-1+e^-RR^2, _0^1(1-t)e^tRdt= e^R-1-RR^2. (69) This proves (22), with the value 1/21/2 obtained by continuity at R=0R=0. ∎ The same integral representations show that c−c_- is nonincreasing and c+c_+ is nondecreasing on [0,∞)[0,∞). For every fixed t∈[0,1]t∈[0,1], (1−t)e−tR(1-t)e^-tR decreases with R, whereas (1−t)etR(1-t)e^tR increases; integration preserves these inequalities. This justifies the monotonicity step below. For query-only adaptation, define GF(u)=β2K(u)[diag(p∗(u))−p∗(u)p∗(u)]K(u)G_F(u)=β^2K(u) T[diag(p_*(u))-p_*(u)p_*(u) T]K(u) (70) and QF(A)=hAGF(u)AhQ_F(A)=Eh TA TG_F(u)Ah. Let r,R0C_r,R_0 be the rank-r candidates whose score differences from the target have range almost surely at most R0R_0, and define ℰr,R0=infM∈r,R0KL(p∗∥pM),Tr,R0F=infM∈r,R0QF(M−Δ∗).E_r,R_0= _M _r,R_0EKL(p_*\|p_M), T^F_r,R_0= _M _r,R_0Q_F(M- _*). (71) Corollary E.1 (Constrained Fisher rank–KL bounds). If r,R0C_r,R_0 is nonempty, then c−(R0)Tr,R0F≤ℰr,R0≤c+(R0)Tr,R0F.c_-(R_0)T^F_r,R_0 _r,R_0≤ c_+(R_0)T^F_r,R_0. (72) Proof. Apply Theorem 5.1 pointwise to every admissible candidate and use monotonicity of c−c_- and c+c_+. Taking the infimum gives the lower bound. For the upper bound, apply the inequality to an η-minimizing sequence for Tr,R0FT^F_r,R_0 and let η↓0η 0. ∎ Under independence, put GF=GF(u)G_F=EG_F(u), DF=GF1/2Δ∗Σ1/2D_F=G_F^1/2 _* ^1/2, and TrF=∑j>rσj(DF)2T_r^F= _j>r _j(D_F)^2. Lemma C.3 gives TrFT_r^F as the unrestricted rank-r Fisher tail. Since r,R0C_r,R_0 is a subset of the unrestricted class, ℰr,R0≥c−(R0)TrF.E_r,R_0≥ c_-(R_0)T_r^F. (73) If the Fisher-weighted SVD optimizer belongs to r,R0C_r,R_0, it also attains the constrained quadratic infimum, yielding the matching upper bound ℰr,R0≤c+(R0)TrFE_r,R_0≤ c_+(R_0)T_r^F. Appendix F High-mass lower bounds Lemma F.1 (Conditional KL decomposition). For positive distributions p,qp,q and nonempty S, put s=p(S)s=p(S) and t=q(S)t=q(S). Then KL(p∥q)=klBern(s∥t)+sKL(pS∥qS)+(1−s)KL(pSc∥qSc).KL(p\|q)=kl_ Bern(s\|t)+sKL(p^S\|q^S)+(1-s)KL(p^S^c\|q^S^c). (74) If p=softmax(z)p=softmax(z) and q=softmax(w)q=softmax(w), then pS=softmax(zS)p^S=softmax(z_S) and qS=softmax(wS)q^S=softmax(w_S). Proof. For i∈Si∈ S, write log(pi/qi)=log(s/t)+log(piS/qiS) (p_i/q_i)= (s/t)+ (p_i^S/q_i^S) and sum with weights pip_i. Repeat on ScS^c. Conditional softmax follows because the full normalizer cancels after conditioning. ∎ Proof of Theorem 5.2. By Lemma F.1, the full KL is at least s∗KL(p∗S∥pMS)s_*KL(p_*^S\|p_M^S), where s∗=p∗(S)s_*=p_*(S). Apply Theorem 3.1 to the restricted logits. Their target floor is aS=mini∈Sp∗,is∗.a_S= _i∈ S p_*,is_*. (75) Thus, for each candidate, KL(p∗∥pM)≥s∗aS2e2ψ(∥ΠS(zM−z∗)S∥2).KL(p_*\|p_M)≥ s_*a_S2e^2ψ(\| _S(z_M-z_*)_S\|_2). (76) Use s∗≥1−δs_*≥ 1-δ and aS≥acorea_S≥ a_core, take expectations, and then take the infimum over the complete unrestricted rank-r class. ∎ For the spectral form, let KS(u)K_S(u) contain the selected key rows and define Gcore(u)=β2KS(u)ΠS(u)KS(u),Dcore=Gcore1/2Δ∗Σ1/2,G_core(u)=β^2K_S(u) T _S(u)K_S(u), D_core=G_core^1/2 _* ^1/2, (77) where Gcore=Gcore(u)G_core=EG_core(u). Under independence, activation hypercontractivity, and Gcore(u)⪯ΛcoreGcoreG_core(u) _coreG_core, the same proof as Theorem 4.1 gives ℰr≥(1−δ)acore2e2(1+Λcoreκh)ψ(Trcore),E_r≥ (1-δ)a_core2e^2(1+ _core _h)ψ\! ( T_r^core ), (78) where Trcore=∑j>rσj(Dcore)2T_r^core= _j>r _j(D_core)^2. Under two-sided dependent geometry λcoreG¯core⪯Gcore(u)⪯ΛcoreG¯core _core G_core G_core(u) _core G_core, apply Theorem 4.3 to obtain ℰr≥(1−δ)acore2e2(1+(Λcore/λcore)κh)ψ(λcoreT¯rcore).E_r≥ (1-δ)a_core2e^2(1+( _core/ _core) _h)ψ\! ( _core T_r^core ). (79) Appendix G Saturation constructions and lower bound For a boundary target matrix P∞P_∞ and an allowed centered score space L, define its softmax closure rank as the least r for which there is a sequence ZmZ_m satisfying rank(Zm)≤rrank(Z_m)≤ r, every column of ZmZ_m lies in L, and columnwise softmax converges to P∞P_∞. G.1 The linear-token achievable construction Write k=3q+sk=3q+s, s∈0,1,2s∈\0,1,2\, and choose the least t such that n=4t≥k+1n=4^t≥ k+1. Identify 22tF_2^2t with 4tF_4^t. The one-dimensional 4F_4 subspaces partition nonzero vectors into triples a,b,a+b\a,b,a+b\. Select q complete triples and s characters from one further triple. For token row x, set χa(x)=(−1)⟨a,x⟩ _a(x)=(-1) a,x and form the normalized character matrix K. Its columns are centered and orthonormal. Context j has hj=ejh_j=e_j, zero pretrained logits, and target logits TKejTKe_j. At every finite T, centered-softmax injectivity and KK=IK TK=I force an exact update to equal TIkTI_k, hence to have rank k. On a complete triple, write the characters as x,y,xyx,y,xy, put g=T1/2g=T^1/2 and α=T−3/4α=T^-3/4, and define b2=(T,T+g,−T),b3=(T,−T,T+g),b1=α(b2+b3),BT=[b1b2b3].b_2=(T,T+g,-T) T, b_3=(T,-T,T+g) T, b_1=α(b_2+b_3), B_T=[b_1\ b_2\ b_3]. (80) BTB_T has rank two. On the first context’s target support x=1x=1, its scores are 2T1/4+2T−1/4y2T^1/4+2T^-1/4y, whereas on x=−1x=-1 the score is −2T1/4-2T^1/4. The within-support spread vanishes and the support gap diverges. For the second context the score is xT+y(T+g)−xyTxT+y(T+g)-xyT: it equals T+gT+g on y=1y=1 and is at most T−gT-g on y=−1y=-1. The third context is symmetric. Scalar T1/4T^1/4 blocks handle leftovers. The block-diagonal update has rank 2q+s=k−⌊k/3⌋2q+s=k- k/3 . For the seven-character strengthening, take the seven nonzero characters of 23F_2^3, set ρ=T−2ρ=T^-2 and L=T5L=T^5, and define Cρ=[1ρ200ρ000ρ1ρ2ρ200000ρ2100ρ20ρ2001]C_ρ= bmatrix1&ρ^2&0&0&ρ&0&0\\ 0&ρ&1&ρ^2&ρ^2&0&0\\ 0&0&0&ρ&ρ^2&1&0\\ 0&ρ^2&0&ρ^2&0&0&1 bmatrix (81) and Fρ=[0000−ρ−ρ20−ρ20−ρ−ρ2−ρ2−10−ρ2000−1−ρ−ρ2−ρ2−ρ00−100−ρ200−1],ZT=LFρCρ.F_ρ= bmatrix0&0&0&0\\ -ρ&-ρ^2&0&-ρ^2\\ 0&-ρ&-ρ^2&-ρ^2\\ -1&0&-ρ^2&0\\ 0&0&-1&-ρ\\ -ρ^2&-ρ^2&-ρ&0\\ 0&-1&0&0\\ -ρ^2&0&0&-1 bmatrix, Z_T=LF_ρC_ρ. (82) The factorization gives rank(ZT)≤4rank(Z_T)≤ 4. For completeness, the product before the scalar factor L is FρCρ=[0000000−ρ−2ρ3−ρ4−ρ2−2ρ4−ρ2−ρ40−ρ20−ρ2−ρ4−ρ−2ρ3−ρ4−ρ3−ρ4−ρ2−ρ2−1−ρ20−ρ3−ρ−ρ4−ρ200−ρ30−ρ−ρ3−ρ2−1−ρ−ρ2−ρ3−ρ4−ρ2−ρ2−ρ4−2ρ3−ρ4−ρ00−ρ−1−ρ2−ρ200−ρ2−ρ2−ρ40−ρ2−ρ30−1].F_ρC_ρ= bmatrix0&0&0&0&0&0&0\\ -ρ&-2ρ^3-ρ^4&-ρ^2&-2ρ^4&-ρ^2-ρ^4&0&-ρ^2\\ 0&-ρ^2-ρ^4&-ρ&-2ρ^3-ρ^4&-ρ^3-ρ^4&-ρ^2&-ρ^2\\ -1&-ρ^2&0&-ρ^3&-ρ-ρ^4&-ρ^2&0\\ 0&-ρ^3&0&-ρ-ρ^3&-ρ^2&-1&-ρ\\ -ρ^2&-ρ^3-ρ^4&-ρ^2&-ρ^2-ρ^4&-2ρ^3-ρ^4&-ρ&0\\ 0&-ρ&-1&-ρ^2&-ρ^2&0&0\\ -ρ^2&-ρ^2-ρ^4&0&-ρ^2&-ρ^3&0&-1 bmatrix. (83) In column a, the four rows in the positive halfspace SaS_a are precisely those with entries between −3ρ3-3ρ^3 and 00; every remaining entry is at most −ρ2-ρ^2 for 0<ρ<1/30<ρ<1/3. Multiplying by L gives, for the positive halfspace SaS_a of every character a, −3ρ3L≤ZT(x,a)≤0(x∈Sa),ZT(x,a)≤−ρ2L(x∉Sa).-3ρ^3L≤ Z_T(x,a)≤ 0 (x∈ S_a), Z_T(x,a)≤-ρ^2L (x∉ S_a). (84) Thus the within-support spread is at most 3/T3/T and the support gap is at least T−3/T-3/T. Column centering preserves softmax and does not increase rank. Identify 23tF_2^3t with 8tF_8^t and pack its nonzero vectors into seven-character one-dimensional 8F_8 subspaces. If k=7m+sk=7m+s, 0≤s<70≤ s<7, the resulting path has rank at most 4m+s=k−3⌊k/7⌋4m+s=k-3 k/7 and uses n<8(k+1)n<8(k+1) tokens. It remains to verify KL convergence rather than only weak convergence of supports. For each target halfspace, the target off-support mass is ϵT=(1+e2T/n)−1 _T=(1+e^2T/ n)^-1. Let ηT _T be the candidate off-support mass. In both constructions, ηT→0 _T→ 0, within-support KL vanishes, and the candidate score range is polynomial in T. By Lemma F.1, the only nontrivial term satisfies ϵTlog(ϵT/ηT)≤ϵT[−logηT]=ϵTpoly(T)⟶0. _T ( _T/ _T)≤ _T[- _T]= _Tpoly(T) 0. (85) The conditional complement contribution obeys the same bound. Taking the better of the triple and septuple packings proves the linear-token statement in Section 6. G.2 Proof of the exact closure-rank theorem Proof of Theorem 6.1. Write k=3q+sk=3q+s, s∈0,1,2s∈\0,1,2\. Work in a binary vector space V whose size is the least power of two above 2k2+22k^2+2. We construct =T1∪˙⋯∪˙Tq∪˙R,Tℓ=aℓ,bℓ,aℓ+bℓ,|R|=s,A=T_1 ∪·s ∪T_q ∪R, T_ =\a_ ,b_ ,a_ +b_ \, |R|=s, (86) so these are the only additive triples in A. Given a current set AjA_j, let Fj=Aj+AjF_j=A_j+A_j. Choose u∉Fju∉ F_j and then v∉Fj∪(u+Fj)∪0,uv∉ F_j∪(u+F_j)∪\0,u\. Fewer than 2k2+22k^2+2 vectors are forbidden, so the construction continues. At most two leftovers can be added outside the current sumset. The number of token rows satisfies n<4k2+8n<4k^2+8. Use normalized Walsh characters χa _a for a∈a as centered orthonormal key columns. The finite targets are again TKeaTKe_a, so every exact finite-logits update is TIkTI_k and has rank k. Applying the rank-two path in (80) independently to each TℓT_ , with scalar paths on leftovers, gives closure rank at most 2q+s2q+s. For the reverse inequality, consider any coefficient sequence MmM_m whose score softmaxes converge to the target boundary distributions. Absorb the fixed Walsh normalization into MmM_m. For context a, write fm,a(x)=∑b∈mb,a(m)χb(x),tm,a=ma,a(m).f_m,a(x)= _b m_b,a^(m) _b(x), t_m,a=m_a,a^(m). (87) On Sa=x:χa(x)=1S_a=\x: _a(x)=1\, the restrictions of χb _b and χb+a _b+a agree. Orthogonality of the restricted characters and convergence to the uniform target give mb,a(m)+mb+a,a(m)⟶0for b∉0,a.m_b,a^(m)+m_b+a,a^(m) 0 b∉\0,a\. (88) All support scores equal tm,a+o(1)t_m,a+o(1), whereas the mean complement score is −tm,a-t_m,a. The diverging support gap forces tm,a→∞t_m,a→∞. Triple isolation and (88) make every cross-block coefficient o(1)o(1). If a,b,c=a+b\a,b,c=a+b\ is the block containing a, put sm,a=mb,a(m)s_m,a=m_b,a^(m). Then mc,a(m)=−sm,a+o(1)m_c,a^(m)=-s_m,a+o(1). On the negative halfspace, χc=−χb _c=- _b and both signs occur, so maxSacfm,a=−tm,a+2|sm,a|+o(1). _S_a^cf_m,a=-t_m,a+2|s_m,a|+o(1). (89) The diverging gap implies |sm,a|≤tm,a+o(tm,a)|s_m,a|≤ t_m,a+o(t_m,a). Normalize each column by its positive diagonal coefficient: Cm=Mmdiag(tm,a−1:a∈).C_m=M_mdiag(t_m,a^-1:a ). (90) This preserves rank. Diagonal entries become one, cross-block entries converge to zero, and within-block entries remain bounded. Pass to a convergent subsequence. The limit is block diagonal, with singleton blocks [1][1] and triple blocks of the form B(α,β,γ)=[1βγα1−γ−α−β1].B(α,β,γ)= bmatrix1&β&γ\\ α&1&-γ\\ -α&-β&1 bmatrix. (91) Every real block (91) has rank at least two. If it had rank one, its 2×22× 2 minors would force αβ=1αβ=1, −αγ=1-αγ=1, and βγ=1βγ=1. The first and third equalities imply α=γα=γ, contradicting −αγ=1-αγ=1 over the reals. The limiting normalized matrix therefore has rank at least 2q+s=k−⌊k/3⌋2q+s=k- k/3 . Since the set of matrices of rank at most r is closed, every approximating sequence has rank at least 2q+s2q+s. This matches the construction. ∎ Appendix H Fused multi-head proofs For nonnegative numbers x1,…,xHx_1,…,x_H, put s=(∑hxh2)1/2s=( _hx_h^2)^1/2. We first prove ψ(∑hxh2)≤∑hψ(xh)≤Hψ(∑hxh2)ψ\! ( _hx_h^2 )≤ _hψ(x_h)≤ H\,ψ\! ( _hx_h^2 ) (92) for all xh≥0x_h≥ 0. If s≤1s≤ 1, every xh≤1x_h≤ 1 and ∑hψ(xh)=s2=ψ(s) _hψ(x_h)=s^2=ψ(s). If s>1s>1, then ψ(xh)≥xh2/sψ(x_h)≥ x_h^2/s: for xh≤1x_h≤ 1 this follows from s≥1s≥ 1, and for xh>1x_h>1 it follows from xh≤sx_h≤ s. Summing gives the left inequality. Moreover, ψ(xh)≤xhψ(x_h)≤ x_h, so Cauchy–Schwarz gives ∑hψ(xh)≤∑hxh≤Hs=Hψ(s) _hψ(x_h)≤ _hx_h≤ Hs= Hψ(s). Proof of Theorem 7.1. Applying Theorem 3.1 headwise and summing, then using Equation (92), gives the lower bound amin2e2ψ(XMH)≤∑hDh,XMH2=∑hXh2. a_ 2e^2ψ(X_MH)≤ _hD_h, X_MH^2= _hX_h^2. (93) For the upper bound, the headwise inequalities give directly ∑hDh≤∑hminXh2/4,2Xh≤minXMH2/4,2HXMH. _hD_h≤ _h \X_h^2/4, 2X_h\≤ \X_MH^2/4, 2HX_MH\. (94) Both constants in the final aggregation inequality are attained: the quadratic branch when all XhX_h are small, and the linear branch when the errors are equal across heads. The block definitions in (29) and independence give XMH2=‖GMH1/2(M−Δ∗MH)Σ1/2‖F2.EX_MH^2=\|G_MH^1/2(M- _*^MH) ^1/2\|_F^2. (95) Weighted Eckart–Young–Mirsky makes the infimum of this quadratic objective equal to TrMHT_r^MH. The leverage and fourth-moment assumptions imply XMH4≤ΛMH2κh(XMH2)2EX_MH^4≤ _MH^2 _h(EX_MH^2)^2 by the argument of Lemma C.2. Lemma C.1 therefore gives the robust lower bound ψ(XMH)≥ψ(TrMH)1+ΛMHκh.Eψ(X_MH)≥ ψ( T_r^MH)1+ _MH _h. (96) For the weighted-SVD candidate, take expectations in Equation (94), use Cauchy–Schwarz on XMHEX_MH, and substitute XMH2=TrMHEX_MH^2=T_r^MH. This gives the stated upper bound and completes the proof of (31). ∎ For independently parameterized head adapters, the feasible set is a Cartesian product once integer ranks rhr_h are fixed, so inf∑hrh≤Rrank(Mh)≤rh∑hDh=min∑h∑hrh≤Rℰh,rh. _ subarrayc _hr_h≤ R\\ rank(M_h)≤ r_h subarray _hED_h= _ _hr_h≤ R _hE_h,r_h. (97) This proves the allocation identity. Finally, let yM,hy_M,h and y∗,hy_*,h be the value-weighted head outputs, let h(u)D_h(u) be the diameter of the head’s value vectors, and let OhO_h be its output-projection block. Total variation and Pinsker give ‖yM,h−y∗,h‖2≤h(u)TV(pM,h,p∗,h)≤h(u)Dh/2.\|y_M,h-y_*,h\|_2 _h(u)TV(p_M,h,p_*,h) _h(u) D_h/2. (98) By Cauchy–Schwarz across heads, ‖∑hOh(yM,h−y∗,h)‖22≤H2∑h‖Oh‖op2h(u)2Dh(u). \| _hO_h(y_M,h-y_*,h) \|_2^2≤ H2 _h\|O_h\|_op^2D_h(u)^2D_h(u). (99) Appendix I Joint query/key proofs For ordinary dot-product attention, define C(A,B)=K0A+B(Q0+A).C(A,B)=K_0 TA+B T(Q_0+A). (100) The two summands have ranks at most rank(A)rank(A) and rank(B)rank(B), proving Equation (33). The bound is sharp: for p=rp=r, d=2rd=2r, take K0=A=[Ir 0],Q0=B=[0Ir].K_0=A=[I_r\ 0], Q_0=B=[0\ I_r]. (101) Then C(A,B)C(A,B) has rank 2r2r. Every joint candidate also satisfies rank(K0Q0+C(A,B))=rank((K0+B)(Q0+A))≤p.rank(K_0 TQ_0+C(A,B))=rank((K_0+B) T(Q_0+A))≤ p. (102) Thus the actual joint class is contained in the effective rank-s class, s=rQ+rKs=r_Q+r_K, and in its width-aware subclass. Let dA,B(u)=Πn(u)βX(u)[C(A,B)−C∗]h(u)d_A,B(u)= _n(u)β X(u)[C(A,B)-C_*]h(u) (103) and let ΨrQ,rKQK ^QK_r_Q,r_K be the infimum of ψ(‖dA,B‖2)Eψ(\|d_A,B\|_2) over the actual factor class. Applying the lower half of Theorem 3.1 to each candidate gives a2e2ΨrQ,rKQK≤ℰrQ,rKQK. a2e^2 ^QK_r_Q,r_K ^QK_r_Q,r_K. (104) Proof of Theorem 8.1. By rank containment, every effective error C(A,B)−C∗C(A,B)-C_* is a rank-s candidate relative to C∗C_*. Weighted Eckart–Young–Mirsky therefore gives ‖dA,B(u)‖22≥TseffE\|d_A,B(u)\|_2^2≥ T_s^eff (105) for every feasible pair. The assumed fourth-moment condition and Lemma C.1 imply ψ(‖dA,B‖2)≥ψ(Tseff)1+κ.Eψ(\|d_A,B\|_2)≥ ψ( T_s^eff)1+ κ. (106) Take the infimum and use (104) for the lower bound. Let ssvdS_s svd be the nonempty set of weighted rank-s optimizers and define ρrQ,rK=infCs∈ssvdinfrank(A)≤rQrank(B)≤rK‖GX1/2[C(A,B)−Cs]Σ1/2‖F. _r_Q,r_K= _C_s _s svd _ subarraycrank(A)≤ r_Q\\ rank(B)≤ r_K subarray\|G_X^1/2[C(A,B)-C_s] ^1/2\|_F. (107) Choose jointly η-minimizing sequences in these two infima. The weighted seminorm triangle inequality gives ‖dA,B(u)‖221/2≤Tseff+ρrQ,rK+o(1). \E\|d_A,B(u)\|_2^2 \^1/2≤ T_s^eff+ _r_Q,r_K+o(1). (108) Apply the pointwise global upper bound and Equation (63), then let η↓0η 0. ∎ The realizability price vanishes in several checkable cases. If rank(C)≤rQrank(C)≤ r_Q and col(C)⊆col(K0)col(C) (K_0 T), then A=(K0)†C,C(A,0)=C,rank(A)≤rank(C).A=(K_0 T) C, C(A,0)=C, (A) (C). (109) The symmetric statement for keys uses B=CQ0†B T=CQ_0 when row(C)⊆row(Q0)row(C) (Q_0). More generally, suppose C=CQ+CKC=C_Q+C_K, choose AQA_Q with K0AQ=CQK_0 TA_Q=C_Q and rank(AQ)≤rQrank(A_Q)≤ r_Q, and assume row(CK)⊆row(Q0+AQ)row(C_K) (Q_0+A_Q) with rank(CK)≤rKrank(C_K)≤ r_K. Then BK=CK(Q0+AQ)†B_K T=C_K(Q_0+A_Q) (110) has rank at most rKr_K and satisfies C(AQ,BK)=CQ+CKC(A_Q,B_K)=C_Q+C_K. These projector identities prove the one-sided and sequential realization claims. An always-valid factor-dependent upper bound follows from C(A,B)−C(A∗,B∗)=(K0+B)(A−A∗)+(B−B∗)(Q0+A∗).C(A,B)-C(A_*,B_*)=(K_0+B) T(A-A_*)+(B-B_*) T(Q_0+A_*). (111) Submultiplicativity yields ‖GX1/2[C(A,B)−C∗]Σ1/2‖F \|G_X^1/2[C(A,B)-C_*] ^1/2\|_F (112) ≤‖GX1/2(K0+B)‖op‖(A−A∗)Σ1/2‖F ≤\|G_X^1/2(K_0+B) T\|_op\|(A-A_*) ^1/2\|_F (113) +‖(Q0+A∗)Σ1/2‖op‖GX1/2(B−B∗)‖F. +\|(Q_0+A_*) ^1/2\|_op\|G_X^1/2(B-B_*) T\|_F. (114) Combining this with (63) and the global softmax upper bound gives the constructive target-factor upper bound. Weighted truncated SVDs of the actual target factors make the two Frobenius errors equal to their corresponding spectral tails. For fused multi-head joint adaptation, every head block satisfies rankCh(Ah,Bh)≤srankC_h(A_h,B_h)≤ s. Applying weighted Eckart–Young headwise and then the aggregate moment bridge gives ℰrQ,rKMH,QK≥amin2e2(1+κMH)ψ(∑hTh,seff).E^MH,QK_r_Q,r_K≥ a_ 2e^2(1+ _MH)ψ\! ( _hT_h,s^eff ). (115) This bound deliberately grants every head the full allowance s and is therefore safe but potentially loose.