Paper deep dive
Fine-Tuning of Transformer models with Frames
Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, Vikas Singh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/28/2026, 3:43:27 AM
Summary
The paper introduces FrameFT, a Parameter-Efficient Fine-Tuning (PEFT) method for Transformer models that utilizes Fusion Frames to model parameter updates. Unlike LoRA, which uses dense low-rank matrices, FrameFT decomposes the parameter space into overlapping subspaces using algorithmically generated Tight Fusion Frames (via Spectral Tetris). It optimizes only sparse coefficient matrices within these subspaces, significantly reducing memory footprint and storage requirements while maintaining or exceeding the performance of state-of-the-art PEFT techniques like LoRA. The method offers theoretical convergence guarantees and improved inference throughput due to inherent sparsity.
Entities (8)
Relation Signals (6)
FrameFT → appliedto → Transformer Models
confidence 95% · Fine-Tuning of Transformer models with Frames
FrameFT → uses → Fusion Frames
confidence 95% · Our proposal, FrameFT, models the parameter update ΔW with a sparse coefficient matrix in a Fusion Frame basis.
FrameFT → reducesmemoryfootprint → LoRA
confidence 92% · needs far fewer trainable parameters... reduces the checkpoint size to just 1.28 MB (vs. 67.1 MB for LoRA rank-32)
Spectral Tetris → generates → Tight Fusion Frames
confidence 90% · we use the Spectral Tetris algorithm... to construct a (k,ρ,d) uniform tight fusion frame
FrameFT → outperformsorequals → LoRA
confidence 90% · Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques...
FrameFT → appliedto → RoBERTa-large
confidence 85% · Figure 3 shows this sparse structure with an example from RoBERTa-Large.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Tags
Links
- Source: https://arxiv.org/abs/2608.26430v1
- Canonical: https://arxiv.org/abs/2608.26430v1
Trouble viewing inline? Open PDF directly →
Full Text
90,306 characters extracted from source content.
Expand or collapse full text
Fine-Tuning of Transformer models with Frames Harshavardhan Adepu Affiliation: University of Wisconsin-Madison Correspondence to: adepu@wisc.edu Li Zhang Affiliation: Google DeepMind Sanjiv Kumar Affiliation: Google DeepMind Vikas Singh Affiliation: University of Wisconsin-Madison Abstract Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, (dr)O(dr), where d is the model’s hidden dimension and r is the rank. Our proposal, FrameFT, models the parameter update ΔW W with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters. Keywords: Machine Learning, ICML 1 Introduction Foundation models show impressive capabilities across a range of domains, including language (Touvron et al., 2023; Team et al., 2024a; OpenAI, 2025), vision (Dehghani et al., 2023), and other modalities (Łajszczak et al., 2024; Fang et al., 2025). While increasing model size gives improved performance, many downstream applications benefit from an additional fine-tuning step to specialize the model for specific tasks. However, fine-tuning all parameters of a large model is expensive. This becomes worse when fine-tuning is required for multiple tasks, each needing to store its own dedicated copy of the model. To address these difficulties, Parameter-Efficient Fine-Tuning (PEFT) methods provide an alternative (Hu et al., 2022; Zhang et al., 2025; Dettmers et al., 2023). These techniques seek to minimize the number of trainable parameters during fine-tuning, significantly reducing resource needs. Recent work has shown that PEFT methods can match the performance of full-model fine-tuning while updating only a small subset of the parameters (Bensaïd et al., 2025; Zhang et al., 2026). LoRA and Sparse Fine-tuning. The most widely used PEFT technique is Low-Rank Adaptation (LoRA) (Hu et al., 2022), which keeps the pre-trained weights frozen and injects task-specific low-rank matrices into each layer. These matrices provide the flexibility to adapt the model while maintaining low memory/compute costs. LoRA’s success has inspired a range of variants that offer improvements in convergence, efficiency, and storage overhead (Dettmers et al., 2023; Hayou et al., 2024; Zhu et al., 2024; Yaras et al., 2024; Xia et al., 2024). One line of work uses the Singular Value Decomposition (SVD) of pre-trained weights to construct fixed bases for adaptation (Lingam et al., 2024; Bałazy et al., 2025). However, this needs dense singular vectors, which increases the memory footprint and affects inference throughput. This overhead is problematic in multi-tenant serving scenarios where there is a single pretrained model and its many fine-tuned versions. In contrast, sparse fine-tuning (SFT) methods use a different strategy: they selectively update a sparse subset of the model’s original parameters. The selection rule varies: some approaches rely on magnitude-based pruning (Lu et al., 2024), while others use sensitivity measures such as Fisher information (Guo et al., 2021), Fourier-domain analyses (Gao et al., 2024), or parameter change measures (Ansell et al., 2022). SFT methods can face scalability issues when applied to large models, especially in identifying sparse patterns and tuning hyperparameters (Guo et al., 2021). Some proposals also rely on access to training dynamics (Ansell et al., 2022) that may not be easily available. Motivation. LoRA’s dense singular vectors and asymmetric A/BA/B structure were inherited from matrix approximation methods and shown to work well for fine-tuning. SFT methods similarly inherited ideas from model pruning, i.e., identify task-relevant parameters through gradient or magnitude-based proxies. We instead take a use case driven approach and ask: how should we parameterize weight updates if we want expressivity, parameter efficiency, stable optimization, and compute scalability simultaneously for fine-tuning (in particular)? Frame Theory provides a natural answer. Briefly, Finite Frames and Fusion Frames are used to study the spanning sets and spanning subspaces of a given finite-dimensional vector space, respectively. They have many applications in coding theory, compressed sensing (Boufounos et al., 2009), quantization (Adepu et al., 2024; Czaja and Na, 2024), and dictionary learning (Hwang et al., 2019; Cai et al., 2014). Their ability to encode updates in overlapping subspaces offers robustness, flexibility, and efficiency – properties that are relevant for model adaptation. FrameFT partitions the model’s parameter space into fusion-frame-based subspaces, within which updates are learned in a highly sparse and structured manner. Our proposal: Fine-tuning with Frames. We propose a new PEFT strategy inspired by Fusion Frames (Christensen, 2018; Casazza et al., 2008; Waldron, 2019). Fusion Frames allow structured representations by decomposing a space into overlapping subspaces at multiple scales. Unlike strategies that rely on either low-rank or sparse updates, FrameFT performs fine-tuning across multiple structured subspaces, naturally combining the global adaptability of LoRA with the precision of sparse updates. Contributions. Our main contributions are: (a) A new fine-tuning framework, FrameFT, that utilizes highly structured subspaces (Fusion Frames) to capture parameter updates efficiently; (b) Extensive empirical evaluations across multiple foundation models and tasks, demonstrating FrameFT’s effectiveness compared to state-of-the-art PEFT baselines; (c) A technical analysis of the numerical stability, convergence, and efficiency benefits offered by FrameFT. Conflict of Interest Disclosure. Two of the authors (LZ, SK) work at Google DeepMind, where the Gemma models were developed. The other two authors (HA, VS) previously worked at Google DeepMind. 2 Frames and Subspace decompositions We provide a brief review of Finite Frame theory for Hilbert spaces (Waldron, 2019; Christensen, 2018) . Readers familiar with these concepts can skip to the next section, where we describe how we apply this idea to fine-tuning. Spanning sets and Frames. A spanning set is a collection of vectors that spans a finite-dimensional Hilbert space or vector space, and orthonormal bases are a canonical example. Frames generalize this concept by allowing redundancy, enabling stable and often more robust representations. Figure 1: Examples of tight frames with k=5k=5 vectors in ℝ3R^3. Let ℋdH^d denote a d-dimensional Hilbert space. A sequence of vectors ϕ=(φi)i=1k [rgb]0.1,0.05,0.96 =( [rgb]0.1,0.05,0.96 _i)_i=1^k in ℋdH^d is called a frame if there exist constants 0<A≤B<∞0<A≤ B<∞ such that for all x∈ℋdx ^d, A‖x‖2≤∑i=1k|⟨x,φi⟩|2≤B‖x‖2, A||x||^2≤ _i=1^k| x, [rgb]0.1,0.05,0.96 _i |^2≤ B||x||^2, (1) where ⟨⋅,⋅⟩ ·,· is the inner product in ℋdH^d, and A and B are known as the frame bounds. Typically, k≥dk≥ d allows the frame to represent vectors with redundancy. An example is illustrated in Figure 1 with k=5k=5 vectors in ℝ3R^3. We can project any vector in ℝ3R^3 onto these frame vectors without distorting the signal (Casazza and Kutyniok, 2012). Throughout the paper, we operate in real-valued Euclidean spaces ℝdR^d instead of a more general Hilbert space for simplicity. Fusion frames extend the above idea to settings where subspace decomposition is also desired (instead of redundancy alone). Given a collection of subspaces ii=1k\ [rgb]0,0,1W_i\_i=1^k in ℝdR^d and a corresponding set of positive weights wii=1k\w_i\_i=1^k, the collection (i,wi)i=1k( [rgb]0,0,1W_i,w_i)_i=1^k forms a fusion frame if there exist constants 0<A≤B<∞0<A≤ B<∞ such that for all x∈ℝdx ^d, A‖x‖2≤∑i=1kwi2‖Pix‖2≤B‖x‖2, A||x||^2≤ _i=1^kw_i^2|| [rgb]0.1,0.05,0.96P_ix||^2≤ B||x||^2, where Pi [rgb]0.1,0.05,0.96P_i denotes the orthogonal projection onto the subspace i [rgb]0.1,0.05,0.96W_i. The weights wiw_i adjust the influence of each subspace, and A, B are fusion frame bounds. A fusion frame is tight if A=BA=B, and Parseval if A=B=1A=B=1. We will focus on Parseval fusion frames. If all weights wiw_i are equal to 11, we write the fusion frame simply as ii=1k\ [rgb]0,0,1W_i\_i=1^k. Frame bounds will help later in our convergence analysis. 2.1 Operators in Fusion Frames Let ((i,wi))i=1k(( [rgb]0,0,1W_i,w_i))_i=1^k be a Fusion Frame for ℝdR^d, with orthogonal projection matrices (Pi)i=1k( [rgb]0.1,0.05,0.96P_i)_i=1^k respectively. We define three operators: The analysis operator :ℝd→⨁i=1kiT_ [rgb]0,0,1W:R^d→ _i=1^k [rgb]0,0,1W_i maps a vector to its projections across all subspaces: (x)=(wiPiTx)i=1k.T_ [rgb]0,0,1W(x)=(w_i [rgb]0.1,0.05,0.96P_i^Tx)_i=1^k. (2) The synthesis operator ∗:⨁i=1ki→ℝdT^*_ [rgb]0,0,1W: _i=1^k [rgb]0,0,1W_i ^d “re-builds” a vector from its fusion frame representation: ∗((yi)i=1k)=∑i=1kwiPiyi.T^*_ [rgb]0,0,1W(( [rgb]0,1,0y_i)_i=1^k)= _i=1^kw_i [rgb]0.1,0.05,0.96P_i [rgb]0,1,0y_i. (3) The fusion frame operator is the composition of the two operators above, =∗S_ [rgb]0,0,1W=T^*_ [rgb]0,0,1WT_ [rgb]0,0,1W, defined by: (x)=∑i=1kwi2PiPiTx.S_ [rgb]0,0,1W(x)= _i=1^kw_i^2 [rgb]0.1,0.05,0.96P_i [rgb]0.1,0.05,0.96P_i^Tx. (4) This operator is self-adjoint, positive semi-definite, and bounded. For Parseval fusion frames, we have =IdS_ [rgb]0,0,1W=I_d (allow exact reconstruction). These operators help in mapping vectors in ℝdR^d to their fusion frame representations and back. 3 Fine-tuning in Fusion Frame Subspaces We will start by analyzing the subspace updates in LoRA (Hu et al., 2022). The key idea is representing weight updates through a low-rank decomposition. For a layer l with pre-trained parameters Wl∈ℝm×nW_l ^m× n, LoRA decomposes the weight update δWlδ W_l as a product of two rank-deficient matrices: Wl′=Wl+δWl=Wl+BlAl W_l =W_l+δ W_l=W_l+B_lA_l (5) where Bl∈ℝm×rB_l ^m× r, Al∈ℝr×nA_l ^r× n, and r<min(m,n)r< (m,n). To better understand the structure of these updates, we can check the Singular Value Decomposition (SVD) of δWlδ W_l: δWl=UΣVT=UrΣrVrT δ W_l=U V^T=U_r _rV_r^T (6) where U∈ℝm×mU ^m× m and V∈ℝn×nV ^n× n are best viewed as orthogonal matrices spanning the output and input spaces respectively, and Σ∈ℝm×n ^m× n is a rectangular diagonal matrix. Since the rank of δWlδ W_l is constrained to r, we can simplify this product to UrΣrVrTU_r _rV_r^T, where Σr _r contains only the top r singular values and UrU_r, VrV_r are the corresponding left and right singular vectors. This decomposition provides insight into LoRA updates: VrV_r defines a subspace in the input space ℝnR^n where the input is first projected. Then, Σr _r scales these projections, and UrU_r maps them to a subspace of the output space ℝmR^m. These input and output subspaces emerge implicitly as a by-product of training the matrices B and A, without direct control over their properties or interactions. Asymmetricity. Recent results indicate that the contribution of matrices A and B is asymmetric, which can lead to instability (Hayou et al., 2024), or failure to converge to the optimal solution (Malinovsky et al., 2024). This suggests two key opportunities for improvement. (i) Can we explicitly choose and work with multiple subspaces in both input space ℝnR^n and output space ℝmR^m to better capture parameter updates? (i) Can we maintain precise control over how these subspaces interact during training? While existing work has explored defining subspaces using Orthogonal bases derived from the pretrained weights (Lingam et al., 2024; Sun et al., 2024a) or fixed spectral bases such as the Fourier basis (Gao et al., 2024), there is a significant cost in storage and impact on inference throughput due to dense operations. We will see how Fusion frames address both questions directly. Specifically, we can decompose the input and output spaces into potentially overlapping subspaces with controlled relationships. Further, we can achieve good storage efficiency and inference throughput due to the inherent sparsity of the generated frames for free. 3.1 Fusion Frames for Finetuning Weight Updates For the first problem, we want to decompose the input and output spaces into multiple subspaces. The Fusion Frame operators in §2.1 can be used. In particular, let (Pn,iT)i=1k( [rgb]0.1,0.05,0.96P^T_n,i)_i=1^k and (Pm,iT)i=1k( [rgb]0.1,0.05,0.96P^T_m,i)_i=1^k be the orthogonal projection matrices for decomposing the input (ℝnR^n) and output (ℝmR^m) spaces into k subspaces with dimensions ρn _n and ρm _m, respectively. Let, Pn=[Pn,1,Pn,2,…,Pn,k]Pm=[Pm,1,Pm,2,…,Pm,k] split [rgb]0.1,0.05,0.96P_n= [ [rgb]0.1,0.05,0.96P_n,1, [rgb]0.1,0.05,0.96P_n,2,…, [rgb]0.1,0.05,0.96P_n,k ]\\ [rgb]0.1,0.05,0.96P_m= [ [rgb]0.1,0.05,0.96P_m,1, [rgb]0.1,0.05,0.96P_m,2,…, [rgb]0.1,0.05,0.96P_m,k ] split (7) We can model rich interactions between these subspaces by using trainable coefficient matrices, in contrast to the diagonal matrix Σr _r in LoRA. To be specific, we define (Cl,i)i=1k(C_l,i)_i=1^k where each Cl,i∈ℝρm×ρnC_l,i _m× _n encodes the relationships within the i-th input and output subspaces. These matrices can be neatly organized into a block diagonal structure: Cl=diag(Cl,1,Cl,2,…,Cl,k)C_l=diag(C_l,1,C_l,2,…,C_l,k) which yields our simple parameter update equation: Wl′=Wl+δWl=Wl+αmnPmClPnTW_l =W_l+δ W_l=W_l+ α mn [rgb]0.1,0.05,0.96P_mC_l [rgb]0.1,0.05,0.96P_n^T Expanding this expression with the subspaces, we get: Wl′=Wl+αmn∑i=1kPm,iCl,iPn,iTW_l =W_l+ α mn _i=1^k [rgb]0.1,0.05,0.96P_m,iC_l,i [rgb]0.1,0.05,0.96P_n,i^T Here, α is a scaling hyperparameter, normalized by mn mn (for dimension independence). Fig. 2 (left) shows the difference between SVD in LoRA and FrameFT. The projection matrices Pn [rgb]0.1,0.05,0.96P_n and Pm [rgb]0.1,0.05,0.96P_m are obtained from Tight Fusion Frames (TFF) discussed in §3.2. Under certain conditions, these TFFs exhibit Equichordal and Equi-Isoclinic properties (Fickus et al., 2023a), maximizing subspace separation for efficient representations. These TFFs remain fixed and only the coefficient matrices (Cl,i)i=1k(C_l,i)_i=1^k are trainable. Fig. 2 (right) shows FrameFT update for a single layer (also see Alg. 1). 0: Frozen Parameters WlW_l, Fusion Frame number of subspaces k, subspace dimensions ρm,ρn _m, _n, number of non-zero coefficients ncn_c, non-zero coefficients G, scalar α 1: m,n = shape(WlW_l) // get the shape of the parameter matrix 2: Pm [rgb]0.1,0.05,0.96P_m, Pn [rgb]0.1,0.05,0.96P_n = TFF(k,ρm,mk, _m,m), TFF(k,ρn,nk, _n,n) // generate the TFFs 3: Coeffs = Init(m, n, size=ncn_c) // initialize the coefficients matrix 4: ClC_l = rearrange(Coeffs, m, n, G) // re-arrange the coefficients in dense format 5: δWl=αmnPmClPnTδ W_l= α mn [rgb]0.1,0.05,0.96P_mC_l [rgb]0.1,0.05,0.96P_n^T //compute the update term 6: Wl′=Wl+δWlW_l^ =W_l+δ W_l // update the parameter matrix Algorithm 1 FrameFT: Finetuning with Frames Figure 2: (left) How parameter update δWδ W is modeled by LoRA and by FrameFT. (right) The parameter update rule in FrameFT. We freeze the Fusion Frame projection matrices and train only the coefficients ClC_l. 3.2 Constructing Tight Fusion Frames We use uniform tight fusion frames, where all subspaces have the same dimension. In practice, one can easily use different subspace dimensions across layers. To construct a (k,ρ,d)(k,ρ,d) uniform tight fusion frame where k is the number of subspaces, ρ is the dimension of each subspace, and d is the dimension of the entire space, we use the Spectral Tetris algorithm described in (Casazza et al., 2011), which is summarized as: (1) Construct a unit norm tight frame (UNTF) in ℂρC^ρ with d vectors. (2) Modulate these vectors using the kthk^th roots of unity to form k subspaces of dimension ρ in ℂdC^d. (3) Use the method in (Fickus et al., 2023b) to extend the result to real-valued spaces. This process ensures that the resultant set satisfies the tight fusion frame conditions. An example is included in Appendix §H. 3.3 Structured Coefficient Matrix Prior work on Sparse Fine-Tuning methods (Ansell et al., 2022) suggests that we can boost parameter efficiency by introducing sparsity into the coefficient matrices (Cl,i)i=1k(C_l,i)_i=1^k. This sparsification nicely complements our fusion frame architecture: while fusion frames provide structured subspace decompositions of the parameter space, sparse coefficients identify the most useful subspace interactions (see off-diagonal terms in Fig. 2). Note that the Fusion Frames themselves are sparse, which is distinct from the sparsity in the coefficient matrix. Both are exploited in our implementation. Our experiments indicate that a simple random sparsity pattern performs well: we randomly designate a small subset of entries in ClC_l as non-zero, distributing them uniformly across the block-diagonal matrices (Cl,i)i=1k(C_l,i)_i=1^k. Figure 3 shows this sparse structure with an example from RoBERTa-Large. Figure 3: An example of the coefficient matrix ClC_l from a Query layer (RoBERTa-L) with different subspace dimensions ρm=ρn=ρ _m= _n=ρ. The memory requirements for storing the updates reduce from O(kρmρn)O(k _m _n) to O(s)O(s), where s≪kρmρns k _m _n. To reduce memory overhead even further, we share these sparsity patterns across layers, i.e., supp(Cl)=supp(Cl′)supp(C_l)=supp(C_l ) for layers l and l′l , where supp(⋅)supp(·) represents the sets of index locations. The cross-layer sharing of sparsity patterns not only reduces memory requirements but also suggests that subspace interactions necessary for task adaptation may share structural patterns across network layers. An ablation study we present in §4.6 confirms this empirically: shared and per-layer random masks perform similarly (plus/minus noise of each other). 3.4 System Efficiency and Scalability Our design choices, specifically the use of algorithmically generated Fusion Frames and sparse coefficients, give a number of advantages over alternatives. Storage efficiency. Since Spectral Tetris generates Fusion Frames algorithmically, we eliminate the need to store basis matrices. Only the sparse coefficient matrices must be serialized. For a Llama-2-7B model, this reduces the checkpoint size to just 1.28 MB (vs. 67.1 MB for LoRA rank-32), facilitating ultra-low-latency “hot-swapping” of adapters in multi-tenant serving scenarios where a single backbone supports thousands of fine-tuned models for different users. Both numbers above assume FP32 precision. The frozen backbone operates in FP16/BF16 during training, but adapter parameters can be maintained in FP32 to preserve dynamic range/gradient fidelity. Memory Overhead. Unlike SVD-based methods such as SVFit (Sun et al., 2024a) and LoRA-XS (Bałazy et al., 2025), which require unique basis vectors for each layer, Fusion Frames generated by the Spectral Tetris algorithm depend only on the layer dimensions. This allows a single set of frames to be shared across all layers with the same dimensions, significantly reducing the initialization time (see Table 11) and runtime memory footprint. Inference throughput. While the orthogonal bases generated by SVD based methods are dense, Spectral Tetris directly constructs frames with inherent block-sparsity (see §10). This structural advantage directly translates to higher inference throughput (see Table 2). Beyond these practical efficiencies, the construction of FrameFT also offers useful theoretical advantages for optimization. Next, we analyze how preserving the smoothness of the loss landscape allows stable convergence properties. 3.5 Analysis of FrameFT Convergence properties Our formulation of the parameter updates using Tight Fusion Frames and sparse coefficients enjoys theoretical benefits compared to LoRA. The analysis in Sun et al. (2024b) describes how, for Lipschitz-smooth loss functions, LoRA can create a non-smooth landscape when projected onto the parameters in A and B. Via Lemma 3.1, we show that FrameFT preserves Lipschitz-smoothness: when a function exhibits Lipschitz smoothness w.r.t. W, this property holds when remapped onto our coefficient space ClC_l. Lemma 3.1. For any differentiable function f(W)f(W) that is L-Lipschitz smooth under Frobenius norm, with fusion frames characterized by frame bounds (Am,Bm)(A_m,B_m) and (An,Bn)(A_n,B_n) for output and input spaces, respectively, the transformed function f(Cl)=f(W0+PmClPnT)f(C_l)=f(W_0+ [rgb]0.1,0.05,0.96P_mC_l [rgb]0.1,0.05,0.96P_n^T) obeys: ‖∇f(Cl1)−∇f(Cl2)‖F≤LBmBn‖Cl1−Cl2‖F ||∇ f(C_l^1)-∇ f(C_l^2)||_F≤ LB_mB_n||C_l^1-C_l^2||_F Note that f(W)f(W) denotes the global loss function of the entire deep neural network, defined over the collection of all layer weights W=W1,…,WDW=\W_1,…,W_D\ just like in (Sun et al., 2024b, see p. 13). Thus, f implicitly encodes the composition of all network layers. During fine-tuning, with pre-trained weights frozen, the loss landscape is reparameterized purely as a function of the coefficient matrices f(C)f(C), where C=C1,…,CDC=\C_1,…,C_D\ represents the set of learnable Frame coefficients across all layers. Since FrameFT preserves smoothness, we can immediately invoke a broad set of convergence results for Lipschitz-smooth functions (Bubeck, 2015; Zhou and Cong, 2017). We highlight one result: gradient descent constrained to learning rates η≤1/L~η≤ 1/ L achieves the expected 1/T1/T convergence rate to first-order stationary points (in Theorem 3.2 below). Additional details are provided in §B. Theorem 3.2. Consider minimizing f(Cl)=f(W0+PmClPnT)f(C_l)=f(W_0+ [rgb]0.1,0.05,0.96P_mC_l [rgb]0.1,0.05,0.96P_n^T) where: f(Cl)f(C_l) is L~ L-Lipschitz smooth (but potentially non-convex) and f(Cl)f(C_l) is lower bounded by f∗f^*. For gradient descent with step size η≤1/L~η≤ 1/ L, running gradient descent for T iterations satisfies: (1/T)∑t=0T−1‖∇f(Clt)‖F2≤2(f(Cl0)−f∗)/(ηT) (1/T) _t=0^T-1||∇ f(C_l^t)||_F^2≤ 2(f(C_l^0)-f^*)/(η T) This analysis offers a unified view of subspace-based fine-tuning. Our results can be extended to methods such as SVFit and FourierFT by substituting the appropriate projection matrices and bounds. Lemma 3.1 explicitly links the Lipschitz constant of the reparameterized loss f(C)f(C) to the upper frame bounds Bm,BnB_m,B_n. This result directly guides our design choice: to maximize the smoothness of the loss landscape, we normalize our Frame bounds to unity (Parseval frames) for all experiments. To summarize, Lemma 3.1 informs the design choices via the bound L~=LBmBn L=LB_mB_n. 4 Experiments In this section, we compare the performance of Transformer-based models fine-tuned with FrameFT. All our experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory each. Individual subsections include more details on the experimental setup. Additional experiments on varying hyperparameters and discussions on subspace dimensions are available in the Appendix §D, §E, and §F. 4.1 Natural Language Understanding Capabilities Table 1: Fine-Tuning RoBERTa Base and Large models on GLUE benchmark. * indicates the results reported in prior work. FrameFT performs better than full fine-tuning and LoRA, using 10×10× fewer parameters. Model Method Params SST-2 MRPC CoLA QNLI RTE STS-B Avg. RoBERTa Base F* 125125M 94.894.8 90.290.2 63.663.6 92.8 92.8 78.778.7 91.291.2 85.285.2 SMT 5555K 94.294.2 89.589.5 63.963.9 91.891.8 78.178.1 90.490.4 84.784.7 LoRA* 0.30.3M 95.1 95.1 89.789.7 63.463.4 93.393.3 78.478.4 91.5 91.5 85.285.2 SVFTd=2RSVFT^R_d=2 9292K 94.394.3 89.489.4 62.462.4 92.092.0 78.378.3 91.191.1 84.684.6 AdaLoRA* 0.30.3M 94.594.5 88.788.7 62.062.0 93.193.1 81.0 81.0 90.590.5 85.085.0 FourierFT* 2424K 94.294.2 90.090.0 63.863.8 92.292.2 79.179.1 90.890.8 85.085.0 SVFit* 1818K 92.492.4 90.090.0 63.863.8 90.890.8 78.078.0 92.492.4 85.185.1 FrameFT 2424K 94.394.3 92.3 92.3 66.8 66.8 92.492.4 79.879.8 90.990.9 86.1 86.1 RoBERTa Large F* 356356M 96.4 96.4 90.990.9 68.068.0 94.794.7 86.686.6 92.4 92.4 88.288.2 SMT 1.51.5M 96.096.0 89.789.7 69.469.4 93.693.6 84.184.1 91.891.8 87.487.4 LoRA* 0.80.8M 96.296.2 90.290.2 68.268.2 94.8 94.8 85.285.2 92.392.3 88.288.2 SVFTd=2RSVFT^R_d=2 0.250.25M 96.196.1 90.290.2 66.766.7 94.394.3 83.083.0 92.192.1 87.187.1 VeRA 6161K 96.196.1 90.990.9 68.068.0 94.494.4 85.985.9 91.791.7 87.887.8 LoRA-XS* 6060K 96.396.3 91.291.2 68.668.6 94.394.3 89.5 89.5 92.292.2 88.7 88.7 RoseLoRA* 53.453.4K 95.295.2 90.290.2 69.269.2 94.794.7 89.289.2 92.092.0 88.588.5 FourierFT* 4848K 96.096.0 90.990.9 67.167.1 94.494.4 87.487.4 91.991.9 88.088.0 SVFit* 3636K 96.296.2 90.990.9 71.4 71.4 94.494.4 86.386.3 92.092.0 88.588.5 FrameFT 4848K 96.296.2 92.6 92.6 69.869.8 93.493.4 88.1 88.1 91.991.9 88.7 88.7 Evaluation framework: We evaluate the performance of FrameFT by fine-tuning the base and large variants of RoBERTa (Liu et al., 2019) across multiple GLUE benchmark tasks (Wang et al., 2019). This suite of benchmarks covers sentiment classification, paraphrase detection, and entailment recognition. It is a standardized setup for testing. Fine-tuning strategy: We use the original LoRA (Hu et al., 2022) recipe by fine-tuning the Query and Value matrices across network layers. For FrameFT, we use 10001000 non-zero coefficient entries with their positions determined randomly and shared across layers. For tight fusion frame construction, we use a subspace dimension ρ=2ρ=2, and the subspace count k is calibrated so that kρ=nkρ=n (n is layer dimension). Performance analysis: We report the Pearson correlation coefficient (PCC) for the STS-B task, Matthews correlation coefficient (MCC) for CoLA and accuracy for the remaining tasks. For the baseline methods, we report LoRA (Hu et al., 2022), AdaLoRA (Zhang et al., 2023), FourierFT (Gao et al., 2024), SVFit (Sun et al., 2024a), SMT (He et al., 2025), SVFT (Lingam et al., 2024) and RoseLoRA (Wang et al., 2024). The results are presented in Table 1. We see that FrameFT performs on par or better than LoRA and full fine-tuning on individual tasks, and performs better than all the baselines on average. FrameFT achieves this with 10×10× fewer parameters when compared to LoRA. We also note that even though the number of parameters of SVFit (Sun et al., 2024a) is slightly lower than FrameFT, SVFit trains the singular values, keeping the singular vectors fixed. So, in practice, one would need to save the singular vectors for each layer after training, which increases the storage requirements. Also, for RoseLoRA (Wang et al., 2024), only the parameter count is reported in Table 1, but they use a different mask for each layer, so the storage cost is 3×3× the number of parameters if we account for the location of the non-zero coefficients. 4.2 Instruction Tuning Benchmarking Framework: We evaluate FrameFT for fine-tuning LLMs to follow instructions. We finetune 7B and 13B variants of Llama2 models (Touvron et al., 2023), Gemma2 models (Team et al., 2024b) with 2B and 9B parameters and Llama 3.1 (Grattafiori et al., 2024) 8B model on the Alpaca instruction dataset (Taori et al., 2023). We evaluate the performance of the fine-tuned models on the LM-evaluation harness (Gao et al., 2023) by Eleuther AI. We use eight distinct challenge categories spanning reasoning, world knowledge, and generalization capabilities. FrameFT configuration: We apply FrameFT with n=5000n=5000 non-zero coefficients and adapt the Query and Value matrices for all the Transformer blocks. We determine the position of the non-zero coefficients at random and share them across all the layers. We use α=200α=200 across all of our experiments. A sensitivity analysis sweeping α∈10,100,400,600α∈\10,100,400,600\ for Llama-2-7B is in the Appendix I. We set the subspace dimension ρ=2ρ=2 and as before, calculate the number of subspaces k such that kρ=nkρ=n for each layer. For all baselines, we choose the hyperparameters suggested for the respective method. Performance Analysis: We compare FrameFT with LoRA (Hu et al., 2022), (IA)3(IA)^3 (Liu et al., 2022a), DoRA (Liu et al., 2024), S2FTS^2FT (Yang et al., 2024), SVFT (Lingam et al., 2024), and FourierFT (Gao et al., 2024). Our results are shown in Table 3. We observe that FrameFT performs on par or better compared to the baselines while using the fewest number of parameters across all models. In addition, FrameFT also provides other compute benefits we discuss in Section 4.4. 4.3 Performance on Vision transformer models We performed a set of experiments to determine whether FrameFT was effective only for Language models. To check this, we applied FrameFT to fine-tune Vision Transformers on 8 image classification tasks, which include remote sensing, fine-grained classification, and texture recognition. Performance Analysis: Section C in the appendix shows the performance of FrameFT across these tasks for ViT-L and ViT-B models. Table 2: Tokens per second for different PEFT methods. Method #params #toks/sec Llama-2-7b LoRA 262k 23.6k DoRA 262k 14.1k SVFit 1k 9.0k FourierFT 1k 1.8k FrameFT 1k 39.4 39.4k Gemma-2-9b LoRA 245k 22.1k DoRA 245k 16.0k SVFit 1k 12.4 FourierFT 1k 1.4k FrameFT 1k 29.6 29.6k Llama-3.1-8b LoRA 163k 22.3k DoRA 163k 18.7k SVFit 1k 11.5k FourierFT 1k 1.6k FrameFT 1k 27.3 27.3k We observe that FrameFT performs better than the baseline methods (which include full-finetuning) on average. Moreover, we observe performance improvements across the majority of the tasks when we increase the number of parameters for FrameFT. These results indicate that FrameFT generalizes well across both Vision and Language models. More experimental details are presented in Appendix C and Table 8. Figure 4: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients. The red line indicates the performance of full fine-tuning. 4.4 Operational Efficiency/Latency Analysis In this section, we analyze the operational efficiency of FrameFT in terms of storage requirements and inference throughput. In practice, when serving LLMs to a large user base, the pretrained model is held fixed, and various specialized variants are maintained simultaneously. Hence, we measure the throughput of the adapter layer introduced by different PEFT methods, with the idea that the pretrained layer throughput remains the same in the pretrained and fine-tuned models. Inference throughput: Table 2 displays the tokens per second for various methods across different model families. All measurements are performed on an NVIDIA A100 machine. We observe that FrameFT performs better than LoRA, which in turn performs better than FourierFT and other methods. These measurements are near-deterministic on fixed hardware. Across 100 iterations the coefficient of variation is below 12%, and the FrameFT–LoRA gap is large enough that variance does not affect the conclusion (e.g., on Llama-2-7B: 37.9±4.537.9± 4.5k for FrameFT vs. 24.4±2.724.4± 2.7k tokens/sec for LoRA). This highlights FrameFT’s efficiency benefits due to sparse coefficients and sparse projection matrices as described in §3.3 and Appendix §H Storage and Memory Footprint. A key advantage of FrameFT is its minimal storage footprint for task-specific adapter checkpoints. Since Fusion Frames are generated algorithmically once and shared across layers of the same dimension, the basis vectors (Pm,PnP_m,P_n) do not need to be stored or transmitted. We only store the sparse coefficient matrices. This is a clear advantage compared to SVD-based methods such as SVFT and LoRA-XS, where one needs to store the dense orthogonal basis for each layer, increasing their storage/memory requirements. We note that constructing the Fusion Frames takes a fixed amount of time. Specifically, the time complexity for constructing a (k,ρ,d)(k,ρ,d) TFF is (kd)O(kd). However, this generation step is performed only once during initialization, and the resulting projection matrices are shared across all network layers, amortizing the one-time compute cost across multiple layers and forward calls (see §H.3 for wall clock time). So, we focus on per-layer throughput measurements. Table 3: Performance of LLMs fine-tuned on the Alpaca dataset by various methods and then evaluated on the LM-evaluation-harness. FrameFT performs competitively with all the baseline methods under comparison, using a 30×30× fewer number of parameters than LoRA. FrameFT needs minimal hyperparameter tuning. Model Method #Params ARC-c ARC-e BoolQ HellaSwag OBQA PIQA RTE WinoGrande Avg. Llama-2-7b LoRA 16.716.7M 45.8245.82 77.0277.02 78.8178.81 58.0858.08 35.235.2 78.8378.83 61.0161.01 70.7270.72 63.1863.18 (IA)3(IA)^3 614K 45.3845.38 76.6076.60 77.8977.89 57.7457.74 34.634.6 78.7878.78 64.2664.26 68.8268.82 63.0163.01 DoRA 16.716.7M 45.1445.14 76.3976.39 78.3578.35 57.7857.78 34.034.0 78.7378.73 67.1567.15 67.1767.17 63.0963.09 S2FTS^2FT 56.656.6M 46.1646.16 77.4477.44 78.3878.38 58.1058.10 32.832.8 78.8478.84 61.0161.01 68.5868.58 62.6662.66 SVFTd=8BSVFT^B_d=8 4.54.5M 45.7945.79 77.0777.07 78.8378.83 57.9857.98 34.634.6 78.5678.56 61.5261.52 70.6470.64 63.1263.12 FourierFT 320320K 44.9644.96 77.1477.14 79.0579.05 58.2158.21 34.634.6 78.8978.89 62.4562.45 70.4870.48 63.2263.22 FrameFT 320320K 45.2245.22 76.9376.93 78.6278.62 58.0858.08 34.234.2 78.6278.62 66.0666.06 71.1971.19 63.62 63.62 Llama-2-13b LoRA 26.226.2M 50.7750.77 80.3580.35 81.4481.44 61.1361.13 36.036.0 79.3879.38 69.6869.68 72.6172.61 66.4266.42 (IA)3(IA)^3 963963K 50.2550.25 79.9679.96 80.5280.52 60.5460.54 34.234.2 79.2779.27 68.5968.59 72.2272.22 65.6965.69 DoRA 26.226.2M 51.7951.79 80.1380.13 80.2180.21 61.4461.44 35.635.6 79.8179.81 71.4871.48 72.2172.21 66.58 66.58 S2FTS^2FT 111111M 20.7320.73 35.1035.10 46.1746.17 32.9432.94 14.614.6 58.4858.48 55.2355.23 50.0450.04 39.1639.16 SVFTd=8BSVFT^B_d=8 7.17.1M 50.4550.45 80.4280.42 80.5180.51 60.8260.82 34.834.8 79.2279.22 69.5869.58 72.0672.06 65.9865.98 FourierFT 400400K 49.7449.74 80.1780.17 80.8280.82 60.9860.98 35.635.6 79.7679.76 64.6264.62 72.0672.06 65.4765.47 FrameFT 400400K 50.3450.34 79.7579.75 81.1981.19 60.8760.87 35.835.8 80.0880.08 68.5968.59 72.2972.29 66.1166.11 Gemma-2-2B LoRA 6.46.4M 48.6348.63 79.7679.76 76.6176.61 55.8555.85 31.631.6 78.9478.94 62.0962.09 68.1968.19 62.7062.70 (IA)3(IA)^3 292K 46.7646.76 80.1380.13 70.7070.70 55.7955.79 31.231.2 78.1378.13 59.2159.21 69.7769.77 61.4661.46 DoRA 6.46.4M 48.7248.72 80.3580.35 72.2672.26 55.9655.96 33.633.6 78.5678.56 67.1567.15 68.9868.98 63.2063.20 S2FTS^2FT 16.916.9M 45.9045.90 78.4178.41 59.2459.24 54.8054.80 31.431.4 77.8077.80 57.4057.40 56.2756.27 57.6557.65 SVFTd=8BSVFT^B_d=8 1.41.4M 48.7248.72 81.1081.10 71.1371.13 56.3756.37 33.633.6 78.8478.84 65.7065.70 70.4070.40 63.2363.23 FourierFT 260260K 45.9845.98 79.1279.12 73.8873.88 55.5755.57 32.832.8 79.1679.16 67.8767.87 67.8767.87 62.7862.78 FrameFT 260260K 48.3748.37 81.0681.06 75.1375.13 55.7155.71 34.634.6 79.0579.05 69.6769.67 69.3069.30 64.11 64.11 Gemma-2-9B LoRA 17.917.9M 64.7664.76 88.2288.22 86.3986.39 62.9662.96 36.236.2 82.4882.48 70.4070.40 75.3075.30 70.83 70.83 (IA)3(IA)^3 774K 61.9561.95 87.2187.21 85.0585.05 62.1162.11 35.835.8 81.5681.56 68.9568.95 74.2774.27 69.6169.61 DoRA 17.917.9M 62.6262.62 87.1687.16 86.0286.02 62.9162.91 35.235.2 81.6181.61 71.4871.48 73.7173.71 69.0969.09 S2FTS^2FT 74.474.4M 53.4153.41 80.5580.55 80.7380.73 59.0259.02 32.832.8 79.5479.54 69.6769.67 70.3270.32 65.7665.76 SVFTd=8BSVFT^B_d=8 3.93.9M 63.2563.25 87.8887.88 86.5486.54 63.1163.11 36.236.2 81.7281.72 71.1271.12 73.6473.64 70.4370.43 FourierFT 420420K 64.1664.16 88.2188.21 86.3686.36 62.7862.78 36.436.4 81.6681.66 70.0370.03 74.2774.27 70.4870.48 FrameFT 420420K 65.0165.01 88.0088.00 86.0286.02 63.0963.09 37.237.2 81.7281.72 69.3169.31 74.8274.82 70.6570.65 Llama-3.1-8B LoRA 13.613.6M 55.3855.38 83.3883.38 82.0882.08 61.7361.73 35.235.2 81.2381.23 74.0174.01 75.5375.53 68.56 68.56 (IA)3(IA)^3 524K 54.2754.27 82.7982.79 82.2082.20 61.0461.04 34.434.4 80.5880.58 68.9568.95 74.5974.59 67.3567.35 DoRA 13.613.6M 54.0154.01 82.0782.07 81.3181.31 61.0761.07 34.634.6 81.2381.23 67.5167.51 72.7772.77 66.8266.82 S2FTS^2FT 65.465.4M 31.5731.57 60.0260.02 59.8259.82 44.2444.24 20.020.0 68.8868.88 56.6856.68 53.6753.67 49.3649.36 SVFTd=8BSVFT^B_d=8 2.82.8M 53.1253.12 82.3982.39 82.3282.32 60.5860.58 36.236.2 80.5380.53 73.3673.36 74.0274.02 67.8167.81 FourierFT 320320K 51.3651.36 80.5180.51 81.4981.49 60.5760.57 34.034.0 80.4180.41 70.3970.39 73.9573.95 66.5866.58 FrameFT 320320K 53.4153.41 82.5382.53 82.2982.29 60.7960.79 34.634.6 80.9080.90 75.0975.09 73.4073.40 67.8767.87 4.5 Performance vis-a-vis non-zero coefficients We evaluate the performance of FrameFT as we increase the number of non-zero coefficients. We choose the RoBERTa base model for this experiment and check the performance of the model fine-tuned with FrameFT on different tasks in the GLUE benchmark. Figure 4 shows expected trends: the performance of the fine-tuned model on different tasks increases as we increase the number of non-zero terms in the coefficient matrix. Table 6 also confirms this behavior as we increase n from 10001000 to 50005000. These results show that FrameFT is able to utilize the additional parameters to improve the fine-tuning performance for both vision and language tasks. Additional experiments on more tasks from the GLUE benchmark are included in appendix E. 4.6 Sparsity Pattern and Cross-Layer Sharing Ablation In this experiment, we ablate the choice of random shared sparsity by comparing four masking strategies – a random shared mask across all layers, a per layer random mask, magnitude-based non-zero coefficients selection and a gradient-based mask for FrameFT. We evaluate the performance of each of these methods for finetuning RoBERTA-base model on the GLUE benchmark. For the random mask strategies, we average the performance accross five different seeds each. Results are shown in Table 4. Table 4: Sparsity mask ablation on RoBERTa-Base/GLUE (averaged over five seeds). Shared and per-layer random masks perform within noise of each other while adaptive heuristics perform strictly worse. Method SST-2 MRPC CoLA QNLI RTE STS-B Avg. Shared random mask 94.40±0.1794.40±0.17 92.31±0.4692.31±0.46 66.58±0.7766.58±0.77 92.35±0.1292.35±0.12 80.21±0.8280.21±0.82 90.96±0.1190.96±0.11 86.18±0.1886.18±0.18 Per-layer random mask 94.10±0.1694.10±0.16 93.53±0.3293.53±0.32 66.44±0.6166.44±0.61 92.31±0.1492.31±0.14 80.34±0.3980.34±0.39 90.86±0.0990.86±0.09 86.26±0.1786.26±0.17 Magnitude-based mask 94.1994.19 92.4992.49 65.8265.82 91.6191.61 78.4078.40 91.0191.01 85.6085.60 Gradient-based mask 89.3989.39 74.5974.59 60.3060.30 84.8084.80 63.4963.49 76.4076.40 74.8374.83 We see that cross-layer sharing incurs no expressivity cost: shared and per-layer masks are indistinguishable within variance. This aligns with Lemma 3.1: the optimization geometry depends on the Fusion Frame bounds and not on the specific entries of the coefficient matrix, which are non-zero. Interestingly, adaptive heuristics perform strictly worse. Gradient-based masks collapse to 74.83 average, lower than random. We suspect this is due to task-relevant subspace interactions being broadly distributed across the coefficient matrix rather than concentrated in high-magnitude or high-gradient pretrained weights. Random sampling provides better coverage of this distributed structure than greedy selection. 4.7 Training Time and GPU Memory We measured training time and peak GPU memory across all five models evaluated in Section 4.2. Table 5 reports these measurements. FrameFT trains 5-7% faster than LoRA and within 0.6 GiB of its peak GPU memory. The modest memory gap reflects the fact that the frozen pretrained backbone dominates the total GPU footprint. Both these measurements use standard PyTorch without custom kernels. Optimized kernels will be available on the GitHub repository shortly. Table 5: Training time (minutes) and peak GPU memory (GiB) for LoRA and FrameFT on the Alpaca instruction tuning task. Metric Method Llama-2-7B Llama-2-13B Gemma-2-9B Gemma-2-2B Llama-3.1-8B Training time (min) LoRA 238 363 323 126 231 FrameFT 225 345 301 115 212 GPU memory (GiB) LoRA 30.1 34.8 37.8 26.9 38.2 FrameFT 29.5 34.7 37.7 26.8 37.6 5 Related work Model adaptation for downstream tasks has been studied extensively in recent years, and has provided various efficient methods that reduce computation and storage needs while maintaining performance. Here, we describe different variants of PEFT methods briefly introduced in §1. Adapters: Adapters introduce specialized modules between pre-existing layers within the pretrained model. These adapter layers are trained during fine-tuning while keeping the pretrained model parameters frozen (Houlsby et al., 2019; Karimi Mahabadi et al., 2021; He et al., 2022). By keeping the parameters of the original model frozen, they preserve the knowledge acquired during pretraining while reducing the risk of overfitting. Low-Rank Matrix Factorization: LoRA techniques reparameterize the weight updates on selected layers of the model through low-rank factorizations (Hu et al., 2022). This framework of training only the decomposition matrices while freezing pretrained parameters has led to many variants exploring asymmetric chaining (Malinovsky et al., 2024), quantization-aware formulations (Dettmers et al., 2023), different learning rates for the update terms (Hayou et al., 2024) , and compressing the adapters into a single-matrix update (Bensaïd et al., 2025). Prefix and Prompt Tuning. Prefix-tuning strategies prepend learnable vector sequences before transformer layer inputs, creating controllable input modifications while keeping the pretrained model fixed (Wang et al., 2025; Li and Liang, 2021; Qin and Eisner, 2021; Lester et al., 2021b; Liu et al., 2022c). Closely related, Prompt-tuning approaches learn different prompt representations to guide model behaviors without architectural modification (Xiao et al., 2025; Lester et al., 2021a; Liu et al., 2022b; Ge et al., 2022). Unlike prefix tuning, prompt tuning operates exclusively at the input embedding level, making it particularly efficient and easy to integrate (Lester et al., 2021a). This approach has been widely extended to domain adaptation (Ge et al., 2022), vision-language models (Zhou et al., 2022), and diffusion models (Dong et al., 2023; Chung et al., 2023). Sparse Fine-Tuning: Sparse fine-tuning methodologies exploit natural parameter redundancy by targeting only critical components while freezing the rest (Iurada et al., 2025; Khaki et al., 2025; He et al., 2025; Lu et al., 2024; Gao et al., 2024; Guo et al., 2021). These approaches, whether through low-rank operations (Lu et al., 2024; Dragomir et al., 2026) or spectral compression via Fourier transformations (Gao et al., 2024; Zhang et al., 2026) – achieve competitive/superior performance compared to full fine-tuning but reducing trainable parameter count. 6 Conclusions We describe FrameFT, a parameter-efficient fine-tuning framework that leverages structured subspace decompositions based on Fusion Frames to fine-tune transformer models for vision and language tasks. Our extensive empirical validation across both vision transformers and state-of-the-art language models (including the Llama and Gemma families), shows that substantial compute and parameter efficiency gains are achievable without sacrificing performance across many evaluation benchmarks. We also provide analysis showing that FrameFT preserves the Lipschitz smoothness of the loss landscape, and so achieves desirable convergence properties. We believe that one key advantage of parameter efficiency of FrameFT will be in situations where the use case requires a set of fine-tuned models, each fine-tuned on a specific task and invoked on a case-by-case basis. We note that support for structured sparsity (beyond 2:42:4 sparsity) remains limited but this provides a concrete opportunity for higher efficiency gains if specialized kernels are implemented. The code is available at https://github.com/vsingh-group/FrameFT. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning by introducing more efficient methods for adapting foundation models. FrameFT aims to lower the computational cost and storage requirements for fine-tuning, which can facilitate broader accessibility and reduced energy usage in AI applications. We are not aware of any specific negative societal consequences or ethical issues unique to this work that require further highlighting. References Adepu et al. (2024) H. Adepu, Z. Zeng, L. Zhang, and V. Singh FrameQuant: flexible low-bit quantization for transformers. In Forty-first International Conference on Machine Learning, Cited by: §1. Ansell et al. (2022) A. Ansell, E. Ponti, A. Korhonen, and I. Vulić Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. Cited by: §1, §3.3. Bałazy et al. (2025) K. Bałazy, M. Banaei, K. Aberer, and J. Tabor LoRA-xs: low-rank adaptation with extremely small number of parameters. External Links: 2405.17604, Link Cited by: §1, §3.4. Bensaïd et al. (2025) D. Bensaïd, N. Rotstein, R. Velich, D. Bensaïd, and R. Kimmel SingLoRA: low rank adaptation using a single matrix. External Links: 2507.05566, Link Cited by: §1, §5. Boufounos et al. (2009) P. Boufounos, G. Kutyniok, and H. Rauhut Compressed sensing for fusion frames. Proceedings of SPIE - The International Society for Optical Engineering. External Links: Document Cited by: §1. Bubeck (2015) S. Bubeck Convex optimization: algorithms and complexity. External Links: 1405.4980 Cited by: §3.5. Cai et al. (2014) J. Cai, H. Ji, Z. Shen, and G. Ye Data-driven tight frame construction and image denoising. Applied and Computational Harmonic Analysis. Cited by: §1. Casazza et al. (2008) P. G. Casazza, G. Kutyniok, and S. Li Fusion frames and distributed processing. Applied and computational harmonic analysis 25 (1). Cited by: §1. Casazza et al. (2011) P. G. Casazza, M. Fickus, D. G. Mixon, Y. Wang, and Z. Zhou Constructing tight fusion frames. Applied and Computational Harmonic Analysis 30 (2). External Links: ISSN 1063-5203, Document, Link Cited by: §H.2, Appendix H, §3.2. Casazza and Kutyniok (2012) P. G. Casazza and G. Kutyniok Finite frames, theory and applications. External Links: Link Cited by: §2. Cheng et al. (2017) G. Cheng, J. Han, and X. Lu Remote sensing image scene classification: benchmark and state of the art. Proceedings of the IEEE 105 (10), p. 1865–1883. Cited by: Appendix C. Christensen (2018) O. Christensen An introduction to frames and riesz bases. External Links: Link Cited by: §1, §2. Chung et al. (2023) H. Chung, J. C. Ye, P. Milanfar, and M. Delbracio Prompt-tuning latent diffusion models for inverse problems. External Links: 2310.01110 Cited by: §5. Cimpoi et al. (2014) M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Cited by: Appendix C. Czaja and Na (2024) W. Czaja and S. Na Frame quantization of neural networks. arXiv preprint arXiv:2404.08131. Cited by: §1. Dehghani et al. (2023) M. Dehghani, J. Djolonga, B. Mustafa, et al. Scaling vision transformers to 22 billion parameters. External Links: 2302.05442 Cited by: §1. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. Cited by: §1, §1, §5. Dong et al. (2023) W. Dong, S. Xue, X. Duan, and S. Han Prompt tuning inversion for text-driven image editing using diffusion models. External Links: 2305.04441 Cited by: §5. Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, Cited by: Appendix C. Dragomir et al. (2026) A. Dragomir, I. Pintilie, A. Barbalau, M. Dragoi, F. Brad, C. D. Paduraru, A. Tifrea, E. Burceanu, and R. T. Ionescu JumpLoRA: sparse adapters for continual learning in large language models. External Links: 2604.16171, Link Cited by: §5. Fang et al. (2025) W. Fang, J. Wang, Y. Lu, S. Liu, Y. Wu, Y. Ma, and Z. Xie A survey of circuit foundation model: foundation ai models for vlsi circuit design and eda. External Links: 2504.03711, Link Cited by: §1. Fickus et al. (2023a) M. Fickus, J. W. Iverson, J. Jasper, and D. G. Mixon Harmonic grassmannian codes. Applied and Computational Harmonic Analysis 65, p. 1–39. Cited by: item 3, Appendix H, §3.1. Fickus et al. (2023b) M. Fickus, J. W. Iverson, J. Jasper, and D. G. Mixon Harmonic grassmannian codes. Applied and Computational Harmonic Analysis 65. External Links: ISSN 1063-5203, Document, Link Cited by: item 3. Gao et al. (2023) L. Gao, J. Tow, B. Abbasi, et al. A framework for few-shot language model evaluation. Zenodo. External Links: Link Cited by: §4.2. Gao et al. (2024) Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li Parameter-efficient fine-tuning with discrete fourier transform. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, Cited by: Appendix C, §1, §3, §4.1, §4.2, §5. Ge et al. (2022) C. Ge, R. Huang, M. Xie, Z. Lai, S. Song, S. Li, and G. Huang Domain adaptation via prompt learning. External Links: 2202.06687 Cited by: §5. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, et al. The llama 3 herd of models. External Links: 2407.21783 Cited by: §4.2. Guo et al. (2021) D. Guo, A. M. Rush, and Y. Kim Parameter-efficient transfer learning with diff pruning. External Links: 2012.07463 Cited by: §1, §5. Hayou et al. (2024) S. Hayou, N. Ghosh, and B. Yu LoRA+: efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning, Cited by: §1, §3, §5. He et al. (2025) H. He, J. B. Li, X. Jiang, and H. Miller SMT: fine-tuning large language models with sparse matrices. In The Thirteenth International Conference on Learning Representations, Cited by: §4.1, §5. He et al. (2022) J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig Towards a unified view of parameter-efficient transfer learning. External Links: 2110.04366 Cited by: §5. Helber et al. (2018) P. Helber, B. Bischke, A. Dengel, and D. Borth Introducing eurosat: a novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, p. 204–207. Cited by: Appendix C. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, et al. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research. Cited by: §5. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: §1, §1, §3, §4.1, §4.1, §4.2, §5. Hwang et al. (2019) W. Hwang, P. Huang, B. Kung, J. Ho, and T. Jong Frame-based sparse analysis and synthesis signal representations and parseval k-svd. IEEE Transactions on Signal Processing 67. Cited by: §1. Iurada et al. (2025) L. Iurada, M. Ciccone, and T. Tommasi Efficient model editing with task-localized sparse fine-tuning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Karimi Mahabadi et al. (2021) R. Karimi Mahabadi, S. Ruder, M. Dehghani, and J. Henderson Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. Association for Computational Linguistics. Cited by: §5. Khaki et al. (2025) S. Khaki, X. Li, J. Guo, L. Zhu, C. Xu, K. N. Plataniotis, A. Yazdanbakhsh, K. Keutzer, S. Han, and Z. Liu SparseLoRA: accelerating llm fine-tuning with contextual sparsity. External Links: 2506.16500, Link Cited by: §5. Lester et al. (2021a) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Cited by: §5. Lester et al. (2021b) B. Lester, R. Al-Rfou, and N. Constant The power of scale for parameter-efficient prompt tuning. External Links: 2104.08691 Cited by: §5. Li and Liang (2021) X. L. Li and P. Liang Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Cited by: §5. Lingam et al. (2024) V. Lingam A. Tejaswi et al. SVFT: parameter-efficient fine-tuning with singular vectors. In Advances in Neural Information Processing Systems, Cited by: §1, §3, §4.1, §4.2. Liu et al. (2022a) H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. arXiv preprint arXiv:2205.05638. Cited by: §4.2. Liu et al. (2024) S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen DoRA: weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, Cited by: §4.2. Liu et al. (2022b) X. Liu, K. Ji, Y. Fu, et al. P-tuning: prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Cited by: §5. Liu et al. (2022c) X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang P-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks. External Links: 2110.07602 Cited by: §5. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov RoBERTa: a robustly optimized bert pretraining approach. External Links: 1907.11692, Link Cited by: §4.1. Lu et al. (2024) X. Lu, A. Zhou, Y. Xu, et al. SPP: sparsity-preserved parameter-efficient fine-tuning for large language models. External Links: 2405.16057 Cited by: §1, §5. Maji et al. (2013) S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi Fine-grained visual classification of aircraft. External Links: 1306.5151 Cited by: Appendix C. Malinovsky et al. (2024) G. Malinovsky, U. Michieli, H. A. A. K. Hammoud, et al. Randomized asymmetric chain of lora: the first meaningful theoretical framework for low-rank adaptation. External Links: 2410.08305 Cited by: §3, §5. OpenAI (2025) OpenAI GPT-4 technical report. External Links: 2303.08774 Cited by: §1. Parkhi et al. (2012) O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: Appendix C. Pu et al. (2023) G. Pu, A. Jain, J. Yin, and R. Kaplan Empirical analysis of the strengths and weaknesses of peft techniques for llms. External Links: 2304.14999, Link Cited by: item a. Qin and Eisner (2021) G. Qin and J. Eisner Learning how to ask: querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599. Cited by: §5. Sun et al. (2024a) C. Sun, J. Wei, Y. Wu, Y. Shi, S. He, Z. Ma, N. Xie, and Y. Yang SVFit: parameter-efficient fine-tuning of large pre-trained models using singular values. External Links: 2409.05926, Link Cited by: §3, §3.4, §4.1. Sun et al. (2024b) Y. Sun, Z. Li, Y. Li, and B. Ding Improving loRA in privacy-preserving federated learning. In The Twelfth International Conference on Learning Representations, Cited by: §3.5, §3.5. Taori et al. (2023) R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, and P. Liang Alpaca: a strong, instruction-following language model. Note: https://crfm.stanford.edu/2023/03/13/alpaca.html Cited by: §4.2. Team et al. (2024a) G. Team, R. Anil, S. Borgeaud, et al. Gemini: a family of highly capable multimodal models. External Links: 2312.11805 Cited by: §1. Team et al. (2024b) G. Team, M. Riviere, S. Pathak, et al. Gemma 2: improving open language models at a practical size. External Links: 2408.00118 Cited by: §4.2. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, et al. Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288 Cited by: §1, §4.2. Waldron (2019) S. F. D. Waldron An introduction to finite tight frames. External Links: Link Cited by: §1, §2. Wang et al. (2019) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, Cited by: §4.1. Wang et al. (2025) H. Wang, B. Chen, S. Li, X. Liang, H. K. Lee, K. Kawaguchi, and T. Hu Prefix-tuning+: modernizing prefix-tuning by decoupling the prefix from attention. External Links: 2506.13674, Link Cited by: §5. Wang et al. (2024) H. Wang, T. Liu, R. Li, M. Cheng, T. Zhao, and J. Gao RoseLoRA: row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. External Links: 2406.10777, Link Cited by: §4.1. Wistuba et al. (2024) M. Wistuba, P. T. Sivaprasad, L. Balles, and G. Zappella Choice of peft technique in continual learning: prompt tuning is not all you need. External Links: 2406.03216, Link Cited by: item a. Xia et al. (2024) W. Xia, C. Qin, and E. Hazan Chain of loRA: efficient fine-tuning of language models via residual learning. In ICML 2024 Workshop on LLMs and Cognition, Cited by: §1. Xiao et al. (2025) Z. Xiao S. Yan et al. DynaPrompt: dynamic test-time prompt tuning. External Links: 2501.16404, Link Cited by: §5. Yang et al. (2024) X. Yang, J. Leng, G. Guo, J. Zhao, R. Nakada, L. Zhang, H. Yao, and B. Chen S2FT: efficient, scalable and generalizable llm fine-tuning by structured sparsity. In The 38th Conference on Neural Information Processing Systems (NeurIPS), Cited by: §4.2. Yaras et al. (2024) C. Yaras, P. Wang, L. Balzano, and Q. Qu Compressible dynamics in deep overparameterized low-rank learning & adaptation. In Forty-first International Conference on Machine Learning, Cited by: §1. Zhang et al. (2026) B. Zhang, Z. Yu, L. Zhang, K. Lin, T. Chen, Y. Sun, Y. Ye, and Y. He S2FT: parameter-efficient fine-tuning in sparse spectrum domain. External Links: 2605.08589, Link Cited by: §1, §5. Zhang et al. (2025) D. Zhang, T. Feng, L. Xue, Y. Wang, Y. Dong, and J. Tang Parameter-efficient fine-tuning for foundation models. External Links: 2501.13787, Link Cited by: §1. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao AdaLoRA: adaptive budget allocation for parameter-efficient fine-tuning. External Links: 2303.10512, Link Cited by: §4.1. Zhou and Cong (2017) F. Zhou and G. Cong On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimization. arXiv preprint arXiv:1708.01012. Cited by: §3.5. Zhou et al. (2022) K. Zhou, J. Yang, C. C. Loy, and Z. Liu Learning to prompt for vision-language models. International Journal of Computer Vision. Cited by: §5. Zhu et al. (2024) J. Zhu K. Greenewald et al. Asymmetry in low-rank adapters of foundation models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. Cited by: §1. Łajszczak et al. (2024) M. Łajszczak, G. Cámbara, and Y. L. others BASE tts: lessons from building a billion-parameter text-to-speech model on 100k hours of data. External Links: 2402.08093 Cited by: §1. Appendix In this appendix, we provide proofs for the theorems and additional details on the experiments presented in the main paper. In Section A, we provide a detailed proof for the lemma showing that FrameFT preserves the Lipschitz smoothness of the loss function. Section B presents the results for the convergence guarantee of Gradient Descent to a stationary point for Lipschitz smooth functions. Section C evaluates the performance of FrameFT for Image classification tasks. In Section D, we measure the performance of FrameFT as we vary the subspace dimension. In Section E, we show the performance of FrameFT with an increasing number of non-zero coefficients. We observe that FrameFT can leverage the additional parameters to improve the performance. We provide some additional discussions in Section F on the Frames and different subspace dimensions in Fusion Frames. Section G lists the hyperparameters used for our experiments. In Section H, we describe an algorithm for constructing Tight Fusion Frames along with an example. Finally, in Section I, we analyze the sensitivity of FrameFT to the scaling factor α and observe that FrameFT performs well above the baselines across a wide range of α values. Appendix A FrameFT preserves Lipschitz smoothness Here, for working through the proof, assume the following dimensions for the matrices involved: PmP_m is m×pm× p, PnP_n is n×qn× q, Cl⋅C_l^· is p×qp× q and W0W_0 is m×nm× n. Proof. Let us consider how the loss function behaves at two different points. Take any two coefficient matrices Cl1C_l^1 and Cl2C_l^2. From our composition rule, these map to: W1=W0+PmCl1PnTW^1=W_0+P_mC_l^1P_n^T and W2=W0+PmCl2PnTW^2=W_0+P_mC_l^2P_n^T. Scalar is omitted for notation simplicity. By the L-Lipschitz smoothness assumption on f(W)f(W): ‖∇f(W1)−∇f(W2)‖F≤L‖W1−W2‖F ||∇ f(W^1)-∇ f(W^2)||_F≤ L||W^1-W^2||_F where we use the notation ∇f(W)=∂f∂W∇ f(W)= ∂ f∂ W, the derivative of f with respect to W. We also have W1−W2=Pm(Cl1−Cl2)PnTW^1-W^2=P_m(C_l^1-C_l^2)P_n^T and we will show shortly that ‖W1−W2‖F||W^1-W^2||_F is upper-bounded by terms involving (Cl1−Cl2)(C_l^1-C_l^2) and other constants. The chain rule for matrix derivatives means the derivative of f with respect to the coefficients ClC_l is, ∇f(Cl) ∇ f(C_l) =∂f∂W⋅∂W∂Cl=∇f(W)⋅∇W(Cl). = ∂ f∂ W· ∂ W∂ C_l=∇ f(W)·∇ W(C_l). The term ∇W(Cl)∇ W(C_l) denotes how the weight matrix W changes with respect to a change in the coefficient matrix ClC_l. We will use directional derivatives to compute this. Consider a matrix ζΔ∈ℝkρm×kρn _ ^k _m× k _n which represents a direction. The directional derivative of W with respect to ClC_l in the direction of ζΔ _ is given by ∇ζΔW(Cl) _ _ W(C_l) =limt→0W(Cl+tζΔ)−W(Cl)t=limt→0PmζΔPnTt=PmζΔPnT = _t→ 0 W(C_l+t _ )-W(C_l)t= _t→ 0 P_m _ P_n^Ttt=P_m _ P_n^T Hence, the derivative of f with respect to ClC_l in the direction of ζΔ _ is given by ∇ζΔf(Cl) _ _ f(C_l) =∇f(W)⋅∇ζΔW(Cl) =∇ f(W)· _ _ W(C_l) =Tr((∇f(W))T(PmζΔPnT)) =Tr((∇ f(W))^T(P_m _ P_n^T)) =(a)Tr(PnT∇f(W)TPmζΔ) (a)=Tr(P_n^T∇ f(W)^TP_m _ ) =(b)Tr((PmT∇f(W)Pn)TζΔ) (b)=Tr((P_m^T∇ f(W)P_n)^T _ ) =⟨PmT∇f(W)Pn,ζΔ⟩ = P_m^T∇ f(W)P_n, _ The equality (a) follows from Tr(AB)=Tr(BA)Tr(AB)=Tr(BA) and the equality (b) follows from (ABC)T=CTBTAT(ABC)^T=C^TB^TA^T. From the definition of the directional gradient, we have that ∇ζΔf(Cl)=⟨∇f(Cl),ζΔ⟩ _ _ f(C_l)= ∇ f(C_l), _ . But from above, we have ∇ζΔf(Cl)=Tr((PmT∇f(W)Pn)TζΔ) _ _ f(C_l)=Tr((P_m^T∇ f(W)P_n)^T _ ). These two expressions are equal for all ζΔ _ . Therefore, ∇f(Cl)=PmT∇f(W)Pn∇ f(C_l)=P_m^T∇ f(W)P_n Hence, for two coefficient matrices Cl1C_l^1, Cl2C_l^2: ‖∇f(Cl1)−∇f(Cl2)‖F ||∇ f(C_l^1)-∇ f(C_l^2)||_F =‖PmT(∇f(W1)−∇f(W2))Pn‖F =||P_m^T(∇ f(W^1)-∇ f(W^2))P_n||_F ≤‖Pm‖op‖(∇f(W1)−∇f(W2))Pn‖F ≤||P_m||_op||(∇ f(W^1)-∇ f(W^2))P_n||_F ≤‖∇f(W1)−∇f(W2)‖F‖Pm‖op||Pn||op ≤||∇ f(W^1)-∇ f(W^2)||_F||P_m||_op||P_n||_op ≤L‖W1−W2‖FBmBn ≤ L||W^1-W^2||_F B_m B_n (8) We can bound the operator norm of PmP_m using the upper frame bounds. ‖PmTx‖22≤Bm‖x‖2⟹||Pm||op=‖PmT‖op≤Bm||P_m^Tx||_2^2≤ B_m||x||^2 ||P_m||_op=||P_m^T||_op≤ B_m Now, we need to bound ‖W1−W2‖F||W^1-W^2||_F. We know that W1=W0+PmCl1PnTW^1=W_0+P_mC_l^1P_n^T and W2=W0+PmCl2PnTW^2=W_0+P_mC_l^2P_n^T. Therefore: W1−W2=Pm(Cl1−Cl2)PnT W^1-W^2=P_m(C_l^1-C_l^2)P_n^T (9) Now ‖W1−W2‖F ||W^1-W^2||_F =‖Pm(Cl1−Cl2)PnT‖F =||P_m(C_l^1-C_l^2)P_n^T||_F ≤(a)‖Pm‖op⋅‖(Cl1−Cl2)PnT‖F (a)≤||P_m||_op·||(C_l^1-C_l^2)P_n^T||_F ≤(b)‖Pm‖op⋅‖Cl1−Cl2‖F⋅‖PnT‖op (b)≤||P_m||_op·||C_l^1-C_l^2||_F·||P_n^T||_op =(BmBn)⋅‖Cl1−Cl2‖F = (B_mB_n)·||C_l^1-C_l^2||_F (10) (a) and (b) follow from ‖AB‖F≤‖A‖op||B||F||AB||_F≤||A||_op||B||_F and ‖AB‖F≤‖A‖F||B||op||AB||_F≤||A||_F||B||_op respectively. Substituting this bound in (8), we get ‖∇f(Cl1)−∇f(Cl2)‖F ||∇ f(C_l^1)-∇ f(C_l^2)||_F ≤BmBnL(BmBn‖Cl1−Cl2‖F) ≤ B_mB_nL ( B_mB_n||C_l^1-C_l^2||_F ) =LBmBn‖Cl1−Cl2‖F =LB_mB_n||C_l^1-C_l^2||_F Hence, f(Cl)f(C_l) is L~ L-Lipschitz smooth with L~=LBmBn L=LB_mB_n. Appendix B Convergence Guarantee based on Lipschitz smoothness The gradient descent update at iteration t is: Clt+1=Clt−η∇f(Clt)C_l^t+1=C_l^t-η∇ f(C_l^t). Let L~ L be the Lipschitz constants derived in Section A. By L~ L-Lipschitz smoothness: f(Clt+1)≤f(Clt)+⟨∇f(Clt),Clt+1−Clt⟩+(L~/2)‖Clt+1−Clt‖F2f(C_l^t+1)≤ f(C_l^t)+ ∇ f(C_l^t),C_l^t+1-C_l^t +( L/2)||C_l^t+1-C_l^t||_F^2 Substituting Clt+1=Clt−η∇f(Clt)C_l^t+1=C_l^t-η∇ f(C_l^t) we get: f(Clt+1) f(C_l^t+1) ≤f(Clt)+⟨∇f(Clt),−η∇f(Clt)⟩+(L~/2)η2||∇f(Clt)||F2 ≤ f(C_l^t)+ ∇ f(C_l^t),-η∇ f(C_l^t) +( L/2)η^2||∇ f(C_l^t)||_F^2 (11) =f(Clt)−η‖∇f(Clt)‖F2+(L~/2)η2‖∇f(Clt)‖F2 =f(C_l^t)-η||∇ f(C_l^t)||_F^2+( L/2)η^2||∇ f(C_l^t)||_F^2 (12) Progress per step is f(Clt)−f(Clt+1)≥η‖∇f(Clt)‖F2(1−(L~η)/2)f(C_l^t)-f(C_l^t+1)≥η||∇ f(C_l^t)||_F^2(1-( Lη)/2) Summing from t=0t=0 to T−1T-1: ∑t=0T−1[f(Clt)−f(Clt+1)]≥η(1−(L~η)/2)∑t=0T−1‖∇f(Clt)‖F2 _t=0^T-1[f(C_l^t)-f(C_l^t+1)]≥η(1-( Lη)/2) _t=0^T-1||∇ f(C_l^t)||_F^2 The left side telescopes: f(Cl0)−f(ClT)≥η(1−(L~η)/2)∑t=0T−1‖∇f(Clt)‖F2f(C_l^0)-f(C_l^T)≥η(1-( Lη)/2) _t=0^T-1||∇ f(C_l^t)||_F^2 Since f(ClT)≥f∗f(C_l^T)≥ f^*, we have: f(Cl0)−f∗≥η(1−(L~η)/2)∑t=0T−1‖∇f(Clt)‖F2f(C_l^0)-f^*≥η(1-( Lη)/2) _t=0^T-1||∇ f(C_l^t)||_F^2 Dividing both sides by T and rearranging: (1/T)∑t=0T−1‖∇f(Clt)‖F2≤(f(Cl0)−f∗)/(η(1−(L~η)/2)T)(1/T) _t=0^T-1||∇ f(C_l^t)||_F^2≤(f(C_l^0)-f^*)/(η(1-( Lη)/2)T) For η≤1/L~η≤ 1/ L, we have 1−(L~η)/2≥1/21-( Lη)/2≥ 1/2, and so (1/T)∑t=0T−1‖∇f(Clt)‖F2≤2(f(Cl0)−f∗)/(ηT).(1/T) _t=0^T-1||∇ f(C_l^t)||_F^2≤ 2(f(C_l^0)-f^*)/(η T). So, this descent reaches an ϵε-first order stationary point. Appendix C Image Classification with Vision Transformers Evaluation Framework: We investigate the effectiveness of FrameFT for fine-tuning Vision Transformers. To this end, we fine-tune the base and large variants of the Vision Transformer (ViT) architecture (Dosovitskiy et al., 2021). We chose the ImageNet-21K pre-trained ViT models available on the Hugging Face Hub for our base model and evaluated across a diverse set of image classification challenges. The test suite includes fine-grained classification tasks – Oxford Pets (Parkhi et al., 2012), Stanford Cars, FGVC Aircraft (Maji et al., 2013), CIFAR10, CIFAR100, texture recognition – DTD (Cimpoi et al., 2014), and remote sensing applications – EuroSAT (Helber et al., 2018), RESISC45 (Cheng et al., 2017). Figure 5: Performance of FrameFT on CoLA and MRPC for different values of subspace dimension ρ FrameFT configuration: For this experiment, we utilized sparse block diagonal coefficient matrices with n=1000n=1000 and n=5000n=5000 non-zero elements, and a scaling factor α=250α=250 across all the tasks. We share the positions of non-zero positions across all the layers. Similar to the natural language understanding experiment, we set ρ=2ρ=2 while adjusting the number of subspaces k to satisfy kρ=nkρ=n throughout the network. Again, following LoRA, we adapt only the Query and Value matrices in the self-attention blocks in all layers of the model. Performance analysis: Table 6 shows the performance of FrameFT for fine-tuning Vision Transformers on diverse datasets. On average, with 50005000 non-zero coefficients per layer, FrameFT performs better than all the baseline methods except full fine-tuning. FrameFT is able to achieve this with the lowest number of parameters compared to all the baseline methods, which is 5−20×5-20× lower than LoRA. We also see an improvement in performance as we increase the number of non-zero coefficients from 10001000 to 50005000. As an additional note, the hyperparameters for FrameFT are held fixed for all the tasks in this benchmark. The hyperparameters for FourierFT were adjusted on a task-by-task basis, as noted by (Gao et al., 2024). Table 6: Performance of different methods for Fine-Tuning ViT Base and Large models on different datasets. * indicates the results reported in previous work. We highlight non PEFT methods in gray. FrameFT performs better than all the baseline methods on average, using a 30x lower number of parameters than LoRA. Method Params Pets Cars CIFAR10 DTD EuroSAT FGVC RESISC45 CIFAR100 Avg. ViT-B Full finetune* 85.885.8M 93.1493.14 79.7879.78 98.9298.92 77.6877.68 99.0599.05 54.8454.84 96.1396.13 92.3892.38 86.4986.49 LinearProbe* −- 90.2890.28 25.7625.76 96.4196.41 69.7769.77 88.7288.72 17.4417.44 74.2274.22 84.2884.28 68.3668.36 LoRA* 581581K 93.1993.19 45.3845.38 98.78 98.78 74.9574.95 98.4498.44 25.1625.16 92.7092.70 92.02 92.02 77.5877.58 FourierFT 7272K 93.2193.21 46.1146.11 98.5898.58 75.0975.09 98.2998.29 27.5127.51 91.9791.97 91.2091.20 77.7577.75 FourierFT 239239K 93.0593.05 56.3656.36 98.6998.69 77.3077.30 98.7898.78 32.4432.44 94.26 94.26 91.4591.45 80.2980.29 FrameFT (ours) 2424K 93.71 93.71 70.8470.84 98.5098.50 77.8077.80 98.5298.52 44.4544.45 92.3092.30 90.8090.80 83.3683.36 FrameFT (ours) 120120K 93.5593.55 78.16 78.16 98.5098.50 79.96 79.96 98.91 98.91 52.35 52.35 94.1094.10 90.9090.90 85.80 85.80 ViT-L Full finetune* 303.3303.3M 94.4394.43 88.9088.90 99.1599.15 81.7981.79 99.0499.04 68.2568.25 96.4396.43 93.5893.58 90.2090.20 LinearProbe* −- 91.1191.11 37.9137.91 97.7897.78 73.3373.33 92.6492.64 24.6224.62 82.0282.02 84.2884.28 72.9672.96 LoRA* 1.571.57M 94.82 94.82 73.2573.25 99.13 99.13 81.7981.79 98.6398.63 42.3242.32 94.7194.71 94.87 94.87 84.9484.94 FourierFT 144144K 94.4694.46 69.5669.56 99.1099.10 80.8380.83 98.6598.65 39.9239.92 93.8693.86 93.3193.31 83.7183.71 FourierFT 480480K 94.8494.84 79.1479.14 99.0899.08 81.88 81.88 98.6698.66 51.2851.28 95.20 95.20 93.3793.37 86.6886.68 FrameFT (ours) 4848K 94.2794.27 78.4878.48 99.0299.02 79.9479.94 98.4998.49 52.6352.63 93.7293.72 92.6892.68 86.1586.15 FrameFT (ours) 240240K 94.2094.20 82.83 82.83 98.9098.90 81.5481.54 98.87 98.87 59.63 59.63 94.9694.96 92.7192.71 87.95 87.95 Appendix D Performance versus subspace dimension In this experiment, we measure the performance of FrameFT for different values of subspace dimension, ρ. For this purpose, we fine-tune the RoBERTa-base model on two of the tasks in the GLUE benchmark - CoLA and MRPC, for different values of ρ. We keep the hyperparameters that we obtain for ρ=2ρ=2 as shown in Table 7. Figure 5 shows the performance of FrameFT as the subspace dimension is varied from 22 to 256256. We observe that FrameFT performs well even with a subspace dimension of 22. We do not see a strong relationship between the performance and the subspace dimension. Given that the two key choices we have for adjustments – the number of coefficients and the subspace dimension – to fill the space, the number of coefficients is a more consistent way of improving performance as indicated in Figure 6. An alternative and promising direction for future work is to make the subspace dimension a trainable parameter, potentially enabling more adaptive and efficient optimization. Appendix E Performance as a function of number of non-zero coefficients Figure 6 shows the performance of RoBERTa base fine-tuned with FrameFT on two of the tasks in the GLUE benchmark, in addition to the tasks shown in the main text. As noted earlier, we observe that FrameFT improves as the number of non-zero coefficients are increased, performing on par or better than full-finetuning. Figure 6: Performance of RoBERTa base model fine-tuned with FrameFT versus the number of non-zero coefficients on the GLUE benchmark. The red line indicates the performance of full fine-tuning. Appendix F Additional Discussions We cover a few relevant points not discussed in detail so far. (a) Comparison with Prompt Tuning. Prior literature has extensively analyzed the trade-offs between prompt tuning and Parameter-Efficient Fine-Tuning (PEFT) methods (Wistuba et al., 2024; Pu et al., 2023). Notably, (Pu et al., 2023) provides a quantitative comparison of prompt tuning against methods like LoRA and (IA)3(IA)^3, reporting that PEFT strategies generally yield superior performance. Since FrameFT demonstrates performance comparable to or exceeding that of LoRA, we infer that it maintains this advantage over prompt tuning. Hence, FrameFT serves as a more robust alternative to prompt tuning, delivering the reliability of PEFT methods while requiring a very low number of parameters. (b) Standard Frames versus Fusion Frames. A natural question is whether standard Frames (characterized by a subspace dimension of 11) could be employed instead of Fusion Frames. While feasible in principle, the specific construction algorithm we outline in §3.2 enforces a minimum subspace dimension of 22. This constraint arises because we first generate the frame in the Complex domain; when transforming these components to the Real domain, the dimension inherently doubles. Consequently, if an application strictly requires a subspace dimension of 11 (while maintaining kρ=nkρ=n), the complexity of fusion frames is unnecessary. In such scalar cases, one can simply rely on classical orthonormal bases, such as Fourier bases or Wavelets. (c) How does performance vary as a function of subspace dimension? We performed experiments changing the subspace dimension and report the results in §D. We observe that as the subspace dimension is increased (with kρ=nkρ=n), the sparsity in Fusion Frames increases, thereby reducing the degrees of freedom for the parameter update. This leads to a small drop in performance. Appendix G Hyperparameters used for the experiments We present the hyperparameters used for different experiments in Tables 7, 8, 9. As for the baselines, we use the hyperparameters suggested by the respective authors. Optimizer AdamW LR scheduler Linear schedule with warmup Batch size 32 Head learning rate 3E-3 Adapter learning rate 0.12 Warmup ratio 0.06 Max. Seq. length 512 α 20.0 Table 7: Hyperparameters for the GLUE benchmark Optimizer AdamW LR scheduler Linear schedule with warmup Batch size 50 Head learning rate 3E-2 Adapter learning rate 0.33 Warmup ratio 0.06 α 250.0 Table 8: Hyperparameters for the Image classification task Optimizer AdamW LR scheduler Linear schedule with warmup Batch size 128 Learning rate 0.1 Warmup ratio 0.03 α 200.0 Table 9: Hyperparameters for the Instruction tuning task Appendix H Fusion Frames Construction: an example In this section, we briefly outline an algorithm for generating Tight Fusion Frames with the help of an example. (Casazza et al., 2011) formalized a systematic framework for identifying the (k,ρ,d)(k,ρ,d) values for which a Tight Fusion Frame exists and generating the TFF whenever it exists. Their algorithm generates a TFF in ℂdC^d. Since we are mainly interested in real vector spaces, we use the simple extension in (Fickus et al., 2023a) to adapt the TFF to the real domain. The overall algorithm can be divided into three parts. 1. Construct a UNTF for ℂρC^ρ with d elements by playing Spectral Tetris. 2. Modulate these vectors with kthk^th roots of unity to form k subspaces of dimension ρ in ℂdC^d. 3. Use the method in (Fickus et al., 2023a) to extend the result to real-valued spaces. We describe these steps in detail by walking through the construction of a (6,3,11) TFF, a Tight Fusion Frame spanning ℂ11C^11 with k=6k=6 subspaces where each subspace has a dimension of ρ=3ρ=3. H.1 Spectral Tetris The first step is to generate a “smaller” frame and in the next step, we modulate the smaller frame to generate a “larger” Tight Fusion Frame. After generating a TFF for ℂdC^d we can easily extend it to the Real Field by applying the entrywise map x+iy↦[x−yyx]x+iy bmatrixx&-y\\ y&x bmatrix. So, k=6,ρ=3,d=11k=6,ρ=3,d=11. As the name suggests UNTFs are Tight frames where each frame vector has a unit norm. We construct a 4×114× 11 matrix F whose columns are the frame vectors for ℂ4C^4 which satisfies In this first step, we generate a “smaller” frame - a Unit Norm Tight Frame (UNTF) for ℂ3C^3 with 1111 vectors. We arrange these vectors in the columns of a matrix F. This UNTF is characterized by • Columns with unit norm • Rows are Orthogonal and have a constant norm, that is FF∗F^* is a constant multiple of the Identity matrix ( here the constant being 113 113) We start by filling the first two entries in F with 1 F=[11???????????????????????????????]F= bmatrix1&1&?&?&?&?&?&?&?&?&?\\ ?&?&?&?&?&?&?&?&?&?&?\\ ?&?&?&?&?&?&?&?&?&?&?\\ bmatrix The remaining norm left to be filled is 113−2=53 113-2= 53. We continue to fill in 1s until the required norm is less than 1. Here, we can do this only once, yielding F=[111??????????????????????????????]F= bmatrix1&1&1&?&?&?&?&?&?&?&?\\ ?&?&?&?&?&?&?&?&?&?&?\\ ?&?&?&?&?&?&?&?&?&?&?\\ bmatrix This leaves a norm of 23 23 to be filled. This can be added with a 2×22× 2 matrix T(x)T(x). T(x)T(x) here is defined as follows: T(x):=12[x2−x−2−x],T(x)T∗(x)=[x002−x]T(x) = 1 2 bmatrix x& x\\ 2-x&- 2-x bmatrix, T(x)T^*(x)= bmatrixx&0\\ 0&2-x bmatrix After substituting T(x)T(x) with x=23x= 23, F is now F=[111131300000000023−23??????0000???????]F= bmatrix1&1&1& 1 3& 1 3&0&0&0&0&0&0\\ 0&0&0& 2 3&- 2 3&?&?&?&?&?&?\\ 0&0&0&0&?&?&?&?&?&?&?\\ bmatrix Now, we continue adding ones in row two until the norm becomes less than 1 again. F=[111131300000000023−2311????0000???????]F= bmatrix1&1&1& 1 3& 1 3&0&0&0&0&0&0\\ 0&0&0& 2 3&- 2 3&1&1&?&?&?&?\\ 0&0&0&0&?&?&?&?&?&?&?\\ bmatrix Now we insert T(x)T(x) with the remaining norm. We repeat this process until all the rows are filled. The Final F is given by F=[111131300000000023−2311161600000000056−5611]F= bmatrix1&1&1& 1 3& 1 3&0&0&0&0&0&0\\ 0&0&0& 2 3&- 2 3&1&1& 1 6& 1 6&0&0\\ 0&0&0&0&0&0&0& 5 6&- 5 6&1&1\\ bmatrix H.2 Modulation In the second step of TFF construction, the F matrix is modulated with complex roots of unity, one subspace at a time. For each ki=0,1,2,…k−1k_i=0,1,2,… k-1, we construct a row vector wki=[(ei2πkik)0(ei2πkik)1(ei2πkik)2…(ei2πkik)d−1]w_k_i= [ (e i2π k_ik )^0 (e i2π k_ik )^1 (e i2π k_ik )^2… (e i2π k_ik )^d-1 ] Each row of F is multiplied by wkiw_k_i to produce the orthogonal basis for the subspace indexed by kik_i. Theorem 1414 by Casazza et al. (2011) proves that the Fusion Frames generated by this algorithm are Tight. The Final Fusion Frame generated is shown in Table 10. [11113130000001ω213ω313ω40000001ω2ω41313ω20000001ω3113ω3130000001ω4ω21313ω40000001ω5ω413ω313ω200000000023−231116160000023ω3−23ω4ω5116ω16ω20000023−23ω2ω4116ω216ω40000023ω3−23ω3116ω3160000023−23ω4ω2116ω416ω20000023ω3−23ω2ω1116ω516ω400000000056−5611000000056ω−56ω2ω3ω4000000056ω2−56ω41ω2000000056ω3−56ω31000000056ω4−56ω21ω4000000056ω5−56ω4ω3ω2] bmatrix[l]1&1&1& 1 3& 1 3&0&0&0&0&0&0\\ 1&ω&ω^2& 1 3ω^3& 1 3ω^4&0&0&0&0&0&0\\ 1&ω^2&ω^4& 1 3& 1 3ω^2&0&0&0&0&0&0\\ 1&ω^3&1& 1 3ω^3& 1 3&0&0&0&0&0&0\\ 1&ω^4&ω^2& 1 3& 1 3ω^4&0&0&0&0&0&0\\ 1&ω^5&ω^4& 1 3ω^3& 1 3ω^2&0&0&0&0&0&0\\ 0&0&0& 2 3&- 2 3&1&1& 1 6& 1 6&0&0\\ 0&0&0& 2 3ω^3&- 2 3ω^4&ω^5&1& 1 6ω& 1 6ω^2&0&0\\ 0&0&0& 2 3&- 2 3ω^2&ω^4&1& 1 6ω^2& 1 6ω^4&0&0\\ 0&0&0& 2 3ω^3&- 2 3&ω^3&1& 1 6ω^3& 1 6&0&0\\ 0&0&0& 2 3&- 2 3ω^4&ω^2&1& 1 6ω^4& 1 6ω^2&0&0\\ 0&0&0& 2 3ω^3&- 2 3ω^2&ω^1&1& 1 6ω^5& 1 6ω^4&0&0\\ 0&0&0&0&0&0&0& 5 6&- 5 6&1&1\\ 0&0&0&0&0&0&0& 5 6ω&- 5 6ω^2&ω^3&ω^4\\ 0&0&0&0&0&0&0& 5 6ω^2&- 5 6ω^4&1&ω^2\\ 0&0&0&0&0&0&0& 5 6ω^3&- 5 6&ω^3&1\\ 0&0&0&0&0&0&0& 5 6ω^4&- 5 6ω^2&1&ω^4\\ 0&0&0&0&0&0&0& 5 6ω^5&- 5 6ω^4&ω^3&ω^2\\ bmatrix Table 10: (,,) (6,3,11)-TFF for ℂ11C^11. Here, ω=eiπ/3ω=e^iπ/3. A pair of rows belongs to the same subspace if their indices differ by a multiple of 6 H.3 Fusion Frames construction time Table 11, presents the time taken to construct the Fusion Frames using the Spectral Tetris method. As a reference, we provide the time taken to construct a random orthonormal basis using two methods, namely the Gram-Schmidt algorithm and QR decomposition on a random matrix. These times are measured on an NVIDIA A100 machine. Since the Gram-Schmidt process is sequential, it cannot be parallelized on GPUs/TPUs. Hence, the time taken to perform this process is quite high. We can see that Spectral Tetris takes a fraction of the time to construct than QR decomposition on modern GPUs. Though there is some latency in generating Fusion Frames, it is a minor issue, as it is amortized across multiple model calls. Table 11: Basis generation time. Spectral Tetris takes a fraction of time to construct the Basis on an NVDIA A100 gpu Model Method Bases construction time (s) Llama-2-7B Gram-Schmidt 112.1 QR decomposition 0.31 Spectral Tetris 0.09 Gemma-2-9B Gram-Schmidt 228.8 QR decomposition 0.64 Spectral Tetris 0.21 Llama-3.1-8B Gram-Schmidt 119.6 QR decomposition 0.53 Spectral Tetris 0.11 Appendix I Scaling Factor Sensitivity Table 12 reports performance of Llama-2-7B instruction-tuned on the Alpaca dataset with FrameFT as the scaling factor α is varied from 10 to 600. FrameFT outperforms LoRA’s average of 63.18 across the entire sweep. For α≥100α≥ 100, it consistently matches or exceeds the full fine-tuning upper bound of 63.39. The results demonstrate a major practical advantage: FrameFT is highly insensitive to this factor. Unlike PEFT methods requiring exhaustive searches to prevent instability, FrameFT operates reliably out-of-the-box. Table 12: Scaling factor α sensitivity on Llama-2-7B/Alpaca. FrameFT outperforms LoRA (63.18) across the full sweep. α ARC-c ARC-e BoolQ HellaSwag OBQA PIQA RTE WinoGrande Avg. 10 45.9 77.4 78.75 58.01 34.0 78.67 61.01 70.0 62.97 100 45.47 77.1 78.53 58.28 34.4 78.78 63.89 70.71 63.40 200 (default) 45.22 76.93 78.62 58.08 34.2 78.62 66.06 71.19 63.62 400 45.47 77.31 79.2 58.32 34.0 78.62 62.45 70.32 63.21 600 45.56 77.02 78.99 58.31 34.6 78.73 62.82 70.56 63.32 LoRA 45.82 77.02 78.81 58.08 35.2 78.83 61.01 70.72 63.18 Full FT 47.52 77.73 78.96 58.99 33.6 78.61 62.09 69.61 63.39