Paper deep dive
Recovering the Pre-Fine-Tuning Weights of Generative Models
Eliahu Horwitz, Jonathan Kahana, Yedid Hoshen
Models: Mistral-7B, Stable Diffusion 1.5, ViT
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 6:30:56 PM
Summary
The paper introduces 'Spectral DeTuning', a novel, unsupervised, and data-free method to recover the original pre-fine-tuning weights of generative models (such as Stable Diffusion and Mistral) that have been fine-tuned using Low-Rank Adaptation (LoRA). By leveraging multiple LoRA-fine-tuned versions of the same source model, the method uses iterative low-rank matrix factorization to reverse the fine-tuning process, posing a significant security and safety vulnerability.
Entities (5)
Relation Signals (3)
Spectral DeTuning â recovers â Pre-Fine-Tuning Weights
confidence 100% ¡ Spectral DeTuning, a method that can recover the weights of the pre-fine-tuning model
LoWRA Bench â evaluates â Pre-Fine-Tuning Weight Recovery
confidence 95% ¡ LoWRA Bench, a comprehensive suite of datasets and metrics designed for the evaluation of pre-fine-tuning weight recovery methods.
Spectral DeTuning â exploits â LoRA
confidence 95% ¡ Our approach exploits this new vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The dominant paradigm in generative modeling consists of two steps: i) pre-training on a large-scale but unsafe dataset, ii) aligning the pre-trained model with human values via fine-tuning. This practice is considered safe, as no current method can recover the unsafe, pre-fine-tuning model weights. In this paper, we demonstrate that this assumption is often false. Concretely, we present Spectral DeTuning, a method that can recover the weights of the pre-fine-tuning model using a few low-rank (LoRA) fine-tuned models. In contrast to previous attacks that attempt to recover pre-fine-tuning capabilities, our method aims to recover the exact pre-fine-tuning weights. Our approach exploits this new vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral.
Tags
Links
- Source: https://arxiv.org/abs/2402.10208
- Canonical: https://arxiv.org/abs/2402.10208
Trouble viewing inline? Open PDF directly â
Full Text
81,822 characters extracted from source content.
Expand or collapse full text
Recovering the Pre-Fine-Tuning Weights of Generative Models Eliahu Horwitz Jonathan Kahana Yedid Hoshen School of Computer Science and Engineering The Hebrew University of Jerusalem, Israel https://vision.huji.ac.il/spectral_detuning/ eliahu.horwitz, jonathan.kahana, yedid.hoshen@mail.huji.ac.il Abstract The dominant paradigm in generative modeling consists of two steps: i) pre-training on a large-scale but unsafe dataset, i) aligning the pre-trained model with human values via fine-tuning. This practice is considered safe, as no current method can recover the unsafe, pre-fine-tuning model weights. In this paper, we demonstrate that this assumption is often false. Concretely, we present Spectral DeTuning, a method that can recover the weights of the pre-fine-tuning model using a few low-rank (LoRA) fine-tuned models. In contrast to previous attacks that attempt to recover pre-fine-tuning capabilities, our method aims to recover the exact pre-fine-tuning weights. Our approach exploits this new vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral. 1 Introduction A key paradigm in deep learning is to first pre-train a foundation model [48, 37] on a large, general-purpose dataset and then fine-tune the model for a specific task. Fine-tuning is used for critical applications including model safety [32], alignment to human preferences and values [31, 8, 34], providing privacy guarantees [58], personalization [38], and more [4, 61]. In this paper, we identify a vulnerability in fine-tuned models, wherein the pre-fine-tuning (Pre-FT) weights, i.e., the model weights before the fine-tuning stage, can be recovered using a small number of models fine-tuned via low-rank adaptation (LoRA) [20]. To illustrate our setting, let us consider a Large Language Model (LLM). While the pre-trained version of the LLM exhibits advanced language understanding and generation capabilities, it is unaligned with human preference and is often deemed unsafe [31, 48]. These unsafe models can be used for example to get instructions for building a bomb or other malicious activities. To improve instruction following and enhance safety, model creators perform an alignment fine-tuning stage. Usually, only the aligned version of the LLM is published, and the recovery of the original Pre-FT unsafe weights, is implicitly assumed to be impossible. While for existing models the recovery of the Pre-FT weights poses a security and safety vulnerability; for future superhuman models, it may lead to catastrophic consequences. Figure 1: Pre-Fine-Tuning Weight Recovery Attack Setting: We uncover a vulnerability in LoRA fine-tuned models wherein an attacker is able to undo the fine-tuning process and recover the weights of the original pre-trained model. The setting for the vulnerability is as follows: (a) The attacker only has access to n different LoRA fine-tuned models. (b) The attacker assumes that all n models originated from the same source model. Note: The attacker has no access to the low-rank decomposition of the fine-tuned models. (c) Using only the n visible models, the attacker attempts to recover the original source model. Our method, Spectral DeTuning, can perform the attack in an unsupervised and data-free manner on real models such as Stable Diffusion and Mistral. For simplicity, we illustrate the attack on a single layer, in reality, the attack is carried out independently on all the fine-tuned layers. Best viewed in color Motivated by the above, we propose the task of Pre-Fine-Tuning Weight Recovery. In this paper, we tackle this task in cases where multiple LoRA fine-tuned flavors of the same source model are available. We present an overview of our setting in Fig. 1. This task is particularly timely due to two trends: i) Popular foundation models come in multiple flavors. E.g., LLaMA 2, Code LLaMA, Code LLaMA-Python, Code LLaMA-Instruct. i) LoRA is becoming a key component for creating SoTA models [27, 46]. These two trends have not yet merged, i.e, we are not aware of multi-flavored foundational models that use LoRA alignment fine-tuning. Here, we bring to the attention of the community the risks and perils involved in merging these trends. We present Spectral DeTuning, a method that recovers the Pre-FT weights with remarkably high precision using iterative low-rank matrix factorization. To enhance optimization stability and accelerate convergence, we introduce a rank scheduler that progressively increases the rank of the factorized matrices during optimization. A key distinction from prior attacks on model alignment [5, 53, 64] is that Spectral DeTuning prioritizes restoring the exact Pre-FT weights over Pre-FT functionalities. It also does not require running inference through the model. This is advantageous as i) it does not require training data i) it is highly parallelizable, e.g., on a cluster of desktop GPUs such as RTX2080 our method can recover the Pre-FT weights of a Mistral-7B model in under five minutes. We demonstrate the effectiveness of our method by uncovering the vulnerability of real and widely used NLP and Vision models. Our approach achieves remarkable precision on an aligned Mistral model, effectively reversing the alignment training and restoring the original model (See Fig. 2). Similarly, on Stable-Diffusion, we recover the original modelâs weights with a vanishingly small error, showcasing almost perfect reconstruction of the original generation capabilities (See Fig. 3). This work aims to stimulate research into preventing Pre-FT weight leakage and the associated risks in terms of model safety and alignment. To facilitate this research, we introduce LoWRA Bench, a comprehensive benchmark comprising datasets and evaluation metrics, designed for assessing Pre-FT weight recovery methods. To summarize, our main contributions are: 1. Introducing the task of Pre-Fine-Tuning Weight Recovery, a new attack vector against fine-tuned models. 2. Presenting Spectral DeTuning, a highly effective method for pre-fine-tuning weight recovery attacks against state-of-the-art models. 3. Providing LoWRA Bench, a comprehensive suite of datasets and metrics designed for the evaluation of pre-fine-tuning weight recovery methods. Figure 2: Mistral DPO Results: Our method, Spectral DeTuning, recovers the pre-fine-tuning generation capabilities with high precision, essentially undoing the DPO alignment LoRA fine-tuning. In green exact recovery, in red unrecovered words. Best viewed in color 2 Related Works 2.1 Model Fine-tuning Model fine-tuning, crucial in deep learning research [61, 60, 2], can be resource-intensive. Parameter-Efficient Fine-tuning (PEFT) methods [20, 10, 19, 26, 25, 29, 18, 28, 23, 62, 52, 22] aim to economize and broaden access to fine-tuning. These methods approximate full fine-tuning with fewer parameters. Some recent works combine multiple PEFT models [56, 17, 44, 33, 21], hoping to leverage the strengths of individual models. LoRA [20] is perhaps the most popular PEFT method and is known for its effectiveness across various tasks and modalities [51, 57, 39, 1], sometimes even outperforming full fine-tuning. Given its popularity, in this paper, we focus on recovering Pre-FT weights of LoRA fine-tuned models. 2.2 Model Safety and Security Deep learning models have various safety and security vulnerabilities. Membership inference attacks aim to detect if specific data samples were used in training [45, 42]. Model inversion attempts to generate the samples used during training [15, 14]. Machine unlearning protects against attacks by removing the effect of specific training samples without retraining the entire model [3]. Model extraction, or model stealing, involves stealing a target model hidden behind an API by querying it multiple times [49, 43]. In contrast, Pre-FT weight recovery aims to recover the exact weights of the pre-trained model, compromising the entire model rather than just a subset of capabilities. Additionally, our method, Spectral DeTuning, operates in an unsupervised and data-free manner. 2.3 Model Red-Teaming and Adversarial Attacks One of the primary methods for ensuring model safety involves incorporating human feedback through a reward model trained on annotator preferences, followed by reinforcement learning to fine-tune the model [34, 8, 32, 16, 41, 47]. However, Wolf et al. [54] argue that these alignment processes may leave undesired behavior partially intact and are thus vulnerable to adversarial prompting attacks. This has been demonstrated by red teaming [32, 16] and adversarial attacks [5, 53, 64] approaches. Unlike targeted attacks, Pre-FT weight recovery compromises the entire model by restoring the pre-trained weights. Moreover, our method, Spectral DeTuning, does not require running inference through the model. 3 Preliminaries - LoRA Fine-tuning deep networks traditionally consisted of training all the network weights initialized by a pre-trained model. As this is costly for large-scale models, Hu et al. [20] recently introduced Low Rank Adaptation (LoRA). The authors postulate that the change in weights during fine-tuning often has a âlow intrinsic rankâ. They therefore introduced LoRA, which transforms each parameter matrix by the addition of a low-rank matrix. To create this low-rank matrix they multiply two full-rank matrices with suitable dimensions. This reparametrization drastically reduces the number of parameters being optimized. Specifically, for a pre-trained weight matrix WââdĂksubscriptsuperscriptâW_P ^dĂ kWcaligraphic_P â blackboard_Rd Ă k, the update Îâ˘WÎ WÎ W can be decomposed into a rank r decomposition Îâ˘W=Bâ˘AÎ W=BAÎ W = B A where BââdĂr,AâârĂkformulae-sequencesuperscriptâsuperscriptâB ^dĂ r,A ^rĂ kB â blackboard_Rd Ă r , A â blackboard_Rr Ă k and râŞmâ˘iâ˘nâ˘(d,k)much-less-thanr min(d,k)r ⪠m i n ( d , k ). During fine-tuning, WsubscriptW_PWcaligraphic_P is frozen and only A and B are fine-tuned. This results in the following forward pass Wâ˘x+Îâ˘Wâ˘x=Wâ˘x+Bâ˘Aâ˘xsubscriptÎsubscriptW_Px+ Wx=W_Px+BAxWcaligraphic_P x + Î W x = Wcaligraphic_P x + B A x, where x is the outcome of the previous layer. Since LoRA is linear by design, it is possible to merge the fine-tuned matrices back into the original matrix Wâ˛=W+Bâ˘Asuperscriptâ˛subscriptW =W_P+BAWⲠ= Wcaligraphic_P + B A (1) , thus introducing no additional parameters or inference latency to the original model. Originally, LoRA was applied to the query and value layers of attention blocks; however, it has been demonstrated that LoRA can be effectively extended to additional layers. Once merged, current models implicitly assume that recovering WsubscriptW_PWcaligraphic_P and Bâ˘ABAB A from WⲠis impossible. Throughout the paper, whenever we refer to the weights of a LoRA fine-tuned model, we assume the weights have been merged back as seen in Eq. 1. 4 Problem Definition We introduce the task of Pre-Fine-Tuning Weight Recovery. Its goal is to recover the Pre-FT weights of a given model, i.e., the weights of the original, pre-trained model. Specifically, in this work we assume that the fine-tuning was performed using LoRA. Notation. Formally, consider a model âąsubscriptâąF_PFcaligraphic_P with m fine-tuned layers that were fine-tuned via a rank r LoRA and originated from the source model PP. We denote the weight matrices of âąsubscriptâąF_PFcaligraphic_P by Wâ˛âŁ(j)j=1msuperscriptsubscriptsuperscriptâ˛1\W (j)\_j=1^m WⲠ( j ) j = 1m and those of PP by W(j)j=1msuperscriptsubscriptsubscriptsuperscript1\W^(j)_P\_j=1^m W( j )P j = 1m where both Wâ˛âŁ(j)superscriptâ˛W (j)WⲠ( j ) and W(j)subscriptsuperscriptW^(j)_PW( j )P are ââdĂkabsentsuperscriptâ ^dĂ kâ blackboard_Rd Ă k. Throughout the paper we assume the attacker does not have access to PP (nor to its weights W(j)j=1msuperscriptsubscriptsubscriptsuperscript1\W^(j)_P\_j=1^m W( j )P j = 1m). Attack setting. The attacker has access to the weights of n different âąsubscriptâąF_PFcaligraphic_P models, all LoRA fine-tuned from the same pre-trained source model PP. The attack succeeds with precision Ďľitalic-ϾξϾ if the attacker can accurately recover the weights of the pre-trained source model PP up to an Ďľitalic-ϾξϾ precision. Formally, given Wiâ˛âŁ(j)j=1mi=1nsuperscriptsubscriptsuperscriptsubscriptsuperscriptsubscriptâ˛11 \\W_i (j)\_j=1^m \_i=1^n Witalic_iⲠ( j ) j = 1m i = 1n, the attacker needs find Wâ(j)j=1msuperscriptsubscriptsuperscriptabsent1\W^*(j)\_j=1^m Wâ ( j ) j = 1m such that âj=1mâW(j)âWâ(j)â<Ďľsuperscriptsubscript1normsubscriptsuperscriptsuperscriptabsentitalic-Ďľ _j=1^m \|W^(j)_P-W^*(j) \|<Îľâj = 1m ⼠W( j )P - Wâ ( j ) ⼠< Ďľ (2) We present an overview of this setting in Fig. 1. Success criteria. We measure the success of the attack by the distance between the recovered weights and the original weights, in addition, in Sec. 6 we discuss a number of ways to measure the success of the attack semantically. 5 Spectral DeTuning We now describe our method for carrying out a Pre-FT weight recovery attack. We start by introducing our optimization objective, followed by our optimization method and finally, a rank scheduler that stabilizes the optimization and results in better convergence. For simplicity, assume for now that all n LoRA fine-tuned models used the same rank r, and that the value of r is known to the attacker, in Secs. 5.3 and 5.4 we relax these assumptions. For brevity, we omit the layer index superscript (j)(j)( j ) and perform the same optimization across all layers independently. 5.1 Optimization Objective To recover the Pre-FT weights, we need to predict WsubscriptW_PWcaligraphic_P given n fine-tuned weight matrices Wiâ˛i=1nsuperscriptsubscriptsuperscriptsubscriptâ˛1 \W_i \_i=1^n Witalic_iⲠi = 1n. Leveraging their difference of up to r principal components, we formulate the task as an optimization problem, where each LoRA provides additional constraints on WsubscriptW_PWcaligraphic_P. Specifically, recall that according to Eq. 1, Wiâ˛subscriptâ˛W_i Witalic_iⲠcan be decomposed into WââdĂksuperscriptâW ^dĂ kW â blackboard_Rd Ă k and a rank r matrix which we will denote by MiââdĂksubscriptsuperscriptâM_i ^dĂ kMitalic_i â blackboard_Rd Ă k. Taking into account all n different LoRA weights, we define the following objective aâ˘râ˘gâ˘minW,Mi1â¤iâ¤nâ˘âi=1nâWiâ˛â(W+Mi)â22â˘s.t.râ˘aâ˘nâ˘kâ˘Miâ¤rformulae-sequencesubscriptsubscript1superscriptsubscript1subscriptsuperscriptnormsuperscriptsubscriptâ˛subscript22subscriptarg _ subarraycW,M_i\\ 1⤠i⤠n subarray _i=1^n \|W_i -(W+M_i)% \|^2_2~~~s.t.~~~rank~M_i⤠ra r g minstart_ARG start_ROW start_CELL W , M start_POSTSUBSCRIPT i end_CELL end_ROW start_ROW start_CELL 1 ⤠i ⤠n end_CELL end_ROW end_ARG end_POSTSUBSCRIPT âi = 1n ⼠Witalic_iⲠ- ( W + Mitalic_i ) âĽ22 s . t . r a n k Mitalic_i ⤠r (3) Where WââdĂksuperscriptâW ^dĂ kW â blackboard_Rd Ă k is the matrix we are optimizing to estimate WsubscriptW_PWcaligraphic_P. Intuitively, the objective optimizes the decomposition of each fine-tuned weight matrix into a shared weight matrix which is the approximated Pre-FT matrix and an independent low rank residual matrix. This objective exhibits desirable properties for an attacker. First, it is training-free, meaning, it requires no data, nor does it make any assumptions with regards to the data used to train the model. Moreover, the optimization is performed on a per-layer basis, enabling high parallelization of the attack. Finally, the objective is unsupervised, allowing an attacker to recover a model even when they have no prior knowledge regarding the source model. Figure 3: Stable Diffusion Results: Spectral DeTuning recovers the Pre-Fine-Tuning images with high precision, even when using âin the wildâ LoRAs, essentially reversing the personalization fine-tuning of the LoRA model 5.2 Pre-FT Weight Recovery Algorithm We propose Spectral DeTuning, an iterative, gradient-free algorithm for Pre-FT weight recovery. The method is fast (even on CPU) and is easily parallelizable. The core idea is that while the optimization problem in Eq. 3 is non-convex, it can be iteratively broken down into a set of simple sub-problems which have closed-form solutions. Our procedure has three major components: initialization, M-step and W-step. Note, solving Eq. 3 requires optimizing n+11n+1n + 1 matrices, i.e., W and M1,M2,âŚ,Mnsubscript1subscript2âŚsubscriptM_1,M_2,...,M_nM1 , M2 , ⌠, Mitalic_n. Initialization. At iteration 00, we set WâsuperscriptW^*Wâ as the average of all the fine-tuned matrices, i.e., Wâ=1nâ˘âi=1nWiâ˛1superscriptsubscript1superscriptsubscriptâ˛W^*= 1n _i=1^nW_i Wâ = divide start_ARG 1 end_ARG start_ARG n end_ARG âi = 1n Witalic_iâ˛. M-step. We solve the optimization problem by coordinate descent [55]. We first fix WâsuperscriptW^*Wâ and solve for Mii=1nsuperscriptsubscriptsubscript1\M_i\_i=1^n Mitalic_i i = 1n. Note that when WâsuperscriptW^*Wâ is given, the optimization problems for each M1,..,MnM_1,..,M_nM1 , . . , Mitalic_n are decoupled. Specifically, at each iteration, the optimization problem for MisubscriptM_iMitalic_i is: Miâ=aâ˘râ˘gâ˘minMiâĄâ(Wiâ˛âWâ)âMiâ22â˘s.t.râ˘aâ˘nâ˘kâ˘Miâ¤rformulae-sequencesuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptnormsuperscriptsubscriptâ˛subscript22subscriptM_i^*=arg _M_i\|(W_i -W^*)-M_i\|_2^2~~~s.t.% ~~~rank~M_i⤠rMitalic_iâ = a r g minitalic_M start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ⼠( Witalic_iⲠ- Wâ ) - Mitalic_i âĽ22 s . t . r a n k Mitalic_i ⤠r (4) Luckily, the solution to this optimization problem is available in closed-form and is given by the âSingular Value Decompositionâ (SVD) of Wiâ˛âWâsuperscriptsubscriptâ˛W_i -W^*Witalic_iⲠ- Wâ. The optimal value of MisubscriptM_iMitalic_i is: Ui,ÎŁi,ViT=SVDâ˘(Wiâ˛âWâ)subscriptsubscriptÎŁsubscriptsuperscriptSVDsuperscriptsubscriptⲠU_i, _i,V^T_i=SVD(W_i -W^*)Uitalic_i , ÎŁitalic_i , Vitalic_Titalic_i = SVD ( Witalic_iⲠ- Wâ ) (5) Miâ=Uiâ˘ÎŁi|râ˘ViTsubscriptsuperscriptsubscriptsubscriptÎŁconditionalsubscriptsuperscript M^*_i=U_i _i|rV^T_iMâitalic_i = Uitalic_i ÎŁitalic_i | r Vitalic_Titalic_i Where ÎŁi|rsubscriptÎŁconditional _i|rÎŁitalic_i | r represents the top r singular values of ÎŁisubscriptÎŁ _iÎŁitalic_i. W-step. By fixing the values of M1â,..,MnâM^*_1,..,M^*_nMâ1 , . . , Mâitalic_n, we can easily compute the optimal value of W. The optimization problem is given by: Wâ=aâ˘râ˘gâ˘minWâ˘âi=1nâ(Wiâ˛âMiâ)âWâ22superscriptsubscriptsuperscriptsubscript1superscriptsubscriptnormsuperscriptsubscriptâ˛subscriptsuperscript22W^*=arg _W _i=1^n\|(W_i -M^*_i)-W\|_2^2Wâ = a r g minitalic_W âi = 1n ⼠( Witalic_iⲠ- Mâitalic_i ) - W âĽ22 (6) By simple calculus, the closed-form solution is: Wâ=1nâ˘âi=1n(Wiâ˛âMiâ)superscript1superscriptsubscript1superscriptsubscriptâ˛subscriptsuperscriptW^*= 1n _i=1^n (W_i -M^*_i )Wâ = divide start_ARG 1 end_ARG start_ARG n end_ARG âi = 1n ( Witalic_iⲠ- Mâitalic_i ) (7) We iterate between the M-step and W-step until convergence. As shown in Alg. 1, the algorithm can be easily implemented in as little as 8888 lines of python. 5.3 Rank Scheduler The algorithm proposed in Sec. 5.2 tends to perform well in general. However, we empirically found that solving the optimization problem with high ranks can result in slow and inaccurate convergence. We therefore introduce a rank scheduler. The idea of the rank scheduler is to start by forcing MisubscriptM_iMitalic_i to be of rank râ<rsuperscriptr^*<râ < r, allowing Spectral DeTuning to focus on the most significant principal components first. râsuperscriptr^*râ is increased according to a schedule until finally râ=rsuperscriptr^*=râ = r. Specifically, we use an âIncrease on Plateauâ type of scheduler where the rank is increased whenever the loss term from Eq. 3 plateaus. When not all LoRAs have the same rank, we assign a distinct rank scheduler to each LoRA. The rank scheduler requires knowing the LoRA rank; we show how to estimate it in Sec. 5.4. For more details see App. F. We show empirically in Sec. 8 that there are cases where the rank scheduler improves the rate and quality of convergence significantly. Algorithm 1 PyTorch Pseudocode for Spectral DeTuning ⏠# W_ps: List of n fine-tuned weight matrices # steps: Number of optimization steps # r: LoRA rank # Initialize W_star W_s = torch.mean(torch.stack(W_ps), axis=0) # Perform optimization for step in range(steps): # M-step # Approximate each M^*_i (Eq. 5) M_s = [W_p - W_s for W_p in W_ps] # Truncate each M^*_i to rank <= r (Eq. 5) for i in range(len(M_s)): (U, S, V) = torch.svd_lowrank(M_s[i], q=r) M_s[i] = (U @ torch.diag_embed(S)) @ V.T # W-step # Approximate W_star (Eq. 7) W_s = [W_p - M_si for (W_p, M_si) in zip(W_ps, M_s)] W_s = torch.mean(torch.stack(W_s), axis=0) 5.4 LoRA Rank Estimation We propose an effective heuristic for estimating LoRA rank. Assume we have two LoRA fine-tuned models Wiâ˛=W+Misubscriptsuperscriptâ˛subscriptW _i=W+M_iWâ˛italic_i = W + Mitalic_i and Wjâ˛=W+Mjsubscriptsuperscriptâ˛subscriptW _j=W+M_jWâ˛italic_j = W + Mitalic_j, where the ranks of Mi,MjsubscriptsubscriptM_i,M_jMitalic_i , Mitalic_j are ri,rjsubscriptsubscriptr_i,r_jritalic_i , ritalic_j respectively. While it is not trivial to recover the rank of MisubscriptM_iMitalic_i solely by observing Wiâ˛subscriptsuperscriptâ˛W _iWâ˛italic_i, there is a trick. Subtracting the two fine-tuned models obtains Wiâ˛âWjâ˛=MiâMjsubscriptsuperscriptâ˛subscriptsuperscriptâ˛subscriptsubscriptW _i-W _j=M_i-M_jWâ˛italic_i - Wâ˛italic_j = Mitalic_i - Mitalic_j. Importantly, the rank Wiâ˛âWjâ˛subscriptsuperscriptâ˛subscriptsuperscriptâ˛W _i-W _jWâ˛italic_i - Wâ˛italic_j is upper bounded by ri+rjsubscriptsubscriptr_i+r_jritalic_i + ritalic_j, i.e., râ˘aâ˘nâ˘kâ˘(Wiâ˛âWjâ˛)â¤ri+rjsuperscriptsubscriptâ˛subscriptâ˛subscriptsubscriptrank(W_i -W_j )⤠r_i+r_jr a n k ( Witalic_iⲠ- Witalic_jⲠ) ⤠ritalic_i + ritalic_j. Given n LoRAs, there are nâ˘(nâ1)212 n(n-1)2divide start_ARG n ( n - 1 ) end_ARG start_ARG 2 end_ARG distinct inequalities for the n unknown ranks r1,r2,..,rnr_1,r_2,..,r_nr1 , r2 , . . , ritalic_n. We can formulate this as a linear programming problem as follows: minimizeTâ˘subject toâĽriâĽ1,âiminimizesuperscript1subject tomissing-subexpressionsubscript1for-all array[]l rminimize&1^T% r\\ subject to&Ar \\ &r_i⼠1, â i arraystart_ARRAY start_ROW start_CELL underr start_ARG minimize end_ARG end_CELL start_CELL 1italic_T r end_CELL end_ROW start_ROW start_CELL subject to end_CELL start_CELL Ar ⼠b end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ritalic_i ⼠1 , â i end_CELL end_ROW end_ARRAY where: ⢠â0,1n2,nsuperscript01superscript2Aâ\0,1\^n^2,nA â 0 , 1 n start_POSTSUPERSCRIPT 2 , n end_POSTSUPERSCRIPT so that Anâ˘i+j,i=1subscript1A_ni+j,i=1Aitalic_n i + j , i = 1 and Anâ˘i+j,j=1subscript1A_ni+j,j=1Aitalic_n i + j , j = 1 and 00 elsewhere. ⢠âRn2superscriptsuperscript2bâ R^n^2b â Ritalic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT so that bnâ˘i+j=râ˘aâ˘nâ˘kâ˘(Wiâ˛âWjâ˛)subscriptsubscriptsuperscriptâ˛subscriptsuperscriptâ˛b_ni+j=rank(W _i-W _j)bitalic_n i + j = r a n k ( Wâ˛italic_i - Wâ˛italic_j ). Figure 4: Motivation for the Log in W-Error: We visualize the convergence of all layers using Spectral DeTuning and the Mean LoRAs baselines. Spectral DeTuning clearly converges to a much better solution for almost all layers. Note that MSE does not summarize the convergence well as it yields the value of the poorly converging outlier layers. The W-Error better conveys the actual convergence by working in log-space. Results for a random subset of 5555 Stable Diffusion LoRAs In practice, we populate b using a numerical rank computed via the multiplicative gap following a similar protocol to [6]. Using an off-the-shelf linear programming solver accurately retrieves the ranks. We demonstrate the accuracy of this method in Sec. 8, the unknown ranks were recovered perfectly in all cases. 6 LoWRA Bench We present LoRA Weight Recovery Attack (LoWRA) Bench, a comprehensive benchmark designed to evaluate Pre-FT weight recovery methods. 6.1 Dataset Our dataset encompasses three pre-trained representative source models: a Vision Transformer (ViT) [12] trained on ImageNet-1K [40], Mistral-7B-v0.1 [24], and Stable Diffusion 1.5 [36]. These models collectively cover supervised and self-supervised objectives, spanning both vision and natural language processing (NLP) domains, as well as generative and discriminative tasks. Notably, these models are widely used and deployed in numerous production systems. See Tab. 1 for an overview of the dataset. For each source model, we curate 15151515 LoRA models fine-tuned on diverse datasets, tasks, and objectives. The dataset comprises a diverse array of layer types, including self-attention, cross-attention, and MLPs. This diversity enables us to assess the generalization capabilities of Pre-FT methods. The evaluation can be conducted on a per-model basis, per layer type, or per layer depth, allowing for a comprehensive analysis of Pre-FT methods. Overall, our dataset includes 544544544544 source model layers. When taking into account the fine-tuned LoRA layers, the dataset includes over 8,00080008,0008 , 000 layers. For further details see App. E. 6.2 Numeric Evaluation Metrics Weight Error (W-Error). We measure numeric convergence by the mean squared weight error (as defined in Eq. 2) and average across all layers in log space: 1mâj=1m(log10(MSE(W(j)âWâ(j))) 1m _j=1^m ( _10 (MSE(W^(j)_P-W^*(j% ) ) )divide start_ARG 1 end_ARG start_ARG m end_ARG âj = 1m ( log10 ( M S E ( W( j )P - Wâ ( j ) ) ) (8) We use log-space as when errors are very small, the average mean squared weight error is determined by outliers, e.g., a single non-converging layer when all other layers converge. Log transforming the mean squared error is robust to such outliers. We visualize this in Fig. 4 where Spectral DeTuning clearly converges to a much better solution. Despite the outstanding convergence, the small number of outliers create a false impression where the MSE shows a significantly higher error. In App. C we show that the W-Error is strongly correlated with the recovery of the Pre-FT semantic capabilities (Ď=0.8800.880Ď=0.880Ď = 0.880 for W-Error vs. LPIPS). Table 1: LoWRA Bench Dataset Summary: The dataset covers widely used models spanning vision and language modalities. It includes over 540540540540 Pre-FT layers and over 8,00080008,0008 , 000 fine-tuned layers Pre-FT Model Task Fine-tuning Task # Pre-FT Layers # FT Layers ViT Classific. VTAB-1K 24242424 360360360360 SD1.5 T2I Gen. Personalization 264264264264 3960396039603960 Mistral Text Gen. UltraChat SFT 128128128128 1920192019201920 Mistral Text Gen. UltraFeedback DPO 128128128128 1920192019201920 6.3 Semantic Evaluation Metrics We design model specific metrics focusing on the Pre-FT task from a semantic perspective. ViT Activation Distance (Act.-Dist.). We take the cosine distance between the activations of the Pre-FT model and those of the recovered one. Specifically, we take the mean of all transformer tokens at the end of the last transformer block. We use a subset of 5000500050005000 images from the ImageNet validation set. Stable Diffusion LPIPS (LPIPS). The LPIPS [63] distance between images generated by the Pre-FT model and by the recovered model. We report the mean LPIPS for the first 100100100100 prompts of the COCO Captions validation dataset [7]. Mistral SBERT (SBERT). The log cosine distance between the Sentence-BERT [35] (SBERT) textual embeddings of text generated by the Pre-FT model and by the recovered model. We report the mean log cosine for the first 100100100100 prompts of the Alpaca Farm evaluation benchmark [13]. 6.4 Experimental Setup Subsets. In each experiment, we specify a number of LoRA fine-tuned models L, which is often lower than the total number of LoRAs available in the datasets. We do this by randomly sampling a set of L models from the datasets. We then perform the Pre-FT weight recovery method on this subset. We repeat this experiment (including subset sampling) 10101010 times. The reported performance metrics are the average and standard deviation over the experiments. Baselines. The two baseline methods are i) using one of the fine-tuned LoRA models; we average the results over all models in the sampled subset. i) averaging the weights across all LoRA fine-tuned models in the sampled subset and reporting the results of the weight averaged model. The motivation behind the mean LoRA baseline, is the assumption that the mean of the residuals is the zero matrix, i.e., 1nâ˘âi=1nMi=01superscriptsubscript1subscript0 1n _i=1^nM_i=0divide start_ARG 1 end_ARG start_ARG n end_ARG âi = 1n Mitalic_i = 0. In this case the optimum of Eq. 3 becomes the average of all the weights. 7 Experiments Table 2: ViT Results: As expected, the LoRA fine-tuned models have drifted away from the initial weights and activations. The mean of the LoRAs is slightly better, but is still far from the Pre-FT model. In contrast, Spectral DeTuning achieves an almost perfect semantic convergence. Reported results use n=55n=5n = 5 fine-tuned LoRAs Method W-Error â â Act.-Dist. â â LoRA FT -4.602 Âą0.110plus-or-minus0.110Âą 0.110Âą 0.110 1e-1 Âą9â˘eâ2plus-or-minus92Âą 9e-2Âą 9 e - 2 Mean LoRA -5.214 Âą0.114plus-or-minus0.114Âą 0.114Âą 0.114 5e-2 Âą1â˘eâ2plus-or-minus12Âą 1e-2Âą 1 e - 2 Spectral DeTuning -15.942 Âą1.889plus-or-minus1.889 1.889Âą 1.889 1e-6 Âą3e-6plus-or-minus3e-6 3e-6Âą 3e-6 7.1 Preliminary Investigation on ViT We begin our exploration of Pre-FT weight recovery using ViT, due to its simple architecture with consistent weight dimensions and relatively small model size. While this is our simplest task, it is not a âtoy exampleâ but a real model that is widely used and deployed in countless production settings. In Tab. 2 we show the results for n=55n=5n = 5 fine-tuned LoRAs. As expected, the LoRA fine-tuned models are indeed different from the Pre-FT model. Averaging over several LoRA models slightly improves the results, but is still far from recovering the Pre-FT activations. Our method, Spectral DeTuning, performs much better and attains an almost perfect semantic convergence, outperforming the baselines by a wide margin. 7.2 In the Wild Weight Recovery of Stable Diffusion Having shown the vulnerability of an image classification model, we now test the vulnerability of Stable Diffusion, a multi-modal text-to-image model. To this end, we used publicly fine-tuned LoRAs found on civitai, allowing us to validate our method âin the wildâ. As in the case of ViT, the baselines perform poorly on all metrics. In contrast, Spectral DeTuning recovers the Pre-FT weights with high precision. This results in a significant improvement of the recovered semantic capabilities of the Pre-FT model while using as little as n=55n=5n = 5 fine-tuned LoRAs (See Tab. 3 and Fig. 3). Implication: SoTA personalization methods using LoRA are vulnerable to Pre-FT weight recovery attacks. 7.3 Pre-FT Weight Recovery of an Aligned LLM Having achieved success with mid-sized image models, we now investigate the ability of our method to scale up to a large-scale aligned LLM. Specifically, we use Mistral-7B, a top performing open-source LLM. Following common practice, we fine-tune the model in two stages, first performing supervised fine-tuning (SFT) followed by a direct preference optimization (DPO) alignment fine-tuning stage [34]. We report the results of both stages in Tab. 4, as we can see, Spectral DeTuning successfully recovers the weights with high precision. This high quality recovery is also expressed in recovering the semantic capabilities of the Pre-FT model. I.e., the estimated weights yield a model which provides responses that are very similar to the Pre-FT model and much more so than the LoRA fine-tuned model (See Fig. 2). Implication: SoTA LLMs that use LoRA for alignment fine-tuning are vulnerable to Pre-FT weight recovery attacks. Table 3: Stable Diffusion Results: Spectral DeTuning is almost three times better than the baselines, recovering a large portion of the semantic capabilities of the pre-fine-tuning Stable Diffusion. Reported results use n=55n=5n = 5 fine-tuned LoRAs taken from an online LoRA marketplace Method W-Error â â LPIPS â â LoRA FT -6.921 Âą1.080plus-or-minus1.080Âą 1.080Âą 1.080 0.514 Âą0.047plus-or-minus0.047Âą 0.047Âą 0.047 Mean LoRA -7.540 Âą1.099plus-or-minus1.099Âą 1.099Âą 1.099 0.482 Âą0.012plus-or-minus0.012Âą 0.012Âą 0.012 Spectral DeTuning -17.816 Âą2.126plus-or-minus2.126 2.126Âą 2.126 0.009 Âą0.006plus-or-minus0.006 0.006Âą 0.006 Table 4: Mistral Results: Spectral DeTuning recovers the Pre-FT weights and semantic capabilities with high precision, both in the supervised fine-tuning (SFT) stage and the alignment fine-tuning stage (DPO). Reported results use n=1212n=12n = 12 fine-tuned LoRAs for SFT and n=88n=8n = 8 fine-tuned LoRAs for DPO Method W-Error â â SBERT â â SFT LoRA FT -8.677 Âą0.153plus-or-minus0.153Âą 0.153Âą 0.153 -0.994 Âą0.731plus-or-minus0.731Âą 0.731Âą 0.731 Mean LoRA -9.299 Âą0.222plus-or-minus0.222Âą 0.222Âą 0.222 -1.007 Âą0.726plus-or-minus0.726Âą 0.726Âą 0.726 Spectral DeTuning -16.502 Âą1.855plus-or-minus1.855 1.855Âą 1.855 -9.324 Âą6.942plus-or-minus6.942 6.942Âą 6.942 DPO LoRA FT -9.903 Âą0.166plus-or-minus0.166Âą 0.166Âą 0.166 -3.058 Âą4.763plus-or-minus4.763Âą 4.763Âą 4.763 Mean LoRA -10.757 Âą0.178plus-or-minus0.178Âą 0.178Âą 0.178 -3.455 Âą5.171plus-or-minus5.171Âą 5.171Âą 5.171 Spectral DeTuning -22.062 Âą1.180plus-or-minus1.180 1.180Âą 1.180 -14.708 Âą3.123plus-or-minus3.123 3.123Âą 3.123 8 Ablations Rank Scheduler Ablation. We ablate the rank scheduler introduced in Sec. 5.3 using the Stable Diffusion experiment. Based on Fig. 5 we observe three phenomena, i) The rank scheduler drastically accelerates the convergence, i) When using the rank scheduler, there is much less variance between the convergence of different layers, and i) Using the rank scheduler results in a higher precision convergence. Fig. 6 visualizes phenomena (i) and (i) by showing the cumulative percent of layers (y axis) that converge to a given W-Error level (x axis). When using the rank scheduler, over 95%percent9595\%95 % of the layers converge with a precision of at least â1616-16- 16, in contrast to less than 40%percent4040\%40 % when not using the scheduler. Moreover, by using the rank scheduler, some layers converge to a more precise solution. Robustness to Unknown and Varying Ranks. We tested the LoRA rank estimation heuristic presented in Sec. 5.4 on hundreds of combinations of LoRAs with different ranks. The heuristic achieved an accuracy of 100%percent100100\%100 %. We further tested the idea of using a dedicated rank scheduler for each LoRA model as described in Secs. 5.3 and 5.4. We use n=66n=6n = 6 fine-tuned LoRAs with ranks [8,32,32,32,64,100]832323264100[8,32,32,32,64,100][ 8 , 32 , 32 , 32 , 64 , 100 ] taken from CivitAI. Spectral DeTuning is robust to the varying ranks, exhibiting only a minor decrease in performance despite the higher rank of the LoRAs (See Tab. 5). Robustness to Different Models. We demonstrate the robustness of Spectral DeTuning to cases where a fine-tuned LoRA from a different Pre-FT model (with the same architecture) was mixed into the set of fine-tuned LoRAs. Using the same heuristic presented in Sec. 5.4, the difference between the mixed model weights and any other LoRA should be of full rank (since the Pre-FT model is different) and trivial to detect. We validated this solution using Stable Diffusion. We added to the set of fine-tuned LoRA models a model that originated from Stable Diffusion 1.4 (all the others originated from Stable Diffusion 1.5). Indeed, the above steps indicated the LoRA that originated from Stable Diffusion 1.4 has a full rank difference from any other LoRA (while the pairwise rank between the LoRAs that used the same Pre-FT model were low rank, as expected). This allows us to detect the LoRA that got mixed up into the set and remove it. Figure 5: Rank Scheduler Convergence Speed: Using the rank scheduler has three benefits, i) accelerated convergence , i) less variance between layers, and i) higher precision convergence. Here we visualize i), see Fig. 6 for a layer-wise visualization Figure 6: Rank Scheduler Convergence Quality: When using the rank scheduler, over 95%percent9595\%95 % of the layers converge with a precision of at least â1616-16- 16, in contrast to less than 40%percent4040\%40 % without the scheduler Table 5: Robustness to Unknown and Varying Ranks Results: We test the robustness to LoRAs with varying ranks. Spectral DeTuning is robust to varying ranks, exhibiting only a minor decrease in performance. We use n=66n=6n = 6 fine-tuned LoRAs with ranks [8,32,32,32,64,100]832323264100[8,32,32,32,64,100][ 8 , 32 , 32 , 32 , 64 , 100 ] taken from an online LoRA marketplace Method W-Error â â LPIPS â â LoRA FT -5.882 0.462 Mean LoRA -6.969 0.307 Spectral DeTuning -14.453 0.073 W-Error vs. Loss. In reality an attacker has no access to the error and can only measure the loss in Eq. 3. To show the loss accurately reflects the error defined in Eq. 2, we measure their relation and find they are almost perfectly correlated (Ď=0.9940.994Ď=0.994Ď = 0.994). For further details see App. B. 9 Discussion and Limitations Number of LoRAs. Spectral DeTuning requires several LoRAs to recover the Pre-FT weights. In Fig. 7 we illustrate the impact of the number of fine-tuned LoRA models on the W-Error convergence. Note that different W-Error values are not comparable across models, e.g., Mistral DPO obtains a lowest W-Error but only semantically converges when using 8 LoRAs (See Fig. 11). In App. A we study the effects of the number of LoRAs on the semantic convergence for all LoWRA Bench subsets. We anticipate that future methods will incorporate additional constraints to reduce the required number of LoRAs. Public Availability of LoRA Fine-tuned Models. We assume the availability of multiple LoRA fine-tuned models originating from the same pre-fine-tuning model. This is a reasonable assumption as there are model âmarketplacesâ such as Hugging Face and civitai, where many LoRA fine-tuned models are publicly available. These LoRA models often share the same source Pre-FT model, which fits our proposed setting perfectly. Other Types of Fine-tuning. While our focus has been on exposing the vulnerability of LoRA fine-tuned models, numerous other parameter-efficient fine-tuning methods exist. The general case of Pre-FT weight recovery of fully fine-tuned models is the most general and probably hardest case. Extending the scope of our attack to encompass these methods presents an exciting avenue for research. Pre-FT Weight Recovery Defense. We do not know of a defense against this attack. Also, as this attack targets publicly available models, once a vulnerability is identified, there is no option to retract the model. However, we remain optimistic that a defense will be discovered in the future. For instance, modifying training such that an infeasible high number of LoRAs will be required for accurate recovery. 10 Conclusion In this paper, we unveiled a new vulnerability in LoRA fine-tuned models, allowing attackers to recover the Pre-FT weights using multiple models. Our method, Spectral DeTuning, demonstrates this vulnerability on large-scale models like Mistral and Stable Diffusion. We introduced LoWRA Bench and discussed future directions to promote further research. By highlighting this vulnerability, we hope to encourage the research community to develop better defenses against such attacks. 11 Acknowledgements This work was supported in part by the âIsrael Science Foundationâ (ISF), the âCouncil for Higher Educationâ (Vatat), and the âCenter for Interdisciplinary Data Science Researchâ (CIDR). 12 Broader Impact This work uncovers a significant vulnerability in fine-tuned models, allowing attackers to access pre-fine-tuning weights. While this discovery reveals potential security risks, our primary objective is to advance the field of Machine Learning and raise awareness within the research community about the existing vulnerabilities in current models. Instead of using the findings of this study to execute attacks, we advocate for their use by model creators to enhance the safety and security of their models. By acknowledging and addressing vulnerabilities, creators can proactively safeguard against potential threats. Furthermore, in the discussion section, we outline potential future directions and mitigation strategies. Following established practices in the cyber security community, we emphasize the importance of open discussion and encourage the reporting of vulnerabilities. By fostering transparency and collaboration, we can collectively create a safer environment for deploying machine learning models. Figure 7: Effect of the Number of LoRAs on W-Error Convergence: For the semantic equivalent see App. A References Avrahami et al. [2023a] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers. Association for Computing Machinery, 2023a. Avrahami et al. [2023b] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b. Bourtoule et al. [2021] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141â159. IEEE, 2021. Burns et al. [2023] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. Carlini et al. [2023] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447, 2023. Carlini et al. [2024] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of a production language model. arXiv preprint arXiv:2403.06634, 2024. Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr DollĂĄr, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Cui et al. [2023] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. Ding et al. [2023] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. Dubois et al. [2023] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. Fredrikson et al. [2014] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An \End-to-End\ case study of personalized warfar in dosing. In 23rd USENIX security symposium (USENIX Security 14), pages 17â32, 2014. Fredrikson et al. [2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322â1333, 2015. Ganguli et al. [2022] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. Gu et al. [2023] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305.18292, 2023. He et al. [2021] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. ArXiv, abs/2110.04366, 2021. Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790â2799. PMLR, 2019. Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Huang et al. [2023] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023. Hyeon-Woo et al. [2021] Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. arXiv preprint arXiv:2108.06098, 2021. Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709â727. Springer, 2022. Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. Li and Liang [2021] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. Lin et al. [2024] Peiqin Lin, Shaoxiong Ji, JĂśrg Tiedemann, AndrĂŠ FT Martins, and Hinrich SchĂźtze. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303, 2024. Liu et al. [2022] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950â1965, 2022. Liu et al. [2023] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023. Mangrulkar et al. [2022] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022. Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730â27744, 2022. Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. Po et al. [2023] Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models. arXiv preprint arXiv:2312.02432, 2023. Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjĂśrn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684â10695, 2022. Roziere et al. [2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, JĂŠrĂŠmy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a. Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949, 2023b. Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211â252, 2015. Segev et al. [2023] Eliya Segev, Maya Alroy, Ronen Katsir, Noam Wies, Ayana Shenhav, Yael Ben-Oren, David Zar, Oren Tadmor, Jacob Bitterman, Amnon Shashua, et al. Align with purpose: Optimize desired properties in ctc models with a general plug-and-play framework. arXiv preprint arXiv:2307.01715, 2023. Shafran et al. [2021] Avital Shafran, Shmuel Peleg, and Yedid Hoshen. Membership inference attacks are easier on difficult problems. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14820â14829, 2021. Shafran et al. [2023] Avital Shafran, Ilia Shumailov, Murat A Erdogdu, and Nicolas Papernot. Beyond labeling oracles: What does it mean to steal ml models? arXiv preprint arXiv:2310.01959, 2023. Shah et al. [2023] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. 2023. Shokri et al. [2017] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3â18. IEEE, 2017. Sidahmed et al. [2024] Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, et al. Perl: Parameter efficient reinforcement learning from human feedback. arXiv preprint arXiv:2403.10704, 2024. Sun et al. [2023] Simeng Sun, Dhawal Gupta, and Mohit Iyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. arXiv preprint arXiv:2309.09055, 2023. Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. Tramèr et al. [2016] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction \APIs\. In 25th USENIX security symposium (USENIX Security 16), pages 601â618, 2016. Tunstall et al. [2023] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, ClĂŠmentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023. Wang et al. [2023a] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. ArXiv, abs/2305.16213, 2023a. Wang et al. [2023b] Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter-efficient transfer learning. arXiv preprint arXiv:2303.02861, 2023b. Wei et al. [2023] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. Wolf et al. [2023] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082, 2023. Wright [2015] Stephen J Wright. Coordinate descent algorithms. Mathematical programming, 151(1):3â34, 2015. Yadav et al. [2023] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. arXiv preprint arXiv:2306.01708, 2023. Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. Yu et al. [2021] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021. Zhai et al. [2019] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019. Zhai et al. [2022] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123â18133, 2022. Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836â3847, 2023a. Zhang et al. [2023b] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023b. Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586â595, 2018. Zou et al. [2023] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. Appendix A The Effect of the Number of LoRAs on Semantic Convergence We visualize the effect of the number of LoRAs on the semantic convergence for each of the LoWRA Bench subsets, results are shown in Figs. 11, 11, 11 and 11. Figure 8: Number of LoRAs vs. Semantic Convergence - ViT Figure 9: Number of LoRAs vs. Semantic Convergence - Stable Diffusion Figure 10: Number of LoRAs vs. Semantic Convergence - Mistral SFT Figure 11: Number of LoRAs vs. Semantic Convergence - Mistral DPO Appendix B W-Error vs. Loss We visualize the relation between the W-Error and the log loss and find they are almost perfectly correlated (Ď=0.9940.994Ď=0.994Ď = 0.994), see Fig. 13 for a visualization over 200200200200 iterations using Stable Diffusion. Figure 12: W-Error vs. Loss - Stable Diffusion Figure 13: W-Error vs. LPIPS Appendix C W-Error vs. LPIPS We visualize the relation between the W-Error and LPIPS and find they are strongly correlated (Ď=0.8800.880Ď=0.880Ď = 0.880), see Fig. 13 for a visualization over 200200200200 iterations using Stable Diffusion. Appendix D LoRA Rank vs. W-Error In Tabs. 7 and 7 we show the results for the ViT model when using different LoRA ranks and fixing the number of LoRAs. Table 6: Using 5555 LoRAs Rank W-Error 8 â15.63615.636-15.636- 15.636 12 â15.55015.550-15.550- 15.550 16 â13.48013.480-13.480- 13.480 32 â4.8174.817-4.817- 4.817 Table 7: Using 5555 LoRAs Rank W-Error 8 â15.82215.822-15.822- 15.822 12 â15.77315.773-15.773- 15.773 16 â15.25815.258-15.258- 15.258 32 â9.6399.639-9.639- 9.639 Appendix E LoWRA Bench Dataset We now elaborate on the implementation details of the LoWRA Bench dataset. E.1 ViT Models As the Pre-FT model we use âvit-base-patch16-224â found on hugging face (https://huggingface.co/google/vit-base-patch16-224). We fine-tune the model using the PEFT library [30]. For each LoRA we use a different VTAB-1k [59] dataset, the datasets we use are: cifar100, caltech101, dtd, flower102, pet37, svhn, patch_camelyon, clevr-count, clevr-distance, dmlab, kitti, dsprites-location, dsprites-orientation, smallnorb-azimuth, smallnorb-elevation. We pre-process the datasets according to the protocol of Jia et al. [23] found on their github page https://github.com/KMnP/vpt/blob/main/VTAB_SETUP.md. We use an 80/20802080/2080 / 20 train/validation split and choose the checkpoint with the best validation loss. We use a rank r=1616r=16r = 16 and LoRA fine-tune the query and value layers. This protocol results in 24242424 Pre-FT model layers and a total of 24â 15=360â 241536024¡ 15=36024 â 15 = 360 LoRA fine-tuned layers. See Tab. 8 for the fine-tuning hyper-parameters. For semantic evaluation we use a subset of the ImageNet-1K [40] validation set. We construct the subset by taking the first 5555 images of each class, resulting in a subset of 5000500050005000. Table 8: ViT Hyper-parameters Name Value lora_rank (r) 16161616 lora_alpha (Îą) 16161616 lr 9â˘eâ3939e-39 e - 3 batch_size 128128128128 epochs 20202020 datasets cifar100, caltech101, dtd, flower102, pet37, svhn, patch_camelyon, clevr-count, clevr-distance, dmlab, kitti, dsprites-location, dsprites-orientation, smallnorb-azimuth, smallnorb-elevation E.2 Mistral Models As the Pre-FT model we use âMistral-7B-v0.1â found on hugging face (https://huggingface.co/mistralai/Mistral-7B-v0.1). We fine-tune the model following the protocol of Tunstall et al. [50], note that unlike Tunstall et al. [50], we perform LoRA fine-tuning as found on their official github repo https://github.com/huggingface/alignment-handbook. Following the original LoRA setting, we make a minor adjustment to the original hyper-parameters of the repo and use a LoRA alpha of 64646464 instead of 16161616 (i.e. Îą=6464Îą=64Îą = 64), this leads to faster and better convergence. To fine-tune 15151515 different models, we use different random subsets of 80%percent8080\%80 % of the fine-tuning dataset. We use seeds of 0â140140-140 - 14 for the different fine-tuned models. We follow this protocol for both the supervised fine-tuning stage (SFT) and the direct preference optimization (DPO) alignment stage. Following Tunstall et al. [50], the SFT stage uses the UltraChat dataset [11] and the DPO stage uses the UltraFeedback dataset [9]. We first fine-tune the 15151515 SFT models, and then fine-tune the 15151515 DPO models, where each DPO model continues the training of the SFT model with the corresponding seed. Following the original setup, use a rank r=6464r=64r = 64 and LoRA fine-tune the q_proj, k_proj, v_proj, and o_proj layers. This protocol results in 128128128128 Pre-FT model layers and a total of 128â 15=1920â 128151920128¡ 15=1920128 â 15 = 1920 LoRA fine-tuned layers for both the SFT and DPO stages. See Tabs. 10 and 10 for the fine-tuning hyper-parameters. For inference we use the following decoding hyper-parameters: max_new_tokens=50, do_sample=True, temperature=0.7, top_k=50, top_p=0.95. For evaluation we use the first 100100100100 prompts from the AlpacaFarm benchmark [13] found in the following link https://huggingface.co/datasets/tatsu-lab/alpaca_farm/viewer/alpaca_farm_evaluation. We provide these prompts in the SM. Table 9: Mistral SFT Hyper-parameters Name Value lora_rank (r) 64646464 lora_alpha (Îą) 64646464 lora_dropout 0.10.10.10.1 lr 2â˘eâ5252e-52 e - 5 batch_size 4444 gradient_accumulation_steps 128128128128 learning_rate_scheduler Cosine epochs 1111 warmup_ratio 0.10.10.10.1 data_type bfloat16 dataset random 80%percent8080\%80 % of UltraChat seeds 0â150150-150 - 15 Table 10: Mistral DPO Hyper-parameters Name Value lora_rank (r) 64646464 lora_alpha (Îą) 64646464 lora_dropout 0.10.10.10.1 lr 5â˘eâ6565e-65 e - 6 batch_size 2222 gradient_accumulation_steps 32323232 learning_rate_scheduler Cosine epochs 1111 warmup_ratio 0.10.10.10.1 data_type bfloat16 dataset random 80%percent8080\%80 % of UltraFeedback seeds 0â150150-150 - 15 E.3 Stable Diffusion Models As the Pre-FT model we use âStable Diffusion 1.5â found on hugging face (https://huggingface.co/runwayml/stable-diffusion-v1-5). We collect 15151515 personalization fine-tuned models from civitai.com, a public and widely used LoRA models marketplace. This allows us to examine our method in a real world setting, for the full list of LoRAs see Tab. 11. After examining the downloaded models, we deduce that their LoRA rank is r=3232r=32r = 32 and that their fine-tuned layers are: to_q, to_v, to_k, to_out, proj_out, proj_in, and f. Resulting in 192192192192 Pre-FT model layers for and a total of 192â 15=2880â 192152880192¡ 15=2880192 â 15 = 2880 LoRA fine-tuned layers. For inference we use the default Stable Diffusion 1.51.51.51.5 generation pipeline (i.e. 50505050 sampling steps). For evaluation we use a the first 100100100100 captions from the COCO Captions [7] validation dataset found in the following link https://github.com/tylin/coco-caption/blob/master/annotations/captions_val2014.json. We provide these prompts in the SM. Table 11: Stable Diffusion Fine-tuned LoRA Links https://civitai.com/models/186716/smol-animals-lora-15sdxl?modelVersionId=241137 https://civitai.com/models/189905/pastry-lora-15sdxl?modelVersionId=241955 https://civitai.com/models/191203/bastet-egypt-cat-style-lora-15sdxl?modelVersionId=243232 https://civitai.com/models/190176/fur-pirates-lora-15sdxl?modelVersionId=241976 https://civitai.com/models/211973/cigarette-style-lora-15sdxl?modelVersionId=247079 https://civitai.com/models/233316/smol-dragons-lora-15sdxl?modelVersionId=263316 https://civitai.com/models/234324/polygon-style-lora-15sdxl?modelVersionId=264506 https://civitai.com/models/202128/overgrowth-style-lora-15sdxl?modelVersionId=264449 https://civitai.com/models/218327/mythical-creatures-lora-15sdxl?modelVersionId=289861 https://civitai.com/models/203169/lava-style-lora-15sdxl?modelVersionId=265372 https://civitai.com/models/197998/chocolate-coffee-style-lora-15sdxl?modelVersionId=259150 https://civitai.com/models/180780/crystals-lora-15sdxl?modelVersionId=238435 https://civitai.com/models/196040/transparent-glass-body-lora-15sdxl?modelVersionId=245630 https://civitai.com/models/199968/liquid-flow-style-lora-15sdxl?modelVersionId=259228 https://civitai.com/models/206783/christmas-critters-lora-15sdxl?modelVersionId=275204 Appendix F Spectral DeTuning Implementation Details For all semantic evaluations we use a seed of 00 for all baselines and for our results. For both the ViTs and Stable Diffusion (SD) experiments we run Spectral DeTuning for 300300300300 optimization steps. For the Mistral SFT and DPO experiments we use 1000100010001000 optimization steps. We base our rank scheduler implementation on the official PyTorch implementation of a the ReduceLROnPlateau learning rate scheduler 111https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html. We expand on the hyper-parameters of the rank scheduler in Tab. 12. Table 12: Spectral DeTuning Rank Scheduler Hyper-parameters Name Value Used Explanation total_steps 200200200200 for ViT and SD, 1000100010001000 for Mistral The total number of optimization steps start_rank 1111 The rank to start the optimization from (i.e. râsuperscriptr^*râ) end_rank 16161616 for ViTs, 32323232 for SD, 64646464 for Mistral The final rank of the scheduler (i.e. r, the actual rank of the LoRA models) factor 2222 The multiplicative factor to increase the rank by patience 15151515 Number of scheduler steps with no improvement after which rank will be increased. force_end_rank_percent 0.50.50.50.5 Percent of the total_steps after which end_rank will be forced Appendix G Runtime and Compute Since Spectral DeTuning does not pass any gradients through the model, it is highly parallelizable and can recover the weights of even large models (e.g., Mistral 7B) in minutes using a cluster of desktop-grade GPUs or even CPUs. For example, using a cluster of RTX2080 it can recover Mistral-7B in under five minutes. Appendix H Detecting the Fine-Tuned Layers We note that it is easy to detect which layers were fine-tuned. This can simply be done by comparing the layers weights of n different fine-tuned versions. The layers which have not been fine-tuned will be equal across all n models, while the fine-tuned layers will have some variation between them. Appendix I Algorithm with Rank Scheduler In Alg. 2 we present pytorch-like pseudocode for Spectral DeTuning that includes that rank scheduler. Algorithm 2 PyTorch Pseudocode for Spectral DeTuning ⏠# W_ps: List of n fine-tuned weight matrices # steps: Number of optimization steps # r: LoRA rank # Initialize rank scheduler current_lora_rank = 1 rank_scheduler = LoRARankScheduler(start_rank=current_lora_rank, end_rank=r) # Initialize W_star W_s = torch.mean(torch.stack(W_ps), axis=0) # Perform optimization for step in range(steps): # M-step # Approximate each M^*_i (Eq. 5) M_s = [W_p - W_s for W_p in W_ps] # Truncate each M^*_i to rank <= r (Eq. 5) for i in range(len(M_s)): (U, S, V) = torch.svd_lowrank(M_s[i], q=current_lora_rank) M_s[i] = (U @ torch.diag_embed(S)) @ V.T # W-step # Approximate W_star (Eq. 7) W_s = [W_p - M_si for (W_p, M_si) in zip(W_ps, M_s)] W_s = torch.mean(torch.stack(W_s), axis=0) # Compute the current loss iteration_losses = [torch.mean((W_ps[i] - (W_s + M_s[i])) ** 2) for i in range(len(M_s))] loss = torch.mean(torch.stack(iteration_losses), axis=0) # Step the rank scheduler rank_scheduler.step(loss) current_lora_rank = rank_scheduler.current_rank Appendix J Mistral Additional Results For the list of mistral prompts see supplementary material (SM). In Fig. 17 we show side-by-side results for 10101010 randomly (random_seed=42) sampled prompts from our evaluation dataset, using the Pre-FT recovered weights of the DPO fine-tuned Mistral model. See SM for the rest of the DPO results and for the SFT results. Appendix K Stable Diffusion Additional Results For the list of stable diffusion prompts see SM. In Figs. 14, 15 and 16 we show side-by-side results for the entire dataset. Note, images are compressed to reduce file size, for the full resolution images see the SM. Figure 14: Stable Diffusion Results: Note, images are compressed to reduce file size, for the full resolution images see the SM. Figure 15: Stable Diffusion Results: Note, images are compressed to reduce file size, for the full resolution images see the SM. Figure 16: Stable Diffusion Results: Note, images are compressed to reduce file size, for the full resolution images see the SM. Figure 17: Non Cherry-picked Mistral DPO Results: We display side-by-side results for 10101010 randomly (random_seed=42) sampled prompts from our evaluation dataset. For the rest of the results see supplementary material.