Paper deep dive
FISGuard: Defending Against Membership Inference via Fixed Input Subspaces
Haocheng Jiang, Hua Shen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/31/2026, 3:28:13 AM
Summary
The paper introduces FISGuard, a defense mechanism against Membership Inference Attacks (specifically ProjRes) in Federated Learning with Large Language Models. FISGuard uses a fixed input subspace constructed from public data to restrict how private representations are exposed via gradients, thereby reducing the projection-residual discrepancy that attackers exploit. It outperforms existing defenses by maintaining model utility while achieving near-random guessing attack performance.
Entities (10)
Relation Signals (7)
FISGuard → defendsagainst → ProjRes
confidence 95% · We evaluate FISGuard against five representative defense methods... The results show that FISGuard reduces the ProjRes attack AUC to near the random-guessing level
ProjRes → exploits → Gradient Structure
confidence 93% · ProjRes... exploits the geometric structure of gradients... based on the projection residual between a candidate representation and the subspace induced by server-observable gradients.
FISGuard → isappliedto → Federated Learning
confidence 92% · As large language models are increasingly adopted in federated learning... To address this issue, we propose FISGuard
FISGuard → uses → Public Data
confidence 90% · Its key idea is to construct and fix a low-dimensional representation subspace using independent public data
FISGuard → improves → Privacy-Utility Trade-off
confidence 88% · thereby achieving a favorable privacy--utility trade-off.
Adapter → isvulnerableto → ProjRes
confidence 85% · This attack is not limited to Adapters... The Down projection in a standard Adapter... exhibit this basic structure.
LoRA → isvulnerableto → ProjRes
confidence 85% · For LoRA, the input-side low-rank factor can be fixed... Any fully connected mapping... may create a similar input-side gradient attack surface.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As large language models are increasingly adopted in federated learning, protecting user privacy while performing parameter-efficient fine-tuning on distributed private data has become an important challenge. Although clients only share gradients instead of directly uploading raw data, the shared gradients may still leak membership information about training samples. ProjRes (S&P, 2026) further increases this risk: with less information and without accessing model outputs, an attacker can effectively distinguish members from non-members solely based on the projection residual between a candidate representation and the subspace induced by server-observable gradients. Existing defenses against membership inference mostly rely on gradient perturbation or regularization, which can not only degrade model utility but also fail to effectively defend against the membership inference attack introduced by ProjRes, which exploits the geometric structure of gradients. To address this issue, we propose FISGuard, a lightweight defense. Its key idea is to construct and fix a low-dimensional representation subspace using independent public data, thereby restricting the space through which private representations are exposed via gradients while preserving the primary information required for downstream tasks. This substantially reduces the projection-residual discrepancy between members and non-members. We evaluate FISGuard against five representative defense methods across three NLP datasets, two LLMs, and two fine-tuning strategies, Adapter and LoRA. The results show that FISGuard reduces the ProjRes attack AUC to near the random-guessing level of 0.5 in most settings, while maintaining downstream task performance close to that of the undefended model and introducing only limited computational overhead, thereby achieving a favorable privacy--utility trade-off.
Tags
Links
- Source: https://arxiv.org/abs/2608.27836v1
- Canonical: https://arxiv.org/abs/2608.27836v1
Trouble viewing inline? Open PDF directly →
Full Text
69,240 characters extracted from source content.
Expand or collapse full text
FISGuard: Defending Against Membership Inference via Fixed Input Subspaces Haocheng Jiang Affiliation: Hubei University of Technology jianghaocheng@hbut.edu.cn Hua Shen Affiliation: Hubei University of Technology nancy78733@126.com Abstract As large language models are increasingly adopted in federated learning, protecting user privacy while performing parameter-efficient fine-tuning on distributed private data has become an important challenge. Although clients only share gradients instead of directly uploading raw data, the shared gradients may still leak membership information about training samples. ProjRes (S&P, 2026) further increases this risk: with less information and without accessing model outputs, an attacker can effectively distinguish members from non-members solely based on the projection residual between a candidate representation and the subspace induced by server-observable gradients. Existing defenses against membership inference mostly rely on gradient perturbation or regularization, which can not only degrade model utility but also fail to effectively defend against the membership inference attack introduced by ProjRes, which exploits the geometric structure of gradients. To address this issue, we propose FISGuard, a lightweight defense. Its key idea is to construct and fix a low-dimensional representation subspace using independent public data, thereby restricting the space through which private representations are exposed via gradients while preserving the primary information required for downstream tasks. This substantially reduces the projection-residual discrepancy between members and non-members. We evaluate FISGuard against five representative defense methods across three NLP datasets, two LLMs, and two fine-tuning strategies, Adapter and LoRA. The results show that FISGuard reduces the ProjRes attack AUC to near the random-guessing level of 0.5 in most settings, while maintaining downstream task performance close to that of the undefended model and introducing only limited computational overhead, thereby achieving a favorable privacy–utility trade-off. Index Terms: Federated Learning, Large Language Models, Fine-tuning, Membership Inference Attacks. I Introduction Federated large language models enable multiple clients to collaboratively fine-tune a pretrained model without sharing their local data [9, 15]. To reduce the training and communication costs on clients, existing systems commonly adopt parameter-efficient fine-tuning methods, such as Adapters and LoRA, which update only a small number of additional parameters [6]. However, despite their compact size and structured form, the gradients of these parameters may still reveal information about clients’ training data[7]. Membership inference attacks aim to determine whether a candidate record was used to train a target client’s model. Membership itself may constitute sensitive information, as it can reveal whether a user appears in a medical, financial, or other privacy-sensitive dataset [25]. ProjRes performs membership inference against federated large language models by exploiting the gradient structure of trainable linear layers. Its key observation is that the gradient of a linear layer is jointly determined by local sample representations and backpropagated error signals. Consequently, the representations of training samples contribute to the construction of the gradient row space. ProjRes [6] extracts this space from server-visible gradients and infers membership based on the projection residual of a candidate representation with respect to the gradient space. Because member samples directly participate in gradient computation, their representations can typically be explained more accurately by the gradient space. In contrast, non-member representations are more likely to retain larger components outside that space. This attack is not limited to Adapters. Any fully connected mapping that directly receives private representations, is trained by the client, and exposes its gradients to the server may create a similar input-side gradient attack surface. The Down projection in a standard Adapter, the input-side low-rank factor in LoRA, and ordinary fully connected layers involved in private fine-tuning all exhibit this basic structure. The original ProjRes attack achieves strong membership discrimination across all evaluated model–dataset combinations. Nevertheless, existing research still lacks a defense specifically designed around the general linear-gradient structure exploited by ProjRes. Existing defenses against membership inference primarily include output protection, training regularization, gradient perturbation, and secure aggregation [11]. Methods such as output perturbation, confidence modification, and knowledge distillation mainly restrict an attacker’s ability to extract membership information from prediction probabilities or loss values. ProjRes, however, does not rely on final model outputs and instead directly analyzes client-uploaded gradients. Even when a model produces similar predictions for members and non-members, the structural relationship exploited by ProjRes may persist as long as private representations continue to participate in the construction of the visible gradient space. Training regularization [18] and early stopping [21] generally reduce membership leakage by mitigating overfitting. However, the basis of ProjRes is not entirely attributable to model memorization. Once a private sample participates in even a single gradient computation, its representation may become embedded in the resulting gradient structure. Therefore, reducing overfitting alone cannot fundamentally prevent private representations from forming a gradient space observable to the server. Gradient clipping [5] and differential privacy [8] provide more general privacy protection, but they operate primarily by perturbing client updates. To disrupt the gradient directions and subspace structure exploited by ProjRes, the added noise must substantially alter the geometry of the gradients, which may further intensify the privacy–utility trade-off in federated large language models. Secure aggregation [4] changes the attacker’s observation model by hiding individual client updates. However, it relies on the additional protocol assumption that the server can observe only the aggregate of multiple client updates, and it does not alter the leakage structure inherent in an individual client’s gradient. Overall, existing defenses either fail to directly address the gradient structure observed by ProjRes or require random perturbation or additional system assumptions. These limitations motivate a different question: instead of concealing information after gradients have already been generated, can we redesign the parameterization of linear layers before private training begins, such that the system no longer produces the gradient structure required by ProjRes? This question follows directly from the mechanism of ProjRes. ProjRes can distinguish members from non-members because a client uses a limited number of local samples to form an incomplete gradient subspace within a high-dimensional representation space. Member representations participate in constructing this subspace and are therefore more likely to lie within it, whereas non-member representations are more likely to occupy uncovered directions. Accordingly, the effectiveness of ProjRes depends on two conditions. First, the original private representations must directly enter server-visible gradients. Second, the gradient space must cover only part of the candidate representation space, thereby producing different projection residuals for members and non-members. An effective defense must therefore address both conditions. First, it should prevent the original high-dimensional private representations from directly generating server-visible input-side gradients. Second, after removing this direct attack surface, it should constrain the remaining gradients to a sufficiently low-dimensional and fixed space that can be fully covered by the gradient span. In this setting, both member and non-member representations can be fully represented within the remaining gradient space, leaving the original ProjRes attack without projection residuals that can be used for membership ranking. Based on this insight, we propose a fixed input-subspace reparameterization framework for trainable linear mappings. A task-specific linear update is expressed as ΔW=UQ, W=UQ, where Q is an input-side projection determined before private training begins and permanently frozen thereafter, while U is the only output-side mapping trained by the client. The original representation is first transformed by Q into a fixed low-dimensional coordinate space, and private client data are used only to learn how these coordinates should be mapped to the output space. The framework reconstructs the attack surface of ProjRes through three consecutive steps. First, it freezes the input-side projection, preventing the original high-dimensional private representations from directly producing observable input-side weight gradients. Second, it moves the remaining server-visible gradients from the original high-dimensional representation space into a fixed low-dimensional space. Third, it ensures that this low-dimensional space can be fully covered by the remaining gradient span, thereby eliminating the out-of-space residuals used by the original ProjRes attack to distinguish members from non-members. This design is not a local modification specific to a particular Adapter parameter. Rather, it is a general principle applicable to trainable fully connected mappings. For a bottleneck Adapter, the framework can be instantiated as a Fixed-Down Adapter, in which the Down projection is frozen and only the Up projection is trained. For LoRA, the input-side low-rank factor can be fixed while only the output-side factor is trained. For a general fully connected layer, the gradient space can be reconstructed in the same way by introducing a low-rank residual branch with a fixed input projection. We use the Fixed-Down Adapter and LoRA as the primary implementations and experimental vehicles, but the theoretical formulation and scope of the method are not limited to these two architectures. To preserve useful representation information within the fixed subspace, we construct the input projection using an independent public dataset rather than an arbitrary random matrix. The public data determine which input directions the model is allowed to use, while the private data are used only to learn how these directions should be combined for the downstream task. This separation ensures that the input-side geometry is determined before private training begins and does not adapt to client samples. We also employ function-preserving initialization so that the newly introduced low-rank branch does not alter the original forward behavior of the pretrained model at the start of training. Compared with existing defenses, the proposed framework offers three advantages. First, it directly targets the gradient attack surface exploited by ProjRes, rather than modifying model outputs or attack scores after the leakage signal has already formed. Second, it does not rely on random gradient noise, thereby avoiding the need to select a noise magnitude and the corresponding utility degradation caused by perturbation. Third, it changes only the organization of trainable parameters and does not require modifications to the client-side task loss, the server aggregation rule, or the model inference interface. In the Adapter instantiation considered in this work, public data are used to construct a fixed Down projection, while only the Up projection and task head are trained, making the method compatible with existing federated training pipelines. The main contributions of this work are as follows: • We provide a unified analysis of ProjRes from the perspective of general trainable fully connected layers and show that its attack capability originates from private input representations directly participating in the construction of a server-observable gradient space. Based on this analysis, we identify two key conditions required for the original ProjRes attack to achieve membership discrimination: direct gradient exposure of the original private representations and incomplete coverage of the candidate representation space by the gradient space. • We propose a fixed input-subspace reparameterization framework that decomposes a linear update into a fixed input projection and a trainable output mapping. The framework first removes the original high-dimensional input-side gradient and then constrains the remaining gradient to a fixed low-dimensional space that can be fully covered. We further instantiate the framework as a Fixed-Down Adapter and provide a unified formulation for extending it to LoRA and general fully connected residual updates. • Using the Fixed-Down Adapter as a representative implementation, we conduct experiments on BERT, GPT-2, and multiple text classification tasks to evaluate the proposed framework in terms of membership privacy protection, task performance, and applicability across different models and datasets. I Preliminary I-A Federated Fine-tuning of Large Language Models Federated fine-tuning of large language models (FedLLM fine-tuning) refers to the application of federated learning to the downstream adaptation of pre-trained large language models (LLMs), enabling multiple clients to collaboratively fine-tune a shared model without directly exchanging their local raw data [17, 10]. A central server first initializes a pre-trained language model and distributes it to the participating clients. Each client independently fine-tunes the received model using its local private dataset and uploads only the updated model parameters or parameter updates to the server. The server then aggregates the updates from participating clients to obtain a new global model, which is subsequently redistributed for the next communication round. This process iteratively follows the paradigm of global model distribution–local fine-tuning–parameter uploading–server aggregation until the global model converges. Formally, consider a federated learning system consisting of K clients. The k-th client owns a local dataset kD_k containing nkn_k samples, and the total number of samples across all clients is N=∑k=1Knk.N= _k=1^Kn_k. The objective of federated LLM fine-tuning is to optimize a global model without centralizing the local datasets, which can be formulated as minΘF(Θ)=∑k=1KnkNFk(Θ), _ F( )= _k=1^K n_kNF_k( ), where Θ denotes the model parameters involved in fine-tuning, and Fk(Θ)F_k( ) denotes the local training objective of client k. Specifically, Fk(Θ)=1nk∑(xi,yi)∈kℓ(fΘ(xi),yi),F_k( )= 1n_k _(x_i,y_i) _k (f_ (x_i),y_i ), where fΘf_ denotes the model parameterized by Θ , and ℓ(⋅) (·) denotes the task-specific loss function. At communication round t, the server distributes the current global model Θt ^t to the participating clients. Each client initializes its local model with Θt ^t and performs local fine-tuning on kD_k, yielding the locally updated model Θkt+1 _k^t+1. Under the Federated Averaging (FedAvg) framework, the server aggregates the local models according to their local data sizes: Θt+1=∑k=1KnkNΘkt+1. ^t+1= _k=1^K n_kN _k^t+1. The resulting Θt+1 ^t+1 is then used as the global model for the next communication round. For modern LLMs, directly fine-tuning and communicating all model parameters can incur substantial computational and communication overhead. Therefore, parameter-efficient fine-tuning (PEFT) is commonly integrated into federated LLM fine-tuning. In this setting, the model parameters can be decomposed as Θ=θ,ϕ, =\θ,φ\, where θ denotes the frozen parameters of the pre-trained backbone, and ϕφ denotes a small set of trainable parameters introduced or selected for downstream adaptation. Accordingly, the federated optimization objective becomes minϕF(ϕ,θ)=∑k=1KnkNFk(ϕ,θ), _φF(φ;θ)= _k=1^K n_kNF_k(φ;θ), where θ remains fixed throughout the fine-tuning process and only ϕφ is locally optimized. At round t, client k receives the global PEFT parameters ϕtφ^t and performs local optimization on its private dataset: ϕkt+1=LocalTrain(θ,ϕt,k). _k^t+1=LocalTrain (θ,φ^t,D_k ). The locally updated PEFT parameters are then uploaded to the server and aggregated as ϕt+1=∑k=1KnkNϕkt+1.φ^t+1= _k=1^K n_kN _k^t+1. Therefore, instead of repeatedly communicating the entire LLM, parameter-efficient federated fine-tuning only requires the trainable parameters ϕφ to participate in local optimization and federated aggregation. In this work, we consider two representative PEFT mechanisms, namely Adapter and Low-Rank Adaptation (LoRA). For Adapter-based fine-tuning, a lightweight bottleneck module is inserted into the pre-trained Transformer while the backbone parameters remain frozen. Given a hidden representation ∈ℝdh ^d, an Adapter can be expressed as Adapter()=+Wupσ(Wdown),Adapter(h)=h+W_upσ (W_downh ), where Wdown∈ℝr×d,Wup∈ℝd×r,r≪d,W_down ^r× d,W_up ^d× r,r d, and σ(⋅)σ(·) denotes a nonlinear activation function. In this case, the trainable parameter set ϕφ mainly consists of the parameters of the inserted Adapter modules. For LoRA-based fine-tuning, the pre-trained weight matrix W0∈ℝdout×dinW_0 ^d_out× d_in is kept frozen, while its task-specific update is parameterized by two low-rank matrices: W=W0+ΔW=W0+BA,W=W_0+ W=W_0+BA, where A∈ℝr×din,B∈ℝdout×r,r≪min(din,dout).A ^r× d_in,B ^d_out× r,r (d_in,d_out). Thus, only the low-rank matrices A and B are optimized during local fine-tuning and communicated during federated aggregation. Under this unified formulation, ϕφ represents the trainable Adapter parameters in Adapter-based fine-tuning or the low-rank parameters in LoRA-based fine-tuning. I Related Work Existing defenses against membership inference attacks mainly reduce privacy leakage by mitigating model overfitting, introducing differential privacy, and perturbing membership-related information. These methods have also been gradually extended to federated learning settings. However, attack signals have expanded from prediction outputs to gradients, training trajectories, and internal representations. Existing defenses therefore still face trade-offs among privacy protection, model utility, and system overhead. In particular, they have difficulty addressing gradient leakage caused by parameter-efficient fine-tuning in federated large language models. One category of membership inference defenses reduces the behavioral gap between members and non-members by mitigating model overfitting and memorization of training samples. Ying et al.[28] combine L2 regularization with differential privacy. Their method reduces membership leakage by suppressing overfitting and perturbing model outputs. Nasr et al.[20] further propose Adversarial Regularization. They formulate training of the target model and the membership inference attack model as a min-max adversarial game. This allows the target model to maintain classification performance while learning output distributions that make members and non-members harder to distinguish. Li and Ribeiro[16] propose MMD+Mix-up. It uses Mix-up to reduce the generalization gap and applies MMD regularization to constrain the prediction distributions of the training and validation sets. For federated learning, Ahmed et al.[2] propose MemberShield. It combines soft labels with early stopping to reduce overconfidence and sample memorization in client models. These methods are relatively easy to implement and require only limited changes to the model architecture. However, their effectiveness largely depends on the correlation between membership leakage and model overfitting. They also lack rigorous privacy guarantees. As a result, exploitable membership information may still be exposed under white-box, trajectory-based, or adaptive membership inference attacks. Another category of methods reduces membership leakage by weakening the direct association between sensitive samples and the final model, or by limiting the model’s memorization of specific training samples. Hu et al.[12] propose DMIG. It trains a generative model on sensitive data and then uses synthetic data to train the final classifier. This prevents the target model from directly learning real training samples. Jarin and Eshete[14] propose MIAShield, which partitions the training data across multiple submodels. During inference, it actively excludes models that have been trained on the queried sample. This weakens membership signals caused by sample memorization. Bai et al.[3] further propose CoFedMID for multi-round trajectory-based MIAs in federated learning. It combines class-guided data partitioning, utility-aware sample compensation, and aggregation-cancelable perturbation. These mechanisms jointly reduce the persistent exposure of sensitive samples during training across multiple clients. Such methods can directly weaken the relationship between the model and sensitive samples. However, they often require additional generative models, model ensembles, sample management, or client coordination mechanisms. These requirements increase training, storage, and communication costs, while also making system implementation more complex. Differential privacy is a widely used formal privacy-preserving mechanism in federated learning. Naseri et al.[19] systematically evaluate local differential privacy (LDP) and central differential privacy (CDP). Their results show that gradient clipping and noise injection can reduce the success rates of membership inference and some backdoor attacks. CDP generally provides a better privacy-utility trade-off. For different federated learning settings, Hu et al.[13] propose PDP-FLames. It exploits the sparsity of knowledge graph gradients to privately select and perturb active gradients. Zhao et al.[30] propose ADPF, which combines particle swarm optimization with a Stackelberg game. It dynamically adjusts the privacy budget according to client heterogeneity and attack behavior. To reduce the performance degradation caused by gradient clipping and random noise in DP-FL, Shi et al.[24] propose DP-FedSAM and DP-FedSAM-topk. These methods use Sharpness-Aware Minimization and update sparsification to improve robustness to DP noise. Yuan et al.[29] further propose DP-FedPUAC. It uses adaptive clipping thresholds and dynamic local iterations to improve the privacy-utility trade-off and communication efficiency. From a robustness perspective, Qi et al.[22] point out that DP noise may interfere with the detection of malicious updates. They therefore propose Robust-DPFL, which identifies poisoning updates based on differences in gradient element distributions. Although some of these studies do not specifically target membership inference, they share the goal of improving the practicality and robustness of DP-FL. Nevertheless, differential privacy still involves a significant privacy-utility trade-off. Stronger privacy protection usually requires stricter clipping and larger amounts of random noise. This can degrade model accuracy, convergence speed, and training stability. In addition to directly applying differential privacy, some studies reduce the information exposed during training or apply targeted perturbations to potential membership signals. Abourayya et al.[1] propose FEDCT. Clients share only hard labels for public unlabeled data, without exchanging model parameters or soft confidence scores. Its extension, DP-FEDCT, further randomizes the discrete labels. This reduces membership leakage while lowering communication overhead. Yang et al.[27] propose Client-level Input Perturbation (CIP). It learns personalized input perturbations for different clients and keeps them locally. The goal is to make the model behavior of members and non-members more similar. At the model-parameter level, Shen et al.[23] propose MemDefense. It identifies parameters that have little effect on classification performance but carry strong membership signals. Pruning-based perturbation is then applied to weaken white-box MIAs. Another work, Accuracy-Lossless Model Perturbation[26], allows the server to perturb the distributed model using secret random parameters. The perturbations are exactly canceled after aggregation. This hides the true model information while preserving model updates that are nearly identical to those of the original FedAvg. Compared with generic random noise, these methods can suppress membership information in a more targeted manner and often achieve a better privacy-utility balance. However, they usually rely on specific information-sharing schemes, perturbation structures, or system assumptions. Their generalizability across different federated learning architectures and against stronger adaptive attacks therefore remains limited. Beyond conventional attacks, Deng et al.[6] reveal a new structural membership privacy risk in federated large language models (FedLLMs). They propose ProjRes, a passive membership inference attack based on projection residuals. ProjRes exploits the relationship between sample hidden representations and client-uploaded gradients. It can infer membership from only a single round of gradients. This makes it particularly effective for FedLLMs that use parameter-efficient fine-tuning methods such as Adapters and LoRA. ProjRes shows that the hidden representations of training samples can be indirectly identified through the subspace formed by gradients of trainable parameters. Existing defenses mainly rely on random noise, gradient clipping, or sparsification. However, weakening this type of attack often comes at the cost of model performance. Therefore, effectively disrupting the relationship between hidden representations and gradients while preserving model utility remains an important open problem in membership privacy protection for FedLLMs. Fig. 1: Overview of the proposed FISGuard. IV FISGuard Figure 1 provides an overview of FISGuard. The proposed method is not tied to a particular Adapter or LoRA design. It applies to any linear layer that is trained on the client, receives representations produced from private client samples, and exposes its update to the server. The method first identifies such layers from their common gradient structure, then builds a fixed input projection from public data, and finally rewrites each task-specific update as a fixed input projection followed by a trainable output mapping. Adapter and LoRA are two concrete implementations of this general construction. Step 1: Construct Fixed Input Subspaces and Reparameterize Target Linear Updates. Before federated training begins, the server performs a one-time model preparation step using only public data. The key idea is to remove trainable mappings that directly receive high-dimensional private representations, replace them with fixed public projections, and leave only the subsequent output mappings trainable on clients. In this way, clients can still adapt the model to their local tasks without learning or uploading the input-side mappings exploited by ProjRes. Step 1.1: The server first identifies trainable linear layers whose updates can directly expose private input representations and collects them in ℒtar.L_tar. Consider a linear layer yℓ=Wℓxℓ+bℓ,y_ =W_ x_ +b_ , where xℓ∈ℝdℓx_ ^d_ is the representation entering the layer and Wℓ∈ℝmℓ×dℓW_ ^m_ × d_ is its weight matrix. For a local batch of B samples, the weight gradient is ∇Wℓℒ=∑i=1Bδℓ,ixℓ,i⊤, _W_ L= _i=1^B _ ,ix_ ,i , where δℓ,i _ ,i is the error signal backpropagated to the layer output. This gradient structure is important because each private representation xℓ,ix_ ,i directly contributes to the uploaded update. As a result, the row space of the gradient is related to the space spanned by the private representations. ProjRes exploits exactly this relation: it measures how well a candidate representation lies in the gradient row space and uses the resulting projection residual for membership inference. We therefore protect a linear layer when three conditions hold: (1) the layer is trained on the client, (2) it directly receives representations generated from private samples, and (3) its update is observable by the server. This criterion is independent of a specific fine-tuning architecture and can apply to attention projections, feed-forward linear layers, adaptation modules, and linear task heads. For each target layer ℓ , the input dimension dℓd_ is determined by the model architecture, while the server selects a smaller subspace dimension rℓr_ , with rℓ<dℓ.r_ <d_ . Step 1.2: After identifying where protection is needed, the server must construct a replacement for the original trainable input-side mapping. Rather than choosing this mapping randomly, the server uses public data to identify the dominant representation directions of each target layer. Specifically, the server prepares a public dataset pub,D_pub, which is independent of all client data. It freezes the pretrained model, performs forward passes on the public samples, and records the representation entering each target layer. For layer ℓ , the representation produced by the j-th public sample is denoted by xℓ,jpub∈ℝdℓ.x_ ,j^pub ^d_ . The collected representations are stacked into Xℓpub=[(xℓ,1pub)⊤(xℓ,2pub)⊤(xℓ,Npubpub)⊤]∈ℝNpub×dℓ.X_ ^pub= bmatrix (x_ ,1^pub ) \\ (x_ ,2^pub ) \\ \\ (x_ ,N_pub^pub ) bmatrix ^N_pub× d_ . At this stage, the server only collects representations. No client gradients are used and no model parameters are updated. The matrix XℓpubX_ ^pub simply provides a public view of the representation space at the target layer and serves as the basis for constructing the fixed projection in the next step. Step 1.3: The server next extracts the dominant directions from the public representations. It applies singular value decomposition to Xℓpub=PℓΣℓVℓ⊤,X_ ^pub=P_ _ V_ , and selects the right singular vectors associated with the largest rℓr_ singular values. These directions are stacked to form Qℓ=Vℓ,1:rℓ⊤∈ℝrℓ×dℓ.Q_ =V_ ,1:r_ ^r_ × d_ . Because the selected directions are orthonormal, QℓQℓ⊤=Irℓ.Q_ Q_ =I_r_ . For any representation xℓx_ , the projection zℓ=Qℓxℓ∈ℝrℓz_ =Q_ x_ ^r_ keeps only its coordinates along these dominant public directions. Thus, QℓQ_ can be viewed as a fixed dimensionality-reduction mapping derived from public representations. It does not store individual public samples. Instead, it captures the main directions along which the pretrained representations vary. Compared with a random fixed projection, this construction is intended to preserve more of the information already encoded by the pretrained model while reducing the representation from dℓd_ dimensions to rℓr_ dimensions. Step 1.4: Once QℓQ_ has been constructed, the server uses it to replace the trainable input-side mapping at the target layer. Instead of learning an unrestricted task-specific update ΔWℓ W_ , we parameterize it as ΔWℓ=UℓQℓ, W_ =U_ Q_ , where Uℓ∈ℝmℓ×rℓU_ ^m_ × r_ is the only layer-specific mapping optimized on private client data. The resulting transformation becomes yℓ=W0,ℓxℓ+UℓQℓxℓ,y_ =W_0, x_ +U_ Q_ x_ , where W0,ℓW_0, is the frozen pretrained weight. Equivalently, the computation can be understood in two stages zℓ=Qℓxℓ,z_ =Q_ x_ , followed by Δyℓ=Uℓzℓ. y_ =U_ z_ . The roles of the two matrices are therefore separated clearly. The fixed matrix QℓQ_ determines which directions of the high-dimensional input are retained, while the trainable matrix UℓU_ learns how to use those retained directions for the downstream task. The server freezes both W0,ℓW_0, and QℓQ_ , and only UℓU_ is optimized on clients. For an ordinary fully connected layer that does not already contain a low-rank adaptation structure, the server adds UℓQℓU_ Q_ as a task-specific branch on top of the frozen pretrained weight yℓ=W0,ℓxℓ+UℓQℓxℓ.y_ =W_0, x_ +U_ Q_ x_ . For a module that already contains an input projection followed by an output projection, the same principle is applied directly: the input-side mapping is replaced by the fixed public projection QℓQ_ , while the output-side mapping remains trainable. To ensure that this additional branch does not change the pretrained model at initialization, the server sets Uℓ0=0.U_ ^0=0. Then Uℓ0Qℓxℓ=0,U_ ^0Q_ x_ =0, and therefore yℓ=W0,ℓxℓ.y_ =W_0, x_ . The model thus starts from the original pretrained function and gradually learns task-specific corrections through UℓU_ . Bottleneck Adapter. A standard Bottleneck Adapter computes Δyℓ=Wup,ℓσ(Wdown,ℓxℓ). y_ =W_up, σ\! (W_down, x_ ). Wdown,ℓW_down, is the first matrix that acts on the high-dimensional representation xℓx_ . It determines how the original representation is compressed into the bottleneck space. The following matrix Wup,ℓW_up, then maps this low-dimensional representation back to the output space. FISGuard uses the identity activation, σ(a)=aσ(a)=a, in the protected Adapter branch. We replace the trainable down projection with the fixed public projection, Wdown,ℓ=Qℓ,W_down, =Q_ , and keep the up projection trainable, Wup,ℓ=Uℓ.W_up, =U_ . The Adapter branch therefore becomes Δyℓ=UℓQℓxℓ. y_ =U_ Q_ x_ . The public projection QℓQ_ fixes the retained input directions. The trainable mapping UℓU_ learns how to use the resulting low-dimensional representation. This yields the Fixed-Down Adapter instantiation of FISGuard. LoRA. Standard LoRA writes the task-specific weight update as ΔWℓ=BℓAℓ, W_ =B_ A_ , which gives Δyℓ=BℓAℓxℓ. y_ =B_ A_ x_ . The factor AℓA_ is the first matrix applied to the high-dimensional input and maps it into the low-rank space. The factor BℓB_ then maps this low-rank representation to the output space. Thus, AℓA_ and BℓB_ play the same input-side and output-side roles as QℓQ_ and UℓU_ , respectively. We therefore set Aℓ=Qℓ,Bℓ=Uℓ,A_ =Q_ ,B_ =U_ , which yields Δyℓ=UℓQℓxℓ. y_ =U_ Q_ x_ . Again, the client no longer learns the input-side low-rank projection. The public projection QℓQ_ remains fixed, while only UℓU_ is optimized during federated training. This gives the LoRA instantiation of the same general reparameterization principle. Both Bottleneck Adapter and LoRA therefore reduce to the common form ΔWℓ=UℓQℓ. W_ =U_ Q_ . They are two examples of the proposed method rather than its scope. More generally, the same construction can be applied whenever a task-specific linear update can be expressed as a fixed input projection followed by a trainable output mapping. Step 2: Distribute Global Trainable Parameters. After constructing the fixed input subspaces, the system enters multi-round federated training. At the beginning of each round, the server selects participating clients and sends the current trainable parameters Φ=Uℓ∣ℓ∈ℒtar∪Θaux, = \U_ _tar \∪ _aux, where Θaux _aux denotes other task parameters that are not target linear weights. If any linear weight in this set also satisfies the conditions in Step 1.1, it must be included in ℒtarL_tar rather than left unprotected. The base weights W0,ℓW_0, and fixed projections QℓQ_ are shared across clients and remain frozen throughout training. Each client initializes its local model with the received global parameters. Step 3: Perform Local Training in the Fixed Input Subspace. Each participating client trains on its private dataset kD_k. For every target layer ℓ , the client computes zℓ=Qℓxℓ,Δyℓ=Uℓzℓ,z_ =Q_ x_ , y_ =U_ z_ , which gives yℓ=W0,ℓxℓ+UℓQℓxℓ.y_ =W_0, x_ +U_ Q_ x_ . The client updates only UℓU_ and Θaux _aux, while ∇W0,ℓℒ=0,∇Qℓℒ=0. _W_0, L=0, _Q_ L=0. Private data therefore no longer train an input matrix that acts directly on the original high-dimensional representation. They only train how the fixed low-dimensional representation should be mapped to the task output. The gradient of the output mapping is ∇Uℓℒ=∑i=1Bδℓ,i(Qℓxℓ,i)⊤. _U_ L= _i=1^B _ ,i (Q_ x_ ,i ) . The representation appearing in this gradient is the low-dimensional projected representation rather than the original high-dimensional input. Step 4: Upload Local Updates. After local training, client k uploads only the updates of the output mappings and other trainable parameters ΔΦk=ΔUℓ,k∣ℓ∈ℒtar∪ΔΘaux,k. _k= \ U_ ,k _tar \∪ _aux,k. Because QℓQ_ remains fixed, the client does not upload its gradient or parameter change. The input-side parameters that directly receive high-dimensional private representations are therefore absent from the server-visible update set. For Adapter, the client uploads only the up-projection update; for LoRA, it uploads only the output-side low-rank factor; other target linear layers upload their corresponding UℓU_ updates. Step 5: Aggregate Client Updates. The server receives the local updates and applies an existing federated aggregation rule. Under FedAvg, the aggregation weight of client k is pk=|k|∑j∈|j|,p_k= |D_k| _j |D_j|, where S is the set of participating clients in the current round. The server updates Φ←Φ+∑k∈pkΔΦk. ← + _k p_k _k. The fixed projections QℓQ_ and base weights W0,ℓW_0, remain unchanged during aggregation. The server then starts the next round and repeats parameter distribution, local training, update upload, and aggregation until the training budget is exhausted or the model converges. In summary, Step 1 is executed once before federated training, whereas Steps 2–5 are repeated in every round. The method does not change the basic federated optimization and aggregation workflow; it changes only how target linear layers are parameterized and which parameters clients train and upload. V Experiments Experimental Environment. All experiments were conducted on a workstation running Windows 11, equipped with a 13th-generation Intel Core i5-13600KF CPU operating at 3.50 GHz, an NVIDIA GeForce RTX 3090 Ti GPU with 24 GB of VRAM, and 64 GB of system memory. V-A Experimental Setup Datasets. We conduct experiments on three widely used NLP datasets: CoLA, SST-5, and IMDb. CoLA is a binary linguistic acceptability classification dataset consisting of English sentences labeled as grammatically acceptable or unacceptable. We use 8,551 samples for training and 1,043 samples for validation. SST-5, derived from the Stanford Sentiment Treebank, is a fine-grained sentiment classification dataset with five sentiment categories ranging from very negative to very positive. We use 8,544 training samples and 1,101 validation samples. IMDb is a binary sentiment classification dataset composed of movie reviews labeled as positive or negative. From its official 25,000-sample training set, we randomly split 22,500 samples for training and 2,500 samples for validation. Models. We consider two representative Transformer-based language models: BERT-Base and GPT-2 Large. BERT-Base is a bidirectional Transformer encoder with 12 layers, a hidden size of 768, and 12 attention heads, containing approximately 110M parameters. GPT-2 Large is an autoregressive Transformer language model with 36 layers, a hidden size of 1,280, and approximately 762M parameters. Baselines. We consider five representative defense methods as baselines: DP-SGD, L2 Regularization, Min-Max, Dropout, and Label Smoothing. DP-SGD limits the influence of individual training samples on model updates through gradient clipping and noise injection. L2 Regularization constrains the magnitude of PEFT parameters to mitigate overfitting. Min-Max jointly optimizes the target model and a membership inference adversary to explicitly reduce the distinguishability between members and non-members. Dropout randomly masks intermediate features during training to reduce feature co-adaptation and sample-specific memorization, while Label Smoothing softens the training targets to alleviate overconfident predictions on training samples. Collectively, these baselines mitigate membership information leakage from complementary perspectives, including gradient perturbation, regularization, and adversarial training, and serve as representative defenses for comparison with FISGuard. Hyperparameter Settings. We use a unified training configuration across all defense methods to ensure a fair comparison. Specifically, the federated learning system consists of 3030 clients, all of which participate in every communication round, and training proceeds for 5050 communication rounds. Each client performs one local update step per round with a local batch size of 1616. We use a learning rate of 2×10−42× 10^-4 for the PEFT modules. The learning rate of the task-specific classification head is set to 5×10−45× 10^-4 for CoLA and IMDB and 1×10−31× 10^-3 for SST-5. Gradients are clipped to a global norm of 1.01.0. In addition, we use a public reference set containing 2,0002,000 samples. For each defense baseline, we sweep its primary defense-specific hyperparameter while keeping all other training configurations unchanged. For DP-SGD, we fix the clipping norm to C=1.0C=1.0 and δ=10−5δ=10^-5, and vary the noise multiplier over σ∈0.1,0.25,0.5,1,2σ∈0.1,0.25,0.5,1,2. For L2 Regularization, we sweep the regularization coefficient over λ∈10−4,10−3,10−2,10−1λ∈10^-4,10^-3,10^-2,10^-1. For Min-Max, we vary the adversarial loss weight over λ∈0.1,0.3,1,3,10λ∈0.1,0.3,1,3,10, set the adversary learning rate to 3×10−33× 10^-3, perform five adversary updates per mini-batch, use a hidden dimension of 128128, and sample 256256 held-out examples as non-member references. For Dropout, we consider dropout rates p∈0.05,0.1,0.2,0.4p∈0.05,0.1,0.2,0.4, while for Label Smoothing, we vary the smoothing coefficient over α∈0.02,0.05,0.1,0.2,0.4α∈0.02,0.05,0.1,0.2,0.4. For membership inference evaluation, we construct a balanced evaluation set containing 100100 member samples and 100100 non-member samples. Fig. 2: Projection residual score distributions of member and non-member samples with and without FISGuard. Fig. 3: Overall comparison of ProjRes attack AUC and downstream model accuracy. Fig. 4: Training time comparison across different datasets under Adapter- and LoRA-based fine-tuning. V-B Defense Effectiveness and Model Utility We compare FISGuard against five representative defenses against membership inference attacks across three datasets, two LLMs, and two fine-tuning strategies, resulting in 12 experimental configurations. Figure 3 summarizes their overall privacy and utility performance across these configurations. Detailed comparisons under different hyperparameter settings are provided in Appendix A. Privacy Protection. ProjRes performs membership inference using the projection residual score s(x)=‖h(x)−h(x)VGVG⊤‖1s(x)= \|h(x)-h(x)V_GV_G \|_1, where VGV_G denotes an orthonormal basis of the subspace induced by the observed gradient and h(x)h(x) is the representation of candidate x. A smaller residual indicates that the candidate representation is better covered by the gradient subspace and is therefore more likely to be inferred as a member. As shown in Fig. 2, the undefended setting exhibits a clear separation between member and non-member scores. After applying FISGuard, the gap between the two distributions is substantially reduced and their overlap becomes much larger, indicating that the projection residual provides much weaker membership-discriminative information. The AUC results in Fig. 3 further confirm this observation. FISGuard produces an attack AUC distribution that is substantially more concentrated around 0.5 than those of the competing defenses. Across the 12 configurations, its attack AUC remains close to the random-guessing level in most cases, demonstrating consistent protection across different datasets, models, and fine-tuning strategies. This behavior follows directly from the design of FISGuard. By freezing the input-side projection, the original high-dimensional private representations no longer directly contribute to the server-visible input-side gradient structure. Both member and non-member representations are instead mapped into the same fixed r-dimensional public subspace, which can be more sufficiently covered by the row space of the remaining observable gradients. As the coverage increases, the projection residuals of members and non-members become increasingly similar. When the gradient row space fully spans the projected space, both are completely covered, eliminating the residual difference exploited by ProjRes. Model Utility. The accuracy results in Fig. 3 further show that the improved privacy protection of FISGuard does not incur substantial degradation in model utility. Its accuracy remains close to the undefended baseline and is generally more favorable than those of the competing defenses across the 12 configurations. Unlike defenses that directly introduce noise or strong optimization constraints, FISGuard constructs its fixed input projection from public representations before private training. Specifically, singular value decomposition is applied to representations collected from public data, and the top-r dominant representation directions are retained to form the fixed projection Q. These directions preserve the principal information encoded in the pretrained representation space. During private training, Q remains frozen while only the output-side mapping U is optimized, allowing the model to adapt the retained representations to downstream tasks while restricting the gradient information exploitable by ProjRes. Taken together, these results show that FISGuard effectively suppresses the membership-discriminative signal exploited by ProjRes while preserving competitive downstream task performance, achieving a favorable privacy–utility trade-off across different datasets, models, and fine-tuning strategies. V-C Computation Overhead Analysis As shown in Fig.4, across different datasets, FISGuard introduces only limited computational overhead. Under Adapter-based fine-tuning, FISGuard achieves the lowest overall training time among the evaluated defense methods, with runtime remaining close to the undefended baseline. Under LoRA-based fine-tuning, its training time also stays close to the baseline, indicating that the additional privacy protection incurs only modest computational cost. This efficiency is mainly because the public-data-derived input projection is constructed before private training and remains frozen throughout fine-tuning, so no gradient computation or parameter update is required for this component. Moreover, FISGuard does not introduce additional per-step procedures such as gradient perturbation or adversarial optimization. Overall, the runtime results show that FISGuard provides effective privacy protection without substantially increasing the computational cost of parameter-efficient fine-tuning. VI Security Analysis This section analyzes how fixed public-subspace reparameterization affects ProjRes. The guarantee established below is specific to the projection-residual score used by ProjRes. It is neither a differential-privacy guarantee nor a universal guarantee against all membership-inference attacks. VI-A Setting Consider a protected linear layer whose task-specific update is reparameterized as ΔW=UQ W=UQ, so that the output corresponding to sample i is yi=W0xi+UQxiy_i=W_0x_i+UQx_i. xi∈ℝdx_i ^d is the high-dimensional representation entering the layer, W0W_0 is the frozen base weight, Q∈ℝr×dQ ^r× d is a fixed input projection constructed from public data, and U∈ℝm×rU ^m× r is the output mapping trained and uploaded by the client. We assume that r<dr<d. Define the projected representation as zi:=Qxi∈ℝrz_i:=Qx_i ^r. Let δi∈ℝm _i ^m denote the backpropagated error signal associated with sample i. Absorbing any nonzero batch-normalization factor into δi _i, the gradient of the loss with respect to U is GU:=∇Uℒ=∑i=1Bδizi⊤∈ℝm×r.G_U:= _UL= _i=1^B _iz_i ^m× r. Because Q is excluded from the client-side optimization variables, the client uploads neither a gradient nor a parameter difference for Q. The server therefore observes GUG_U, rather than an input-side update directly constructed from the original high-dimensional representations xix_i. For a candidate sample x, let z:=Qxz:=Qx. The ProjRes score computed from the observed update GUG_U is sPR(x,GU):=‖z−Πrow(GU)z‖p,s_PR(x;G_U):= \|z- _row(G_U)z \|_p, where Πrow(GU) _row(G_U) denotes the orthogonal projection onto the row space of GUG_U. VI-B Invalidation of the ProjRes Score Theorem VI.1 (ProjRes invalidation under a full-rank fixed subspace). Suppose that the output-side update observed by the server satisfies rank(GU)=rrank(G_U)=r. Then, for every candidate sample x, regardless of whether x belongs to the client’s training data, sPR(x,GU)=0s_PR(x;G_U)=0. Consequently, ProjRes cannot rank or classify members and non-members using the projection-residual score. Proof. Because GU∈ℝm×rG_U ^m× r, its row space is a subspace of ℝrR^r, namely, row(GU)⊆ℝr.row(G_U) ^r. Under the condition rank(GU)=rrank(G_U)=r, the dimension of the row space equals the dimension of the entire projected representation space. Therefore,row(GU)=ℝr.row(G_U)=R^r. Orthogonal projection onto all of ℝrR^r is the identity map, so Πrow(GU)=Ir _row(G_U)=I_r. For any candidate sample x, the projected representation z=Qxz=Qx belongs to ℝrR^r. Πrow(GU)z=Irz=z. _row(G_U)z=I_rz=z. It follows that sPR(x,GU)=‖z−z‖p=0.s_PR(x;G_U)= \|z-z \|_p=0. This conclusion is independent of the membership status of x. Thus, the ProjRes score collapses to a constant and contains no membership-discriminating information. ∎ Corollary VI.2 (Zero membership advantage for projection-residual attacks). Let M∈0,1M∈\0,1\ denote the membership variable, where M=1M=1 represents a member and M=0M=0 represents a non-member. Define the threshold-based ProjRes advantage as AdvPR:=supτ∈ℝ|Pr[sPR(x,GU)≤τ|M=1]−Pr[sPR(x,GU)≤τ|M=0]|.Adv_PR:= _τ | \! [s_PR(x;G_U)≤τ\, |\,M=1 ]\\ - \! [s_PR(x;G_U)≤τ\, |\,M=0 ] |. Under the conditions of Theorem VI.1, AdvPR=0Adv_PR=0. Under the standard convention that tied scores receive half credit, the corresponding ROC AUC is AUC=1/2AUC=1/2. Proof. By Theorem VI.1, sPR(x,GU)=0s_PR(x;G_U)=0 for both members and non-members. Therefore, for every threshold τ, Pr[sPR(x,GU)≤τ|M=1]=Pr[sPR(x,GU)≤τ|M=0]. \! [s_PR(x;G_U)≤τ\, |\,M=1 ]= \! [s_PR(x;G_U)≤τ\, |\,M=0 ]. The two conditional score distributions are identical, so their probability difference is zero for every τ. Consequently, AdvPR=0Adv_PR=0. Moreover, all candidates receive the same score. ProjRes produces no nontrivial ranking between members and non-members. With the standard treatment of tied scores, its ROC AUC is 1/21/2. ∎ VI-C A Sufficient Condition for Full Rank Define the projected-representation matrix Z and the error-signal matrix D as Z:=[z1z2⋯zB]∈ℝr×B,Z:= bmatrixz_1&z_2&·s&z_B bmatrix ^r× B, D:=[δ1δ2⋯δB]∈ℝm×B.D:= bmatrix _1& _2&·s& _B bmatrix ^m× B. The output-side gradient can then be written as GU=DZ⊤G_U=DZ . Proposition VI.3 (Sufficient condition for a full-rank output update). Suppose that rank(Z)=rrank(Z)=r and ker(D)∩col(Z⊤)=0. (D) (Z )=\0\. Then rank(GU)=rrank(G_U)=r. Proof. Consider any vector a∈ℝra ^r satisfying DZ⊤a=0DZ a=0. It follows that Z⊤a∈ker(D)Z a∈ (D). By construction, Z⊤a∈col(Z⊤)Z a (Z ). Therefore, Z⊤a∈ker(D)∩col(Z⊤).Z a∈ (D) (Z ). The assumed intersection condition implies that Z⊤a=0Z a=0. Because rank(Z)=rrank(Z)=r, the matrix Z⊤Z has full column rank and therefore has a trivial null space. a=0a=0. Thus, the linear map DZ⊤:ℝr→ℝmDZ ^r ^m is injective. Consequently, rank(DZ⊤)=r.rank(DZ )=r. Since GU=DZ⊤G_U=DZ , we obtain rank(GU)=rrank(G_U)=r. ∎ VI-D Interpretation and Design Condition Proposition VI.3 has a direct geometric interpretation. The projected batch representations must span all r directions of the fixed public subspace, while the backpropagated error signals must not eliminate any nonzero direction in that span. Because rank(GU)≤minm,r,B,rank(G_U)≤ \m,r,B\, the full-rank condition rank(GU)=rrank(G_U)=r requires r≤mr≤ m and r≤Br≤ B. For sequence models, B may be replaced by the effective number of linearly independent representations contained in the observed update, denoted by BeffB_eff. A practical dimensional condition is therefore r≤minm,Beff.r≤ \m,B_eff\. This dimensional condition is necessary for full rank but is not, by itself, sufficient. The spanning and non-cancellation conditions in Proposition VI.3 must also hold. VII Conclusion In this paper, we presented FISGuard, a lightweight defense against the ProjRes membership inference attack in federated parameter-efficient fine-tuning. FISGuard constructs a fixed low-dimensional input subspace from independent public data and restricts trainable updates to operate within this subspace, thereby reducing the projection-residual discrepancy between members and non-members while preserving the information required for downstream tasks. We evaluated FISGuard across three NLP datasets, two large language models, and two fine-tuning strategies, and compared it with five representative membership-inference defenses. The results show that FISGuard reduces the AUC of the ProjRes attack to near the random-guessing level in most settings, while maintaining downstream task performance close to that of the undefended model and introducing only limited computational overhead. Our analysis further characterizes the conditions under which the server-observable update span fully covers the fixed representation subspace, explaining why the projection-residual signal exploited by ProjRes can be substantially weakened. Overall, FISGuard provides an effective privacy–utility trade-off against projection-residual-based membership inference without requiring heavy gradient perturbation or regularization. References [1] A. Abourayya, J. Kleesiek, K. Rao, E. Ayday, B. Rao, G. I. Webb, and M. Kamp (2025) Little is enough: boosting privacy by sharing only hard labels in federated semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 39, p. 15293–15301. Cited by: §I. [2] F. Ahmed, D. Sanchez, Z. Haddi, and J. Domingo-Ferrer (2025) MemberShield: a framework for federated learning with membership privacy. Neural Networks 181, p. 106768. Cited by: §I. [3] L. Bai, J. Liu, S. Zhang, X. Zhang, Q. Ye, and H. Hu (2026) United we defend: collaborative membership inference defenses in federated learning. arXiv preprint arXiv:2601.06866. Cited by: §I. [4] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth (2016) Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482. Cited by: §I. [5] X. Chen, S. Z. Wu, and M. Hong (2020) Understanding gradient clipping in private sgd: a geometric perspective. Advances in neural information processing systems 33, p. 13773–13782. Cited by: §I. [6] G. Deng, S. Chen, Y. Luo, Y. Liu, S. Wang, Z. Cai, L. Liu, X. Jia, and S. Fu (2026) Toward efficient membership inference attacks against federated large language models: a projection residual approach. arXiv preprint arXiv:2604.21197. Cited by: §I, §I, §I. [7] H. Du, S. Liu, L. Zheng, Y. Cao, A. Nakamura, and L. Chen (2025) Privacy in fine-tuning large language models: attacks, defenses, and future directions. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, p. 326–344. Cited by: §I. [8] C. Dwork (2006) Differential privacy. In International colloquium on automata, languages, and programming, p. 1–12. Cited by: §I. [9] Y. He, B. Li, L. Liu, Z. Ba, W. Dong, Y. Li, Z. Qin, K. Ren, and C. Chen (2025) Towards label-only membership inference attack against pre-trained large language models. In 34th USENIX Security Symposium (USENIX Security 25), p. 1609–1628. Cited by: §I. [10] A. Hilmkil, S. Callh, M. Barbieri, L. R. Sütfeld, E. L. Zec, and O. Mogren (2021) Scaling federated learning for fine-tuning of large language models. In International Conference on Applications of Natural Language to Information Systems, p. 15–23. Cited by: §I-A. [11] H. Hu, Z. Salcic, L. Sun, G. Dobbie, P. S. Yu, and X. Zhang (2022) Membership inference attacks on machine learning: a survey. ACM Computing Surveys (CSUR) 54 (11s), p. 1–37. Cited by: §I. [12] L. Hu, J. Li, G. Lin, S. Peng, Z. Zhang, Y. Zhang, and C. Dong (2022) Defending against membership inference attacks with high utility by gan. IEEE Transactions on Dependable and Secure Computing 20 (3), p. 2144–2157. Cited by: §I. [13] Y. Hu, Y. Wang, J. Lou, W. Liang, R. Wu, W. Wang, X. Li, J. Liu, and Z. Qin (2024) Privacy risks of federated knowledge graph embedding: new membership inference attacks and personalized differential privacy defense. IEEE Transactions on Dependable and Secure Computing 22 (3), p. 2788–2805. Cited by: §I. [14] I. Jarin and B. Eshete (2022) MIAShield: defending membership inference attacks via preemptive exclusion of members. arXiv preprint arXiv:2203.00915. Cited by: §I. [15] W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou (2024) Federatedscope-llm: a comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 5260–5271. Cited by: §I. [16] J. Li, N. Li, and B. Ribeiro (2021) Membership inference attacks and defenses in classification models. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, p. 5–16. Cited by: §I. [17] D. Liu and T. Miller (2020) Federated pretraining and fine tuning of bert using clinical notes from multiple silos. arXiv preprint arXiv:2002.08562. Cited by: §I-A. [18] F. Mireshghallah, H. Inan, M. Hasegawa, V. Rühle, T. Berg-Kirkpatrick, and R. Sim (2021) Privacy regularization: joint privacy-utility optimization in languagemodels. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 3799–3807. Cited by: §I. [19] M. Naseri, J. Hayes, and E. De Cristofaro (2020) Local and central differential privacy for robustness and privacy in federated learning. arXiv preprint arXiv:2009.03561. Cited by: §I. [20] M. Nasr, R. Shokri, and A. Houmansadr (2018) Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, p. 634–646. Cited by: §I. [21] L. Prechelt (2002) Early stopping-but when?. In Neural Networks: Tricks of the trade, p. 55–69. Cited by: §I. [22] T. Qi, H. Wang, and Y. Huang (2024) Towards the robustness of differentially private federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 19911–19919. Cited by: §I. [23] M. Shen, J. Meng, K. Xu, S. Yu, and L. Zhu (2024) MemDefense: defending against membership inference attacks in iot-based federated learning via pruning perturbations. IEEE Transactions on Big Data 11 (5), p. 2135–2147. Cited by: §I. [24] Y. Shi, K. Wei, L. Shen, Y. Liu, X. Wang, B. Yuan, and D. Tao (2025) Towards the flatter landscape and better generalization in federated learning under client-level differential privacy. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §I. [25] R. Shokri, M. Stronati, C. Song, and V. Shmatikov (2017) Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), p. 3–18. Cited by: §I. [26] X. Yang, Y. Feng, W. Fang, J. Shao, X. Tang, S. Xia, and R. Lu (2022) An accuracy-lossless perturbation method for defending privacy attacks in federated learning. In Proceedings of the ACM Web Conference 2022, p. 732–742. Cited by: §I. [27] Y. Yang, H. Yuan, B. Hui, N. Gong, N. Fendley, P. Burlina, and Y. Cao (2023) Fortifying federated learning against membership inference attacks via client-level input perturbation. In 2023 53rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), p. 288–301. Cited by: §I. [28] Z. Ying, Y. Zhang, and X. Liu (2020) Privacy-preserving in defending against membership inference attacks. In Proceedings of the 2020 workshop on privacy-preserving machine learning in practice, p. 61–63. Cited by: §I. [29] J. Yuan, Y. Chen, Z. Wang, C. Wang, X. Hu, and Z. Zeng (2025) DP-fedpuac: federated learning with differential privacy via adaptive gradient clipping and local iteration optimization. Information Sciences, p. 122981. Cited by: §I. [30] Z. Zhao, Z. Lin, and Y. Sun (2025) ADPF: anti-inference differentially private protocol for federated learning. Computer Networks 261, p. 111130. Cited by: §I. Appendix A Detailed Results While the main text provides an aggregate comparison across the 12 experimental configurations, this section presents detailed results under the different hyperparameter settings specified in Section V-A. Specifically, we report the ProjRes attack AUC and the corresponding downstream model accuracy as the defense-specific hyperparameters vary across all combinations of three datasets, two LLMs, and two fine-tuning strategies. For FISGuard, the defense parameter is the rank r, which determines the dimensionality of the fixed public subspace constructed from the top-r dominant directions of public representations. These detailed results provide a more fine-grained view of how different defense parameters affect privacy protection and model utility. Attack AUC. Figure 5 presents the ProjRes attack AUC under the complete set of evaluated hyperparameter settings. The results provide a detailed comparison of how the effectiveness of each defense changes with its defense-specific parameter across different datasets, models, and fine-tuning strategies. For conventional defenses, varying the amount of noise, regularization strength, adversarial penalty, dropout probability, or label-smoothing strength changes the extent to which the original optimization process is perturbed, which in turn affects their ability to suppress the membership signal exploited by ProjRes. For FISGuard, varying r directly changes the dimensionality of the fixed subspace in which the remaining observable gradients are formed. Across the evaluated settings, FISGuard keeps the ProjRes AUC close to the random-guessing level in most configurations, further supporting the aggregate results reported in the main text. The effect of r also admits a direct geometric interpretation. A smaller r compresses both member and non-member representations into a lower-dimensional shared space, making a larger portion of this space easier to cover with the row space of the observable gradient. As this coverage increases, member and non-member representations share increasingly overlapping covered components, and the difference between their projection residuals correspondingly decreases. In the limiting case where the gradient row space spans the entire r-dimensional projected space, both types of candidates are fully covered and the ProjRes residual becomes identical regardless of membership. Conversely, increasing r retains a larger representation space but also increases the dimensionality that must be covered by the observable gradient span. Model Accuracy. Figure 6 reports the downstream model accuracy corresponding to the same hyperparameter sweeps. For several existing defenses, changing the defense strength directly alters the degree of perturbation or regularization applied during training. Stronger perturbation can suppress membership leakage more effectively, but may simultaneously interfere with normal optimization and reduce downstream task performance. The role of the hyperparameter r in FISGuard is different. Rather than controlling the magnitude of noise or gradient perturbation, r determines how many dominant representation directions extracted from public data are retained in the fixed input subspace. The projection matrix Q is constructed from the top-r right singular vectors of the public representation matrix, such that the retained dimensions correspond to the principal directions of variation in the pretrained representation space. Increasing r preserves a larger portion of the original representation space and provides the trainable output-side mapping with richer information for downstream adaptation, whereas reducing r imposes stronger dimensionality reduction. Therefore, r directly governs the balance between retaining task-relevant representation information and reducing the dimensionality of the observable space exploited by ProjRes. The accuracy results across different values of r empirically illustrate how this representation-space constraint affects downstream model utility. Fig. 5: Attack AUC of ProjRes under different defense methods across three datasets, two LLMs, and two fine-tuning strategies. Fig. 6: Model accuracy under different defense methods across three datasets, two LLMs, and two fine-tuning strategies.