Paper deep dive

Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

Qinglin Yang, Chen Qiu, Hongyuan Zhang, Pengdeng Li, Yuan Liu, Zhihong Tian

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 140

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/17/2026, 4:27:56 AM

Summary

This paper presents a comprehensive survey of Federated Prompt Learning (FPL), a unified framework integrating Federated Learning (FL) with Large Language Models (LLMs). It addresses the limitations of centralized LLM training, such as high computational costs, data centralization, and privacy concerns, by leveraging lightweight prompt-based adaptation in a decentralized setting. The survey analyzes FPL across the full model lifecycle (pre-training, fine-tuning, applications), evaluates trade-offs in performance, communication efficiency, and scalability, and discusses security, privacy, and robustness challenges.

Entities (10)

Chen Qiu · person · 99%Hongyuan Zhang · person · 99%Large Language Models · technology · 99%Qinglin Yang · person · 99%Federated Prompt Learning · methodology · 98%Federated Learning · paradigm · 97%Privacy · challenge · 95%Communication Efficiency · metric · 93%Parameter-Efficient Fine-Tuning · technique · 92%LoRA · technique · 90%

Relation Signals (6)

Qinglin Yang → isauthorof → Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

confidence 99% · QINGLIN YANG... Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

Chen Qiu → isauthorof → Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

confidence 99% · CHEN QIU... Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

Large Language Models → faceschallengesfrom → Privacy

confidence 95% · training and deployment are hindered by high computational costs, data centralization, and privacy concerns.

Federated Prompt Learning → isavariantof → Federated Learning

confidence 95% · This paper presents a comprehensive survey of federated prompt learning (FPL) to review recent advances in integrating the federated learning paradigm and large language models

Federated Prompt Learning → utilizes → Parameter-Efficient Fine-Tuning

confidence 92% · prompt learning and parameter-efficient fine-tuning (PEFT) have gained significant attention as lightweight alternatives to full-model fine-tuning... This observation has led to a new paradigm, federated prompt learning (FPL)

Federated Prompt Learning → improves → Communication Efficiency

confidence 90% · This design dramatically reduces communication costs... while maintaining strong privacy guarantees.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) have become core components of cloud-based intelligent services in academia and industry, yet their training and deployment are hindered by high computational costs, data centralization, and privacy concerns. Federated learning (FL) offers a decentralized training paradigm that enables clients to collaboratively train a learning model without sharing raw data, making it a promising solution for privacy-preserving LLM training and reasoning. This paper presents a comprehensive survey of federated prompt learning (FPL) to review recent advances in integrating the federated learning paradigm and large language models, answering the following research questions: RQ1: The fundamental motivations, characteristics, and enabling technologies of FPL, and how it differs from conventional FL and full-model federated fine-tuning; RQ2: The trade-offs FPL approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling; RQ3: The remaining security, privacy, robustness, and system challenges, along with key future research directions. To this end, we systematically examine existing FPL methods across the full model lifecycle: pre-training, fine-tuning, and practical applications, while discussing security, privacy, and robustness issues and summarizing existing defense mechanisms. Finally, we highlight open challenges and future directions, aiming to help readers understand how the insights drive research in FPL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

136,323 characters extracted from source content.

Expand or collapse full text

, , Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions QINGLIN YANG,Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Labora- tory of Industrial Control System Security; Huangpu Research School of Guangzhou University, China CHEN QIU, Iwate Biotechnology Research Center, Japan HONGYUAN ZHANG,Nanjing University of Posts and Telecommunications, China PENGDENGLI,YUANLIU ∗ ,andZHIHONGTIAN,Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, China Large language models (LLMs) have become core components of cloud-based intelligent services in academia and industry, yet their training and deployment are hindered by high computational costs, data centralization, and privacy concerns. Federated learning (FL) offers a decentralized training paradigm that enables clients to collaboratively train a learning model without sharing raw data, making it a promising solution for privacy-preserving LLM training and reasoning. This paper presents a comprehensive survey of federated prompt learning (FPL) to review recent advances in integrating the federated learning paradigm and large language models, answering the following research questions: RQ1: The fundamental motivations, characteristics, and enabling technologies of FPL, and how it differs from conventional FL and full-model federated fine-tuning; RQ2: The trade-offs FPL approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling; RQ3: The remaining security, privacy, robustness, and system challenges, along with key future research directions. To this end, we systematically ex- amine existing FPL methods across the full model lifecycle: pre-training, fine-tuning, and practical applications, while discussing security, privacy, and robustness issues and summarizing existing defense mechanisms. Finally, we highlight open challenges and future directions, aiming to help readers understand how the insights drive research in FPL. CCS Concepts: •Computing methodologies→Natural language processing;Machine learning; •Security and privacy→ Privacy-preserving protocols; •Computer systems organization→Distributed architectures. Additional Key Words and Phrases: Large Language Models, Collaborative Learning, Prompt Learning, Federated Fine-tuning, Parameter- Efficient Fine-tuning, Security ∗ The Corresponding author. Authors’ Contact Information:Qinglin Yang,yangqinglin@gzhu.edu.cn, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, Guangzhou, Guangdong, China; Chen Qiu,c-qiu@ibrc.or.jp, Department of Horticultural Science, Iwate Biotechnology Research Center, Iwate, Japan; Hongyuan Zhang, b23110420@njupt.edu.cn, Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, China; Pengdeng Li; Yuan Liu; Zhihong Tian,pdli, yuanliu,tianzhihong@gzhu.edu.cn, Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University, Guangzhou, Guangdong, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM1 arXiv:2608.13844v1 [cs.LG] 14 Aug 2026 2Q.Yang C.Qiu, H.Zhang et al. ACM Reference Format: Qinglin Yang, Chen Qiu, Hongyuan Zhang, Pengdeng Li, Yuan Liu, and Zhihong Tian. 2026. Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions.J. ACM0, 0, Article 1 (August 2026),35pages.https://doi.org/x 1 Introduction Large language models (LLMs) have become the cornerstone of modern artificial intelligence, demonstrating remark- able capabilities in natural language understanding, reasoning, and generation across a wide range of domains, in- cluding healthcare, finance, software engineering, and education [4]. However, the rapid scaling of LLMs has exposed fundamental limitations of the prevailing centralized training and deployment paradigm. For training, the reliance on massive centralized datasets raises severe privacy and regulatory concerns, while the growing scarcity of high-quality public data and the prohibitive computational and communication costs increasingly constrain further model scaling. These challenges motivate the exploration of new collaborative and privacy-preserving learning paradigms for the development and adaptation of LLMs. FL has emerged as a promising decentralized framework that enables multiple data owners to collaboratively train machine learning models without sharing raw data [ 59]. When FL meets LLMs, it raises some diverse challenges compared with conventional FL patterns. (i) The distinctive significance of FL for LLMs is the stronger demand for multi- source heterogeneous data. (i) Conventional LLMs usually rely on centralized training, in which massive amounts of data are aggregated onto a unified platform before pre-training or fine-tuning is conducted. However, in real-world scenarios, many forms of high-value data cannot be centralized, such as internal enterprise documents, medical data, financial data, legal data, personal device data, and cross-institutional business data. (i) In terms of model architecture, FL promotes the evolution of LLMs from a single unified model toward a structure in which shared capabilities and personalized capabilities coexist. Therefore, the integration of LLMs with FL faces greater technical challenges than traditional deep learning. LLMs usually have enormous parameter scales, often reaching billions or even hundreds of billions of parameters, which is several orders of magnitude larger than conventional deep learning models (e.g., GoogLeNet, AlexNet, VGG, and ResNet). If the full model parameters are directly transmitted, both communication costs and storage pressure become extremely high. Hence, the core issue is no longer merely how to aggregate models from multiple clients, but how to transmit and update only the most critical, minimal, and effective parameters under extremely large model scales. To this end, prompt learning and parameter-efficient fine-tuning (PEFT) have gained significant attention as light- weight alternatives to full-model fine-tuning. By freezing the backbone of a pretrained foundation model and optimiz- ing only a small number of task-specific parameters—such as soft prompts, prefixes, or low-rank adapters (LoRA) [ 21], training cost is drastically reduced while preserving the generalization power of large models. Importantly, the com- pactness and modularity of prompt-based updates make them inherently well-suited to federated environments. This observation has led to a new paradigm, federated prompt learning (FPL), which leverages prompt learning as the primary interface between LLMs and federated optimization. FPL fundamentally reshapes how LLMs are trained, adapted, and deployed in distributed settings. Instead of feder- ating the entire model or large subsets of parameters, clients collaboratively optimize lightweight prompt or adapter modules on top of a shared frozen foundation model. This design dramatically reduces communication costs, mitigates client-side resource constraints, and enhances robustness to non-IID data, while maintaining strong privacy guarantees. In consequence, federated prompt learning has rapidly evolved from early proof-of-concept studies to a rich ecosystem Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions3 Userstrainpromptadapters; local data stay private ParameterServer Aggregateglobalmodel Localmodeltraining Centralserver improvesglobalmodel withpromptupdate Usersdirectlytrainmodels onlocaldata UserA Globalmodeltrainingrequireslocaldata ConventionalFederatedLearning UserC Federated PromptLearning Localmodelupdates Local prompt adapter training Only small prompt adapters are trained UserB Enhancementswith customprompts UserD Private Local Data Prompt adapter updates Fig. 1. The Difference between conventional FL and federated prompt learning. of methods spanning prompt-based fine-tuning, personalization, split and off-site architectures, multimodal learning, and real-world applications. 1.1 Motivations and Contributions Despite this rapid progress, the existing literature remains fragmented. Prior surveys [14,43,46,53,55,56] have pri- marily focused on either federated learning in general, federated fine-tuning of LLMs, or the security and privacy of LLMs. Despite the rapid growth of this field, its literature remains fragmented across machine learning, distributed systems, cloud–edge computing, multimodal learning, and cybersecurity. Existing surveys typically examine general FedLLM architectures, federated fine-tuning, IoT-oriented deployments, or LLM security independently. Prompt- and adapter-based methods are often treated as auxiliary implementation techniques rather than as a distinct distributed optimization and service-delivery paradigm. Moreover, previous reviews rarely connect the complete model-service lifecycle, including training, personalization, inference, deployment, application, and protection, or systematically com- pare methods in terms of computational overhead, communication cost, scalability, dependability, and privacy. A systematic and unified treatment that centers on federated prompt learning as a distinct paradigm, examines its role across the full LLM lifecycle, and synthesizes insights from algorithmic, system, application, and security perspec- tives is still lacking. Without such a comprehensive view, it is difficult for researchers and practitioners to understand the design trade-offs, identify open challenges, and navigate future research directions in this emerging field. To ad- dress this gap, this paper presents a comprehensive survey of FPL for LLMs that develops a unified, lifecycle-oriented understanding of federated prompt learning from both algorithmic and cloud-system perspectives. It organizes ex- isting approaches by their technical mechanisms and deployment roles, and examines how prompt-based adaptation interacts with cloud coordination, heterogeneous edge resources, distributed inference, multimodal applications, and Manuscript submitted to ACM 4Q.Yang C.Qiu, H.Zhang et al. adversarial environments. Particular attention is given to performance, communication efficiency, client-side computa- tion and memory, scalability, personalization, privacy, robustness, and practical deployability. It is critically important to answer the following research questions(RQs) comprehensively: (1)RQ1: What are the fundamental motivations, characteristics, and enabling technologies of federated prompt learning, and how does it differ from conventional federated learning and full-model federated fine-tuning? (2)RQ2: What trade-offs do federated prompt learning approaches exhibit in performance, communication effi- ciency, computational overhead, scalability, personalization, and heterogeneity handling? (3)RQ3: What security, privacy, robustness, and system challenges remain, and what future research directions are most important? Our contributions are summarized as follows: •The core contribution of this survey lies in the experimental validation of representative federated prompt learning models and frameworks, through which we derive key empirical findings and practical insights. These results are intended to provide guidance and inspiration for researchers to advance research in this emerging area. •We analyze the security, privacy, and robustness challenges that arise when prompt learning meets federated optimization, and summarize existing attack models and defense mechanisms tailored to federated LLMs. •We distill key lessons from current research and outline open challenges and promising future directions to guide subsequent studies in federated prompt learning. Through this survey, we aim to provide a structured and coherent reference for researchers and practitioners seeking to understand how prompt learning can effectively bridge large language models and federated learning, enabling scalable, efficient, and privacy-preserving collaboration in the era of foundation models. 1.2 Methods This review combines a systematic literature review with taxonomy-based comparative analysis. Relevant studies are retrieved from databases, such as “Web of Science”, “Google Scholar”, and “dblp”, using predefined search terms related to federated learning and prompt learning. Explicit inclusion and exclusion criteria are applied according to publica- tion period, document type, research relevance, and methodological completeness. After duplicate removal, title and abstract screening, and full-text assessment, 52 high-quality studies were retained for subsequent analysis. The selected studies are then systematically coded in terms of research objectives, core methods, model architectures, datasets, evaluation metrics, empirical results, and reported limitations. Based on their underlying technical mechanisms, the reviewed approaches are organized into a structured taxonomy and compared with respect to performance, efficiency, communication cost, computational overhead, privacy protection, scalability, and application scenarios. Finally, the review synthesizes the major advances, unresolved challenges, and potential directions for future research. 2 Related Work and Background 2.1 Related work Recent research has increasingly investigated the integration of LLMs and FL as a response to the escalating challenges of data scarcity, privacy preservation, communication cost, and the prohibitive cost of centralized model training. Niko- laouet al.[37] establish a fundamental theoretical property of decoder-only Transformer language models, showing that they are almost surely injective and hence exactly invertible with respect to their input prompts. Contrary to Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions5 the prevailing intuition that nonlinearities, normalization, and attention mechanisms inevitably lead to information loss, the authors prove that distinct input prompts map to distinct hidden representations with probability one under standard architectures, continuous parameter initialization, and finite-step gradient-based training. Leveraging tools from real analysis, they show that collisions can only occur on measure-zero parameter sets and that common training procedures cannot reach such pathological configurations. Beyond theory, Nikolaouet al.introduce SIPIT, the first provably correct and efficient algorithm that reconstructs the exact input prompt from internal hidden states in linear time, exploiting the causal structure of Transformers. Cheng et al. [10] provide a comprehensive, structured overview of federated large language models (FedLLMs), elu- cidating motivations, methodologies, and future directions at the intersection of FL and LLMs. They highlight FedLLMs as a promising paradigm for addressing fundamental LLM challenges, including exhaustion of high-quality public data, stringent privacy requirements, continuous model updating, and prohibitive computational and communication costs. By systematically reviewing the full FedLLM lifecycle, from pre-training and fine-tuning to deployment and applica- tion, the paper categorizes existing approaches such as parameter-efficient fine-tuning, prompt-based and split learning methods, personalized federated LLMs, and backpropagation-free techniques, analyzing their performance trade-offs under data, system, and model heterogeneity. The survey further identifies critical challenges, including communica- tion bottlenecks, synchronization and straggler issues, non-IID data, and amplified security and privacy threats such as poisoning, backdoors, and inference attacks, and summarizes current defense mechanisms and their limitations. For data scarcity, Villalobos et al. [ 46] quantitatively demonstrate that the supply of high-quality public human- generated text is finite and likely to be exhausted by frontier LLMs between 2026 and 2032 under current scaling trends. This finding underscores a fundamental limitation of centralized data-driven scaling and highlights the necessity of leveraging decentralized, privately held data sources, thereby motivating privacy-preserving collaborative learning paradigms (e.g., FL). On the systems side, Fanet al.[14] propose FATE-LLM, an industrial-grade FedLLM framework that shows how parameter-efficient fine-tuning (PEFT), off-site tuning, and knowledge distillation can substantially reduce communica- tion overhead while approaching centralized fine-tuning performance. These core techniques of the work demonstrate the practical feasibility of federated LLM training in enterprise environments, but primarily focus on system imple- mentation rather than a unified methodological or conceptual abstraction. Chenet al.[7] systematically review the emerging integration of LLMs and federated learning, highlighting their complementarity in addressing data scarcity, privacy constraints, and scalability. It organizes prior work into a unified framework spanning: (i) incorporating LLM sub-technologies, pre-training and prompt engineering, into FL to improve convergence speed, robustness to non- IID data, personalization, and domain generalization; (i) applying FL sub-technologies, distributed computing and privacy-preserving mechanisms, to mitigate LLMs’ high computational cost, limited data coverage, and privacy risks; and (i) holistic federated LLM (FedLLM) systems that jointly optimize training, instruction tuning, and alignment under decentralized settings. Several surveys have attempted to organize the rapidly growing literature at the intersection of LLMs and FL. For ex- ample, Thakur et al. [43] provide a broad unifying analysis of FL–LLM fusion, categorizing prior work into paradigms that either use LLM capabilities to enhance FL, apply FL to improve LLM adaptation, or jointly design holistic FedLLM systems. Nevertheless, this review adopts a macro-level perspective and does not systematically analyze prompt- or adapter-based learning as a first-class federated optimization interface. Yao et al. [ 56] focus on security and privacy issues of LLMs, offering an extensive taxonomy of risks and defenses, but largely decouple these concerns from fed- erated training dynamics. Yang et al. [55] further expand the scope by reviewing the joint integration of IoT, LLMs, Manuscript submitted to ACM 6Q.Yang C.Qiu, H.Zhang et al. and FL, emphasizing architectural synergies in edge systems, yet their analysis is application-driven and not centered on LLM adaptation mechanisms. At a finer granularity, Yan et al. [53] conduct a comparative study of federated fine- tuning paradigms for LLMs, including FedLLMs, KD-FedLLMs, and Split-FedLLMs. Their results reveal inherent trade- offs between model accuracy, communication cost, and client-side computation, illustrating that no single federated framework is universally optimal. Despite these substantial advances, existing studies and surveys exhibit a notable gap: the lack of a dedicated, sys- tematic treatment of FPL as a distinct and unifying paradigm for LLM–FL integration. Prompt- and adapter-based methods have emerged as the dominant practical mechanism for adapting large language models under resource, com- munication, and privacy constraints, yet they are often discussed only as auxiliary techniques within broader FedLLM frameworks or system implementations. Different from the existing surveys, our review places FPL at the center of analysis, systematically reviewing how prompt- and adapter-based techniques enable efficient, scalable, and privacy-preserving collaboration of large lan- guage models under federated settings. By organizing existing methods along the full LLM lifecycle and jointly exam- ining algorithmic, system-level, application-driven, and security-related dimensions, this survey constructs a unified framework that provides answers to the questions:Why do LLMs need FL?,How does FPL work?,What can FL and LLMs do for each other?,What kinds of challenges can FPL meet?, andWhat are the potential research directions of FPL? 2.2 Background and Fundamentals This subsection introduces the fundamental concepts underlying federated prompt learning. We first describe large language models and their adaptation requirements, followed by conventional federated learning, prompt learning, and parameter-efficient fine-tuning. We then formalize federated prompt learning as a lightweight interface for adapting foundation models using decentralized private data. 2.2.1 Large Language Models and Model Adaptation.LLMs are typically built on transformer [20] architectures and pretrained on large-scale text corpora using self-supervised objectives. Through large-scale pretraining, LLMs acquire general-purpose representations that support a broad range of downstream capabilities, including language under- standing, text generation, reasoning, question answering, and instruction following [ 4]. These pretrained models can subsequently be adapted to specific tasks, domains, organizations, or users. LLMs containing billions of parameters impose substantial computation, memory, communication, and storage re- quirements. In distributed settings, repeatedly transmitting full-model parameters or gradients further creates a major communication bottleneck. Full fine-tuning may also require each participant to possess sufficient hardware resources to store and optimize the complete model. The adaptation of LLMs is further complicated by the decentralized na- ture of high-value data. Enterprise documents, medical records, financial information, source code, personal-device data, and institution-specific knowledge are frequently distributed across independent data owners. Privacy require- ments, regulatory restrictions, and commercial confidentiality often prevent these data from being transferred to a centralized training platform. These requirements motivate the combination of federated learning with lightweight model-adaptation techniques. 2.2.2 Conventional Federated Learning.As illustrated in Fig.1, a typical FL system consists of a central server and a set of participating clients. Each client maintains a local dataset and performs model optimization within its own trusted environment. The server coordinates training by distributing model parameters, collecting local updates, and Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions7 aggregating them into a global model. A conventional FL round generally contains the following three steps: global model broadcast, local model training and transmission, and model aggregation [26]. Let퐾denote the number of clients,퐷 푘 the private dataset of client푘, and푛 푘 = |퐷 푘 |its number of samples. Conven- tional FL can be expressed as the minimization of a weighted global objective: min w 퐹(w) = 퐾 ∑ 푘=1 푛 푘 ∑ 퐾 푗=1 푛 푗 퐹 푘 (w),(1) where푤represents the shared model parameters and퐹 푘 (w)is the local objective evaluated on퐷 푘 . In FedAvg, the server aggregates the locally optimized parameters as: w 푡+1 = ∑ 푘∈푆 푡 푛 푘 ∑ 푗∈푆 푡 푛 푗 w 푡+1 푘 ,(2) where푆 푡 is the set of clients participating in communication round푡. The clients in conventional FL are usually heterogeneous, including non-independent and identically distributed data, computation capacities, storage, and communication. This case becomes more severe when FL is directly applied to LLMs. Transmitting and optimizing an entire LLM may exceed the computation, memory, and bandwidth available to many clients. Therefore, federated LLM systems generally require parameter-efficient adaptation, compression, model partitioning, or off-site training rather than straightforward full-model federated optimization. 2.2.3 Prompt Learning.The prompt plays a role in bridging the downstream task and the knowledge encoded in the pretrained model. Prompt learning adapts a pretrained model by modifying or learning the conditioning context while keeping most or all backbone parameters frozen [63]. This paper treats “prompt tuning” as a class of methods rather than a particular method. Prompts can generally be divided into discrete and continuous forms. A discrete prompt consists of human-readable tokens, task descriptions, demonstrations, or instructions inserted into the model input. Continuous prompt learning instead represents the prompt as a set of trainable embedding vectors. In prompt tuning, learnable embeddings are appended or prepended to the input sequence. Prefix tuning introduces trainable vectors into multiple transformer layers, commonly through the attention [45] mechanism. In vision or vision- language models, visual prompts may be introduced into image patches or intermediate visual representations, while textual prompts condition the language branch. Let푓 휃 denote a pretrained model with frozen parameters휃, and let푝denote a trainable prompt. Prompt learning optimizes: min p ℒ ( 푓 휃 (x;p), 푦 ) ,(3) where(푥, 푦)is a downstream training example. Because휃remains fixed and the dimension of푝is substantially smaller than that of the backbone, prompt learning reduces the computation, storage, and communication required for adap- tation. However, prompt learning remains sensitive to the quality of the pretrained model and the distance between the pretraining and downstream distributions. 2.2.4 Parameter-Efficient Fine-Tuning.Parameter-efficient fine-tuning (PEFT) [15] refers to a broader family of adapta- tion techniques that freeze most pretrained parameters and optimize only a small number of task-specific parameters. Manuscript submitted to ACM 8Q.Yang C.Qiu, H.Zhang et al. Prompt learning (2.2.3) is an important form of PEFT, while other representative techniques include adapters [22], LoRA, prefix-based modules [27], and bias-only tuning [58]. PEFT is particularly suitable for federated LLM adaptation. Because only compact parameter subsets are transmit- ted, communication overhead can be reduced from full-model scale to prompt- or adapter-module scale. Clients can maintain a common frozen backbone while learning local adaptation modules, and the server can aggregate only the lightweight updates. 2.2.5 Federated Prompt Learning (FPL).FPL integrates federated optimization with prompt learning or closely related PEFT methods [11]. It mainly federates lightweight adaptation modules rather than the complete foundation model. FPL can substantially reduce communication and client-side optimization costs because only compact modules are trained and exchanged. In FPL, the frozen foundation model preserves broadly transferable knowledge and reduces the risk of catastrophic forgetting during local adaptation. Meanwhile, modular prompts and adapters enable the co- existence of shared and personalized capabilities. A typical FPL training round contains four stages: (1)The server distributes the current global prompt or adapter parameters to selected clients. (2)Each client attaches the received module to a shared frozen foundation model and performs local optimization using its private data. (3)Clients transmit only their updated prompt or adapter parameters to the server. (4)The server aggregates the lightweight updates and constructs a new global module. Let휙denote the federated prompt or adapter parameters, while휃denotes the frozen backbone. The FPL objective can be represented as: min 휙 퐾 ∑ 푘=1 푛 푘 ∑ 퐾 푗=1 푛 푗 퐹 푘 (휃, 휙).(4) After local optimization, the server may perform weighted aggregation: 휙 푡+1 = ∑ 푘∈푆 푡 푛 푘 ∑ 푗∈푆 푡 푛 푗 휙 푡+1 푘 .(5) Different from conventional FL, in which the communicated variable푤may contain all model parameters, FPL com- municates only휙, whose size is generally much smaller than that of the foundation model. For example, PROMPTFL [ 18] replaces conventional full-model training with cooperative optimization of continuous prompts over a frozen founda- tion model, while FedPrompt [61] extends soft prompt tuning to federated pretrained language models and supports prompt tuning, P-tuning [30], and prefix tuning. Nevertheless, FPL introduces challenges that differ from those of conventional FL. Prompt parameters learned by heterogeneous clients may encode semantically different concepts, making position-wise averaging ineffective. Data heterogeneity can therefore cause prompt misalignment, client drift, and negative transfer. A single global prompt may favor dominant clients and provide limited adaptation to minority domains. Personalized FPL addresses this issue by separating global knowledge from client-specific prompts or by generating customized prompts according to client updates. 2.2.6 Privacy, Security, and Robustness Considerations.Although keeping raw data local is an important privacy prop- erty of FPL, lightweight parameter exchange does not eliminate security risks. Prompt or adapter updates may still Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions9 reveal information about local examples through gradient inversion, membership inference, or representation recon- struction. A malicious client may poison its local data or manipulate its prompt updates to insert a backdoor into the aggregated module. Conversely, a malicious or compromised server may distribute manipulated prompts, infer client properties, or exploit intermediate activations in split architectures. Answers to RQ1:FPL emerges from combining FL with prompt-based tuning of large pretrained (often vision- language) models, motivated primarily by the practical limitations of applying conventional FL to foundation mod- els: high communication overhead, prohibitive on-device compute and memory demands, non-IID data heterogeneity across clients, and risks of overfitting or catastrophic forgetting under full-model fine-tuning. Rather than exchanging full model gradients or weights, FPL keeps the pretrained backbone frozen at both server and clients, and collabora- tively learns only lightweight, task-specific prompts, such as continuous soft-prompt embeddings, prefix vectors, or prompt-generating networks.Then these prompts are aggregated via standard or adapted FL protocols (e.g., FedAvg, FedProx, personalized FL). This design is enabled by advances in parameter-efficient fine-tuning (e.g., prompt tuning, prefix-tuning, P-tuning v2) and foundation models such as CLIP, whose strong pretrained representations can be effec- tively steered through prompts alone. Different from conventional FL, FPL reduces communication and computation by orders of magnitude, supports client-level personalization by decomposing prompts into shared and local compo- nents, and better preserves general pretrained knowledge. Compared to full-model federated fine-tuning, FPL avoids destructive weight-averaging under heterogeneous data, lowers system requirements by eliminating the need to store or transmit full gradients and optimizer states, and offers greater flexibility for supporting multiple tasks or domains without maintaining separate full model copies, positioning FPL as a parameter-efficient, communication-friendly par- adigm for adapting foundation models in federated settings. 3 Federated Learning for Large Language Models From the perspective of the interaction between FL and LLMs, this survey broadly categorizes the existing research into three types: (i) federated learning for large language models; (i) large language model-enhanced federated learning; (i) the synergistic integration of federated learning and large language models. For the first category, federated learning is applied to address the challenges of training, adaptation, and deploy- ment of large language models on distributed private data. In this relationship, the large language model is the primary object of learning and optimization, while FL provides a collaborative training mechanism that keeps data local. Based on the lifecycle of LLMs, this category can be further divided into federated pretraining, federated fine-tuning, feder- ated instruction tuning, federated alignment, federated prompt learning, federated model compression, and federated continual learning, among other directions. 3.1 Larger Language Models Optimization through Federated Learning Chenet al[6] address the fundamental difficulty of applying FL to large language models LLMs, namely the prohib- itive communication and computation costs incurred when updating massive model parameters under non-IID de- centralized data. To overcome the limitations of existing prompt-based FL methods such as performance degradation, inefficient training, and client drift, the authors propose FedPepTAO by integrating parameter-efficient prompt tun- ing with a communication-efficient adaptive optimization strategy. FedPepTAO lies in a layer-importance scoring and lossless selection mechanism, which identifies and synchronizes only a subset of influential prompt layers while keeping other prompts locally updated, thereby significantly reducing communication overhead without sacrificing accuracy. FedPepTAO is evaluated through extensive experiments on 10 NLP benchmarks using RoBERTa-Large and Manuscript submitted to ACM 10Q.Yang C.Qiu, H.Zhang et al. multiple decoder-based LLMs (e.g., GPT-2, LLaMA-3B, LLaMA-7B). The results demonstrate that FedPepTAO consis- tently achieves state-of-the-art accuracy up to 60.8% improvement over baselines, while reducing training time by up to 97.59%. Raje [41] systematically investigates communication-efficient training of LLMs in FL settings, addressing the dual bottlenecks of limited client-side computation and expensive wireless communication that hinder practical federated LLM deployment. Considering that fine-tuning updates of pretrained LLMs are intrinsically low-rank, the work inte- grates LoRA-based parameter-efficient fine-tuning with communication-only sparsification, proposing federated LoRA with simple sparsity (FLoSS). Unlike prior approaches that prune adapters during training, FLoSS applies unstructured top-k sparsity exclusively during download and upload phases, while preserving dense local optimization to maintain model utility. To address the largely overlooked limitation of existing federated instruction tuning methods for large language models, Qinet al.[ 39] propose FedHDS. Considering the resource-constrained edge devices and overfitting to narrow client-specific domains, FedHDS constructs a hierarchical selection strategy that combines cross-layer feature fusion from multiple transformer layers with density-based clustering, enabling both local redundancy removal and global coordination across clients in a privacy-preserving manner. FedHDS establishes the first systematic solution for feder- ated data-efficient instruction tuning and highlights the importance of redundancy-aware data selection for scalable, efficient, and generalizable federated LLM training. Furthermore, FedDQC [ 13] tackles a fundamental yet underexplored challenge in federated instruction tuning of large language models: data quality heterogeneity across decentralized clients, which can severely degrade global model performance despite privacy preservation. To address the lack of global visibility and the impracticality of cen- tralized data filtering in FL, the authors [13] propose FedDQC. It is a novel federated data quality control framework that operates entirely on the client side with minimal overhead. The kernel of FedDQC is the instruction response alignment (IRA) metric, a lightweight and privacy-preserving quality estimator that measures how well an instruction conditions its response by comparing conditional and unconditional inference losses. Building on IRA, FedDQC intro- duces a quality-aware hierarchical federated training strategy that progressively fine-tunes the model from high-IRA (easy, high-quality) samples to lower-IRA (harder, noisier) data, mirroring human curriculum learning. To address the data heterogeneity issue in federated prompt tuning, Chenet al.[9] propose DualFPT, a federated vi- sual prompt-tuning framework that jointly pursues generalized and personalized adaptation by decomposing learnable prompts into shared global prompts and client-specific local prompts. Because feature shifts and class imbalance impair the transferability of globally aggregated prompts while purely personalized methods risk overfitting to client-specific distributions. DualFPT is compose of two mechanisms: (i) feature sharing (FS); (i) prompt composition scheme (PCS). First, FS uses a variational autoencoder to disentangle classification-sensitive features from redundant information and applies differential-privacy noise before sharing selected features, narrowing inter-client distribution gaps to improve global-prompt generalization. Second, the PCS employs a lightweight adaptive network to estimate similarity between each test instance and client distributions, dynamically combining local prompts into an instance-specific composite prompt. A two-stage alternating strategy optimizes global parameters, local prompts, and the adaptive network. Nev- ertheless, DualFPT requires an added feature-extraction stage, exchange of differentially private intermediate features, and training of auxiliary VAE, classifier, and adaptive-network components, increasing implementation complexity relative to prompt-only aggregation. Evaluation is also limited to image classification with a ViT-B/16 backbone, con- trolled feature- and class-heterogeneity settings, fixed prompt configurations, and relatively short federated training schedules. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions11 To address the task and client resource heterogeneity in cross-device FL, Baiet al.[3] propose FlexLoRA, a federated fine-tuning framework for LLMs. Motivated by the “bucket effect” in conventional FL where all clients are constrained by the least-capable participant, the authors allow clients to apply heterogeneous LoRA ranks to match their local resources, enabling more powerful clients to contribute richer, less task-specific knowledge. FlexLoRA aggregates client updates by reconstructing full LoRA weight matrices, averaging them on the server, and applying SVD-based decomposition to redistribute rank-adaptive LoRA parameters back to clients, without additional hyperparameters. Theoretical analysis links higher local ranks and larger client populations to improved generalization bounds. Gaoet al.[16] address the memory overhead and training latency of federated LoRA fine-tuning for LLMs on resource-constrained, system-heterogeneous edge devices by proposing FAH-QLoRA, a framework that combines het- erogeneous base-model quantization with dynamically adjusted LoRA ranks. Its core mechanism follows a two-stage rank-allocation strategy: first determining the average LoRA rank that maximizes loss-reduction rate per unit wall- clock time, then assigning device-specific ranks based on heterogeneous computation and communication capabilities, allocating lower ranks to slower devices to mitigate the straggler effect. Truncation and zero-padding further support local training and global aggregation of heterogeneous LoRA modules. The main contribution lies in jointly optimiz- ing model precision and LoRA rank allocation to improve time and memory efficiency in federated LLM fine-tuning, complemented by a convergence analysis under non-convex, non-IID settings. 3.2 Larger Language Models Application via Federated Learning FedMRG [5] is the first comprehensive framework for LLM-driven medical report generation (MRG) under federated learning, tackling two key obstacles to multi-center collaboration: prohibitive LLM communication costs and severe multi-modal data heterogeneity across institutions. To enable privacy-preserving, scalable training, FedMRG employs LoRA to substantially reduce communication overhead. It further addresses heterogeneity at both visual and textual levels: on the encoder side, Hierarchical Contrasting and Prompting (HCP) combines client-aware contrastive learning with diagnosis-aware prompting to capture globally consistent yet locally distinctive visual features; on the decoder side, a Dual-adapter Mutual Boosting (DMB) mechanism harmonizes global reporting knowledge with client-specific linguistic styles via bidirectional knowledge distillation. Otoumet al.[38] present an LLM-driven federated learning framework for scalable and secure IoT management, motivated by the latency, privacy, energy, and scalability limitations of centralized cloud-based IoT architectures. The authors integrate LLMs with FL in a hybrid edge–cloud architecture, enabling privacy-preserving decision-making directly on IoT devices while leveraging cloud resources for global coordination. A key contribution is the gradient sensing federated strategy (GSFS), which adaptively regulates client participation and asynchronous update uploads based on performance shifts and gradient magnitudes, reducing redundant communication and improving convergence efficiency over classical methods, such as FedAvg and FedOpt. Evaluations on the IoT-23 dataset show GSFS achieves higher accuracy and F1-scores for both central and client models, while significantly cutting response latency (up to 51% on the client side) and improving energy efficiency. Results further indicate that edge-based LLM inference enables real-time IoT analytics with reduced cloud dependency, while federated coordination preserves data privacy across heterogeneous devices. Agarwal et al. [2] provide a critical empirical investigation into whether FL with pretrained language models (PLMs) genuinely achieves domain adaptation and personalization, or whether its apparent success stems largely from pretraining-induced regularization. Through systematic experiments on three NLP tasks: sentiment classification (SST- 2), sequence tagging (OntoNotes), and text generation (Gigaword), the authors analyze key confounders in federated Manuscript submitted to ACM 12Q.Yang C.Qiu, H.Zhang et al. NLP, including the role of pretrained weights, client size imbalance, data partition strategies, and the trade-off between server generalization and client personalization. Results show that with PLMs, the performance gap between federated and centralized training stays small even after aggressively ablating client updates, suggesting FL often fails to mean- ingfully adapt to local client distributions and instead benefits from pretraining’s strong semantic priors. The proposed personalization–generalization slope (푚Δ푃) further indicates that local-data learning provides limited or neutral gains to global generalization, challenging common claims about personalization in federated NLP. While uniform client data distributions improve convergence speed and stability, they do not fundamentally resolve the lack of true domain adaptation. Zhang [ 60] addresses the resource and memory heterogeneity encountered in federated foundation-model fine- tuning, where resource-constrained clients may be unable to update all LoRA layers efficiently. It proposes Fed-HeLLo, a federated LoRA framework that assigns different subsets of trainable LoRA layers to clients according to their compu- tational capabilities and layer importance. Its core design combines Fisher information matrix–based allocation (FIM- HLA) for dynamically estimating layer importance with geometrically defined and randomized allocation strategies (GD-HLA/RGD-HLA) for stabilizing early-stage training. The principal contribution therefore lies in jointly exploiting client resource heterogeneity and layer-wise importance for efficient federated LoRA fine-tuning. Nevertheless, the framework still relies on a server-side proxy dataset, retains non-negligible activation-memory requirements, lacks explicit privacy protection, and does not directly resolve data heterogeneity. 3.3 Empowering the Reasoning Process of LLMs in Federated Settings eFedLLM [12] is an FL-based framework for LLM inference, targeting the prohibitive computational, memory, and bandwidth requirements that limit LLM accessibility. Departing from conventional data-parallel FL, it adopts a transformer- based model-parallel architecture, distributing different transformer layers across heterogeneous participants to enable collaborative inference without requiring any single user to host the full model. To ensure reliability in this chained execution setting, eFedLLM introduces a trust-based incentive and verification mechanism that evaluates intermedi- ate layer outputs and filters malicious or low-quality contributors. It further incorporates transformer-tailored opti- mizations, including hierarchical memory access strategies to reduce global memory reads and SVD-based low-rank compression of weight matrices to lower communication bandwidth while preserving accuracy. Analytical and numer- ical evaluations show these optimizations reduce memory access and bandwidth usage by up to 60% under practical compression ratios, significantly improving inference efficiency. Lin [29] introduces SplitLoRA, the first split learning-based parameter-efficient fine-tuning framework for LLMs, addressing the prohibitive computation and communication costs limiting federated LLM adaptation on distributed private data. Motivated by the scarcity of high-quality public data and the impracticality of full-model federated fine- tuning, SplitLoRA combines split federated learning (SFL) with LoRA-based PEFT, partitioning the LLM between clients and a central server so clients train only shallow layers with lightweight LoRA adapters while the server handles most computation. This substantially reduces client-side resource demands and mitigates data heterogeneity by centralizing deeper representations. SplitLoRA provides a practical balance between performance, efficiency, and scalability. Mean- while, it establishes an open-source benchmark for split LLM fine-tuning. However, open challenges remain in optimal model splitting, handling heterogeneous client resources, and strengthening privacy guarantees against split-learning inference attacks, pointing to promising directions for future research. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions13 Table 1. Research trajectories of FL for LLMs. Research Trajectory Representative Works Core IdeaMain Challenges Parameter- Efficient Adaptation and Communication Compression FedPepTAO [6], FLoSS [41] Replace full-model federated fine-tuning with lightweight prompt- or LoRA-based adaptation. How to minimize communication overhead while preserving the effectiveness and stability of local optimization. Federated Instruction Data Selection and Quality Control FedHDS [39], FedDQC [13] Hierarchical data selection, cross-layer feature fusion, density-based clustering, and quality-aware curriculum learning. How to identify which decentralized training samples are informative and reliable when raw client data cannot be globally inspected. Data Heterogeneity and Personalization DualFPT [9], Agarwal et al. [2] Address non-IID client distributions through global–local prompt decomposition and personalized adaptation. How to balance globally transferable knowledge and client-specific adaptation without sacrificing either generalization or personalization. System and Resource Heterogeneity FlexLoRA [3], FAH-QLoRA [16], Fed-HeLLo [ 60] Allow heterogeneous clients to train different LoRA ranks, quantization precisions, or subsets of LoRA layers according to devices’ capabilities. How to prevent resource-constrained clients from becoming system bottlenecks while fully exploiting the clients with high capabilities. Model Partitioning and Collaborative Inference eFedLLM [12], SplitLoRA [29] Partition LLMs across clients, servers, or multiple participants so that no single resource-constrained device is required to host or execute the full model. How to achieve efficient and trustworthy collaborative execution while controlling intermediate-feature leakage, communication latency, participant heterogeneity, and model-partitioning overhead. Continued on next page Manuscript submitted to ACM 14Q.Yang C.Qiu, H.Zhang et al. Table 1 (continued) Research Trajectory Representative Works Core IdeaMain Challenges Application- Driven Integrated Optimization FedMRG [5], GSFS [38] Integrate parameter-efficient adaptation, heterogeneity handling, communication optimization, and edge–cloud collaboration according to the requirements of specific domains, such as healthcare and IoT. How to adapt general-purpose federated LLM techniques to domain-specific data distributions, system constraints, privacy requirements, and real-time deployment conditions. As summarized in Table1, the research focus of federated LLMs has gradually shifted from simply reducing the cost of full-model federated optimization toward a more fine-grained design space involvingwhat to train,what to com- municate,where to execute, andwhat knowledge should be shared. Early efforts primarily exploit parameter-efficient adaptation and communication compression, whereas more recent studies increasingly consider data quality, client heterogeneity, and collaborative model execution in a unified manner. This evolution suggests that federated LLM optimization is no longer merely an aggregation problem, but rather a joint optimization problem across model param- eters, data, communication, computation, and privacy. 4 Large Language Model-enhanced Federated Learning In LLM-enhanced FL, LLMs serve as knowledge auxiliary tools to improve the performance and automation level of conventional federated learning. In this context, the model being federated is not necessarily a large language model itself, but may instead be a classification model, a vision model, or another lightweight model. LLMs can participate in local data generation, data augmentation, automatic annotation, and pseudo-label construction to alleviate issues of insufficient client data and class imbalance. They can also transfer knowledge to federated models through knowledge distillation, feature transfer, and model initialization. Furthermore, LLMs can assist with client selection, aggregation strategy design, hyperparameter configuration, training log analysis, anomalous update detection, and privacy risk in- terpretation, thereby improving the optimization efficiency, manageability, and intelligence level of federated learning systems. As summarized in Table2, the role of large pretrained models in federated learning is gradually evolving from being directly federated toward serving as reusable knowledge priors for lightweight collaborative adaptation. Accordingly, the main optimization target shifts from complete model parameters to prompts, adapters, low-rank modules, and compressed updates. This evolution further relaxes several conventional assumptions of federated learning. In particular, clients are no longer necessarily required to optimize the same parameter space or even possess the full foundation model. Instead, pretrained knowledge can be accessed through parameter-efficient adaptation, off-site tuning, lightweight emulators, or domain-specific instruction tuning. Consequently, LLM-enhanced federated learning increasingly becomes a joint design problem involving knowledge reuse, parameter efficiency, model accessibility, data quality, and communication efficiency. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions15 Table 2. Research trajectories of large language model-enhanced federated learning. Research Trajectory Representative Works Core IdeaMain Challenges Foundation Model-Assisted Federated Adaptation PROMPTFL [18], FedPrompt [61] Exploit the transferable knowledge encoded in pretrained foundation models while freezing the backbone and collaboratively optimizing only lightweight prompt parameters. How to leverage strong pretrained knowledge to reduce communication and optimization costs while retaining sufficient adaptability to heterogeneous and highly specialized downstream tasks. PEFT-Based Practical Federated Learning FederatedScope- LLM [25], Malaviya et al. [35] Apply parameter-efficient fine-tuning techniques, including LoRA [ 21], Prefix-Tuning, Adapters, BitFit [58], and Prompt-Tuning, to reduce the trainable and communicated parameter space. How to select and configure PEFT mechanisms that provide a favorable trade-off among model utility, communication efficiency, client-side resource consumption, and robustness to non-IID data. Model-Access- Constrained and Off-Site Federated Tuning FedBiOT [51]Relax the assumption that each client must access or deploy the full foundation model. The original model is represented through lightweight emulator and adapter components, while client-side adapter optimization is coordinated with server-side emulator distillation through bi-level optimization. How to simultaneously protect model intellectual property, reduce client-side computation and memory requirements, and preserve adaptation quality under distribution mismatch between server-side public data and client-side private data. Data- and Application- Aware LLM-Assisted Federated Learning FIT with Feature Diversity [ 8] Combine parameter-efficient federated instruction tuning with systematic augmentation of instruction-following data. Foundation-model knowledge and feature diversity are exploited to compensate for scarce annotations. How to exploit pretrained knowledge and diversified training data to improve downstream generalization while preserving privacy and avoiding excessive dependence on application-specific data augmentation strategies. Continued on next page Manuscript submitted to ACM 16Q.Yang C.Qiu, H.Zhang et al. Table 2 (continued) Research Trajectory Representative Works Core IdeaMain Challenges Communication- Aware Lightweight Adaptation FLM-TopK [40]Further compress PEFT updates through joint gradient sparsification and quantization. Intervalized TopK sparsification reduces not only the number of transmitted gradient values but also the position-index overhead associated with conventional TopK compression. How to further reduce communication after the trainable parameter space has already been substantially reduced by PEFT, while controlling the optimization errors introduced by sparsification and quantization. 4.1 Using LLMs to Empower Federated Learning Settings Motivated by the severe communication overhead, slow convergence, data scarcity, and non-IID sensitivity of classi- cal FL, Guoet al.[ 18] leverage the strong generalization capability of large pretrained vision–language models (e.g., CLIP) and propose a paradigm shift in FL, named PROMPTFL. PROMPTFL allows distributed clients to collaboratively optimize only a small set of continuous (soft) prompt parameters on top of a frozen foundation model instead of con- ventional full-model training. This design dramatically reduces trainable parameters and communication cost while preserving privacy, as clients share only prompt updates rather than raw data or full model gradients. Theoretically, PROMPTFL can reach a convergence rate of풪(1/ √ 푇)under the setting where partial clients are selected at each round, and local data distributions are Non-IID. Extensive experiments across diverse vision benchmarks demonstrate that PROMPTFL achieves competitive or superior accuracy and robustness. Particularly in heterogeneous and data- scarce scenarios, it uses orders of magnitude fewer parameters and fewer communication rounds than FedAvg and fine-tuning-based baselines. Its performance depends on the availability of strong foundation models and may be less effective for highly specialized tasks. FedPrompt [61] extends parameter-efficient learning to the federated setting by integrating soft prompt tuning with federated learning for large pre-trained language models (PLMs). Considering the prohibitive communication and memory costs of full-model fine-tuning in FL, FedPrompt freezes the PLM backbone and collaboratively trains only a small set of soft prompt parameters using a split-aggregation strategy, reducing communication overhead to approxi- mately 0.01% of the original model size with minimal accuracy degradation. The method is compatible with standard FedAvg and supports multiple prompt paradigms (e.g., prompt tuning, P-tuning, prefix-tuning). Extensive experiments on diverse NLP tasks under both IID and Non-IID data distributions demonstrate that FedPrompt achieves competi- tive performance compared to full fine-tuning while being significantly more communication-efficient. Furthermore, robustness analyses show that FedPrompt is resistant to backdoor attacks based on data poisoning, as prompt aggre- gation mitigates malicious updates, and optional LDP can be incorporated to further enhance privacy at the cost of moderate accuracy loss. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions17 4.2 Prompt Adapter for Federated Learning FederatedScope-LLM (FS-LLM) [25] is a comprehensive, open-source framework designed to bridge the gap between general-purpose FL systems and the unique requirements of federated fine-tuning of LLMs. Motivated by the pro- hibitive communication cost, computation burden, data heterogeneity, and model intellectual-property constraints inherent to LLMs, FS-LLM provides an end-to-end solution that integrates benchmarking, algorithm support, and system-level optimization. It offers a standardized federated LLM benchmarking pipeline with curated datasets and evaluation tasks, enabling fair comparison across domains such as code generation, general language understanding, and reasoning. To address efficiency, FS-LLM incorporates a rich suite of parameter-efficient fine-tuning (PEFT) meth- ods (e.g., LoRA, P-/prompt-tuning) and supports federated offsite tuning for scenarios where clients cannot access full models, achieving orders-of-magnitude reductions in communication while maintaining competitive performance. At the system level, FS-LLM integrates accelerating and resource-efficient operators (mixed precision, compression, DeepSpeed, offloading) and flexible training modes, making federated LLM fine-tuning feasible on limited hardware. Malaviyaet al.[35] investigate how parameter-efficient fine-tuning (PEFT) can make FL with pretrained language models (PLMs) practical under strict privacy constraints and limited communication budgets. Motivated by the prohib- itive cost of transmitting full PLM parameters and the frequent lack of labeled client data, the authors systematically evaluate four PEFT methods: Prefix-tuning, Adapters, BitFit, and LoRA, under both supervised FedAvg and a more realistic semi-supervised scenario (FedFAME). Only a small subset of trainable “훿” parameters is exchanged between clients and server while the PLM backbone stays frozen, reducing communication from about 420 MB to below 50 MB, and in some cases under 1 MB. Experiments on six GLUE tasks under varying non-IID conditions show PEFT methods achieve competitive performance, with Prefix and Adapter tuning generally outperforming BitFit and LoRA, and Fed- FAME exhibiting substantially higher robustness than FedAvg under severe heterogeneity (훼= 0.1). The paper further demonstrates strong task-level transferability of PEFT parameters, particularly between semantically related tasks, en- abling effective zero-shot adaptation when target-task labels are scarce. While the approach assumes the frozen PLM fits on client devices and is sensitive to hyperparameters like learning rate, the work provides compelling evidence that PEFT is a viable, scalable foundation for communication-efficient, privacy-preserving federated NLP. To relieve the two critical bottlenecks in federated LLM fine-tuning: clients’ lack of access to full model parame- ters and prohibitive computation/communication costs, Wu et al. [ 51] propose FedBiOT, a resource-efficient, privacy- preserving framework that avoids full-model exposure. Building on the off-site-tuning paradigm, FedBiOT compresses the LLM into an emulator, aligned server-side on public data to mimic the original model, and an adapter, which cap- tures domain-specific knowledge and is fine-tuned locally by clients using lightweight LoRA modules. To handle the distribution mismatch between server-side public data and client-side private data, the authors formulate a bi-level op- timization problem alternating between client-side adapter updates and server-side emulator distillation. Experiments on LLaMA-2 across math problem solving, code generation, and question answering under both IID and non-IID set- tings show FedBiOT consistently outperforms offsite-tuning and FedOT, achieving higher accuracy and robustness while cutting client-side computation by up to 50% and avoiding transmission of full model parameters. Though it incurs slightly higher communication overhead than prior offsite methods due to emulator updates, this cost remains negligible compared to full LLM transfer. Chenet al[ 8] propose federated instruction tuning (FIT) with a complementary feature diversity (FD) strategy to enable privacy-preserving and data-efficient deployment of large language models for IoT-based autonomous driving. Motivated by the scarcity of high-quality annotated driving data and stringent privacy constraints in vehicular IoT Manuscript submitted to ACM 18Q.Yang C.Qiu, H.Zhang et al. environments, FIT leverages federated learning to collaboratively fine-tune only lightweight adapter parameters of a frozen multimodal LLM, thereby substantially reducing communication and computational overhead while preserving sensitive local data. To further address limited scenario coverage and improve generalization, the FD strategy system- atically augments instruction-following data along key dimensions, including time, weather, and occlusion, enriching both visual and textual diversity and enhancing robustness in open-world driving conditions. Extensive experiments on the DriveLM benchmark using LLaMA-Adapter backbones and multiple FL algorithms demonstrate that FIT com- bined with FD, consistently outperforms local training and achieves strong gains in accuracy, language quality, and semantic alignment, while synchronizing only a tiny fraction of model parameters. 4.3 Prompt Tuning for Federated Learning Qiuet al.[40] solve the substantial communication overhead incurred when exchanging high-dimensional LoRA up- dates during federated large language model fine-tuning, particularly because conventional TopK sparsification re- quires transmitting position identifiers for every retained gradient. To this end, the authors propose FLM-TopK, a communication-efficient framework that partitions gradients into multiple intervals and independently applies TopK sparsification within each interval, thereby reducing the number of bits required to encode gradient positions. Its core design models both original and intervalized fine-tuning gradients using a zero-mean Gaussian distribution and formulates compression as a joint optimization of interval size, packet allocation, and interval-specific sparsification rates under a fixed communication budget. Because the resulting optimization problem is non-convex, the authors decompose it into alternating convex subproblems and combine sparsification with unbiased quantization to balance discarded-gradient and quantization errors. Nevertheless, the formulation relies on an empirically derived Gaussian- gradient assumption and a fixed per-round communication budget, while the evaluation adopts a single LoRA rank, simulated participation of 10 out of 100 clients, and limited network-level measurements. 5 The Synergistic Integration of FL and LLMs The synergistic integration of FL and LLMs no longer exhibits a one-directional service relationship but instead jointly forms a distributed intelligent system with bidirectional interaction capabilities. As summarized in Table3, the syner- gistic integration of federated learning and large language models extends collaboration beyond conventional model- parameter aggregation. The objects of federation progressively evolve from shared model components toward person- alized prompts, prompt mixtures, cross-task adapters, semantic representations, and black-box optimization statistics. This evolution also relaxes several assumptions commonly adopted in traditional federated learning. Participants are no longer necessarily required to maintain identical model parameterizations, perform the same downstream task, or possess white-box access to the underlying foundation model. Instead, collaboration can occur at the level of adaptation modules, semantic concepts, multimodal task knowledge, or prompt evaluation information. Consequently, synergistic FL–LLM systems exhibit a broader transition fromparameter-level federationtowardknowledge-level federation, where the central problem is no longer merely how to aggregate distributed updates, but how to identify, align, personalize, and securely transfer useful knowledge across heterogeneous participants. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions19 Table 3. Research trajectories of the synergistic integration of FL and LLMs. Research Trajectory Representative Works Core IdeaMain Challenges Personalized Foundation- Model Adaptation FedPerfix [ 42], pFedPG [54] Personalization can further be generated from client updates through a server-side prompt generator instead of relying on a single globally aggregated adaptation module. How to retain transferable knowledge from the shared foundation model while providing communication-efficient personalization for clients with severely non-IID data, limited local samples, and heterogeneous resource constraints. Mixture-Based and Global–Local Prompt Personalization pFedMoAP [33], FedPGP [11] Model personalized adaptation either as a mixture of local and non-local prompt experts or as a decomposition of a globally shared prompt and a locally retained low-rank component. How to benefit from cross-client knowledge transfer without sacrificing local specialization or degrading the generalization capability inherited from pretrained foundation models. Domain- Generalized and Privacy-Aware Prompt Learning PLAN [17], DP-FPL [44] Use lightweight prompts as a medium for transferring domain knowledge across clients and further integrate personalized prompt adaptation with low-rank decomposition and DP to support unseen-domain generalization and formal privacy protection. How to jointly balance personalization, cross-domain generalization, and privacy, since aggressive local adaptation may impair transferable knowledge while stronger privacy protection may introduce substantial utility degradation. Multimodal and Task- Heterogeneous Federated Adaptation Pilot [ 52]Task- and client-specific adapters extract specialized knowledge, while cross-task adapter mixtures and task-aware aggregation promote selective knowledge transfer. How to enable beneficial knowledge sharing among clients performing different multimodal tasks while suppressing negative transfer caused by task, modality, and data heterogeneity. Semantic and Robustness- Oriented Prompt Aggregation PFPT [ 50], Wang et al. [ 49], FOCoOp [28] Move beyond position-wise parameter averaging by aligning local prompts at the semantic or concept level, systematically characterizing aggregation behavior under heterogeneous client distributions, and explicitly optimizing both in-distribution performance and out-of-distribution robustness. How to construct globally useful prompt representations when local prompts may be semantically misaligned, client distributions differ, domain shifts occur, and previously unseen distributions must be handled at inference time. Continued on next page Manuscript submitted to ACM 20Q.Yang C.Qiu, H.Zhang et al. Table 3 (continued) Research Trajectory Representative Works Core IdeaMain Challenges Black-Box Collaborative Prompt Optimization FedPOB [31]Bandit optimization[1], preference feedback, summary-statistic aggregation, and event-triggered communication are used to coordinate black-box prompt search. How to achieve query-efficient, communication-efficient, and privacy-preserving collaborative prompt optimization when the underlying foundation model is proprietary and only black-box access and limited feedback are available. Privacy, Security, and Trusted Execution FL-GLM [62], Huang et al. [23], ROFED-LLM [48] Combine split learning, encryption, TEEs, differential privacy (DP), secure aggregation, model pruning, and communication-layer defenses to protect federated LLM training against information leakage and adversarial attacks. How to strengthen privacy and robustness without introducing prohibitive computation, communication, or system complexity. 5.1 Federated pruning learning for personalized model learning FedPerfix [42] advances personalized federated learning (PFL) for vision transformers by systematically addressing where and how to personalize ViT models under heterogeneous data. Through an empirical sensitivity analysis, the authors identify the self-attention layers and classification head as the most distribution-sensitive components of ViTs, motivating partial personalization rather than full local training. Building on this insight and drawing an analogy to transfer learning, FedPerfix introduces prefix-based personalization with parallel attention, where lightweight, client- specific prefix plugins adapt the global self-attention representations while the backbone remains globally aggregated. Yang et al. [54] propose pFedPG, a personalized federated learning framework that uses client-specific prompt gen- eration to efficiently adapt large pretrained foundation models under severe data heterogeneity, communication con- straints, and limited client resources. Instead of sharing or averaging full model parameters, pFedPG freezes the back- bone (e.g., ViT) and alternates between local personalized prompt adaptation and server-side personalized prompt generation, where a lightweight cross-attention-based generator infers client-specific optimization directions from prompt updates and produces customized prompts for each client. This design addresses key challenges of federated learning with foundation models—performance degradation under non-IID data, excessive communication cost, and overfitting with limited local data—while retaining strong representational power. For the personalization issues under severe data heterogeneity in federated prompt learning, Luoet al.[33] argue that distributing only a single globally aggregated prompt fails to fully exploit the lightweight nature of prompt param- eters. Hence, the authors propose personalized federated mixture of adaptive prompts (pFedMoAP) that reformulates locally trained prompts as specialized experts and lets clients download multiple pre-aggregated prompts from other participants as fixed non-local experts. The design maintains a server-side prompt pool and uses prompt-space K- nearest-neighbor selection to assign relevant non-local experts to each client; a client-specific attention-based gating Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions21 network then takes image features as queries and local/non-local text features as keys and values to produce instance- adaptive text representations. Final predictions combine the mixture-enhanced representation with the locally trained prompt, while only the local prompt is uploaded and the gating network stays private. Ablations show jointly retain- ing local and mixture-based logits improves performance, moderate local-prompt weighting generally suffices, and more experts help but with diminishing returns. Gating-feature dimensions of 128–256 balance representation capac- ity and parameter efficiency. However, the method requires a server-side expert pool, multiple prompt downloads, and a client-specific gating network. Thus, communication, storage, and computation shall grow with expert count despite prompts being lightweight. In federated prompt learning, the personalization and generalization are hard to balance. Because a shared prompt may inadequately represent heterogeneous client distributions while aggressive local adaptation erodes the transfer- able knowledge of pretrained vision-language models. To this end, Cuiet al.[11] design FedPGP, a framework that preserves CLIP’s generalization while adapting the global prompt to client-specific data, representing each person- alized prompt as the sum of an aggregated global prompt and a locally retained low-rank adaptation term. FedPGP further introduces a prompt-wise contrastive objective treating global and handcrafted CLIP prompts as positive pairs to preserve category-agnostic knowledge, while treating global and personalized prompts as negative pairs to encour- age client-specific representations; only the global prompt is communicated and aggregated, while low-rank adaptation parameters remain local. Ablations confirm this design: full-rank adaptation improves local accuracy but substantially degrades base-to-novel generalization, while removing either contrastive relation reduces overall performance. Nev- ertheless, the framework is evaluated only on CLIP-based image classification, relies on fixed prompt and bottleneck dimensions, and lacks a theoretical explanation for why low-rank adaptation achieves the observed generalization– personalization trade-off. Federated domain generalization means that models trained collaboratively on heterogeneous source domains must generalize to unseen domains without exposing sensitive client data. The conventional approaches exchange domain knowledge directly derived from local samples such as feature statistics, class prototypes, or frequency information, which might bring privacy risks. Therefore, Gonget al.[ 17] proposePromptLearning andAggregatioN(PLAN), which utilizes locally learned prompts as a parameter-efficient and potentially less data-revealing medium for cross-client knowledge transfer. PLAN lies in combining reference-guided local prompt learning with attention-based multimodal prompt aggregation to improve unseen-domain generalization without exchanging raw data or directly extracted domain statistics. It follows a two-stage procedure each round. First, every client learns multi-layer text and visual prompts on local data while aligning predictions with the previous round’s global prompts via a KL-divergence regu- larizer, indirectly synchronizing client prompts and reducing local-domain overfitting. Second, lightweight attention- based aggregators estimate the relative importance of local prompts and selectively combine them into global text and visual prompts, rather than relying on fixed or uniform averaging. Only prompt tokens and aggregator parameters are optimized, while the pretrained CLIP backbone stays frozen. However, the privacy advantage rests mainly on the indirect nature of prompt learning rather than formal guarantees or leakage analysis. Distributing all local prompts to clients and performing two communication phases per round may also constrain scalability as participant numbers grow, and evaluation is restricted to CLIP-based image classification with a shared label space between source and target domains. While Tranet al.[ 44] investigate the competing requirements of personalization, generalization, and privacy in FPL for multimodal LLMs. Because personalized prompts can overfit heterogeneous local data and generalize poorly to unseen inputs, while differentially private training may substantially reduce both local and cross-domain performance. Manuscript submitted to ACM 22Q.Yang C.Qiu, H.Zhang et al. Thus, Tranet al.proposes DP-FPL, a differentially private personalized federated prompt-learning framework designed to protect prompts against membership-inference attacks while preserving their utility. In DP-FPL, each personalized prompt combines the global prompt with a local component that is iteratively decomposed into two low-rank factors and a residual term. The low-rank factors restrict the optimization space and improve generalization and robustness to privacy noise, whereas the residual compensates for information lost during factorization and preserves personalized expressiveness. Unlike approaches that perform factorization only once, DP-FPL [44] repeats the decomposition during every train- ing iteration and reconstructs the local-prompt gradient from the low-rank gradients. Privacy is enforced through a hybrid mechanism: LDP and global differential privacy (GDP). Both mechanisms employ clipped gradients and Gauss- ian perturbation, with the noise levels selected to satisfy formal (푒푝푠푙푖표푛, 훿)-LDP and GDP guarantees. The residual is used in the forward computation but is not directly perturbed because it is excluded from the local-prompt gradient reconstruction. The results show that the residual term benefits both local and neighboring-class performance, particu- larly under low-rank and high-noise conditions. They also indicate that moderate privacy noise may act as a regularizer and improve generalization, although excessive noise eventually degrades overall utility. Nevertheless, the evaluation is limited primarily to CLIP-based visual classification, a small set of benchmark datasets, and simulated pathological or Dirichlet client distributions. Moreover, the claimed regularization effect and protective role of the residual term are largely supported empirically rather than through a complete theoretical account. 5.2 Multimodal Fusion through LLMs and FL Pilot [52] introduces the federated multimodal instruction tuning (FedMIT) task and provides a principled framework for collaboratively fine-tuning multimodal LLMs under severe task and data heterogeneity. Unlike prior federated instruction tuning approaches that focus on unimodal NLP tasks, FedMIT addresses the more challenging setting where clients perform different multimodal tasks (e.g., VQA [32], captioning, and visual grounding) on decentralized data. To tackle this heterogeneity, Pilot adopts a two-stage “adapter-on-adapter” design within the vision–language connector: i) a task-specific and client-specific feature mining stage that disentangles task-relevant and personalized visual representations via orthogonality constraints, and i) a cross-task interaction stage that introduces a cross-task mixture-of-adapters (CT-MoA) module to selectively integrate knowledge from heterogeneous tasks. On the server side, Pilot further proposes task-aware aggregation for visual adapters and an adaptive distance-based aggregation strategy for text adapters, mitigating negative interference across clients. In federated prompt tuning under non-IID, locally learned prompt sets may encode concepts in arbitrary orders, causing position-wise averaging to combine semantically unrelated prompts into an uninformative global represen- tation. Therefore, Wenget al.[50] propose probabilistic federated prompt tuning (PFPT), which reframes federated aggregation as a distributed set-modeling problem rather than conventional parameter averaging. PFPT models each client’s prompts as samples from a hierarchical generative model parameterized by a server-maintained set of global summarizing prompts. Each round, a client selects a relevant subset of global prompts via a Bernoulli point-process model and fine-tunes them locally with a frozen pretrained vision transformer and a personalized classification head, with resulting local prompts modeled as Gaussian perturbations around their associated summarizing prompts. The server then infers latent local–global prompt associations and aggregates prompts encoding similar contextual infor- mation, aligning at the concept level rather than by prompt position. The resulting mixed discrete–continuous problem is solved via alternating optimization: (1) Generative-model and summarizing-prompt parameters are updated by gradient descent given associations; (2) Association inference reduces Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions23 to weighted bipartite matching solved via the Hungarian algorithm (cubic complexity in the maximum number of local prompts matched) given parameters. The global prompt pool is non-parametric, unused prompts are pruned. While heterogeneous clients can expand it to represent new concepts. Nevertheless, evaluation is limited to vision classification with a pretrained ViT under largely simulated heterogeneity and imbalance. In addition, matching cost and pool size may grow with client diversity. The prompt capacity needed for larger pretraining and downstream domain shifts is neither quantified nor theoretically bounded. Meanwhile, despite reduced communication relative to full-model fine-tuning, prompt tuning still requires storing intermediate gradients through the frozen backbone, imposing non-trivial client-side memory overhead. Rather than proposing a new aggregation algorithm for federated vision-language model adaptation, Wanget al.[ 49] provide a systematic empirical analysis of federated prompt learning under label skew, domain shift, and their coexis- tence, examining sensitivity to communication rounds, aggregation strategies, client scale, and prompt length. Using a frozen CLIP model with a ViT-B/16 encoder, the study federatively trains textual or visual prompts via sample-weighted aggregation, evaluated on CIFAR-100 (Dirichlet-based label skew) and Office-Home, Office31, and DomainNet (domain shift). The authors also analyze similarity of optimization directions across clients and between local and global prompts to explain the distinct behaviors of LPT and VPT. Results show FPL is relatively stable across communication rounds, aggregation rules, client counts, and prompt lengths. Equal aggregation performs slightly better under label skew, while sample-weighted aggregation is preferable under domain shift; more clients modestly reduce performance un- der label skew but show no consistent effect under domain shift, and longer prompts yield only incremental gains at added computational cost. However, evidence is confined to CLIP-based image classification with a single ViT-B/16 backbone, four datasets, simulated heterogeneity, fixed configurations, and one random seed; evaluation mainly compares internal prompt variants rather than benchmarking against recent federated prompt-learning methods, and the optimization-direction- similarity explanation remains empirical, without establishing a causal or theoretical link to generalization. Considering the trade-off between predictive performance and out-of-distribution (OOD) robustness in federated prompt learning for vision-language models, Liaoet al.[ 28] design FOCoOp, a federated OOD-aware context optimiza- tion framework jointly modeling global in-distribution prompts, client-specific local prompts, and OOD prompts. At each client, Bi-level OOD Separations (BOS) optimizes the three prompt sets to establish class-level separation between categories and distribution-level separation between ID and OOD samples. It employs bi-level distributionally robust optimization to perturb global and OOD prompts within optimal-transport uncertainty sets, thereby exposing the model to challenging prompt distributions without requiring real OOD training images. At the server, Global- view OOD Consistency (GOC) aligns aggregated global prompts with client-generated OOD prompts through semi- unbalanced optimal transport. OOD prompts that closely resemble global ID prompts are used to calibrate the latter, whereas distant prompts are retained and redistributed as globally consistent OOD references. FOCoOp consistently achieves the strongest overall balance between classification and OOD robustness. Under the ten-client pathological setting, it reaches 93.85% ID accuracy, 91.47% covariate-shift accuracy, 19.50% FPR95, and 95.42% AUROC on CIFAR-100, and 88.35% ID accuracy, 83.56% covariate-shift accuracy, 21.73% FPR95, and 96.56% AUROC on TinyImageNet. Ablations confirm BOS and GOC are complementary, with excluding GOC generally causing larger degradation, underscoring the importance of globally consistent ID–OOD discrimination. Similarity visualizations further show strong within-class image prompt alignment versus weaker ID–OOD prompt matching. Manuscript submitted to ACM 24Q.Yang C.Qiu, H.Zhang et al. Luet al.[31] study black-box prompt optimization for large language models under three practical constraints: expensive model queries, limited access to proprietary model parameters, and the need for privacy-preserving col- laboration among multiple users. The authors propose federated prompt optimization via bandits (FedPOB) and a preference-based extension, FedPOB-Pref, which allow multiple agents to search for effective discrete prompts with- out sharing their local prompt-evaluation histories. FedPOB encodes discrete prompts using a pretrained text encoder and models prompt performance with a linear reward function. Each agent selects prompts using a LinUCB exploration- exploitation policy and periodically sends summary statistics to a central server, which aggregates and redistributes this information. Communication is event-triggered, meaning synchronization only happens once newly collected in- formation passes a preset threshold. FedPOB-Pref extends this idea to pairwise feedback using federated linear dueling bandits: it picks one prompt for exploitation and another for exploration, learns from Bradley-Terry-Luce preference comparisons, and uses dynamic regularization to reduce optimization drift across heterogeneous agents. That said, the framework assumes all agents are optimizing prompts for the same task, with heterogeneity captured only through different candidate prompt spaces. Hence, it is unclear how well it would handle agents with different tasks or conflicting preferences. The methods also assume a linear relationship between pretrained prompt embeddings and rewards, and FedPOB-Pref further assumes a Bradley-Terry-Luce preference model; both assumptions may not hold for more complex, non-linear prompt-performance relationships. In addition, preference feedback is simulated from validation scores rather than gathered from real users, so it remains unclear how robust the method is to noisy or manipulated preferences. Finally, while raw prompts and evaluation histories are kept private, agents still send learned statistics to a trusted central server, and the paper does not offer DP, secure aggregation, or any analysis of potential information leakage. Answers to RQ2:FPL approaches exhibit systematic trade-offs across performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling, with no single method dominating on all dimensions simultaneously. The survey’s unified benchmark (Section 7)shows that LoRA achieves the highest and most stable accuracy across IID and non-IID settings but incurs substantially larger communication costs, whereas prompt tuning minimizes communication and trainable parameters at the expense of lower and more variable accuracy. Notably, P-tuning’s larger parameter count does not translate into greater stability, indicating that parameterization structure matters more than raw scale for robustness under heterogeneous optimization. Methods that improve per- sonalization through richer mechanisms (e.g., multi-expert prompt mixtures or global-local prompt decomposition) gain adaptability at the cost of increased communication, storage, and computation. While aggressive local adaptation risks eroding the generalization inherited from pretrained backbones, a tension partially mitigated through low-rank restrictions and contrastive regularization. Similarly, naive position-wise prompt averaging often fails under non-IID data due to semantic misalignment across clients, motivating concept-level or attention-based aggregation strategies that improve heterogeneity handling but introduce additional computational overhead and remain validated mainly under simulated conditions. 6 Security and Privacy Issues in FPL 6.1 Security Issues in FPL Safety alignment can be severely compromised by a simple, low-cost, and highly stealthy data poisoning attack in federated instruction tuning (FedIT) of LLMs [57]. Unlike traditional FL poisoning attacks, the proposed attack does not introduce conflicting optimization signals. Instead, malicious clients are trained on harmful instruction: response Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions25 pairs. These pairs remain optimization-consistent with benign training data. As a result, the attack exploits the intrinsic alignment objective of instruction tuning. This makes existing model-level FL defenses largely ineffective. Extensive experiments demonstrate that such attacks can reduce safety metrics by up to 70%, while classical robust aggrega- tion methods provide only marginal protection. To address this gap, the authors propose a post-hoc, server-side de- fense strategy that decouples safety recovery from client-side training by automatically generating aligned and normal instruction–response data and performing lightweight fine-tuning on the aggregated model. This defense is shown to be plug-and-play, scalable, and capable of restoring safety performance by up to 69% without significantly degrading helpfulness. Zheng et al. [ 62] propose FL-GLM, a secure and efficient split-learning-based federated learning framework for LLMs, addressing the incompatibility of conventional FL (e.g., FedAvg) with the heavy computation and privacy risks of large models. The framework places the embedding layer and the first and last transformer blocks on the client, keeping intermediate blocks on the server to prevent embedding-gradient leakage, while encrypting all client–server “smashed data” via public-key encryption to guard against peer-client eavesdropping. To mitigate split learning’s low efficiency, FL-GLM introduces client-batch and server-hierarchical parallelism, cutting training time by over 48% versus serial execution. While limited to ChatGLM-scale models and still incurring communication overhead, FL-GLM offers a practical blueprint showing that encrypted, parallelized split architectures can jointly reconcile utility, efficiency, and privacy in federated LLM training. Huang et al. [ 23] propose a fast, accurate, and security-preserving distributed training framework for LLMs that addresses parameter and data leakage risks in federated settings. Motivated by the limitations of prior defenses: DP’s accuracy loss and the heavy overhead of multi-party computation, the authors design a model slicing-based secure architecture integrating trusted execution environments (TEEs) with lightweight encryption, via two complementary schemes (i)Method 1; (i)Method 2.Method 1targets consumer-grade, small-memory TEEs (e.g., Intel SGX) by isolating sensitive fine-tuned components (LoRA or P-Tuning v2 embeddings) within client- and server-side TEEs, protecting in- termediate activations exchanged with GPUs via one-time-pad encryption.Method 2improves efficiency and accuracy further by splitting the LLM by layers, deploying deeper layers in a large-memory server-side TEE (Intel TDX/SGX) while freezing client-side layers. A novel sparsification parameter fine-tuning (SPF) strategy selectively updates key attention heads and combines LoRA for MLP layers. FedShield-LLM [36] is a secure, scalable federated fine-tuning framework for LLMs that addresses the trade-off be- tween strong privacy protection and high model utility in cross-silo FL. Motivated by the vulnerability of standard federated LLM fine-tuning to inference attacks (e.g., gradient inversion, membership inference) and the performance degradation caused by noise-based defenses like DP, FedShield-LLM integrates PEFT (LoRA) with unstructured prun- ing and Fully Homomorphic Encryption (FHE). Clients fine-tune only low-rank LoRA adapters on local data, prune less informative adapter weights to reduce attack surface and communication cost, and encrypt updates via the CKKS scheme, enabling the server to aggregate encrypted parameters without accessing plaintext. The authors provide the- oretical convergence guarantees, showing the encrypted, sparsified FedAvg procedure retains sublinear convergence, and formal security arguments demonstrating resistance to gradient inversion and inference attacks under standard cryptographic assumptions. Experiments on LLaMA-2 (7B and 13B) across medical, financial, mathematical, and gen- eral instruction-tuning benchmarks demonstrate FedShield-LLM consistently outperforms Vanilla FL and DP-LoRA in convergence speed, training stability, and text quality, achieving BERTScore gains while approaching the response quality of GPT-4-class models. Manuscript submitted to ACM 26Q.Yang C.Qiu, H.Zhang et al. However, in federated graph learning, the participants may differ not only in graph structures and feature distribu- tions but also in downstream tasks, including node-, edge-, and graph-level prediction. To address this multifaceted heterogeneity in federated graph learning, Guoet al.[19] propose FedGPL, an asymmetric federated graph prompt- learning framework that separates universal graph representation learning from client-specific prompting and pre- diction. FedGPL integrates a server-hosted GNN with locally maintained graph prompts and task heads. Meanwhile, the hierarchical directed transfer aggregator (HiDTA) estimates directional transferability between clients and per- forms personalized aggregation to encourage beneficial cross-task knowledge transfer while suppressing potentially harmful contributions. The Virtual Prompt Graph (VPG) adaptively adds informative virtual structures and removes or neutralizes redundant nodes and edges, thereby preserving domain-specific patterns while reducing graph-data di- vergence. FedGPL additionally employs split learning and differential-privacy noise to protect the graph embeddings and gradients exchanged between clients and the server. The principal contribution therefore lies in jointly addressing graph-task and graph-data heterogeneity through asymmetric transferability-aware aggregation and adaptive struc- tural prompting, while retaining parameter and communication efficiency. The accompanying theoretical analysis pro- vides conditional support for heterogeneity reduction: HiDTA requires positive bidirectional transferability, whereas the VPG analysis assumes particular graph-representation distributions and similar prompting ratios across clients. Nevertheless, the empirical evaluation is restricted to five benchmark graph datasets, generally uses three clients per task level, assumes full participation and one local training step per round, and adopts a semi-honest threat model. Moreover, stronger privacy constraints reduce predictive accuracy, indicating a non-trivial privacy–utility trade-off. Khanet al.[ 24] address the largely unexplored security risks of FPL, showing that communication-efficient prompt updates remain vulnerable to targeted backdoor attacks despite a frozen vision-language backbone. It introduces a learnable, visually imperceptible noise trigger that malicious clients inject into local images to shift their CLIP rep- resentations toward an attacker-chosen target class, preserving accuracy on clean inputs while causing systematic misclassification of triggered samples. To counter this, the paper proposes selective and accurate backdoor rejection for federated prompt learning (SABRE-FL), a lightweight server-side defense that detects and removes poisoned client updates via embedding-space anomaly detection. The attack adapts the centralized BadCLIP mechanism to a federated, prompt-only setting: each compromised client jointly optimizes a local prompt and an additive noise trigger, relabeling triggered images to a fixed target class. Al- though the CLIP encoders stay frozen, the trigger shifts image embeddings toward the target-class text representation, and the resulting malicious prompt updates propagate through FedAvg aggregation. Under the default threat model, the adversary controls 25% of clients, knows the public backbone and training process, and can modify local data and labels but not server aggregation directly. The theoretical argument also depends on triggered and clean embeddings maintaining a stable separation margin and on the auxiliary detector achieving near-zero training error, so robustness against adaptive attackers minimizing this separation, alternative trigger types, and direct model-poisoning attacks remains unestablished. The method fur- ther assumes a boundable number of malicious clients and is evaluated mainly with a single CLIP ViT-B/16 backbone, with the authors flagging model poisoning, broader threat models, and additional vision-language architectures as directions for future work. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions27 6.2 Privacy Issues in FPL Luoet al.[34] systematically examine FL as a privacy-preserving paradigm for fine-tuning LLMs in automated program repair (APR), motivated by the growing reliance of LLM-based repair methods on high-quality industrial code that can- not be centrally shared due to confidentiality constraints. Using a private industrial dataset (TutorCode) for training and the leakage-resistant EvalRepair-Java benchmark for evaluation, the authors investigate three core challenges: whether federated fine-tuning can effectively enhance repair performance, how code heterogeneity (feature-skewed Non-IID data) influences generative repair tasks, and how different FL algorithms impact optimization. By adopting parameter-efficient fine-tuning (QLoRA), the framework significantly reduces communication and memory overhead while preserving data locality. Extensive experiments across six state-of-the-art code LLMs show that federated fine- tuning consistently and substantially improves bug-fixing performance over local fine-tuning and often approaches, or even surpasses, centralized training, with gains of up to 16.6% in Top@10 and 18.4% in Pass@10. Surprisingly, het- erogeneous code distributions do not degrade performance and can even enhance generalization, highlighting the robustness of LLMs in feature-skewed federated generative tasks. While FedAvg emerges as the most stable and effec- tive aggregation strategy, personalized FL remains challenging for LLM-based repair. ROFED-LLM [ 48] is a robust, privacy-preserving federated learning framework for training LLMs in adversarial wireless environments, addressing the vulnerability of conventional federated LLM training to jamming attacks, de- vice heterogeneity, and privacy leakage. It integrates split federated learning with a multi-modal defense spanning both model and communication layers: on the learning side, LLMs are partitioned between clients and server, with local differential privacy (LDP), secure aggregation, and dynamic parameter pruning reducing information leakage and computational burden; on the communication side, adaptive beamforming, jamming detection, and resource al- location optimization enable real-time mitigation of interference and adversarial attacks over wireless channels. The authors provide convergence guarantees and privacy analysis, showing ROFED-LLM achieves sublinear convergence while maintaining formal DP. Experiments across diverse natural language processing (NLP) benchmarks (e.g., gen- eral language understanding evaluation benchmark (GLUE) [47], SQuAD, WikiText-103, FinancialPhraseBank) show it consistently outperforms FedAvg, FedProx, SCAFFOLD, and SplitFed. ROFED-LLM prioritizes FedLLM methods with a 47% lower privacy leakage and significantly greater robustness under severe jamming, with only marginal commu- nication overhead. 7 Experimental Benchmarking and Empirical Insights Besides of the systematic analysis for the existing work on the FPL topic, we conduct a unified empirical benchmark to complement the methodological taxonomy with reproducible quantitative evidence. A major difficulty in this area is that published results are often obtained under disparate task formulations, model backbones, data partitions, and eval- uation protocols, rendering direct comparison unreliable. Therefore, rather than pursuing state-of-the-art performance, this benchmark evaluates representative federated PEFT strategies under a strictly consistent experimental protocol, thereby enabling a fair and controlled comparison across methods originally tested in heterogeneous settings. 7.1 Unified Benchmark Evaluation Table 4reports the core benchmark results on AG News using TinyLlama-1.1B-Chat as the backbone. All methods were trained with four clients over 100 communication rounds and evaluated on the same held-out test set using the post hoc candidate-label scoring protocol. We report the mean and standard deviation over three random seeds. Manuscript submitted to ACM 28Q.Yang C.Qiu, H.Zhang et al. Table 4. Unified benchmark results of federated PEFT methods on AG News using TinyLlama-1.1B-Chat. Values are reported as mean±standard deviation over three random seeds. SettingMethod Predictive PerformanceAdaptation ScaleSystem Efficiency Acc. (%)↑Test Loss↓Params (M)↓Trainable (%)↓Download (MB)↓FL Time (min)↓ IID LoRA88.78 ± 0.63 0.091 ± 0.0011.130.10235.0476.80 ± 0.53 Prompt tuning82.57 ± 7.28 0.133 ± 0.0480.200.01860.11110.23 ± 0.84 P-tuning81.74 ± 1.89 0.115 ± 0.00912.631.13510.3982.46 ± 0.89 LDA훼 = 0.1 LoRA88.85 ± 0.12 0.093 ± 0.0021.130.10235.0478.55 ± 3.19 Prompt tuning83.43 ± 3.85 0.142 ± 0.0490.200.01860.11111.80 ± 5.85 P-tuning77.56 ± 13.24 0.151 ± 0.04912.631.13510.3989.19 ± 13.00 LDA훼 = 0.5 LoRA88.95 ± 0.35 0.091 ± 0.0021.130.10235.0486.95 ± 13.08 Prompt tuning83.10 ± 6.28 0.145 ± 0.0770.200.01860.11113.90 ± 6.48 P-tuning73.45 ± 18.28 0.116 ± 0.01412.631.13510.3982.23 ± 0.44 LDA훼 = 1.0 LoRA88.76 ± 0.08 0.094 ± 0.0041.130.10235.0478.94 ± 3.62 Prompt tuning79.69 ± 11.77 0.142 ± 0.0540.200.01860.11113.43 ± 3.91 P-tuning83.65 ± 3.63 0.116 ± 0.00912.631.13510.3989.56 ± 11.30 Test Loss is the server-side weighted-average test loss reported by the federated framework. Download denotes the average download volume per worker reported by the framework-level system monitor; the original values are converted to MB for consistent presentation. All experiments use four clients, full client participation, 100 communication rounds, and the same held-out evaluation set. Several consistent observations emerge from the benchmark results. First, LoRA provides the most stable perfor- mance among the compared methods. Its accuracy remains around 88–89% across IID and all LDA-based non-IID settings, with consistently small standard deviations. These results suggest that LoRA is comparatively robust to the evaluated data partition changes, although it incurs a larger communication payload than prompt-based methods. Second, prompt tuning achieves the lowest trainable parameter ratio and the smallest average download volume, but its predictive performance is more sensitive to random seed and data partition. This suggests that prompt tuning provides a strong communication-efficiency advantage, but its optimization stability is weaker than LoRA under the current federated setting. Third, P-tuning does not consistently outperform prompt tuning despite using substantially more trainable param- eters. In particular, it exhibits large variance under LDA훼 = 0.1and훼 = 0.5, indicating that increasing the prompt- encoder parameter scale does not automatically translate into more stable federated adaptation. This observation is important because it suggests that the practical value of a parameter-efficient method should be judged jointly by accuracy, stability, trainable scale, and communication behavior rather than by parameter count alone. 7.2 Empirical Analysis of Performance, Robustness, and Reproducibility Fig. 2complements the quantitative results in Table4by visualizing the empirical characteristics of representative fed- erated PEFT methods from three complementary perspectives, namely communication efficiency, robustness under heterogeneous federated settings, and reproducibility across random seeds. Several important empirical observations can be derived from the benchmark results. (1) The compared methods exhibit substantially different robustness profiles under federated heterogeneity. Among all evaluated methods, LoRA consistently demonstrates the strongest stability across IID and non-IID settings. Its task accuracy remains highly stable across different random seeds and LDA parti- tion conditions, while its test loss also remains comparatively well-controlled. This behavior highlights that low-rank parameter adaptation provides a relatively robust optimization path for federated LLM fine-tuning, even when client distributions become heterogeneous. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions29 (a)(b)(c) Fig. 2. Comparison of representative federated PEFT methods under the unified benchmark from three complementary perspectives. (a) Performance–efficiency trade-off between predictive accuracy and communication cost. (b) Robustness under IID and progres- sively heterogeneous federated data partitions. (c) Reproducibility across random seeds, where each point denotes one method under one partition setting and is positioned according to its mean accuracy and cross-seed standard deviation. Error bars indicate the standard deviation over three random seeds. (2) Communication-efficient methods do not necessarily preserve stable predictive behavior under federated opti- mization. Prompt tuning achieves the smallest trainable parameter ratio and the lowest communication volume among all evaluated methods. However, its performance variance becomes noticeably larger under certain non-IID settings, indicating that extreme parameter compression may increase optimization sensitivity in federated environments. In particular, several prompt-tuning runs show substantial seed-dependent fluctuations despite using the same backbone model and evaluation protocol. (3) Larger trainable parameter scales do not automatically guarantee stronger robustness. Although P-tuning intro- duces considerably more trainable parameters than prompt tuning, it exhibits the greatest instability across several heterogeneous settings, including occasional performance collapse with moderate non-IID partitions. This observation suggests that federated prompt learning performance is influenced not only by the scale of trainable parameters but also by how the parameterization interacts with heterogeneous local optimization dynamics. In summary, our benchmark demonstrates that federated PEFT methods must be evaluated as a multi-objective de- sign problem rather than through predictive accuracy alone. Specifically, these findings suggest that communication- efficient adaptation should not be evaluated solely by parameter reduction; indeed, a strategy with superior parameter efficiency may suffer from heightened optimization instability, whereas a more stable method may incur substantially larger communication overhead. Consequently, practical federated deployment requires balancing these distinct trade- offs—spanning effectiveness, optimization robustness, parameter footprint, and communication cost—all of which con- stitute complementary dimensions in the design space of federated prompt learning instead of being optimized in isolation. 8 Insights and Future Research Although we have reviewed many related studies, the integration of FL and LLMs remains at an early stage of maturity when judged against the recurring limitations acknowledged throughout the surveyed studies: simulated heterogeneity, small or moderate client populations, single-backbone evaluation, informally justified privacy claims, and fragmented experimental protocols. These limitations become sharper still as the field moves from static model adaptation toward Manuscript submitted to ACM 30Q.Yang C.Qiu, H.Zhang et al. federated agentic systems, where models act, retrieve, and communicate autonomously rather than simply producing text. Synthesizing these gaps, we organize future research into nine directions spanning evaluation, theory, systems, and increasingly central trustworthiness. 8.1 Standardized, Reproducible, and Multi-Objective Evaluation As our unified benchmark in Section7demonstrates, methods with comparable accuracy can differ sharply in stability, communication cost, and seed-to-seed variance—dimensions that most published results, obtained under disparate backbones, partitions, and protocols, do not jointly report. Future work should move toward shared benchmark suites with multi-seed reporting and standardized non-IID partition generators, so that communication efficiency, robustness, and parameter footprint are treated as co-equal objectives rather than being subordinated to a single accuracy figure. 8.2 Theoretical Foundations for Federated Prompt and PEFT Optimization Convergence guarantees remain largely confined to simplified regimes (e.g., PromptFL’s 풪 (1/ √ 푇) bound [ 18]), while empirical phenomena such as the generalization–personalization trade-off in FedPGP [11], the concept-drift dynamics in PFPT [ 50], or the ID–OOD separation margin in FOCoOp [28] are supported only empirically. A principled theory connecting prompt rank, capacity, and domain shift to generalization bounds under non-IID, nonlinear transformer dynamics is still missing, and would let practitioners choose prompt configurations analytically rather than by grid search. 8.3 Heterogeneity- and Resource-Aware Optimization at Realistic Scale FlexLoRA [3], FAH-QLoRA [16], and Fed-HeLLo [60] all address resource heterogeneity, but under simulated condi- tions, moderate model sizes, and limited client counts. Real deployments will involve thousands of genuinely hetero- geneous devices and clients pursuing different tasks—not merely different label or domain distributions—so validating these mechanisms on larger LLM backbones and authentic edge testbeds is a necessary next step. 8.4 LLMs as Intelligent Controllers of Federated Systems Section IV’s LLM-enhanced FL largely covers data augmentation and pseudo-labeling; a less explored direction is using LLMs as adaptive controllers—for client selection, aggregation-weight design, hyperparameter tuning, and anomalous- update detection—closing the loop between “FL for LLMs” and “LLMs for FL” within a single self-improving system. 8.5 Federated Agentic Ecosystems: Retrieval, Multi-Agent Collaboration, and Verifiable Trust Federated RAG and federated multi-agent systems were introduced in Section I as part of the synergistic-integration category, yet neither is represented by a dedicated study in Section V-B, marking a clear literature gap. Open prob- lems include collaboratively optimizing retrievers or knowledge representations across clients that each retain private knowledge bases, and designing protocols for LLM agents to share task experience or tool-use knowledge without exposing local memory or data. As such systems mature, trust cannot rest on single-round update verification alone: unlike the anomaly-detection defenses developed for prompt-only settings such as SABRE-FL [24] and FedGPL [19], an agent’s trustworthiness must be assessed over extended, multi-step interaction trajectories, where one manipulated tool call or retrieved document can silently corrupt downstream reasoning. Future work should therefore pair federated RAG and multi-agent architectures with trajectory-level verification mechanisms—potentially combining lightweight Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions31 cryptographic attestation, verifiable computation, or provenance logging—that certify agent behavior without requir- ing centralized visibility into private client data. 8.6 Privacy Protection Beyond Training: From Formal Guarantees to Inference-Time and Agentic Leakage Several methods justify privacy only indirectly—e.g., PLAN [17] argues that prompts are a compact, less data-revealing medium without formal leakage bounds, and SABRE-FL [24] requires exchanging raw embeddings without quantify- ing the resulting risk. Even DP-FPL [44], despite offering formal( 푒푝푠푖푙표푛, 훿)guarantees, evaluates them against a single shadow-model attack. This gap is sharpened by the theoretical result of Nikolaou et al. [37] that decoder-only Transformers are almost surely invertible, implying that exchanged hidden representations may be reconstructible in principle. The risk extends beyond training: agentic deployments introduce persistent memory, retrieved docu- ments, and inter-agent messages exchanged at inference time, which can leak sensitive local context even when model weights stay private. Future work should derive leakage bounds specific to prompt-, embedding-, and memory-space exchange, and test defenses—privacy-preserving memory management, encrypted or federated retrieval—against adap- tive, reconstruction-aware adversaries rather than fixed, known attacks. 8.7 Safety Alignment and Accountability in Federated and Agentic Deployments Ye et al. [ 57] show that safety alignment achieved during centralized instruction tuning can be substantially eroded by a small fraction of poisoned federated clients, with conventional robust aggregation providing only marginal pro- tection. This risk is amplified in agentic settings, where a misaligned or jailbroken local model may issue harmful tool calls or propagate unsafe behavior to other agents, rather than merely producing an unsafe text response. Extending post-hoc safety-recovery strategies—such as the server-side realignment fine-tuning in [ 57], to agentic action spaces, and moving from one-time post-training checks to continuous, round-wise safety auditing, remain open problems. This challenge is compounded by an accountability gap: because no single party observes the complete federation- wide interaction trace, attributing a faulty or harmful agent decision to a specific client, prompt, or aggregation step is substantially harder than in centralized settings, particularly in regulated domains such as healthcare (FedMRG [ 5]) and IoT management (Otoum [ 38]) already surveyed in this paper. Future research should explore lightweight, privacy- compatible audit trails and explanation mechanisms that preserve data locality while still enabling post-hoc account- ability and regulatory compliance. 8.8 Incentive Design and Real-World Deployment Most surveyed methods assume full or voluntary client participation, yet clients holding valuable proprietary data or compute have limited reason to contribute without appropriate incentives. Coupling incentive-mechanism design with energy- and communication-aware scheduling, and validating results through real-world case studies rather than simulation alone, is needed to translate the substantial methodological progress reviewed in this survey, together with the trustworthiness safeguards outlined above, into deployable, accountable federated LLM systems. Answers to RQ3:Despite substantial methodological progress, federated prompt learning continues to face sig- nificant security, privacy, robustness, and system challenges. On the security front, federated instruction tuning re- mains vulnerable to stealthy data-poisoning attacks that erode safety alignment while evading conventional robust aggregation, and prompt-only systems remain susceptible to backdoor attacks via imperceptible triggers even when Manuscript submitted to ACM 32Q.Yang C.Qiu, H.Zhang et al. the backbone stays frozen, with existing defenses (e.g., SABRE-FL, FedGPL) yet to be validated against adaptive at- tackers or broader threat models. Privacy protections are often justified only informally—relying on the implicit as- sumption that prompts are less data-revealing—rather than through formal guarantees, and even methods offering DP bounds (e.g., DP-FPL) are tested against limited attack models; this gap is compounded by theoretical evidence that transformer representations may be invertible in principle, and by largely unexamined leakage risks arising from persistent memory and inter-agent communication in agentic deployments. Robustness remains constrained by sim- ulated (rather than real-world) heterogeneity, small client populations, single-backbone evaluation, and fragmented experimental protocols, while accountability is undermined by the absence of any single party observing the complete federation-wide interaction trace. Synthesizing these gaps, the survey outlines future directions spanning standardized multi-objective evaluation, theoretical foundations connecting prompt capacity to generalization under non-IID con- ditions, heterogeneity- and resource-aware optimization at realistic scale, LLMs as intelligent controllers of federated systems, trustworthy federated agentic ecosystems with trajectory-level verification, privacy protection extending be- yond training to inference-time and memory-space leakage, continuous safety auditing and accountability mechanisms for agentic deployments, and incentive design coupled with real-world validation—collectively underscoring that the field remains at an early stage of maturity as it transitions from static model adaptation toward autonomous, agentic federated systems. 9 Conclusion This survey systematically reviewed federated prompt learning across model adaptation, personalization, inference, applications, and security. Existing studies show that prompts and lightweight adapters can substantially reduce the communication, computation, and storage costs of federated LLM fine-tuning while keeping raw data local. However, methods differ considerably in their ability to handle non-IID data, resource heterogeneity, personalization, scalabil- ity, and privacy. The unified benchmark further indicates that federated PEFT should be evaluated as a multi-objective problem. LoRA generally offers more stable performance, whereas prompt-based methods achieve lower communica- tion costs but may be more sensitive to heterogeneous optimization. Future research should focus on resource-aware and adaptive optimization, stronger personalization and generalization, standardized evaluation, and formal protection against privacy leakage, poisoning, backdoor, and reconstruction attacks. Acknowledgments This work was partially supported by National Natural Science Foundation of China under Grant No.U25B2030; Na- tional Natural Science Foundation of China under Grant No.T2522011; GuangDong Basic and Applied Basic Research Foundation under Grant No. 2025B1515020022, 2026A1515010183. References [1]Alekh Agarwal, Dean P Foster, Daniel J Hsu, Sham M Kakade, and Alexander Rakhlin. 2011. Stochastic convex optimization with bandit feedback. Advances in neural information processing systems24 (2011). [2]Ankur Agarwal, Mehdi Rezagholizadeh, and Prasanna Parthasarathi. 2023. Practical takes on federated learning with pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2023 . 454–471. [3]Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. 2024. Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems37 (2024), 14457–14483. [4]Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology15, 3 (2024), 1–45. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions33 [5]Haoxuan Che, Haibo Jin, Zhengrui Gu, Yi Lin, Cheng Jin, and Hao Chen. 2025. Llm-driven medical report generation via communication-efficient heterogeneous federated learning.IEEE Transactions on Medical Imaging(2025). [6]Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou. 2023. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 7871–7888. [7]Chaochao Chen, Xiaohua Feng, Yuyuan Li, Lingjuan Lyu, Jun Zhou, Xiaolin Zheng, and Jianwei Yin. 2024. Integration of large language models and federated learning.Patterns5, 12 (2024). [8]Jiao Chen, Jiayi He, Fangfang Chen, Zuohong Lv, Jianhua Tang, and Yunjian Jia. 2024. Empowering IoT-based autonomous driving via federated instruction tuning with feature diversity.IEEE Internet of Things Journal(2024). [9]Yuliang Chen, Xi Lin, Chao Sang, and Xiu Su. 2025. DualFPT: Handling Data Heterogeneity in Federated Prompt Tuning from both Generalized and Personalized Perspective. InProceedings of the 33rd ACM International Conference on Multimedia. 2890–2899. [10]Yujun Cheng, Weiting Zhang, Zhewei Zhang, Chuan Zhang, Shengjin Wang, and Shiwen Mao. 2024. Towards federated large language models: Motivations, methods, and future directions.IEEE Communications Surveys & Tutorials(2024). [11]Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi. 2024. Harmonizing generalization and personalization in federated prompt learning. InProceed- ings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 381, 16 pages. [12]Shengwen Ding and Chenhui Hu. 2024. efedllm: Efficient llm inference based on federated learning.arXiv preprint arXiv:2411.16003(2024). [13]Yaxin Du, Rui Ye, Fengting Yuchi, Wanru Zhao, Jingjing Qu, Yanfeng Wang, and Siheng Chen. 2025. FedDQC: Data quality control in federated instruction-tuning of large language models. In Findings of the Association for Computational Linguistics: ACL 2025 . 15267–15291. [14]Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. 2023. Fate-llm: A industrial grade federated learning framework for large language models.arXiv preprint arXiv:2310.10049(2023). [15]Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine- tuning. InProceedings of the AAAI conference on artificial intelligence, Vol. 37. 12799–12807. [16]Zhidong Gao, Zhenxiao Zhang, Yuanxiong Guo, and Yanmin Gong. 2025. Federated adaptive fine-tuning of large language models with heteroge- neous quantization and lora. InIeee infocom 2025-ieee conference on computer communications. IEEE, 1–10. [17]Shuai Gong, Chaoran Cui, Chunyun Zhang, Wenna Wang, Xiushan Nie, and Lei Zhu. 2026. Federated domain generalization via prompt learning and aggregation.IEEE Transactions on Information Forensics and Security(2026). [18]Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. 2023. Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.IEEE Transactions on Mobile Computing23, 5 (2023), 5179–5194. [19]Zhuoning Guo, Ruiqian Han, and Hao Liu. 2024. Against multifaceted graph heterogeneity via asymmetric federated prompt learning.arXiv preprint arXiv:2411.02003(2024). [20]Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer.Advances in neural information processing systems34 (2021), 15908–15919. [21]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations.https://openreview.net/forum?id=nZeVKeeFYf9 [22]Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. InProceedings of the 2023 conference on empirical methods in natural language processing. 5254–5276. [23]Wei Huang, Yinggui Wang, Anda Cheng, Aihui Zhou, Chaofan Yu, and Lei Wang. 2024. A fast, performant, secure distributed training framework for llm. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 4800–4804. [24]Momin Khan, Yasra Chandio, and Fatima Anwar. 2026. SABRE-FL: Selective and accurate backdoor rejection for federated prompt learning. In International Conference on Learning Representations, Vol. 2026. 157155–157176. [25]Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5260–5271. [26]Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions.IEEE signal processing magazine37, 3 (2020), 50–60. [27]Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers). 4582–4597. [28]Xinting Liao, Weiming Liu, Jiaming Qian, Pengyang Zhou, Jiahe Xu, Wenjie Wang, Chaochao Chen, Xiaolin Zheng, and Tat-Seng Chua. 2025. FOCoOp: enhancing out-of-distribution robustness in federated prompt learning for vision-language models(ICML’25). JMLR.org, Article 1489, 27 pages. [29]Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao. 2024. Splitlora: A split parameter-efficient fine-tuning framework for large language models.arXiv preprint arXiv:2407.00952(2024). [30]Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine- tuning across scales and tasks. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Manuscript submitted to ACM 34Q.Yang C.Qiu, H.Zhang et al. 61–68. [31]Pingchen Lu, Zhi Hong, Zhiwei Shang, Zhiyong Wang, Yikun Ban, Yao Shu, Min Zhang, Shuang Qiu, and Zhongxiang Dai. 2025. FedPOB: Sample- Efficient Federated Prompt Optimization via Bandits.arXiv preprint arXiv:2509.24701(2025). [32]Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng. 2023. The multi-modal fusion in visual question answering: a review of attention mechanisms.PeerJ Computer Science9 (2023), e1400. [33]Jun Luo, Chen Chen, and Shandong Wu. 2025. Mixture of experts made personalized: Federated prompt learning for vision-language models. In International Conference on Learning Representations, Vol. 2025. 58061–58080. [34]Wenqiang Luo, Jacky Keung, Boyang Yang, He Ye, Claire Le Goues, Tegawende F Bissyande, Haoye Tian, and Xuan Bach D Le. 2024. When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair.ACM Transactions on Software Engineering and Methodology(2024). [35]Shubham Malaviya, Manish Shukla, and Sachin Lodha. 2023. Reducing communication overhead in federated learning for pre-trained language models using parameter-efficient finetuning. InConference on Lifelong Learning Agents. PMLR, 456–469. [36]Md Jueal Mia and M Hadi Amini. 2025. FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model.arXiv preprint arXiv:2506.05640(2025). [37]Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, and Emanuele Rodolà. 2025. Language models are injective and hence invertible.arXiv preprint arXiv:2510.15511(2025). [38]Yazan Otoum, Arghavan Asad, and Amiya Nayak. 2025. Llms meet federated learning for scalable and secure iot management.arXiv preprint arXiv:2504.16032 (2025). [39]Zhen Qin, Zhaomin Wu, Bingsheng He, and Shuiguang Deng. 2025. Federated data-efficient instruction tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2025. 15550–15568. [40]Wenqi Qiu, Yipeng Zhou, Jinzhi Wang, Quan Z Sheng, and Laizhong Cui. 2025. Flm-topk: Expediting federated large language model tuning by sparsifying intervalized gradients. InIEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 1–10. [41]Arian Raje. 2024.Communication-efficient LLM training for federated learning. Ph. D. Dissertation. Carnegie Mellon University Pittsburgh, PA. [42]Guangyu Sun, Matias Mendieta, Jun Luo, Shandong Wu, and Chen Chen. 2023. Fedperfix: Towards partial model personalization of vision trans- formers in federated learning. InProceedings of the IEEE/CVF international conference on computer vision. 4988–4998. [43]Dipanwita Thakur, Antonella Guzzo, and Giancarlo Fortino. 2025. Analyzing the Fusion of Federated Learning and Large Language Model. In2025 IEEE 5th International Conference on Human-Machine Systems (ICHMS). IEEE, 282–288. [44]Linh Tran, Wei Sun, Stacy Patterson, and Ana Milanova. 2025. Privacy-preserving personalized federated prompt learning for multimodal large language models. InInternational Conference on Learning Representations, Vol. 2025. 27362–27380. [45]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017). [46]Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? Limits of LLM scaling based on human-generated data. InForty-first International Conference on Machine Learning. [47]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP. 353–355. [48]Haoyu Wang, Zilong Yin, Bin Chen, Yujie Zeng, Xiyue Yan, Chenyu Zhou, and Anji Li. 2025. Rofed-LLM: robust federated learning for large language models in adversarial wireless environments.IEEE Transactions on Network Science and Engineering(2025). [49]Zhihao Wang, Wenke Huang, Tian Chen, Zekun Shi, Guancheng Wan, Yu Qiao, Bin Yang, Jian Wang, Bing Li, and Mang Ye. 2025. An empirical study of federated prompt learning for vision language model(IJCAI ’25). Article 1188, 9 pages.doi:10.24963/ijcai.2025/1188 [50]Pei-Yau Weng, Minh Hoang, Lam M Nguyen, My T Thai, Tsui-Wei Weng, and Trong N Hoang. 2024. Probabilistic federated prompt-tuning with non-IID and imbalanced data.Advances in neural information processing systems37 (2024), 81933–81958. [51]Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. 2024. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3345–3355. [52]Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, and Changsheng Xu. 2025. Pilot: Building the federated multimodal instruction tuning framework. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 21716–21724. [53]Na Yan, Yang Su, Yansha Deng, and Robert Schober. 2025. Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions.arXiv preprint arXiv:2501.04436(2025). [54]Fu-En Yang, Chien-Yi Wang, and Yu-Chiang Frank Wang. 2023. Efficient model personalization in federated learning via client-specific prompt generation. InProceedings of the IEEE/CVF International Conference on Computer Vision. 19159–19168. [55]Hongming Yang, Hao Liu, Xin Yuan, Kai Wu, Wei Ni, J Andrew Zhang, and Ren Ping Liu. 2025. Synergizing Intelligence and Privacy: A Review of Integrating Internet of Things, Large Language Models, and Federated Learning in Advanced Networked Systems.Applied Sciences15, 12 (2025), 6587. [56]Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.High-Confidence Computing4, 2 (2024), 100211. Manuscript submitted to ACM Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions35 [57]Rui Ye, Jingyi Chai, Xiangrui Liu, Yaodong Yang, Yanfeng Wang, and Siheng Chen. [n. d.]. Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models. InInternational Workshop on Federated Foundation Models in Conjunction with NeurIPS 2024. [58]Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language- models. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 1–9. [59]Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. 2021. A survey on federated learning.Knowledge-Based Systems216 (2021), 106775. [60]Zikai Zhang, Ping Liu, Jiahao Xu, and Rui Hu. 2025. Fed-hello: Efficient federated foundation model fine-tuning with heterogeneous lora allocation. IEEE Transactions on Neural Networks and Learning Systems36, 10 (2025), 17556–17569. [61]Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. 2023. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. [62]Jia-Ying Zheng, HaiNan Zhang, LingXiang Wang, WangJie Qiu, Hong-Wei Zheng, and Zhi-Ming Zheng. 2024. Safely learning with private data: A federated learning framework for large language model. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 5293–5306. [63]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 16795–16804. Manuscript submitted to ACM

Instant research discovery