Paper deep dive
Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization
Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez, Nathaniel D. Bastian, Jia Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/13/2026, 3:35:37 AM
Summary
This paper proposes a Foundation Model-aided Multi-Agent Reinforcement Learning (FM-MARL) framework to optimize wireless Random Access (RA) networks. It addresses the high training overhead and lack of generalization in conventional MARL by integrating a pretrained Foundation Model (FM) as a reward-agnostic backbone within a decentralized actor-critic architecture. The approach utilizes a consensus-based decentralized mechanism for local reward exchange and provides theoretical convergence analysis, demonstrating improved learning speed and performance across multiple optimization objectives (Fair AoI, Max-Rate, Fair-Rate) compared to end-to-end MARL.
Entities (10)
Relation Signals (8)
FM-MARL → publishedin → MobiHoc '26
confidence 99% · Conference: The Twenty-seventh International Symposium... MobiHoc ’26
FM-MARL → optimizes → Random Access (RA)
confidence 98% · Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization
FM-MARL → employs → Actor-Critic
confidence 95% · design an FM-aided actor-critic algorithm
FM-MARL → uses → Foundation Model (FM)
confidence 95% · we propose to leverage a foundation model (FM) to improve MARL efficiency... pretrained FM serves as a reward-agnostic backbone
FM-MARL → uses → Consensus Mechanism
confidence 92% · incorporate an average consensus mechanism... allowing devices to synchronize via local peer-to-peer communication
FM-MARL → supports → Fair AoI
confidence 90% · consider three RA optimization objectives, namely fair age of information (AoI)
FM-MARL → supports → Max-Rate
confidence 90% · consider three RA optimization objectives, namely... maximum sum-rate
FM-MARL → supports → Fair-Rate
confidence 90% · consider three RA optimization objectives, namely... fair rate
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Random access (RA) is one of the most foundational medium access control (MAC) layer scheduling schemes for handling unpredictable data traffic from multiple terminals. While multi-agent reinforcement learning (MARL) has been explored to optimize RA-based wireless networks, its reliance on experience-driven, distributed policy learning incurs significant training overhead for each optimization task, limiting its feasibility in real-world applications. In this work, we propose to leverage a foundation model (FM) to improve MARL efficiency across diverse RA network optimization tasks. Specifically, we design an FM-aided actor-critic algorithm within a consensus-based decentralized MARL architecture and provide its convergence analysis under local reward exchanges and nonlinear value function approximations to show that our algorithm achieves the same convergence order as the conventional MARL with critic model exchanges and linear approximations. Our numerical results show that our FM-based approach significantly enhances MARL speed for RA network optimization.
Tags
Links
- Source: https://arxiv.org/abs/2610.07550v1
- Canonical: https://arxiv.org/abs/2610.07550v1
Trouble viewing inline? Open PDF directly →
Full Text
70,070 characters extracted from source content.
Expand or collapse full text
Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization Conference: The Twenty-seventh International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing; November 23–26, 2026; Tokyo, JapanThe Twenty-seventh International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc ’26), November 23–26, 2026, Tokyo, JapanDOI: 10.1145/3842721.3849640ISBN: 979-8-4007-2975-1/2026/11CCS: Networks Network control algorithms Myeung Suk Oh email: oh.746@osu.edu Affiliation: The Ohio State University, Columbus, Ohio, USA , Zhiyao Zhang email: zhang.15178@osu.edu Affiliation: The Ohio State University, Columbus, Ohio, USA , Alvaro Velasquez email: alvaro.velasquez@colorado.edu Affiliation: University of Colorado Boulder, Boulder, Colorado, USA , Nathaniel D. Bastian email: nbastian@pathfinderappliedresearch.com Affiliation: Pathfinder Applied Research, LLC, Ashburn, Virginia, USA and Jia Liu email: liu@ece.osu.edu Affiliation: The Ohio State University, Columbus, Ohio, USA © rightsretained Abstract. Random access (RA) is one of the most foundational medium access control (MAC) layer scheduling schemes for handling unpredictable data traffic from multiple terminals. While multi-agent reinforcement learning (MARL) has been explored to optimize RA-based wireless networks, its reliance on experience-driven, distributed policy learning incurs significant training overhead for each optimization task, limiting its feasibility in real-world applications. In this work, we propose to leverage a foundation model (FM) to improve MARL efficiency across diverse RA network optimization tasks. Specifically, we design an FM-aided actor-critic algorithm within a consensus-based decentralized MARL architecture and provide its convergence analysis under local reward exchanges and nonlinear value function approximations to show that our algorithm achieves the same convergence order as the conventional MARL with critic model exchanges and linear approximations. Our numerical results show that our FM-based approach significantly enhances MARL speed for RA network optimization. Keywords: Foundation model, multi-agent reinforcement learning (MARL), random access, self-supervised learning, wireless networks 1. Introduction 1) Background and Motivations: Random Access (RA) techniques at the Medium Access Control (MAC) layer are fundamental to modern wireless standards, including Wi-Fi and LTE Licensed Assisted Access (Chen et al., 2016). In RA-based networks (Figure 1), multiple devices contend for a shared communication channel rather than relying on centralized scheduling. By allowing devices to independently determine transmission opportunities, RA eliminates the overhead associated with pre-allocated channel access. This decentralized paradigm is particularly well suited for the Internet of Things (IoT) (Aouedi et al., 2025), machine-type communications (MTC) (Mahmood et al., 2021), and smart grids (Rodriguez et al., 2023), where low-complexity and scalable mechanisms are required to manage highly sporadic and unpredictable traffic. Despite these advantages, the primary challenge in RA is maximizing network efficiency while mitigating collisions. Excessive contention leads to wasted communication resources, often diminishing the benefits of decentralized access. Although extensive research has sought to improve the performance of RA-based networks, existing approaches either rely on heuristic methods without performance guarantees or adopt idealized analytical models that struggle to capture the complexities of real-world environments. . Figure 1. An RA-based wireless network, where devices contend for channel access. Each device makes its own decision on when to transmit. Communication links are established across the devices for exchanging local information.. 2) Foundation Model-Aided Multi-Agent Reinforcement Learning: Recent advances in machine learning have introduced new opportunities for optimizing RA networks (Ahmad et al., 2020; Kulin et al., 2021; Cao et al., 2021). Due to their distributed nature, RA networks can naturally be modeled as multi-agent reinforcement learning (MARL) systems. Many existing approaches (Han et al., 2020; Lee et al., 2024; Yu et al., 2019; Yu et al., 2022; Zhang et al., 2020; Guo et al., 2022; He et al., 2024) utilize the centralized training and decentralized execution (CTDE) paradigm (Amato, 2024) to maintain learning stability. However, CTDE-based methods face fundamental limitations. Their reliance on a central training entity restricts deployment in large-scale or infrastructure-constrained settings. Even when such an entity exists, communication overhead on aggregating observations and actions scales poorly with the number of devices. Also, centralization can introduce single points of failure and raise concerns regarding privacy and adversarial vulnerability. Furthermore, conventional MARL methods often suffer from task-specific overfitting. Models designed for narrow objectives (e.g., collision minimization) typically require retraining or architectural modifications when adapting to new performance metrics, resulting in poor sample efficiency and limited generalization. These limitations motivate the need for MARL frameworks that are both fully decentralized and adaptable to diverse optimization objectives. Meanwhile, foundation models (FMs) pretrained on large-scale datasets and then finetuned for downstream tasks offer a promising path for reinforcement learning (RL). By modeling system dynamics as sequence processes (e.g., using architectures like MAMBA (Gu and Dao, 2023)), self-supervised FMs can learn reward-agnostic representations that allow faster policy adaptation and improved generalization across tasks. While there is growing interest in applying FMs to wireless systems (Du et al., 2024; Fontaine et al., 2024; Chen et al., 2024; Yang et al., 2025b), several open challenges remain. Particularly, it is unclear how to effectively integrate FMs into established RL frameworks (e.g., actor–critic methods) with theoretical performance guarantees, and how to design MARL algorithms that exploit the representational power of FMs without introducing centralized dependencies. This motivates us to pursue FM-aided MARL for wireless random access network optimization. 3) Our Approach and Contributions: In this work, we propose an FM-aided MARL framework for optimizing RA-based wireless networks. To ensure practical viability, we adopt a fully decentralized paradigm based on decentralized training and decentralized execution (DTDE). To maintain stability and guarantee convergence without a central coordinator, we incorporate an average consensus mechanism (Boyd et al., 2006), allowing devices to synchronize via local peer-to-peer communication. Our proposed framework employs a specialized actor-critic (AC) design, where a pretrained FM serves as a reward-agnostic backbone for the critic network. By attaching a task-specific head to this backbone, the critic benefits from expressive latent representations of system dynamics learned through self-supervised pretraining on reward-independent data. This integration accelerates online policy learning while achieving performance comparable to or better than end-to-end MARL training. Our main contributions are summarized as follows. • We consider a consensus-driven, fully decentralized MARL that eliminates the need for global controllers. Relying on local exchanges, the design is well suited for scenarios where centralized control is infeasible or undesirable. Unlike (Oh et al., 2025), we consider three RA optimization objectives, namely fair age of information (AoI), maximum sum-rate, and fair rate, and formulate unified Markov decision process (MDP) models with shared observation and action spaces, as well as locally computable reward functions. • We introduce a new method for integrating FMs into the actor–critic pipeline. Inspired by the SMART architecture (Sun et al., 2023), we pretrain an FM for forward-dynamics prediction and incorporate it as a reward-agnostic backbone within the critic. The resulting latent representations significantly enhance the critic’s ability to evaluate rewards during online learning. • Unlike (Sun et al., 2023), which is purely empirical, we provide a finite-time convergence analysis of our FM-aided decentralized actor–critic algorithm. Our analysis addresses consensus applied to local rewards and nonlinear state-value approximation, offering theoretical insights that differentiate this work from existing decentralized MARL literature. Through extensive numerical simulations, we show that our framework outperforms end-to-end training in learning speed across multiple RA optimization tasks. 2. Related Work 1) Traditional RA Network Optimization: RA techniques are broadly classified into two types: sensing-free and sensing-based. Sensing-free protocols, like ALOHA (Abramson, 1970), are valued for their simplicity in satellite and multi-hop networks (Yu et al., 2023; Baccelli et al., 2006) but suffer from inherently high collision rates. In contrast, sensing-based RA employs a listen-before-talk (LBT) mechanism. The most prominent example, CSMA/CA (Kleinrock and Tobagi, 1975), uses random backoff to mitigate contention. However, its heuristic nature often results in excessive delays and suboptimal channel utilization. Moreover, its theoretical performance guarantees remain unclear. While research into adaptive, queue-length-based CSMA (Jiang et al., 2010) shows that throughput optimality is possible, these findings often rely on idealized models that fail to maintain fairness or low latency as networks scale (Jiang and Walrand, 2009; Jiang et al., 2012; Ni et al., 2010). 2) Learning-driven RA Network Optimization: To overcome the limitations of heuristics, recent work has explored RL for RA. Early efforts focused on optimizing CSMA/CA parameters, such as using Q-learning or SAC-LSTM architectures to tune contention windows and waiting times (Han et al., 2020; Lee et al., 2024). However, since these methods remain dependent on the probabilistic framework of CSMA, their performance gains are fundamentally limited. To move beyond simply parameter tuning, researchers have turned to MARL to design deterministic transmission policies. In the literature, notable approaches include deep Q-networks for achieving α-fairness (Yu et al., 2019), federated learning-based policy training (Zhang et al., 2020), and value factorization or policy gradient methods such as QMIX (Guo et al., 2022) and MAPPO (He et al., 2024). Despite their strong performance, these methods rely on CTDE. The requirement for a central entity introduces significant communication overhead and raises concerns regarding scalability, data privacy, and security. Furthermore, these models are often task-specific. As a result, changing the optimization objective typically requires extensive architectural modification and retraining, leading to poor sample efficiency. These challenges motivate our development of a fully decentralized FM-aided MARL framework that allows efficient wireless RA network optimization under diverse objectives without central coordination. 3) Foundation Models for Wireless Networks: FMs have achieved state-of-the-art performance in domains such as natural language processing and computer vision through large-scale representation learning. In wireless communications, FMs are now being explored for PHY and MAC layer tasks (Du et al., 2024; Fontaine et al., 2024; Chen et al., 2024; Yang et al., 2025b), especially within semantic and integrated sensing and communication (ISAC) frameworks. In particular, their ability to generalize across heterogeneous tasks and domains makes them highly suitable for addressing the challenges posed by dynamic and complex wireless environments. Emerging works such as WirelessGPT (Yang et al., 2025a) and patch-masked FMs (Sheng et al., 2025) have demonstrated that Transformer-based architectures can effectively handle multi-task predictions, such as channel state estimation and traffic forecasting. Similar to this line of work, we investigate the use of FMs for wireless RA network optimization. Unlike existing RL methods that suffer from task-specific overfitting, we utilize an FM as a reward-agnostic backbone within an AC framework. This allows for superior generalization and faster adaptation across diverse networking objectives without the need for centralized coordination. Our approach aims to overcome the limitations of conventional RL methods that often struggle with their lack of adaptability and dependence on centralized training. 3. System Model and Problem Formulation 3.1. Network System and the RA Protocol We consider an RA-based wireless network with K single-antenna devices that contend for channel access to transmit data to an N_ r-antenna access point (AP). Each device k∈=1,2,…,Kk =\1,2,…,K\ maintains a buffer where packets are queued and transmitted in a first-come first-served manner. We consider a saturated traffic condition, i.e., the packet arrival rate at each node is sufficiently high so that all devices consistently have data packets to transmit. The network operates in slotted time over a finite horizon T, with time slots indexed by t∈1,2,…,Tt∈\1,2,…,T\. Similar to the CSMA/CA protocol, the network adopts the LBT mechanism. Before attempting transmission, each device first performs a clear channel assessment (CCA) to detect whether the channel is idle. If the channel remains idle for a specified duration, the device decides whether to transmit a packet to the AP. A device that chooses not to transmit simply returns to the CCA phase. If the device decides to transmit, it sends a single data packet to the AP over τ time slots. A transmission is considered successful if the AP correctly receives the packet without collision, in which case the AP sends out an acknowledgment (ACK) for confirmation. This allows the device to proceed with its next packet transmission. In the event of a collision (i.e., multiple devices have simultaneously transmitted), the AP does not return an ACK, and each involved device will wait for the next transmission opportunity to attempt retransmission. To support fully decentralized network operation, we assume there is no global controller responsible for tasks like information aggregation or global message broadcasting. Instead, RA devices can exchange information with their neighbors according to local communication links defined by an undirected graph =(,ℰ)G -1.42262pt= -1.42262pt(K,E), where ℰE represents the set of device-to-device connectivity links (Boyd et al., 2006). 3.2. Data Transmission Model The single-input multiple-output (SIMO) channel gain between device k and the AP at time slot t is modeled as (1) k,t=ηkk,t, g_k,t= _k h_k,t, -2.84526pt where k,t∈ℂN h_k,t ^N_ r represents the small-scale fading vector whose entries are independent and identically distributed (i.i.d.) complex Gaussian random variables with zero mean and unit variance. The term ηk _k in (1) denotes the large-scale fading factor which depends on the distance between device k and the AP. For example, the magnitude of small-scale fading could follow a Rayleigh distribution, which is well-suited for modeling dense scattering wireless environments (Tse and Viswanath, 2005). Also, our large-scale fading model can be adopted to various realistic propagation scenarios, including the ITU-R indoor (, 2023) and 3GPP urban-micro (UMi) (, 2017) pathloss models. Using (1), the achievable data rate of device k at time slot t can be obtained by the Shannon capacity formula: (2) Rk,t=Blog2(1+pk‖k,t‖22BN0),R_k,t=B _2 (1+ p_k\| g_k,t\|^2_2BN_0 ), -2.84526pt where B is the transmission bandwidth, pkp_k is the transmit power of device k, and N0N_0 is the noise spectral density (Tse and Viswanath, 2005). Assuming a block-fading channel (i.e., the channel remains static during each packet transmission), we express the amount of data which device k can successfully transmit at time slot t as (3) mk,t=Rk,tTτ,m_k,t=R_k,tT_ sτ, -2.84526pt where T_ s denotes the slot duration. Note that (3) plays a crucial role in transmission decision-making. For example, a device with low mk,tm_k,t may lead to suboptimal throughput by potentially blocking other devices with more favorable channel conditions (i.e., higher mk,tm_k,t). Since mk,tm_k,t is directly proportional to Rk,tR_k,t, each RA device needs to carefully consider its channel quality when making transmission decisions to maximize the overall network performance. 3.3. RA Network Optimization Tasks To illustrate the FM’s capability in supporting multiple performance goals in wireless RA network optimization with MARL, in this work, we consider the following three objectives as downstream tasks: 1) Fair-AoI (Worst-case AoI Minimization): AoI (age of information) measures the freshness of information by tracking the time elapsed since the latest data transmission of a device. Let ℓk,t _k,t denote the number of time slots that have elapsed since device k’s last successful packet transmission by time slot t (Guo et al., 2022). This quantity can be interpreted as the anticipated AoI if the AP successfully receives a packet from device k at time slot t. Let k,t⊆1,2,…,tT_k,t \1,2,…,t\ denote the set of time slot indices at which device k has successfully transmitted a packet during the time horizon t≤Tt≤ T. Then, we minimize the worst-case AoI among all devices using (4) minmaxk∈,t∈k,Tℓk,t. _k ,t _k,T _k,t. -2.84526pt Note that this mini-max formulation solely focuses on fairness in transmit frequency and does not account for channel quality or data rates. This task is particularly useful in applications such as real-time system control in wireless drone networks, where timely information updates are important. 2) Max-Rate (Sum-rate Maximization with Minimum Rate Guarantees): As a classical network performance metric, sum-rate quantifies the total amount of data successfully received by the AP for a given time window. Here, we let (5) xk,t=1tT∑t′∈k,tmk,t′x_k,t= 1tT_ s _t _k,tm_k,t -2.84526pt denote the throughput of device k over the time horizon t≤Tt≤ T. To maximize the sum-rate, i.e., ∑k∈xk,T _k x_k,T, it is ideal for a network to prioritize RA devices with higher data rates mk,tm_k,t. However, such a policy can result in poor fairness as it may allow devices with superior channel quality to dominate transmission resources. To prevent this, we impose a minimum rate requirement R_ min and formulate our objective as (6) max∑k∈xk,T 2.84526pt _k x_k,T (7) s.t. xk,T≥R,∀k∈. x_k,T≥ R_ min, 2.84526pt∀ k . -2.84526pt The problem above seeks to maximize the overall throughput while ensuring that each device’s rate maintains at least R_ min. 3) Fair-Rate (Worst-case Rate Maximization): In certain applications such as collaborative sensing, the system performance can be compromised by the device with the poorest throughput. In such cases, it is crucial to promote throughput fairness to prevent any single device from becoming a bottleneck. To ensure balanced per-device rate, we maximize the minimum throughput across all devices over the time horizon T, which can be expressed as (8) maxmink∈xk,T. 2.84526pt _k \,x_k,T. -2.84526pt This formulation ensures that no device is significantly disadvantaged in terms of data rate, thus supporting balanced and reliable system performance. 4. Foundation Model-Aided Decentralized MARL for Wireless RA Network Optimization In this section, we first review the AC framework for MARL. We then formulate the MDP tailored to our wireless RA network optimization. Next, we describe the architecture of our FM, which serves as a reward-agnostic backbone within the AC framework. Finally, we present the proposed FM-aided decentralized MARL algorithm and analyze its convergence properties. 4.1. Actor-Critic Framework for MARL Problem We model the wireless RA network optimization problem as an MARL task defined by the MDP tuple ,kk∈,P,ϱkk∈\S,\A_k\_k ,P,\ _k\_k \, where S is the global state space and kA_k is the action set for device k. The term P:×→[0,1]P:S×A×S→[0,1] denotes the state transition probability, where =∏k∈kA= _k A_k is the joint action set of all agents. Lastly, ϱk:×→ℝ _k:S×A is the local reward function for device k. Since most of time slots 1,…,T\1,…,T\ are occupied by the RA protocol steps, such as CCA, packet transmission, and waiting upon ACK, the actual time slots related to MARL procedure are confined. Hence, we consider our MDP to only progress over time slots, where an action is taken by the devices, which we denote using n. At an MDP step n, each device k at a global state sn∈s_n selects an action ak,n∈ka_k,n _k according to its local policy πk _k, i.e., ak,n∼πk(⋅|sn)a_k,n _k(·|s_n). Due to the coupled nature of the devices’ actions, the state transitions from sns_n to sn+1s_n+1 via P(sn+1|sn,n)P(s_n+1|s_n, a_n), where n=[a1,n,a2,n,…,aK,n] a_n=[a_1,n,a_2,n,…,a_K,n] is the joint action vector. Each device k then receives a reward rk,n=ϱk(sn,n)r_k,n= _k(s_n, a_n). The AC framework employs two sets of function approximators: actors and critics. Each device k is equipped with an actor network parameterized by weights θk _k, which defines its local policy πθk _ _k. Let θ=[θ1⊤,θ2⊤,…,θK⊤]⊤θ=[ _1 , _2 ,…, _K ] denote the joint weight vector of all K actors. The goal of MARL is to optimize θ to maximize the expected long-term discounted cumulative reward defined as J(θ)=πθ[∑n=0∞γnr¯n]J(θ)=E_ _θ\! [ _n=0^∞γ^n r_n ], where r¯n=1K∑k=1Krk,n r_n= 1K _k=1^Kr_k,n is the global average reward, and γ∈[0,1]γ∈[0,1] is the discount factor. For each device k, the state-value function Vθ(s)=πθ[∑n=0∞γnr¯n|s0=s]V_θ(s)=E_ _θ [ _n=0^∞γ^n r_n |s_0=s ] is estimated using a critic network of parameters wkw_k. Applying the Bellman equation (Sutton et al., 1998), the estimated function can be expressed as Vwk(s)=πθ[r¯+γVwk(s′)]V_w_k(s)=E_ _θ [ r+γ V_w_k(s ) ]. During training, each actor selects an action based on its current policy, and the corresponding critic evaluates the resulting reward. The actor policy is then updated using the policy gradient s,a[∇θklogπθk(ak|s)⋅Advθ(s,)]E_s,a[ _ _k _ _k(a_k|s)·Adv_θ(s, a)], where Advθ(s,)=r¯+γVθ(s′)−Vθ(s)Adv_θ(s, a)= r+γ V_θ(s )-V_θ(s) is the advantage function. This update encourages the actor to select actions that yield higher expected future rewards relative to the baseline value. 4.2. MDP Formulation for MARL-based Wireless RA Network Optimization We now design the MDP components for our RA network optimization. First, the state at network time slot t is defined as (9) st=(ξ^kk∈,ℓ^k,tk∈,qt),s_t= (\ ξ_k\_k ,\ _k,t\_k ,q_t ), -2.84526pt where ξ^k=ξk−minkξkmaxkξk−minkξk ξ_k= _k- _k _k _k _k- _k _k is the min-max normalized long-term signal-to-noise ratio (SNR) for device k, ℓ^k,t=ℓk,t/λ _k,t= _k,t/λ is the normalized AoI with λ as a scaling factor, and qt∈0,1q_t∈\0,1\ is the binary channel occupancy indicator. We use dB scale for SNR, i.e., ξk=t[10log10pk‖k,t‖22BN0]=10log10pkηkBN0 _k=E_t [10 _10 p_k\| g_k,t\|^2_2BN_0 ]=10 _10 p_k _kBN_0. For each device k, ξ^k ξ_k and ℓ^k,t _k,t are local information. We assume that long-term SNRs of other devices ξ^k′k′∈∖k\ ξ_k \_k k are available via local information exchanges or broadcast that take place at a much slower rate. Following the assumptions in (Yu et al., 2019; Guo et al., 2022; He et al., 2024), we consider AoIs of other devices ℓ^k′,tk′∈∖k\ _k ,t\_k k to be observable from the broadcast ACK packets. Note that qtq_t is also observable during the CCA phase. Each device operates in a discrete action space k=0,1A_k=\0,1\, where 00 indicates the decision to wait and 11 represents the decision to transmit. Hence, at each given time slot t, device k relies on its policy πk _k and selects its action ak,t∈0,1a_k,t∈\0,1\. We now define local reward functions corresponding to each downstream task. First, for device k at time slot t, the local reward for fair-AoI task is defined as (10) rk,t=−ℓk,tωK,r_k,t FA=- _k,t _ FAK, -2.84526pt where ωK _ FAK is a scaling factor to prevent the reward from increasing with the network size. Since the value of ℓk,t _k,t becomes zero only upon successful transmission, this reward encourages devices to avoid collisions. Moreover, with (10), prioritizing devices with higher AoI values leads to greater marginal reward gains. Hence, maximizing (10) promotes AoI fairness across the network. Next, the local reward of device k at time slot t for max-rate task is given by (11) rk,t=xk,tω+D(xk,t),r_k,t MR= x_k,t _ MR+D(x_k,t), where ω _ MR is a scaling parameter, and D(xk,t)D(x_k,t) is a penalty function: (12) D(xk,t)=log(xk,t+ωR(1+ω)R)ω/K,if xk,t≤R,0,otherwise, -2.84526ptD(x_k,t)= cases ( x_k,t+ _ P1R_ min(1+ _ P1)R_ min ) _ P2/K,&if x_k,t≤ R_ min,\\ 0,&otherwise, cases with scaling factors ω _ P1 and ω _ P2. Note that (12) is bounded in the interval [log(ω/(1+ω),0][ ( _ P1/(1+ _ P1),0] and designed to only penalize devices that fail to meet the threshold R_ min. In other words, this reward drives each device to maintain the minimum rate requirement while maximizing the sum-rate. Lastly, we adopt the α-fairness utility function (Yu et al., 2019) and define the local reward of device k at time slot t for fair-rate task as (13) rk,t=11−α(xk,tω/K+ω)1−α,r_k,t FR= 11- _ FR ( x_k,t _ FR1/K+ _ FR2 )^1- _ FR, where α∈(1,∞) _ FR∈(1,∞) controls the fairness, and ω _ FR1 and ω _ FR2 are scaling factors. Larger values of α _ FR impose stricter fairness. To reflect the condition of fully decentralized MARL, we strictly define our reward functions to be locally computable. For the consensus step, we let rk,t∈rk,t,rk,t,rk,tr_k,t∈\r_k,t FA,r_k,t MR,r_k,t FR\ to be exchanged across the devices through local communication links defined by G. Figure 2. The architecture of FM as a reward-agnostic critic backbone. The model takes a state-action history and extracts embeddings via state and action encoders. Transformer blocks are used to process the embeddings. The output of a prediction head is sent to the reward-specific critic head.. 4.3. Foundation Model Architecture Design We present the detailed design of our FM, which supports the AC framework described in Section 4.1 as a reward-agnostic critic backbone. Typical RL models rely on reward signals as supervision, i.e., rewards serve as targets that guide an agent to train and learn desirable behaviors. In contrast, our FM is pretrained in a self-supervised manner and thus does not require reward labels. This allows our FM to be reward-agnostic and transferable across different RL tasks. In this work, we leverage the FM to enhance our MARL-based optimization for wireless RA-based networks. During pretraining, the FM captures useful feature representations that effectively model the underlying environment dynamics, which can improve learning efficiency over multiple downstream tasks (e.g., three wireless RA objectives defined in Section 3.3). Our FM pretraining is inspired by the SMART framework (Sun et al., 2023), which uses a control transformer architecture to facilitate self-supervised learning across different pretraining goals. Similar to SMART, we employ encoders to extract meaningful embeddings from the inputs and utilize the attention mechanism within transformers to perform a prediction task. In this paper, we specifically adopt the forward dynamics prediction task for self-supervised pretraining, which aligns naturally with the core principle of RL to optimize future outcome predictions. The architecture of our FM is shown in Figure 2, which consists of three main components: i) encoders, i) transformers, and i) a prediction head. The encoders first map the input into a sequence of token embeddings, which are augmented with position embeddings to convey temporal information. The position embeddings are designed to capture both the relative order within the history window and the absolute position within the global time trajectory, allowing the model to effectively understand the environment dynamics. The combined embeddings are then passed through a multi-transformer module, where each block consists of a self-attention layer and a multi-layer perceptron (MLP). The MLP uses the Gaussian error linear unit (GELU) as its activation function. The output from the transformer module is fed into the prediction head, which generates an embedding that represents next-step state. During the pretraining, the FM is optimized such that the loss between the predicted and actual next-state embeddings is minimized. By integrating the self-supervised FM, we allow our MARL agents to utilize effective representations of the environment dynamics during their online learning. Note that the pretraining of FM does not require explicit reward signals, making the model generalizable to different downstream RL tasks. Similar to (Yu et al., 2019; Guo et al., 2022; He et al., 2024), we assume that each device can store the H latest states and actions. Let t~k[h] t_k^[h] denote the time slot when the h-th latest action was taken by device k. Then, the state-action history at time slot t can be written as (14) Ψk,t[H:1]=st~k[H],ak,t~k[H],…,st~k[1],ak,t~k[1],st. _k,t^[H:1]= \s_ t_k^[H],a_k, t_k^[H],…,s_ t_k^[1],a_k, t_k^[1],s_t \. -2.84526pt We integrate (14) as an input to both actor and critic. This enables the model to better reflect the temporal dynamics of the wireless RA environment and improve its learning stability and performance. 4.4. Foundation Model-Aided Decentralized MARL Algorithm Design We now describe the proposed FM-aided decentralized MARL algorithm for RA network optimization. Figure 3 illustrates the overall architecture of our decentralized AC framework. Each device k∈k updates its local actor and critic networks using its own experience. To capture temporal dependencies, each device maintains a history buffer that stores past observation–action pairs and uses this history as input during learning. As described in Section 3, neighboring devices can exchange limited information. In our design, each device shares only scalar local rewards with its neighbors, which incurs significantly lower communication costs compared to exchanging full critic parameters, as in (Zhang et al., 2018; Chen et al., 2022; Hairi et al., 2022). Specifically, the critic network consists of two parts: a reward-agnostic FM and a reward-specific head. The FM is pretrained using observation–action sequences and remains fixed during the RL phase. This design eliminates the need for online updates of the high-capacity backbone, reducing computational and communication overhead while leveraging rich representations of system dynamics. The overall procedure of the proposed algorithm for a K-device RA-based MAC layer is summarized in Algorithm 1. Each device operates under the LBT protocol and continuously monitors the channel state. Meanwhile, each device updates its local status based on transmission outcomes (e.g., successful transmission or collision) (Lines 6 - 10). When the channel is sensed to be idle, the device observes the current state sts_t and makes a transmission decision ak,ta_k,t using its local policy conditioned on the past action-observation history and current observation (i.e., Ψk,t[H:1] _k,t^[H:1]). After executing the action, the tuple st,ak,t,rk,t\s_t,a_k,t,r_k,t\ is stored in the history buffer (Lines 11 - 17). This interaction process is repeated over T time slots. Figure 3. A visual representation of our fully decentralized MARL framework. Both the actor and critic are trained in a decentralized manner. The critic contains two distinct parts: a reward-agnostic backbone for forward dynamics prediction and a reward-specific head for task-specific finetuning.. Algorithm 1 FM-aided Decentralized MARL for RA Network Optimization. Input: device set K, pretrained FM weights Ξkk∈\ _k\_k , time horizon length T, history length H, consensus weight matrix C, consensus iteration count G, number of samples to collect NBN_B, actor rate α, critic rate β 1 Initialize: For all k∈k , actor weights θk _k, critic weights wkw_k, ready-for-update status uk=Falseu_k=False, sample count bk=0b_k=0 2 Load: FM weights Ξk _k for all k∈k 3 for t=1,2…,Tt=1,2…,T do 4 for k∈k do 5 Observe qtq_t via clear channel assessment (CCA) 6 if ACK received then 7 ℓk,t←0 _k,t← 0 8 else 9 ℓk,t←ℓk,t+1 _k,t← _k,t+1 10 if qt=0q_t=0 then 11 Acquire state sts_t and reward rk,tr_k,t 12 Select ak,t∼πθk(⋅|Ψk,t[H:1])a_k,t _ _k(\>·\>| _k,t^[H:1]) 13 if ak,t=1a_k,t=1 then 14 Initiate a packet transmission 15 Store sts_t, ak,ta_k,t, and rk,tr_k,t in the history buffer 16 bk←bk+1b_k← b_k+1 17 if bk=Nbb_k=N_b then 18 uk←Trueu_k 19 if uk=True,∀k∈u_k=True,∀ k then 20 ¯(t)=G(t) R^(t)=C^GR^(t) 21 for k∈k do 22 w¯k,0←wk w_k,0← w_k 23 for b=1,2,…,NBb=1,2,…,N_B do 24 δk,b←¯[k,b](t)+γVw¯k,b−1(ϕΞk(Ψk,t~k[b−1][H+b−1:b]))−Vw¯k,b−1(ϕΞk(Ψk,t~k[b][H+b:b+1])) _k,b← R^(t)_[k,b]+γ V_ w_k,b-1( _ _k( _k, t_k^[b-1]^[H+b-1:b]))-V_ w_k,b-1( _ _k( _k, t_k^[b]^[H+b:b+1])) 25 w~k,b←w¯k,b−1−βδk,b⋅∇Vw¯k,b−1(ϕΞk(Ψk,t~k[b][H+b:b+1])) w_k,b -1.42262pt← w_k,b-1-β _k,b·∇ V_ w_k,b-1( _ _k( _k, t_k^[b]^[H+b:b+1])) 26 w¯k,b←argminw∈ℬ(B)‖w−w~k,b‖2 w_k,b← _w (B)\|w- w_k,b\|_2 27 wk←b+1wk+1b+1w¯k,bw_k← bb+1w_k+ 1b+1 w_k,b 28 θk←θk+αNB∑b=1NBδk,b⋅∇logπθk(ak,t~k[b]|Ψk,t~k[b][H+b:b+1]) _k -1.42262pt← -1.42262pt _k -1.42262pt+ -1.42262pt αN_B -2.84526pt _b=1^N_B _k,b -1.42262pt· -1.42262pt∇ -1.42262pt _ _k(a_k, t_k^[b]| _k, t_k^[b]^[H+b:b+1]) -14.22636pt 29 uk←Falseu_k 30 Output: θk _k for all k∈k Each time the devices are ready to update their model weights, the consensus (Line 21) first takes place for reward exchange. Each consensus step consists of G rounds of communication, where weight-based averaging is performed in each round. Then, each device updates its actor and critic parameters after computing the temporal-difference (TD) error (Lines 23 - 29). While the actor takes the input directly from the observation-action history, the critic processes it via the FM backbone to obtain a latent representation ϕΞk(Ψk,t) _ _k( _k,t), which is then fed into the reward-specific head for value estimation. Extending our prior work (Oh et al., 2025) to FM-aided MARL, we adopt a batch-based update scheme to facilitate the theoretical analysis in Section 4.5. In particular, the critic update incorporates projection onto a bounded parameter set and an averaging step (Lines 27 - 28) to ensure that each update remains within a controlled region and allow convergence guarantees (Xu et al., 2020; Zhang et al., 2025). 4.5. Theoretical Performance Analysis Our prior work on RA network optimization via decentralized MARL (Oh et al., 2025) establishes convergence guarantees under linear value function approximation, which has been the dominant analytical framework for studying MARL convergence (Chen et al., 2022; Zhang et al., 2021; Hairi et al., 2022). However, linear approximations are insufficient to capture the expressive behavior of deep neural networks (DNNs), which are widely used in modern RL applications. To address this gap, we build upon our previous analytical framework in (Zhang et al., 2025) and extend our results to the FM-aided setting where value functions are approximated using DNNs. A key distinction of this work from (Zhang et al., 2025) is that their analysis assumes consensus over the entire critic model, whereas our approach requires only local reward exchange among neighboring agents. By restricting consensus to scalar rewards rather than high-dimensional model parameters, our framework significantly improves practical deployability and reduces communication overhead. Also, we address the impact of integrating pretrained FMs in the actor-critic pipeline, where we perceive the power of self-supervised learning as the prediction capability on next states. We now formally describe the DNN architecture used in our analysis. For each device k, the critic maintains a DNN-based approximation VwkV_w_k of the true value function VθV_θ. We assume the network has width MwM_w and depth MdM_d. Let xin∈ℝdx_in ^d denote the input features vector. The forward pass is defined as: (15) x(0) x^(0) =W(0)xin, =W^(0)x_in, (16) x(h) x^(h) =1Mw(W(h)x(h−1)) for h=1,2,…,Md = 1 M_w ReLU(W^(h)x^(h-1)) for h=1,2,…,M_d (17) y y =b⊤x(Md), =b x^(M_d), where W(0)∈ℝm×dW^(0) ^m× d, W(h)∈ℝm×mW^(h) ^m× m, and b∈ℝmb ^m are the network parameters, and (x):=max0,x ReLU(x):= \0,x\ is the ReLU activation function. The actors share the same architecture; however, a softmax layer is appended to yield a probability distribution over actions. With n denoting the MDP iteration index, we establish the theoretical convergence guarantee of Algorithm 1. First, we provide the required assumptions in the following. Assumption 1 (Bounded Reward). For each downstream task, there exists a constant rmax>0r_max>0 such that |rk,n|≤rmax,∀n≥0,k∈|r_k,n|≤ r_max,∀ n≥ 0,k . Assumption 2 (Mixing Time). There exists a stationary distribution ζ for (s,a)(s,a), and positive constants κ and ρ∈(0,1)ρ∈(0,1), such that sups∈‖P(sn,an|s0=s)−ζθ‖TV≤κρn,∀n≥0 _s \|P(s_n,a_n|s_0=s)- _θ\|_TV≤κρ^n,∀ n≥ 0. Assumption 3 (Lipschitz Continuity). There exist some positive constants LJL_J, LWL_W, LVL_V, and MgM_g such that, for any θ and θ′θ , any w and w′w , any x and x′x , and any s and a, we have |J(θ)−J(θ′)|≤LJ‖θ−θ′‖2|J(θ)-J(θ )|≤ L_J\|θ-θ \|_2, |Vw(x)−Vw′(x)|≤LW‖w−w′‖2|V_w(x)-V_w (x)|≤ L_W\|w-w \|_2, |Vw(x)−Vw(x′)∥≤LV∥x−x′∥2|V_w(x)-V_w(x )\|≤ L_V\|x-x \|_2, and ‖∇θlogπθ(a|s)‖2≤Mg\| _θ _θ(a|s)\|_2≤ M_g. Assumption 4 (Consensus Matrix). The consensus weight matrix C is doubly stochastic. Additionally, for all k,k′∈k,k , there exists a positive constant ν>0ν>0 such that (i) ckk≥νc_k≥ν and (i) ckk′≥νc_k ≥ν whenever devices k and k′k are connected. Assumption 4 ensures that information is exchanged with uniformly positive weights among connected agents, facilitating convergence of the local consensus estimates toward the globally averaged information. Assumption 5. For any policy θ, there exist positive constants σ and ϵbias _bias such that [θ(Ψ,a)−(1−γ)E [Adv_θ( ,a)-(1-γ)((θ)−1(F(θ)^-1∇θJ(θ))⊤ _θJ(θ)) ∇θlogπθ(a|Ψ)]2 _θ _θ(a| ) ]^2 ≤ϵbias≤ _bias, where (Ψ,a)( ,a) follows ν(π∗)ν(π^*), which is a stationary distribution under the optimal policy π∗π^*, and (θ)=ν(θ)[∇θlogπθ(a|Ψ)F(θ)=E_ν(θ)[ _θ _θ(a| ) ∇θlogπθ(a|Ψ)⊤]+σ _θ _θ(a| ) ]+ . Assumption 6 (Universal Approximation). Suppose for any policy θ, the critic estimation V(⋅,w∗)V(·;w^*), where w∗w^* is a stationary point, can be close enough to the ground truth Vθ(⋅)V_θ(·), i.e., there exists a positive constant ϵcritic _critic such that maxθ,x|V(x,w∗)−Vθ(x)|≤ϵcritic _θ,x|V(x;w^*)-V_θ(x)|≤ _critic. We state the main theoretical result as below. Theorem 1. With a pretrained FM of estimation error ϵestFMε^FM_est and approximation error ϵapproxFMε^FM_approx fixed and integrated as a reward-agnostic critic backbone, by choosing step-size parameters α=72μα= 72 μ, αn=αn _n= αn, β=1/NBβ=1/ N_B, and network/algorithm parameters satisfying m=Ω(d3Md−112)m= (d^3M_d^- 112), B=Θ(Mw132Md−6)B= (M_w 132M_d^-6), and NB=Ω(Md4)N_B= (M_d^4), it holds with probability at least 1−exp(−Ω(log2Mw))1- (- ( ^2M_w)) that Algorithm 1 achieves: [J(θ∗)−J(θNT)]=(NT−1)⏟(a)+(K32(1−νK−1)G)⏟(b) [J(θ^*)-J( _N_T) ]= O (N_T^-1 )_(a)+ O(K 32(1-ν^K-1)^G)_(b) +(K12(BMd52NB−14+B43Mw−112Md4)log32Mwlog12NB)⏟(c)+(ϵbias)⏟(d) + O (K 12(BM_d 52N_B^- 14+B 43M_w^- 112M_d^4) 32M_w 12N_B )_(c)+ O ( _bias )_(d) (18) +(K12ϵestFM)⏟(e)+(K12ϵapproxFM)⏟(f)+(K1/2ϵcritic/NB1/2)⏟(g). + O(K 12ε^FM_est)_(e)+ O(K 12ε^FM_approx)_(f)+ O (K^1/2 _critic/N_B^1/2 )_(g). Proof Sketch. To bound the suboptimality gap J(θ∗)−J(θNT)J(θ^*)-J( _N_T), we focus on analyzing the discrepancy between the update direction dnd_n and the true policy gradient ∇J(θn)∇ J( _n) at each iteration. We decompose this discrepancy into four components: (i) value function approximation error, (i) actor update error induced by imperfect reward consensus, (i) the mismatch between TD error and the advantage function, and (iv) intrinsic representation error of the function approximators. Each component is bounded individually to obtain the terms in (18). In particular, the value function approximation error is further decomposed into contributions from imperfect critic updates, consensus inaccuracies, and limitations of the pretrained FM representations. These terms are controlled using the stated assumptions and standard properties of overparameterized DNNs. Due to space limitation, we relegate the full proof to our online technical report (Oh et al., 2026) ∎ Theorem 1 provides several key insights. First, term (b)(b) shows that imperfect reward consensus induces an optimality gap of the same order as approaches that exchange full critic parameters and TD errors (Zhang et al., 2025). This indicates that restricting communication to scalar reward exchange preserves performance while significantly improving practicality and reducing communication overhead. Second, term (c)(c) highlights the role of network architecture. The convergence bound exhibits stronger sensitivity to the network depth than to its width, suggesting that increasing depth is more effective for improving performance, whereas overparameterization in width yields diminishing returns. Third, terms (e)−(g)(e)-(g) characterize the impact of critic approximation errors, which scale with the number of agents K. In particular, terms (e)(e) and (f)(f) correspond to estimation and approximation errors arising from the self-supervised pretraining of the FM, where the learning objective is forward dynamics prediction. These error components are unique to our framework and reflect the quality of the pretrained, reward-agnostic representations used by the critic. Theorem 1 shows that Algorithm 1 converges to a neighborhood of the global optimum at a rate of (1/NT)O(1/N_T). This indicates that operating under nonlinear function approximation and decentralized reward consensus matches the convergence rate of state-of-the-art actor–critic methods with linear function approximation and full critic exchange (Hairi et al., 2022). Note that our result generalizes (Oh et al., 2025), which only considered linear approximation, to a more realistic deep learning setting. Moreover, unlike prior work that guarantees convergence only to stationary points (e.g., in terms of ‖∇J(θ)‖2\|∇ J(θ)\|^2) (Hairi et al., 2022; Oh et al., 2025), our analysis characterizes convergence toward global optimality. (a) Fair-AoI task. (b) Max-rate task. (c) Fair-rate task. Figure 4. Throughput and AoI under end-to-end learning with K=2K=2 devices across different RA optimization tasks. 5. Numerical Evaluation 5.1. Experimental Settings We consider operating a wireless RA-based network over T=600T=600 time slots. As described in Section 3, all devices operate under the LBT mechanism. Following the IEEE 802.11 protocol, the SIFS and DIFS durations are set to 22 and 44 time slots, respectively, with each slot lasting T=9T_ s=9 μsμ s. Each data packet transmission requires τ=10τ=10 time slots, while ACK signals occupy 44 time slots. We set the number of AP antennas to N=2N_ r=2, the bandwidth to B=10B=10 MHz, and the transmit power to pk=0p_k=0 dBm, with the noise spectral density at −174-174 dBm/Hz. For a given number of devices K, device-to-AP distances are uniformly distributed between 22 m and 1414 m. For example, with K=4K=4, the device distances are dkk=1K=4=2,6,10,14\d_k\_k=1^K=4=\2,6,10,14\. The large-scale fading factor ηk _k is modeled as ηk=L(dkd)z _k=L_ o ( d_kd_ o )^z, where L=−40L_ o=-40 dB is the reference pathloss, d=1d_ o=1 m is the reference distance, and z=4z=4 is the pathloss exponent. The network graph G is generated using the Watts-Strogatz model (Watts and Strogatz, 1998), where each device connects to one neighboring device with zero rewiring probability. To generate the consensus weight matrix C, we assign equal weights are assigned to each device’s established links, i.e., for each device k∈k , ckk′=1|k|+1,∀k′∈kc_k = 1|N_k|+1,∀ k _k, where kN_k denotes the neighbor set of device k. Each consensus step consists of G=3G=3 communication rounds. For the critic-backbone FM, we set the embedding size to N=64N_ e=64 such that the encoders, transformer blocks, and prediction head all generate embeddings of dimension 6464. Both the state and action encoders are implemented as two-layer MLPs with width 6464. The FM consists of two transformer blocks, each containing a self-attention layer with 8 heads and an MLP of width 4N4N_ e and depth 11. The prediction head is a linear transformation of size 2N×N2N_ e× N_ e with bias. No dropout is applied in the FM design. Each device’s FM is pretrained using more than 100,000100,000 reward-independent RA-based wireless network samples, each consisting of a tuple t,st,ak,t\t,s_t,a_k,t\. We employ stochastic gradient descent (SGD) with a rate of 0.0050.005, applying 500500 updates with a batch size 256256. The target encoder is updated using a soft update with a rate of 0.0050.005. For both actor and reward-specific critic head, we adopt an MLP block of width Mw=128M_w=128 and depth Md=3M_d=3. The actor and critic weights are updated using the SGD optimizer, with learning rates α=0.001α=0.001 and β=0.002β=0.002. Discount factor is set to γ=0.9γ=0.9. For the end-to-end training baseline, we employ an MLP block of width Mw=128M_w=128 and depth Md=7M_d=7 for both actor and critic to ensure a comparable number of parameters for fair evaluation. Since the state dimension grows proportionally with K, we adjust the history length H for different values K to stabilize the input dimension d of the actor–critic network against the FM embedding size NeN_e. For the MDP parameters, we set λ=60λ=60, ω=15 _ FA=15, ω=24×106 _ MR=24× 10^6, ω=0.5 _ P1=0.5, ω=8 _ P2=8, ω=256×106 _ FR1=256× 10^6, ω=0.7 _ FR2=0.7, and α=12 _ FR=12 for parameter scaling. These values are heuristically selected to bound the reward to a reasonable range. All results are averaged over at least 2020 independent runs. As performance metrics on RA-based wireless networks, we evaluate throughput (in Mbps) and AoI (in ms). In addition, we assess fairness across devices using the normalized gap (N-Gap) defined as N-Gap of x=(maxx−minx)/maxxN-Gap of \x\=( \x\- \x\)/ \x\. (a) Number of Collisions. (b) Normalized reward sum. Figure 5. Collision and reward performance under end-to-end learning with K=2K=2 devices across RA optimization tasks.. 5.2. Experimental Results and Discussion 1) Performance Analysis of the Proposed Decentralized MARL Algorithm on Wireless RA Network Optimization Tasks: We first validate Algorithm 1 and the MDP formulation introduced in Section 4.2 by evaluating end-to-end learning performance (Oh et al., 2025). In Figure 4, we present the throughput and AoI achieved during the learning phase for three downstream tasks. For ease of illustration, we consider K=2K=2 devices. The results demonstrate strong learning performance across all tasks. As expected, the fair-AoI task (Figure 4(a)) successfully balances AoI across devices. The max-rate task (Figure 4(b)) effectively maximizes the throughput of one device while maintaining the other near the minimum rate constraint R_ min. The fair-rate task (Figure 4(c)) is able to jointly improve throughput of both devices while preserving the fairness. These observations confirm that the proposed MDP formulation is capable of supporting diverse RA network optimization objectives. As additional performance metrics, Figures 5(a) and 5(b) show the number of collisions and the normalized reward sum, respectively. For improved readability, all results are smoothed using a moving average over 55 samples. The results in Figure 5(a) show that the fair-AoI task achieves the fastest reduction in collisions, indicating that balancing AoI is the least complex among the three objectives. This is intuitive, as the fair-AoI objective does not require consideration of channel conditions, which directly influence device data rates. By contrast, the fair-rate task exhibits the highest collision counts even after many episodes. This is because its reward function imposes stronger penalties for rate unfairness across devices. Consequently, the policy that reduces collisions emerges later in training, consistent with Figure 4(c) where device rates increase gradually while maintaining fairness gaps. In Figure 5(b), we observe a steady increase in the normalized reward sum over the learning process. This trend validates the effectiveness of our reward functions and confirms that the algorithm successfully optimizes the objectives. (a) Fair-AoI task. (b) Max-rate task. (c) Fair-rate task. Figure 6. Performance comparison of conventional, end-to-end (E-E), and FM-aided MARL across different RA optimization tasks with K=4K=4. The FM-aided MARL achieves faster convergence than the end-to-end baseline.. . Figure 7. Fair-AoI and Max-rate performance comparison with non-MARL baselines across the number of devices K.. 2) Performance Comparison of FM-Aided MARL and Baselines: With the end-to-end (E-E) MARL results as a proof of concepts, we are now ready to assess the effectiveness of our FM-aided MARL algorithm by comparing it with E-E MARL and conventional decentralized MARL (conventional MARL), which performs full critic exchanges (Zhang et al., 2025). Figure 6 presents the AoI and throughput over runtime for each downstream task with K=4K=4. For all tasks, FM-aided learning shows superior learning speed against both baselines, approximately improving the convergence speed by at least 55%. Note that the performance gap with E-E MARL is attributed to the integration of the FM as a reward-agnostic critic backbone. In Figure 7, we compare our algorithm with several baselines, which include 1/K1/K-persistent CSMA (1/K1/K-CSMA) and CSMA/CA with binary exponential backoff (BEB-CSMA), under the Fair-AoI and Max-rate objectives for different numbers of devices. We observe that both E-E MARL and FM-MARL consistently outperform the CSMA-based baselines, which rely on stochastic channel-access mechanisms. Round-robin, where devices are scheduled in turn through centralized control, achieves near-optimal performance for the Fair-AoI objective because it provides balanced access opportunities across devices. However, its performance degrades under the Max-rate objective, since equal transmission allocation does not account for differences in the achievable data rates of individual devices. In contrast, our proposed method, along with end-to-end training, learns to prioritize devices with more favorable channel conditions, resulting in performance closer to the optimal. Lastly, we provide a summary of performance comparison in Table 1. We observe that the algorithms achieve comparable final rate and AoI performance. However, the proposed FM-aided MARL approach arrives at its 95%95\%-optimal performance level substantially faster than the end-to-end training approach. Table 1. Performance Comparison of End-to-End (E-E) and FM-aided MARL for K=4K=4 (top) and K=8K=8 (bottom). AoI is measured for the fair-AoI task, and rate is measured for both max-rate and fair-rate tasks. Task Alg. Mean Min. Max. N-Gap 95%-T (s) Fair-AoI E-E 0.731 ± 0.010 0.726 0.734 0.011 335 (ms) FM 0.769 ± 0.032 0.757 0.772 0.019 108 Max-rate E-E 17.044 ± 0.266 4.092 41.225 0.901 380 (Mbps) FM 17.924 ± 0.345 3.918 43.865 0.911 159 Fair-rate E-E 9.049 ± 0.186 6.903 11.956 0.423 373 (Mbps) FM 8.083 ± 0.131 6.143 11.590 0.463 104 Task Alg. Mean Min. Max. N-Gap 95%-T (s) Fair AoI E-E 0.366 ± 0.014 0.364 0.368 0.003 723 (ms) FM 0.368 ± 0.025 0.365 0.370 0.013 231 Max Rate E-E 7.235 ± 0.375 3.895 30.175 0.871 812 (Mbps) FM 7.152 ± 0.236 4.100 29.685 0.862 345 Fair Rate E-E 6.365 ± 0.352 5.851 6.864 0.148 768 (Mbps) FM 6.335 ± 0.244 5.642 7.025 0.197 225 6. Conclusion In this work, we proposed an FM-aided decentralized MARL framework for RA network optimization. By employing an FM as a reward-agnostic critic backbone, our algorithm effectively leverages self-supervised knowledge on network dynamics and improves learning efficiency. Compared with end-to-end training, the FM-aided approach achieves faster convergence, demonstrating its effectiveness in optimizing throughput and AoI across diverse objectives. Future work includes extending this framework to more complex and sophisticated wireless RA scenarios with larger action spaces (e.g., considering transmit power control). Acknowledgements. This work was supported in part by the National Science Foundation (NSF) under Grant CNS2110259, Grant CNS2112471, and Grant ECCS233104; in part by the Office of Naval Research (ONR) under Grant N000142412729; in part by the Defense Advanced Research Projects Agency (DARPA) under Grant D24AP00265, Grant HR00112520019, and Grant HR0011261E001; and in part by Meta Research Award INB3267726. References Abramson (1970) N. Abramson The ALOHA system: another alternative for computer communications. In Proceedings of Fall Joint Computer Conference, p. 281–285. Cited by: §2. Ahmad et al. (2020) I. Ahmad, S. Shahabuddin, H. Malik, E. Harjula, T. Leppänen, L. Loven, A. Anttonen, A. H. Sodhro, M. M. Alam, M. Juntti, et al. Machine learning meets communication networks: current trends and future challenges. IEEE Access 8, p. 223418–223460. Cited by: §1. Amato (2024) C. Amato An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2409.03052. Cited by: §1. Aouedi et al. (2025) O. Aouedi, T. Vu, A. Sacco, D. C. Nguyen, K. Piamrat, G. Marchetto, and Q. Pham A survey on intelligent Internet of Things: applications, security, privacy, and future directions. IEEE Communications Surveys & Tutorials 27 (2), p. 1238–1292. Cited by: §1. Baccelli et al. (2006) F. Baccelli, B. Blaszczyszyn, and P. Muhlethaler An aloha protocol for multihop mobile wireless networks. IEEE Transactions on Information Theory 52 (2), p. 421–436. Cited by: §2. Boyd et al. (2006) S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah Randomized gossip algorithms. IEEE Transactions on Information Theory 52 (6), p. 2508–2530. Cited by: §1, §3.1. Cao et al. (2021) X. Cao, B. Yang, C. Huang, C. Yuen, M. Di Renzo, Z. Han, D. Niyato, H. V. Poor, and L. Hanzo AI-assisted MAC for reconfigurable intelligent-surface-aided wireless networks: challenges and opportunities. IEEE Communications Magazine 59 (6), p. 21–27. Cited by: §1. Chen et al. (2016) B. Chen, J. Chen, Y. Gao, and J. Zhang Coexistence of LTE-LAA and Wi-Fi on 5 GHz with corresponding deployment scenarios: a survey. IEEE Communications Surveys & Tutorials 19 (1), p. 7–32. Cited by: §1. Chen et al. (2024) Z. Chen, Z. Zhang, and Z. Yang Big AI models for 6G wireless networks: opportunities, challenges, and research directions. IEEE Wireless Communications 31 (5), p. 164–172. Cited by: §1, §2. Chen et al. (2022) Z. Chen, Y. Zhou, R. Chen, and S. Zou Sample and communication-efficient decentralized actor-critic algorithms with finite-time analysis. In Proceedings of International Conference on Machine Learning (ICML), p. 3794–3834. Cited by: §4.4, §4.5. Du et al. (2024) J. Du, T. Lin, C. Jiang, Q. Yang, C. F. Bader, and Z. Han Distributed foundation models for multi-modal learning in 6G wireless networks. IEEE Wireless Communications 31 (3), p. 20–30. Cited by: §1, §2. [12] (2017) Evolved universal terrestrial radio access (E-UTRA); further advancements for E-UTRA physical layer aspects. document TR 36.814, V9.2.0, 3GPP. Cited by: §3.2. Fontaine et al. (2024) J. Fontaine, A. Shahid, and E. De Poorter Towards a wireless physical-layer foundation model: challenges and strategies. In 2024 IEEE International Conference on Communications Workshops (ICC Workshops), p. 1–7. Cited by: §1, §2. Gu and Dao (2023) A. Gu and T. Dao Mamba: linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Cited by: §1. Guo et al. (2022) Z. Guo, Z. Chen, P. Liu, J. Luo, X. Yang, and X. Sun Multi-agent reinforcement learning-based distributed channel access for next generation wireless networks. IEEE Journal on Selected Areas in Communications 40 (5), p. 1587–1599. Cited by: §1, §2, §3.3, §4.2, §4.3. Hairi et al. (2022) F. Hairi, J. Liu, and S. Lu Finite-time convergence and sample complexity of multi-agent actor-critic reinforcement learning with average reward. In Proceedings of International Conference on Learning Representations (ICLR), Cited by: §4.4, §4.5, §4.5. Han et al. (2020) M. Han, S. Khairy, L. X. Cai, Y. Cheng, and R. Zhang Reinforcement learning for efficient and fair coexistence between LTE-LAA and Wi-Fi. IEEE Transactions on Vehicular Technology 69 (8), p. 8764–8776. Cited by: §1, §2. He et al. (2024) Y. He, X. Gang, and Y. Gao Intelligent decentralized multiple access via multi-agent deep reinforcement learning. In Proceedings of IEEE Wireless Communications and Networking Conference (WCNC), p. 1–6. Cited by: §1, §2, §4.2, §4.3. Jiang et al. (2012) L. Jiang, M. Leconte, J. Ni, R. Srikant, and J. Walrand Fast mixing of parallel Glauber dynamics and low-delay CSMA scheduling. IEEE Transactions on Information Theory 58 (10), p. 6541–6555. Cited by: §2. Jiang et al. (2010) L. Jiang, D. Shah, and J. Walrand Distributed random access algorithm: scheduling and congestion control. IEEE Transactions on Information Theory 56 (12), p. 6182–6207. Cited by: §2. Jiang and Walrand (2009) L. Jiang and J. Walrand A distributed CSMA algorithm for throughput and utility maximization in wireless networks. IEEE/ACM Transactions on Networking 18 (3), p. 960–972. Cited by: §2. Kleinrock and Tobagi (1975) L. Kleinrock and F. Tobagi Packet switching in radio channels: part I-carrier sense multiple-access modes and their throughput-delay characteristics. IEEE Transactions on Communications 23 (12), p. 1400–1416. Cited by: §2. Kulin et al. (2021) M. Kulin, T. Kazaz, E. De Poorter, and I. Moerman A survey on machine learning-based performance improvement of wireless networks: PHY, MAC and network layer. Electronics 10 (3), p. 318. Cited by: §1. Lee et al. (2024) C. Lee, S. Park, and T. Cheong Dynamic-persistent CSMA: a reinforcement learning approach for multi-user channel access. IEEE Access 12, p. 178705–178716. Cited by: §1, §2. Mahmood et al. (2021) N. H. Mahmood, S. Böcker, I. Moerman, O. A. López, A. Munari, K. Mikhaylov, F. Clazzer, H. Bartz, O. Park, E. Mercier, et al. Machine type communications: key drivers and enablers towards the 6G era. EURASIP Journal on Wireless Communications and Networking 2021 (1), p. 134. Cited by: §1. Ni et al. (2010) J. Ni, B. Tan, and R. Srikant Q-CSMA: queue-length-based CSMA/CA algorithms for achieving maximum throughput and low delay in wireless networks. IEEE/ACM Transactions on Networking 20 (3), p. 825–836. Cited by: §2. Oh et al. (2025) M. S. Oh, Z. Zhang, F. Hairi, A. Velasquez, and J. Liu Consensus-based decentralized multi-agent reinforcement learning for random access network optimization. In Proceedings of the Twenty-sixth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, p. 241–250. Cited by: 1st item, §4.4, §4.5, §4.5, §5.2. Oh et al. (2026) M. S. Oh, Z. Zhang, A. Velasquez, N. D. Bastian, and J. Liu Technical report on foundation model-aided multi-agent reinforcement learning for random access network optimization. Technical report The Ohio State University. Note: Online technical report External Links: Link Cited by: §4.5. [29] (2023) P series: radiowave propagation; propagation data and prediction methods for the planning of indoor radiocommunication systems and radio local area networks in the frequency range 300 MHz to 450 GHz. document ITU-R P.1238-12, ITU. Cited by: §3.2. Rodriguez et al. (2023) J. C. Rodriguez, F. Grijalva, M. García, D. E. Chérrez Barragán, B. A. Acuña Acurio, and H. Carvajal Wireless communication technologies for smart grid distribution networks. Engineering Proceedings 47 (1), p. 7. Cited by: §1. Sheng et al. (2025) Y. Sheng, J. Wang, X. Zhou, L. Liang, H. Ye, S. Jin, and G. Y. Li A wireless foundation model for multi-task prediction. arXiv preprint arXiv:2507.05938. Cited by: §2. Sun et al. (2023) Y. Sun, S. Ma, R. Madaan, R. Bonatti, F. Huang, and A. Kapoor SMART: self-supervised multi-task pretraining with control transformers. In Proceedings of International Conference on Learning Representations (ICLR), Cited by: 2nd item, 3rd item, §4.3. Sutton et al. (1998) R. S. Sutton A. G. Barto et al. Reinforcement learning: an introduction. MIT Press Cambridge. Cited by: §4.1. Tse and Viswanath (2005) D. Tse and P. Viswanath Fundamentals of wireless communication. Cambridge University Press, New York, NY, USA. Cited by: §3.2, §3.2. Watts and Strogatz (1998) D. J. Watts and S. H. Strogatz Collective dynamics of ‘small-world’ networks. Nature 393 (6684), p. 440–442. Cited by: §5.1. Xu et al. (2020) T. Xu, Z. Wang, and Y. Liang Improving sample complexity bounds for (natural) actor-critic algorithms. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), p. 4358–4369. Cited by: §4.4. Yang et al. (2025a) T. Yang, P. Zhang, M. Zheng, Y. Shi, L. Jing, J. Huang, and N. Li WirelessGPT: a generative pre-trained multi-task learning framework for wireless communication. IEEE Network. Cited by: §2. Yang et al. (2025b) Z. Yang, H. Du, D. Niyato, X. Wang, Y. Zhou, L. Feng, F. Zhou, W. Li, and X. Qiu Revolutionizing wireless networks with self-supervised learning: a pathway to intelligent communications. IEEE Wireless Communications. Cited by: §1, §2. Yu et al. (2023) H. Yu, H. Zhao, Z. Fei, J. Wang, Z. Chen, and Y. Gong Deep-reinforcement-learning-based NOMA-aided slotted ALOHA for LEO satellite IoT networks. IEEE Internet of Things Journal 10 (20), p. 17772–17784. Cited by: §2. Yu et al. (2022) Y. Yu, S. C. Liew, and T. Wang Multi-agent deep reinforcement learning multiple access for heterogeneous wireless networks with imperfect channels. IEEE Transactions on Mobile Computing 21 (10), p. 3718–3730. Cited by: §1. Yu et al. (2019) Y. Yu, T. Wang, and S. C. Liew Deep-reinforcement learning multiple access for heterogeneous wireless networks. IEEE Journal on Selected Areas in Communications 37 (6), p. 1277–1290. Cited by: §1, §2, §4.2, §4.2, §4.3. Zhang et al. (2018) K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Basar Fully decentralized multi-agent reinforcement learning with networked agents. In Proceedings of International Conference on Machine Learning (ICML), p. 5872–5881. Cited by: §4.4. Zhang et al. (2020) L. Zhang, H. Yin, Z. Zhou, S. Roy, and Y. Sun Enhancing WiFi multiple access performance with federated deep reinforcement learning. In Proceedings of IEEE Vehicular Technology Conference, p. 1–6. Cited by: §1, §2. Zhang et al. (2021) X. Zhang, Z. Liu, J. Liu, Z. Zhu, and S. Lu Taming communication and sample complexities in decentralized policy evaluation for cooperative multi-agent reinforcement learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Cited by: §4.5. Zhang et al. (2025) Z. Zhang, M. S. Oh, F. Hairi, Z. Luo, A. Velasquez, and J. Liu Finite-time global optimality convergence in deep neural actor-critic methods for decentralized multi-agent reinforcement learning. In Proceedings of International Conference on Machine Learning (ICML), Cited by: §4.4, §4.5, §4.5, §5.2.