Paper deep dive
SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning
Mohammad Aref Jafari-Raddani, Morteza Mohajjel Kafshdooz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/18/2026, 6:17:49 AM
Summary
The paper introduces SAPE (Sandwich Adapters for Parameter Efficiency), a Parameter-Efficient Fine-Tuning (PEFT) framework that utilizes a 'sandwich-style' hard weight-sharing topology. SAPE isolates input and output boundary layers to prevent gradient interference while sharing adapter weights across balanced groups of intermediate Transformer layers. This approach reduces memory consumption and computational overhead compared to dynamic masking methods, achieving state-of-the-art performance in low-parameter regimes on tasks like natural language understanding and reasoning.
Entities (12)
Relation Signals (9)
SAPE → isa → PEFT
confidence 95% · we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework...
SAPE → usestopology → sandwich-style hard weight-sharing
confidence 95% · SAPE... based on a sandwich-style hard weight-sharing topology.
SAPE → appliedto → Llama-3.2
confidence 90% · On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B)...
SAPE → appliedto → RoBERTa-large
confidence 90% · SAPE outperforms proPETL on RoBERTa-large...
SAPE → improvesmetricon → CommonsenseQA
confidence 90% · and +3.11% on CommonsenseQA.
SAPE → improvesmetricon → GSM8K
confidence 90% · yielding absolute improvements of +4.85% on GSM8K
SAPE → outperforms → AdaLoRA
confidence 90% · SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K...
SAPE → outperforms → proPETL
confidence 90% · SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget.
Subformer → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2608.15360v1
- Canonical: https://arxiv.org/abs/2608.15360v1
Trouble viewing inline? Open PDF directly →
Full Text
64,263 characters extracted from source content.
Expand or collapse full text
While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline’s parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ∼ 0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning. Additionally, ablation studies confirm the robustness of SAPE’s architectural design choices. SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning Mohammad Aref Jafari-Raddani Email: raddaniaref@gmail.com Affiliation: Department of Electrical and Computer Engineering, Qom University of Technology, Qom, Iran Morteza Mohajjel Kafshdooz Email: mohajjel@qut.ac.ir Affiliation: Department of Electrical and Computer Engineering, Qom University of Technology, Qom, Iran keywordsLarge Language Models, Parameter-Efficient Fine-Tuning, Adapter Tuning, Transformer Architectures, Weight Sharing, sandwich Topology 1 Introduction The advent of Transformer-based pre-trained models has fundamentally revolutionized Natural Language Processing (NLP). Fine-tuning these architectures has yielded state-of-the-art performance across a wide range of downstream tasks (Han et al 2024; Raiaan et al 2024), a breakthrough largely attributed to their immense capacity—enabled by billions of parameters—and pre-training on vast datasets. However, the sheer scale of these models renders full fine-tuning prohibitively expensive in terms of both memory and computational resources. To mitigate the computational and memory costs imposed by large-scale Transformer models, Parameter-Efficient Fine-Tuning (PEFT) techniques have been proposed as an effective alternative to full fine-tuning. Adapter tuning, as one of the earlier PEFT methods, was introduced by Houlsby et al 2019 and Pfeiffer et al 2021, which involves training a small, non-linear bottleneck inserted sequentially into feed-forward sublayers within Transformer layers. Additionally, Low-Rank Adaptation (LoRA) (Hu et al 2021) was proposed as another PEFT approach that reparameterizes weight updates as low-rank matrices. Although these methods are highly successful in reducing memory costs, there is a noticeable performance gap between them and the full fine-tuning strategy (Biderman et al 2024). To address the rank collapse problem in LoRA, AdaLoRA (Zhang et al 2023) was proposed to dynamically allocate the parameter budget across different weight matrices based on their relative importance, instead of applying a fixed rank uniformly. However, this dynamic rank allocation relies on continuous importance scoring and iterative masking during training, which introduces a computational overhead. A parallel line of research has demonstrated that cross-layer parameter sharing serves as a regularizer and simultaneously reduces the memory footprint of large-scale Transformer models. For example, ALBERT (Lan et al 2019) demonstrated the viability of sharing entire Transformer blocks to improve parameter efficiency. Inspired by this idea, the Subformer architecture (Reid et al 2021) refined this approach for generative modeling by introducing a “sandwich-style” topology, where intermediate layers share parameters while crucial boundary layers remain independent to preserve representational power. Furthermore, Pires et al 2023 demonstrated that sharing a single, wide Feed-Forward Network (FFN) across encoder layers yields superior accuracy and latency compared to standard, isolated narrow networks. Inspired by these parameter-sharing techniques, recent studies have begun integrating cross-layer parameter sharing into PEFT methods. For instance, ShareLoRA (Song et al 2024) and Tied-LoRA (Renduchintala et al 2024) demonstrated that sharing low-rank projection matrices uniformly across all layers significantly reduces the parameter footprint compared to standard LoRA. However, this homogeneous sharing strategy leads to delayed convergence. Nevertheless, while the LoRA paradigm has seen extensive exploration in parameter sharing, a significant research gap remains: cross-layer sharing within the Adapter paradigm is remarkably underexplored. The few existing attempts, such as proPETL (Zeng et al 2023), share a single prototype network across all Transformer blocks, relying on learnable binary masks to dynamically prune connections. Nonetheless, this masking step introduces a slight overhead during training. Concurrently, within Automatic Speech Recognition (ASR), methods like Hierarchical Recurrent Adapters (HRA) (Munkhdalai et al 2024) and Shared-Adapters (Rolland and Abad 2024) have successfully implemented cross-layer adapter sharing in speech models. However, these ASR approaches also utilize homogeneous sharing. Despite the efficacy of the sandwich-style topology in layer sharing like Subformer (Reid et al 2021), its potential application within Parameter-Efficient Fine-Tuning (PEFT) remains unexplored. Specifically, investigating a sandwich sharing configuration for the additive paradigm is an unaddressed opportunity. Figure 1: Generative Performance vs. Trainable Parameters on ConvAI2. SAPE maintains high semantic fidelity (BERTScore) even at extreme sub-million parameter constraints. To bridge this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), an additive PEFT method that employs a static, weight-sharing topology aligned with the hierarchical representation depth of Transformers. Unlike prior parameter-shared PEFT approaches that either enforce uniform layer-wise tying or rely on dynamic layer-wise masking with additional computational cost, SAPE leverages parameter sharing through a deterministic sandwich structure. Specifically, the proposed framework incorporates a dual-mechanism design: (i) It isolates the critical boundary layers (L0L_0 and LN−1L_N-1) with dedicated, unshared adapters, preserving independent feature adaptation at raw embedding inputs and task-specific projections to prevent boundary gradient interference; and (i) It partitions the intermediate layers into G balanced groups where adapters within each group share hard parameter weights. This deterministic topology drastically compresses trainable parameter complexity while functioning as an effective structural regularizer—completely eliminating the auxiliary memory and latency overhead associated with dynamic rank allocation or mask generation schedules. The primary contributions of this paper are summarized as follows: • Novel Structural PEFT Framework: We propose SAPE, an additive PEFT framework that leverages a boundary-isolated sandwich topology for hard adapter weight sharing. SAPE eliminates parameter redundancy across intermediate layers while preventing boundary gradient interference, achieving high parameter compression without relying on computationally expensive dynamic methods. • State-of-the-Art Low-Parameter Efficiency: We evaluate SAPE across both decoder-only and encoder-only Transformers on diverse downstream tasks, achieving state-of-the-art performance in low-parameter regimes. Notably, when fine-tuning LLaMA-3.2 (3B) with a strict budget of ∼ 0.6M parameters, SAPE outperforms AdaLoRA across major reasoning and generative metrics, yielding substantial absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, SAPE outperforms proPETL Adapter on the GLUE benchmark using RoBERTa-large, despite utilizing only 10% of the baseline’s parameter budget. • Comprehensive Architectural & Empirical Insights: We present a thorough series of topological and mechanistic ablations, establishing four core insights into shared adaptation. First, we show that hard parameter sharing strongly regularizes semantic generalization while marginally smoothing rigid step-by-step arithmetic reasoning. Second, topological ablations confirm that boundary layer isolation (L0,LN−1L_0,L_N-1) is structurally necessary to prevent representational collapse on complex entailment tasks. Third, we show that performance does not scale monotonically with group granularity (G), identifying G=2G=2 as the optimal configuration for DeBERTa-v3-base, retaining over 99.4%99.4\% of unshared baseline performance at one-third of the parameter cost. Finally, we demonstrate that while framework performance is robust to spatial density allocation (DenseEarly vs. DenseLate), it benefits significantly from layer-wise independent dropout masking dynamics. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning (PEFT) To mitigate the high computational cost of full fine-tuning of LLMs, parameter-efficient fine-tuning methods have been proposed to reduce the number of trainable parameters while maintaining full fine-tuning performance. Houlsby et al 2019 pioneered the insertion of an adapter module with a non-linear bottleneck after both the attention and feed-forward sublayers. By only training this modular adapter, not only is the number of trainable parameters reduced significantly, but the modular design also reduces the memory required to store task-specific parameters. Subsequently, Pfeiffer et al 2021 demonstrated that isolating a single adapter after the feed-forward sublayer is more efficient. Furthermore, He et al 2022 introduced SparseAdapter to further reduce parameter redundancy within adapter modules. They prune adapter weight matrices at initialization utilizing different pruning strategies, such as random pruning and SNIP. They demonstrate that adapters remain effective even at high sparsity levels (up to 80%). Building on this insight, their proposed Large-Sparse setting scales up the bottleneck dimension alongside a high sparsity ratio, boosting representation capacity while strictly maintaining a fixed parameter budget. However, while SparseAdapter improves parameter efficiency locally via unstructured intra-module pruning, it does not share parameters across Transformer layers, leaving macro-level layer redundancy unaddressed. Besides bottleneck adapters, soft prompt methods represent another additive PEFT paradigm. Pioneered by Prefix-Tuning (Li and Liang 2021) and Prompt Tuning (Lester et al 2021), these methods prepend a sequence of learnable continuous vectors to input embeddings or intermediate activations, achieving high parameter efficiency while leaving all pre-trained backbone weights completely frozen. In contrast to additive modules, selective PEFT methods modify only a small, specific subset of the model’s pre-trained parameters. For instance, BitFit Zaken et al 2022 fine-tunes solely the internal bias vectors of the network while keeping all weight matrices frozen. Despite its high parameter efficiency, it faces key limitations, such as a strict architectural dependency, and its low parameter capacity bottlenecks performance when adapting to complex domain shifts. Another paradigm of PEFT is decomposition-based methods, which reduce the number of trainable parameters by learning low-rank weight updates that are added to the frozen pre-trained weights of the model. In this case, LoRA (Hu et al 2021) introduces low-rank matrices A and B representing the weight update ΔW=AB W=AB of the query and value projection matrices. LoRA enables augmenting the original weight matrix with a trainable low-rank update and incurs no additional inference overhead. Despite these advantages, standard LoRA allocates a uniform rank r to all reparameterized matrices, ignoring the varying importance of individual weight projections. To improve rank allocation efficiency, AdaLoRA (Zhang et al 2023) utilizes an SVD-style formulation ΔW=PΛQ W=P Q, where singular values in Λ are dynamically pruned during training based on sensitivity-based importance scoring. Building upon this paradigm, subsequent works, including DyLoRA (Valipour et al 2023) and DoRA (Liu et al 2024), explore dynamic rank selection and weight-decomposed adaptation, respectively. (We refer readers to Wang et al 2025 for a comprehensive survey on all these PEFT methods). Although AdaLoRA efficiently determines rank allocation, it introduces minor computational overhead due to iterative importance scoring and dynamic pruning schedules. Nevertheless, despite the parameter efficiency of the LoRA paradigm, Ko 2025 recently demonstrated that LoRA can sometimes be even slower than full fine-tuning. This occurs because GPUs process a single kernel at a time; inserting parallel low-rank adapter branches (A and B) forces the execution of sequential CUDA kernels, creating a computational bottleneck. 2.2 Cross-Layer Parameter Sharing To reduce the overall size of large language models and improve pretraining and memory usage efficiency, parameter sharing across layers has been explored. ALBERT (Lan et al 2019) utilizes cross-layer parameter sharing in BERT to eliminate the increase in parameters caused by growing the depth of the network. By gradually increasing the hidden size, they illustrate that it is possible to surpass BERT even with a lower number of parameters. However, the trade-off is between the overhead of training time due to larger width and better performance and memory consumption. Building upon these parameter-sharing paradigms, Subformer (Reid et al 2021) suggests a sandwich-style sharing strategy, which shares middle layers L1L_1 to LN−2L_N-2 as one group with a wider dimension than the base model dimension, and keeps two separate layers for L0L_0 and LN−1L_N-1 as they interact with raw embeddings and final outputs. This structure-wise sharing not only outperforms uniform sharing across all layers, but also reaches comparable performance compared to a base model with more parameters, achieving even better performance and faster training convergence in some tasks. However, the trade-off of increasing the dimension of the shared layers remains. Despite these successful results, structure-wise sharing is still a gap in fine-tuning methods. 2.3 Parameter Sharing within PEFT In an early foundational work, Compacter (Karimi Mahabadi et al 2021) introduced parameter sharing in bottleneck adapters by leveraging Parameterized Hypercomplex Multiplication (PHM) layers to share global factor matrices AiA_i across all Transformer layers, while employing layer-specific rank-one weight factorizations (siti⊤s_it_i ) for adaptation. Building on the concept of shared prototypes in PEFT, Zeng et al 2023 introduced the proPETL framework. proPETL trains a shared prototype network such as an adapter module, LoRA parameters, or a prefix across all layers, applying layer-specific binary masks to extract individual sub-networks. Notably, their findings demonstrate that utilizing an adapter as the shared prototype outperforms LoRA or prefix sharing, although the requirement for layer-wise mask search and storage introduces non-trivial memory and computational overhead. Additionally, while factorized and masked adapter sharing have been investigated, direct, macro-level hard weight sharing of complete bottleneck adapters across structured layer groups remains remarkably underexplored. In parallel, ShareLoRA (Song et al 2024) explores a variety of sharing approaches in LoRA, including sharing only the A matrix with a unique B matrix per layer, sharing only B with an independent A matrix, or sharing both at the same time. Despite competitive results with shared A, configuration variants that share B or both matrices tend to converge more slowly and underperform. Recently, MASA (Dong et al 2025) leveraged cross-layer parameter sharing in multi-task learning by proposing a LoRA-based framework. Instead of tying parameters across all layers uniformly like ShareLoRA (Song et al 2024), MASA shares an ensemble of A matrices across adjacent layer groups while retaining layer-specific B matrices. Furthermore, rather than relying on dynamic routing mechanisms such as AdapterFusion (Pfeiffer et al 2021), MASA aggregates the outputs of these A matrices via simple summation. These asymmetric, heterogeneous sharing strategies yield superior multi-task adaptation capabilities while maintaining parameter efficiency. However, all of these parameter sharing strategies across both additive adapters and LoRA reparameterizations enforce parameter tying uniformly across all depth levels, thereby overlooking depth-wise structural sharing patterns. SAPE tackles this unaddressed gap by defining a heterogeneous topology based on sandwich sharing for adapters. 3 Methodology As an adapter-based PEFT framework, SAPE employs both shared and isolated adapters across distinct Transformer layers. This section begins by defining the architecture of the base adapter. Next, driven by the hierarchical architecture of Language Models (LMs), the proposed sandwich sharing topology is detailed. Finally, the optimization routing for this paradigm is formulated, in addition to formalizing the total trainable parameter complexity. Figure 2: Architectural overview of the SAPE framework. The diagram illustrates the macro-to-micro structural constraints of the proposed sandwich topology. (a) Sandwich Topology: Network-level parameter sharing across N Transformer layers, depicting isolated boundary adapters ((0)=θL0,(N−1)=θLN−1A(0)= _L_0,A(N-1)= _L_N-1) alongside shared intermediate groups (ℐg⟹(l)=θgI_g (l)= _g). (b) Intra-Layer Placement: Detailing the injection of the bottleneck module ℱθF_θ immediately following the final feed-forward sub-layer, receiving the hidden state lh_l. (c) Adapter Module: Internal tensor operations of the bottleneck adapter, highlighting the down-projection (down∈ℝd×mW_down ^d× m), non-linear ReLU activation σ(⋅)σ(·), dropout regularization, and up-projection (up∈ℝm×dW_up ^m× d). 3.1 Adapter Architecture and Placement SAPE utilizes a bottleneck architecture with non-linear activation, similar to the adapter modules which were introduced by Houlsby et al 2019 and the optimized single-adapter configuration by Pfeiffer et al 2021. Let l∈ℝdh_l ^d denote the hidden state output of the final feed-forward sub-layer at the l-th Transformer layer. The adapter transformation is parameterized by θ=down,up,down,upθ=\W_down,W_up,b_down,b_up\ and is formalized as the mapping ℱθ:ℝd→ℝdF_θ:R^d ^d: ℱθ(l)=l+Φ(σ(ldown+down))up+upF_θ(h_l)=h_l+ (σ(h_lW_down+b_down) )W_up+b_up (1) where down∈ℝd×mW_down ^d× m projects the d-dimensional input to a smaller bottleneck dimension m (m≪dm d), and up∈ℝm×dW_up ^m× d projects it back to the original dimension. The function σ(⋅)σ(·) represents the ReLU activation, and Φ(⋅) (·) denotes a dropout operator with a rate of 10%10\% applied during training to mitigate overfitting. In the SAPE framework, ℱθF_θ is applied once per Transformer layer after the last feed-forward layer. We employ a hybrid adapter injection strategy that structurally separates boundary transformations from intermediate semantic routing. Inspired by the sandwich-style weight sharing paradigm introduced by Reid et al 2021 for generative models, we partition the entire set of Transformer layers ℒL into a set of boundary layers ℬB and a set of intermediate layers ℐI: ℬ=0,N−1,ℐ=ℒ∖ℬ=1,2,…,N−2B=\0,N-1\, =L =\1,2,…,N-2\ (2) To implement the sandwich topology, we define an adapter allocation function :ℒ→A:L→ , mapping each layer l to a specific parameter set θ∈θ∈ . For the critical boundary layers, which capture low-level syntax and high-level task-specific semantics respectively, as corroborated by extensive probing studies (jawahar2019does; Tenney et al 2019; Vulić et al 2020), we allocate strictly isolated parameter sets: (0)=θL0,(N−1)=θLN−1A(0)= _L_0, (N-1)= _L_N-1 (3) For the intermediate layers ℐI, we partition the set into G disjoint subsets (groups), such that ℐ=⋃g=0G−1ℐgI= _g=0^G-1I_g and ℐi∩ℐj=∅I_i _j= for i≠ji≠ j. All layers within a subset ℐgI_g share a single adapter parameter set θg _g: ∀l∈ℐg⟹(l)=θg∀ l _g (l)= _g (4) After partitioning the intermediate layers into G groups, the objective is to distribute the N−2N-2 layers as uniformly as possible among them. Let s=⌊N−2G⌋,r=(N−2)modG,s= N-2G , r=(N-2) G, (5) where s denotes the minimum number of layers assigned to each group, and r represents the number of remaining layers after the uniform allocation. Consequently, each group contains either s or s+1s+1 layers, ensuring that the size difference between any two groups is at most one layer. The distribution of the r remaining layers is controlled by the boolean hyperparameter DenseEarly. When DenseEarly = True, the additional layers are assigned to the first r groups, resulting in denser grouping toward the earlier Transformer layers. Conversely, when DenseEarly = False, the additional layers are assigned to the last r groups, producing denser grouping toward the later Transformer layers. Formally, the cardinality of each group is defined as: |ℐg|=s+1,if DenseEarly=True ∧g<r,s,if DenseEarly=True ∧g≥r,s+1,if DenseEarly=False ∧g≥G−r,s,if DenseEarly=False ∧g<G−r.|I_g|= casess+1,&if DenseEarly=True \ g<r,\\ s,&if DenseEarly=True \ g≥ r,\\ s+1,&if DenseEarly=False \ g≥ G-r,\\ s,&if DenseEarly=False \ g<G-r. cases (6) This allocation guarantees that every intermediate layer is assigned to exactly one group while maintaining balanced group sizes. As a result, the sandwich topology preserves parameter sharing across the intermediate layers while allowing explicit control over whether the larger groups are concentrated near the input or near the output of Transformer. 3.2 Optimization and Parameter Complexity During backpropagation, hard parameter sharing modifies the optimization dynamics by coupling the gradient updates across multiple Transformer layers. SAPE utilizes a dropout layer in all adapter modules, meaning an independent dropout mask l∼Bernoulli(1−p)m_l (1-p) is dynamically generated during the forward pass, regardless of whether the module belongs to a shared group or an isolated boundary layer. The adapter transformation at any specific layer is formally conditioned on its local mask: ℱθg(l,l)=l+(l⊙σ(ldown+down))up+upF_ _g(h_l;m_l)=h_l+ (m_l σ(h_lW_down+b_down) )W_up+b_up (7) where ⊙ denotes the element-wise product. Let (Θ)J( ) denote the task-specific loss function. Since the adapter parameter set θg _g is shared by all layers in the group ℐgI_g, its gradient is obtained by accumulating the contributions from every layer in which it is applied. By the multivariate chain rule, the parameter update is given by: θg←θg−η∑l∈ℐg(∂ℱθg(l,l)∂θg)T∇ℱθg(l,l) _g← _g-η _l _g ( _ _g(h_l;m_l)∂ _g )^T _F_ _g(h_l;m_l)J (8) where η denotes the learning rate. Unlike conventional adapters, where each parameter set is optimized using the gradient of a single Transformer layer, the shared adapter receives the aggregated gradient from all layers within its assigned group. Consequently, each update is jointly influenced by the representations learned at multiple depths of the network, encouraging the shared parameters to capture features that are beneficial across the entire group rather than adapting to a single layer. From a parameterization perspective, hard parameter sharing introduces an equality constraint among the adapters assigned to the layers in ℐgI_g, i.e., all layers in the group are required to use the same parameter set θg _g. This constraint reduces the number of trainable parameters while acting as a form of structural regularization. Because the shared adapter must accommodate the feature distributions of multiple Transformer layers simultaneously, it is encouraged to learn more generalizable transformations instead of becoming highly specialized for the representation of any individual layer. As a result, the proposed sandwich topology improves parameter efficiency while preserving the expressive capacity required for effective adaptation. Given a bottleneck dimension m and a base model dimension d, the total number of trainable parameters PtotalP_total introduced by the SAPE method across the G intermediate groups and 22 boundary layers is strictly constrained to: Ptotal=(2+G)×(2md+m+d)P_total=(2+G)×(2md+m+d) (9) This rigorous parameterization highlights SAPE’s ability to drastically compress the fine-tuning memory footprint while explicitly defining the routing and optimization pathways. 4 Experiments The SAPE framework is implemented for fine-tuning RoBERTa-large (Liu et al 2019) and DeBERTa V3-base (He et al 2020) on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al 2018). In addition, for natural language generation and reasoning, LLaMA-3.2 (3B) (Grattafiori et al 2024) is fine-tuned on a wide range of tasks utilizing SAPE and standard baselines. Finally, we conduct a comprehensive ablation study to evaluate the core design choices within SAPE, including the sandwich placement topology, group granularity (G), spatial density allocation(DenseEarlyDenseEarly), and independent dropout masking. 4.1 Experimental Setup Implementation Details: PyTorch and the Hugging Face transformers library are used for the implementation. We evaluate SAPE by setting DenseEarly = False as the default configuration across the various setups of intermediate groups (G) and bottleneck dimensions (D), as specified in each section. The experiments were conducted on an NVIDIA RTX 3090 and an NVIDIA T4 equipped with 24 GB and 15 GB of VRAM, respectively. Notably, the complete training hyperparameter configurations for all subsequent sections are detailed in Appendix 6. Datasets We evaluate SAPE across the following benchmarks: • GLUE Benchmark (Wang et al 2018): Selected to assess standard Natural Language Understanding (NLU) on encoder-only architectures. • ConvAI2 (Dinan et al 2019): A multi-turn, non-goal-oriented dialogue dataset explicitly designed to evaluate conversational consistency and persona maintenance in open-domain chatbots. • CommonsenseQA (Talmor et al 2019): A multiple-choice question answering dataset designed to evaluate question answering with prior world knowledge across complex semantic relationships. • GSM8K (Cobbe et al 2021): A dataset of 8.5K high-quality, linguistically diverse grade school math word problems designed to evaluate multi-step mathematical reasoning. Baselines SAPE is compared with full fine-tuning and the following PEFT methods: • Adapter: Houlsby et al 2019 pioneered adapter tuning (referred to in this study as HAdapter), which shares similar architectures to ours but injects two distinct adapters per Transformer layer. Additionally, we compare our method to Pfeiffer et al 2021, who introduced a more efficient design with adapters injected only once after the FFN modules, in addition to Adapter Fusion which is used for knowledge composition among adapters in multi-task learning. However, in this study, PAdapter strictly refers to utilizing a single adapter per layer. • LoRA (Hu et al 2021): This method freezes the pre-trained weights and injects trainable low-rank decomposition matrices (A and B) into the attention layers to approximate weight increments. • BitFit (Zaken et al 2022): This method modifies only the model’s internal bias vectors, modifying less than 0.1% of the total parameters. • AdaLoRA (Zhang et al 2023): An extension of LoRA that reparameterizes incremental weight updates using a singular value decomposition (SVD) formulation. AdaLoRA adaptively allocates the parameter budget across layers based on importance scoring. • proPETL Adapter (Zeng et al 2023): This approach shares a single prototype network across layers and tasks while jointly learning layer-specific binary masks to select distinct sub-networks. In the following experiment, the prototype network is considered as an adapter module. 4.2 Natural Language Understanding To evaluate SAPE’s performance on encoder-only models and NLU tasks, DeBERTa-base and RoBERTa-large are fine-tuned. For the DeBERTa-base model, we choose G=2G=2, resulting in two groups of adapters that each contain five layers and with both D=16D=16 and D=64D=64. Additionally, for RoBERTa-large, we set G=2G=2 and D=64D=64, resulting in two groups of 11 shared adapters across its 24 layers. Table 1: Performance comparison of SAPE against full fine-tuning and other PEFT methods on the GLUE benchmark. Best results among PEFT methods are highlighted in bold. Baseline results for DeBERTa-v3-base are sourced from the AdaLoRA paper Zhang et al 2023, while RoBERTa-large baseline results are reported from Xu et al. Xu et al 2023. Results for our proposed SAPE method are independently computed. Method Params MNLI SST-2 CoLA QQP QNLI RTE MRPC STS-B Avg. (m/m) (Acc) (Mcc) (Acc/F1) (Acc) (Acc) (Acc) (Corr) DeBERTa-v3-base Full FT 184M 90.01 95.63 69.19 92.40/89.80 94.03 83.75 89.46 91.60 88.09 HAdapter 0.31M 90.06 95.41 67.65 91.54/88.81 93.52 83.39 89.25 91.31 87.60 PAdapter 0.30M 89.97 94.72 69.06 91.40/88.62 93.87 84.48 89.71 91.38 87.90 LoRA (r=2r=2) 0.33M 90.34 94.95 68.71 91.61/88.91 94.03 85.56 89.71 91.68 88.15 AdaLoRA 0.32M 90.68 95.80 70.04 91.78/89.16 94.49 87.36 90.44 91.63 88.86 SAPE (D=64,G=2D=64,G=2) 0.40M 90.10 95.64 68.89 91.18/88.26 94.03 86.64 91.18 91.65 88.48 BitFit 0.10M 89.64 94.84 66.96 88.41/84.95 92.24 78.70 87.75 91.35 86.02 SAPE (D=16,G=2D=16,G=2) 0.10M 89.93 96.10 68.37 90.33/87.05 93.79 87.36 90.20 91.28 88.22 RoBERTa-large Full FT 355M 89.42 95.54 65.78 89.30/86.68 93.61 81.23 89.22 91.75 86.82 HAdapter 19.7M 91.00 96.37 67.03 92.19/88.50 94.31 85.25 89.94 92.59 88.35 proPETL Adapter 5.40M 91.37 96.27 65.55 90.67/87.74 95.20 88.89 89.71 91.80 88.50 AdaLoRA 2.23M 90.36 94.95 65.85 89.60/86.30 94.62 77.98 89.46 91.92 86.64 LoRA 1.84M 90.76 96.67 64.47 90.15/86.91 95.00 79.78 87.50 91.55 86.78 BitFit 1.32M 89.98 96.10 68.01 89.48/86.43 94.47 87.73 90.93 91.85 88.38 SAPE (D=64,G=2D=64,G=2) 0.55M 90.28 96.44 69.25 90.22/87.20 94.65 86.28 92.25 92.11 88.75 DeBERTa-base: SAPE is evaluated against existing baselines across two distinct trainable parameter budgets, as detailed in Table 1. At the D=64D=64 budget (0.40M parameters), SAPE achieves an average score of 88.48%, outperforming full fine-tuning by +0.39%. Furthermore, it surpasses HAdapter, PAdapter, and LoRA. Although SAPE remains second to AdaLoRA in final peak accuracy at this budget, Figure 3: Epoch-wise validation trajectories for SAPE (D=16D=16) and AdaLoRA across low-resource tasks. To ensure a strict and fair evaluation, AdaLoRA is implemented using the exact hyperparameter configurations and initialization schedules specified in its original paper. Figure 3 demonstrates that AdaLoRA’s convergence speed is visibly constrained by its warmup and rank reallocation schedules. Conversely, SAPE’s fixed-topology design eliminates this iterative ranking overhead entirely, allowing it to achieve significantly higher validation results during the early epochs of training. Under the highly constrained 0.10M budget (D=16D=16), SAPE outperforms BitFit by an average of +2.20% and exceeds HAdapter, PAdapter, and LoRA while utilizing roughly one-third (0.3×0.3×) the parameters. On SST-2, SAPE (D=16D=16) achieves a peak accuracy of 96.10%, outperforming all evaluated baselines alongside highly competitive performance on RTE. RoBERTa-large: As reported in Table 1, SAPE operates under the smallest parameter budget (0.55M) among all evaluated baselines, yet achieves the highest overall average of 88.75%. Notably, SAPE outperforms proPETL Adapter by +0.25%, despite proPETL requiring over 10×10× more trainable parameters and incurring extra computational overhead for mask generation. On individual tasks, SAPE establishes new state-of-the-art results on MRPC (92.25%) and CoLA (69.25%), surpassing the nearest literature baselines by +1.32% and +1.24%, respectively. 4.3 Natural Language Generation We evaluate the generative and reasoning capabilities of SAPE by fine-tuning LLaMA-3.2 (3B) across three distinct tasks. For open-domain dialogue, we utilized a subset of the ConvAI2 Dinan et al 2019 dataset (3,495 training and 100 validation dialogues), repeating each evaluation three times with different random seeds to ensure statistical robustness. Additionally, GSM8K Cobbe et al 2021 and CommonsenseQA Talmor et al 2019 are evaluated entirely on their standard validation partitions. SAPE is implemented with parameter budgets ranging from ∼ 0.6M to ∼ 2.5M by modulating bottleneck dimensions (dim∈16,32,64dim∈\16,32,64\) and the number of shared groups (G∈3,4G∈\3,4\). Within similar budgets, we also evaluated PAdapter, LoRA, and AdaLoRA. All methods across all benchmarks are trained for 3 epochs using FP16 precision, gradient checkpointing, and frozen backbone parameters, ensuring near-optimal performance while avoiding overfitting. Performance on ConvAI2 is measured via BLEU Papineni et al 2002, ROUGE-L Lin 2004, and BERTScore Zhang et al 2020, while the other two benchmarks are evaluated using Exact Match (EM) accuracy. (Note: Extended hyperparameters and evaluation details are provided in Appendix 6.3). Table 2: Generative and reasoning evaluation of LLaMA-3.2 (3B) across iso-parameter budget tiers. SAPE demonstrates dominance in semantic tasks (ConvAI2, CSQA) while indicating an architectural trade-off in rigid reasoning (GSM8K). Method Topology & Bottleneck Params ConvAI2 CSQA GSM8K BLEU ROUGE-L BERTScore EM (%) EM (%) Zero-Shot Baseline 0 0.0100 0.0947 0.6818 53.48 9.55 High Budget Tier (∼2.0M−2.5M 2.0M-2.5M) LoRA R=8R=8 ∼2.29M 2.29M 0.0263 0.1563 0.7075 79.28 40.86 PAdapter d=13d=13 ∼2.32M 2.32M 0.0316 0.1636 0.7125 78.21 41.55 AdaLoRA final R=7R=7 ∼2.00M 2.00M 0.0239 0.1409 0.7340 74.20 32.37 SAPE (Ours) G=3G=3, d=64d=64 ∼1.98M 1.98M 0.0275 0.1551 0.7086 78.87 40.30 SAPE (Ours) G=4G=4, d=64d=64 ∼2.38M 2.38M 0.0262 0.1424 0.7072 78.54 38.74 Medium Budget Tier (∼1.0M−1.5M 1.0M-1.5M) PAdapter d=8d=8 ∼1.46M 1.46M 0.0320 0.1459 0.7053 79.12 40.86 LoRA R=4R=4 ∼1.14M 1.14M 0.0243 0.1643 0.7150 78.46 40.71 AdaLoRA final R=4R=4 ∼1.14M 1.14M 0.0163 0.1320 0.7340 79.94 32.07 SAPE (Ours) G=3G=3, d=32d=32 ∼1.00M 1.00M 0.0160 0.1467 0.7393 79.12 38.59 SAPE (Ours) G=4G=4, d=32d=32 ∼1.20M 1.20M 0.0305 0.1528 0.7096 78.71 39.27 Low Budget Tier (∼0.6M 0.6M) LoRA R=2R=2 ∼0.57M 0.57M 0.0186 0.1385 0.7046 79.12 39.50 AdaLoRA final R=2R=2 ∼0.60M 0.60M 0.0199 0.1332 0.7322 75.43 32.45 SAPE (Ours) G=3G=3, d=16d=16 ∼0.51M 0.51M 0.0197 0.1477 0.7339 78.46 36.32 SAPE (Ours) G=4G=4, d=16d=16 ∼0.61M 0.61M 0.0196 0.1496 0.7376 78.54 37.30 (G=2G=2) ConvAI2: The results on the ConvAI2 dataset demonstrate that SAPE surpasses prior PEFT methods in generative semantic tasks, especially in low-parameter regimes. In the highly constrained ∼ 0.6M parameter budget, SAPE (4 Groups, dim=16dim=16) significantly outperforms LoRA and AdaLoRA, yielding a superior ROUGE-L of 0.1496 and a BERTScore F1 of 0.7376. Furthermore, as the budget scales to the ∼ 1.0M range, SAPE (3 Groups, dim=32dim=32) achieves a BERTScore of 0.7393, outperforming all comparable baselines. Notably, in the high-budget tier with dim=64dim=64, SAPE starts to overfit due to the large bottleneck dimension. Conversely, PAdapter shows lower performance in the medium-budget tier than in the high-budget tier because its dimension must be reduced significantly, whereas SAPE maintains a suitable dimension and reduces overall parameters through cross-layer sharing. Overall, these results indicate that cross-layer parameter sharing effectively learns generalized semantic representations for conversational tasks under strict parameter constraints. CommonsenseQA: This architectural efficiency strongly extends to the CommonsenseQA results. A notable observation is SAPE’s ability to prevent representational degradation at extreme low-parameter constraints. While AdaLoRA suffers a catastrophic performance drop to 75.43% at the ∼ 0.6M tier, SAPE retains a robust 78.54% Exact Match accuracy. Moreover, the medium budget is once again a sweet spot (∼ 1.0M) for SAPE, as it matches the 79.12% accuracy that LoRA reaches with a double parameter budget. Overall, in world knowledge question answering, AdaLoRA achieves a better accuracy of 79.94%, but SAPE closely chases the best result across all budgets by showing robust performance. GSM8K: Conversely, evaluating SAPE on the GSM8K dataset reveals a fundamental architectural trade-off regarding rigid mathematical reasoning. While SAPE demonstrates competitive performance across all budgets and outperforms AdaLoRA in this benchmark, it is ultimately surpassed by LoRA and PAdapter. For instance, PAdapter achieves 41.55%, whereas SAPE plateaus at 40.30%. This reveals that while forcing layers to share bottleneck representations acts as a powerful regularizer that boosts semantic generalization, it reduces step-by-step arithmetic capability compared to non-sharing adapters. In other words, although the sandwich sharing topology shows competitive performance in this task, reaching peak accuracy in step-by-step reasoning still requires layer-wise adaptation. Figure 4: Architectural ablation of the SAPE model comparing bottleneck capacity (D) and topological granularity (G) across rigid reasoning (GSM8K), semantic reasoning (CSQA), and generative fidelity (ConvAI2). Sensitivity Analysis of G and D: As illustrated by Figure 4, scaling both variables simultaneously would lead to a higher capacity, which could lead to overfitting depending on the type of task. For instance, in rigid arithmetic reasoning (GSM8K), increasing the bottleneck dimension to D=64D=64 provides a higher score, but only under a moderate group division (G=3G=3). Conversely, in conversational tasks (ConvAI2) and generalized semantics (CSQA), SAPE reaches its peak performance at tighter bottlenecks (D=16D=16 or D=32D=32). Ultimately, a balanced configuration of G=3G=3 and D=32D=32 serves as the optimal architectural sweet spot. Overall, at a lower D=16D=16, a higher G=4G=4 yields a better result, but at a higher D=64D=64, a lower G=3G=3 performs better. 4.4 Ablation Study and Empirical Analysis To rigorously evaluate the individual design choices within the SAPE framework, we conduct a comprehensive series of ablation studies on DeBERTa-v3-base across four representative GLUE benchmarks (RTE, QNLI, CoLA, and SST-2). We examine three core architectural dimensions: structural sharing topology, group granularity and spatial density allocation, and layer-wise dropout masking dynamics. To ensure controlled experimental conditions, the sharing topology ablation in Section 4.4.1 evaluates all structural variants under fixed optimization parameters (LR=1×10−4LR=1× 10^-4 with a linear decay schedule) as detailed in Appendix 6.4, strictly isolating structural efficacy from hyperparameter search bias. The remaining ablations are evaluated under the framework’s task-optimized hyperparameter configurations outlined in Appendix 6.1. 4.4.1 Impact of Structural Topology To understand the structural necessity of the sandwich topology, We first isolate the necessity of the sandwich topology by comparing four placement configurations, (1) Normal Sharing: Uniformly sharing across all layers; (2) Bottom-Slice: Isolating only the first layer (L0L_0); (3) Top-Slice: Isolating only the final layer (LN−1L_N-1); and (4) Sandwich-Style (Ours): Isolating both the first and final layers.To ensure a fair comparison, four adapters with an identical bottleneck dimension (dim=16dim=16) were utilized across all four configurations. Table 3: Ablation study on adapter placement topologies across GLUE benchmark tasks using DeBERTa-base. The Sandwich-Style configuration prevents boundary interference and yields the highest average. Topology Distinct L0L_0 Distinct LN−1L_N-1 SST-2 CoLA QNLI RTE Avg. Normal Sharing No No 95.18 67.65 49.46 82.31 73.65 Bottom-Slice Yes No 94.50 66.85 92.99 82.31 84.16 Top-Slice No Yes 95.18 66.80 81.93 82.67 81.65 Sandwich-Style Yes Yes 95.18 66.44 92.82 84.48 84.73 As demonstrated in Table 3, the Normal Sharing topology suffers from severe representational degradation, failing on QNLI task (49.46%). Forcing the network to process raw lexical embeddings and deeper logical entailment using same adapter . Isolating the input layer (Bottom-Slice) successfully restores QNLI performance to 92.99%. Ultimately, the Sandwich-Style topology establishes the optimal architectural equilibrium. By concurrently isolating the input embeddings and the task-specific output projections, it successfully prevents boundary interference while routing general semantic representations through the shared intermediate adapters. 4.4.2 Impact of Group Granularity (G) and Density Allocation We analyze performance sensitivity with respect to the number of intermediate shared groups G∈1,2,4,5,7,10G∈\1,2,4,5,7,10\, which directly governs the total trainable parameter complexity Ptotal=(2+G)×(2md+m+d)P_total=(2+G)×(2md+m+d). In addition, for setups where residual layers exist such that r=(N−2)modG≠0r=(N-2) G≠ 0 (specifically when G=4G=4 and G=7G=7), the allocation logic assigns non-uniform capacity toward either the initial input layers (DenseEarly) or deeper output layers (DenseLate). We empirically evaluate this structural density distribution as well. Throughout this experiment, the inner bottleneck dimension is held constant at d=16d=16. Table 4: Ablation on intermediate group granularity (G) and structural density allocation across GLUE benchmarks. Configuration / Group (G) Params SST-2 CoLA QNLI RTE Avg. G=1G=1 76K 95.41 68.81 93.63 85.92 85.94 G=2G=2 101K 96.10 68.37 93.79 87.36 86.41 G=4G=4 (DenseLate) 152K 95.76 69.24 93.68 84.84 85.88 G=4G=4 (DenseEarly) 152K 96.10 69.27 93.52 85.56 86.11 G=5G=5 178K 96.44 67.31 93.77 85.56 85.77 G=7G=7 (DenseLate) 228K 95.99 68.28 93.43 85.56 85.82 G=7G=7 (DenseEarly) 228K 95.53 67.15 93.57 85.92 85.54 G=10G=10 (PAdapter Baseline) 304K 95.53 70.08 93.67 88.44 86.93 As illustrated in Table 4, performance does not scale monotonically with group granularity G. Remarkably, G=2G=2 achieves the optimal configuration among all SAPE variants, reaching an average accuracy of 86.41%86.41\%. Specifically, SAPE with G=2G=2 captures over 99.4%99.4\% of the performance of the full unshared baseline (G=10G=10, 86.93%86.93\%), despite using three times fewer parameters (101K101K vs. 304K304K) and dispensing with per-layer individual adapters. Notably, while the PAdapter baseline reported in Table 1 reflects standard published configurations from prior work, our re-tuned G=10G=10 setup provides an even stronger, highly optimized baseline for unshared individual adapters under identical training conditions. Furthermore, spatial density allocation (DenseEarly vs. DenseLate) demonstrates marginal and mixed impact across granularities: DenseEarly performs slightly better at G=4G=4 (86.11%86.11\% vs. 85.88%85.88\%), whereas DenseLate leads at G=7G=7 (85.82%85.82\% vs. 85.54%85.54\%). This indicates that SAPE is structurally robust and does not critically depend on precise layer-density placement. Table 5: Comparison of parameter reduction strategies under an equivalent budget (∼101K 101K parameters). Strategy / Method Config Params SST-2 CoLA QNLI RTE Avg. Severe Bottleneck (PAdapter) G=10,d=5G=10,d=5 101K 95.76 66.06 93.59 82.67 84.52 Aggressive Sharing (SAPE) =,=G=2,d=16 101K 96.10 68.37 93.79 87.36 86.41 Finally, to isolate the effect of parameter reduction mechanics under a strict parameter budget (∼101K 101K parameters), we compare an unshared baseline (G=10G=10) using a choked bottleneck dimension (d=5d=5) against SAPE (G=2G=2) using a wider bottleneck dimension (d=16d=16). As presented in Table 5, SAPE consistently outperforms the unshared baseline across the board, achieving a higher average accuracy (86.41%86.41\% vs. 84.52%84.52\%). Most notably, SAPE achieves a 4.69% improvement on the RTE benchmark (87.36% vs. 82.67%). This indicates that excessively reducing the bottleneck dimension (d=5d=5) limits adapter capacity on challenging entailment tasks. In contrast, SAPE maintains parameter efficiency at an equivalent budget by sharing higher-capacity adapters (d=16,G=2d=16,G=2) across layers. 4.4.3 Role of Independent Dropout Masking Because shared adapter modules within group ℐgI_g reuse identical weight matrices across distinct Transformer layers, a critical architectural decision is whether to broadcast a single static dropout mask across the entire shared group or apply random, independent masks for each layer invocation within a group. Table 6 evaluates these strategies against a zero-dropout (p=0.00p=0.00). Table 6: Ablation study on dropout masking mechanics within shared adapter layers. Masking Strategy Dropout (p) SST-2 CoLA QNLI RTE Avg. Shared Mask (across group) 0.10 95.76 66.98 93.52 85.56 85.46 No Dropout 0.00 95.99 66.64 93.33 83.75 84.93 Independent Mask (SAPE) 0.10 96.10 68.37 93.79 87.36 86.41 As indicated in Table 6, the proposed Independent Masking strategy achieves superior performance across all four evaluated benchmarks, yielding an average accuracy of 86.41%86.41\% (+1.48% over zero dropout). Applying identical dropout masks across shared layer adapters forces coupled co-adaptation between shared instances. Independent masks break this intra-group dependency, forcing the single shared weight matrix θg _g to learn representations resilient to diverse layer-wise noise configurations. Notably, static Shared Masking degrades performance relative to zero dropout on SST-2 (95.76%95.76\% vs. 95.99%95.99\%), demonstrating that shared mask broadcasting can lead to rigid spatial constraints rather than effective stochastic regularization. 5 Conclusion and Future Work In this paper, we introduce SAPE, a highly efficient Parameter-Efficient Fine-Tuning method designed to optimize adapter-based learning. By utilizing a sandwich-style structure, we isolate boundary transformations at the initial and final layers from a shared group of intermediate adapters. This topology not only prevents destructive gradient interference but also acts as a powerful structural regularizer. Furthermore, SAPE is a highly versatile framework across a wide range of NLP tasks. At extremely constrained parameter budgets, SAPE delivers state-of-the-art efficiency in natural language understanding, open-domain generative tasks (ConvAI2), and semantic world knowledge retrieval (CommonsenseQA), successfully surpassing even dynamic methods by completely eliminating their computational overhead. Furthermore, while SAPE maintains competitive functionality across all evaluated benchmarks, we acknowledge a distinct architectural trade-off. Because cross-layer parameter sharing inherently generalizes features, it marginally smooths the sharp transformations required for highly rigid tasks. Consequently, in multi-step arithmetic reasoning (GSM8K), SAPE yields respectable results but is ultimately outperformed by unshared baselines. Future work will explore the deployment of the SAPE framework within Federated Learning (FL) environments. Given the critical communication bottlenecks inherent to FL, SAPE’s drastically minimized parameter footprint provides a significant structural advantage. Finally, extending the boundary-isolated sandwich topology beyond standard adapters to other PEFT methods such as developing a boundary-aware Shared LoRA architecture presents a compelling direction for future parameter-efficient research. Statements and Declarations Funding The authors did not receive support from any organization for the submitted work. Competing Interests The authors have no relevant financial or non-financial interests to disclose. Data Availability The datasets analyzed during the current study (GLUE, ConvAI2, CommonsenseQA, and GSM8K) are available in the public domain. References Biderman et al (2024) Biderman D, Portes J, Ortiz JJG, Paul M, Greengard P, Jennings C, King D, Havens S, Chiley V, Frankle J, et al (2024) Lora learns less and forgets less. arXiv preprint arXiv:240509673 Cobbe et al (2021) Cobbe K, Kosaraju V, Bavarian M, Chen M, Jun H, Kaiser L, Plappert M, Tworek J, Hilton J, Nakano R, et al (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:211014168 Dinan et al (2019) Dinan E, Logacheva V, Malykh V, Miller A, Shuster K, Urbanek J, Kiela D, Szlam A, Serban I, Lowe R, et al (2019) The second conversational intelligence challenge (convai2). In: The NeurIPS’18 Competition: From Machine Learning to Intelligent Conversations, Springer, p 187–208 Dong et al (2025) Dong Q, Tang Y, Jia H, Shen Y, Jia B, Huang W, Zhang L, Xie J, Lin S, Ji R (2025) Masa: Rethinking the representational bottleneck in lora with multi-a shared adaptation. arXiv preprint arXiv:251006005 Grattafiori et al (2024) Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Vaughan A, et al (2024) The llama 3 herd of models. arXiv preprint arXiv:240721783 Han et al (2024) Han Z, Gao C, Liu J, Zhang J, Zhang SQ (2024) Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:240314608 He et al (2020) He P, Liu X, Gao J, Chen W (2020) Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:200603654 He et al (2022) He S, Ding L, Dong D, Zhang J, Tao D (2022) Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. In: Findings of the Association for Computational Linguistics: EMNLP 2022, p 2184–2190 Houlsby et al (2019) Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, De Laroussilhe Q, Gesmundo A, Attariyan M, Gelly S (2019) Parameter-efficient transfer learning for nlp. In: International conference on machine learning, PMLR, p 2790–2799 Hu et al (2021) Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021) Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:210609685 Karimi Mahabadi et al (2021) Karimi Mahabadi R, Henderson J, Ruder S (2021) Compacter: Efficient low-rank hypercomplex adapter layers. Advances in neural information processing systems 34:1022–1035 Ko (2025) Ko S (2025) Lora is slower than you think. arXiv preprint arXiv:250708833 Lan et al (2019) Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:190911942 Lester et al (2021) Lester B, Al-Rfou R, Constant N (2021) The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 conference on empirical methods in natural language processing, p 3045–3059 Li and Liang (2021) Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers), p 4582–4597 Lin (2004) Lin CY (2004) Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out, p 74–81 Liu et al (2024) Liu SY, Wang CY, Yin H, Molchanov P, Wang YCF, Cheng KT, Chen MH (2024) Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:240209353 Liu et al (2019) Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190711692 Munkhdalai et al (2024) Munkhdalai T, Chen Y, Sim KC, Biadsy F, Sainath T, Mengibar PM (2024) Hierarchical recurrent adapters for efficient multi-task adaptation of large speech models. arXiv preprint arXiv:240319709 Papineni et al (2002) Papineni K, Roukos S, Ward T, Zhu WJ (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, p 311–318 Pfeiffer et al (2021) Pfeiffer J, Kamath A, Rücklé A, Cho K, Gurevych I (2021) Adapterfusion: Non-destructive task composition for transfer learning. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume, p 487–503 Pires et al (2023) Pires T, Lopes AV, Assogba Y, Setiawan H (2023) One wide feedforward is all you need. In: Proceedings of the Eighth Conference on Machine Translation, p 1031–1044 Raiaan et al (2024) Raiaan MAK, Mukta MSH, Fatema K, Fahad NM, Sakib S, Mim MMJ, Ahmad J, Ali ME, Azam S (2024) A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE access 12:26839–26874 Reid et al (2021) Reid M, Marrese-Taylor E, Matsuo Y (2021) Subformer: Exploring weight sharing for parameter efficiency in generative transformers. In: Findings of the Association for Computational Linguistics: EMNLP 2021, p 4081–4090 Renduchintala et al (2024) Renduchintala A, Konuk T, Kuchaiev O (2024) Tied-lora: Enhancing parameter efficiency of lora with weight tying. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p 8694–8705 Rolland and Abad (2024) Rolland T, Abad A (2024) Shared-adapters: A novel transformer-based parameter efficient transfer learning approach for children’s automatic speech recognition. In: Proc. Interspeech 2024, p 2370–2374 Song et al (2024) Song Y, Zhao J, Harris IG, Jyothi SA (2024) Sharelora: Parameter efficient and robust large language model fine-tuning via shared low-rank adaptation. arXiv preprint arXiv:240610785 Talmor et al (2019) Talmor A, Herzig J, Lourie N, Berant J (2019) Commonsenseqa: A question answering challenge targeting commonsense knowledge. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p 4149–4158 Tenney et al (2019) Tenney I, Das D, Pavlick E (2019) Bert rediscovers the classical nlp pipeline. In: Proceedings of the 57th annual meeting of the association for computational linguistics, p 4593–4601 Valipour et al (2023) Valipour M, Rezagholizadeh M, Kobyzev I, Ghodsi A (2023) Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, p 3274–3287 Vulić et al (2020) Vulić I, Ponti EM, Litschko R, Glavaš G, Korhonen A (2020) Probing pre-trained language models for lexical semantics. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p 7222–7240 Wang et al (2018) Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S (2018) Glue: A multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, p 353–355 Wang et al (2025) Wang L, Chen S, Jiang L, Pan S, Cai R, Yang S, Yang F (2025) Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review 58(8):227 Xu et al (2023) Xu L, Xie H, Qin SZJ, Tao X, Wang FL (2023) Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:231212148 Zaken et al (2022) Zaken EB, Goldberg Y, Ravfogel S (2022) Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p 1–9 Zeng et al (2023) Zeng G, Zhang P, Lu W (2023) One network, many masks: Towards more parameter-efficient transfer learning. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p 7564–7580 Zhang et al (2023) Zhang Q, Chen M, Bukharin A, Karampatziakis N, He P, Cheng Y, Chen W, Zhao T (2023) Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:230310512 Zhang et al (2020) Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y (2020) Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (ICLR) 6 Comprehensive Hyperparameter Specifications To ensure complete experimental transparency and reproducibility, the detailed hyperparameter configurations, optimization dynamics, and decoding setups utilized across all evaluated benchmarks and model backbones are provided below. 6.1 GLUE Benchmark Setup: DeBERTa V3-Base Table 7 outlines the task-specific optimization parameters for our primary GLUE experiments under both the D=16D=16 and D=64D=64 bottleneck constraints. Table 7: Detailed hyperparameter configurations for SAPE fine-tuning on DeBERTa V3-base across all GLUE benchmark tasks. Across all experiments, the random seed is fixed to 4242, the learning rate schedule follows a cosine annealing curve, and early stopping is applied with a patience budget of 22 epochs. Task Learning Rate Weight Decay Warmup Ratio Max Epochs Best Epoch Batch Size Configuration: SAPE Bottleneck Dimension D=16D=16 MNLI 5×10−45× 10^-4 0.01 0.10 5 5 32 QQP 5×10−45× 10^-4 0.01 0.10 5 5 32 QNLI 1.5×10−31.5× 10^-3 0.01 0.06 5 3 32 SST-2 1.5×10−41.5× 10^-4 0.01 0.10 15 7 32 CoLA 5×10−45× 10^-4 0.01 0.16 20 8 32 STS-B 1×10−31× 10^-3 0.01 0.10 15 7 32 MRPC 8×10−48× 10^-4 0.01 0.10 10 6 32 RTE 1×10−31× 10^-3 0.01 0.10 15 8 32 Configuration: SAPE Bottleneck Dimension D=64D=64 MNLI 4×10−44× 10^-4 0.01 0.10 7 6 32 QQP 5×10−45× 10^-4 0.01 0.10 5 5 32 QNLI 8×10−48× 10^-4 0.01 0.06 5 5 32 SST-2 1.8×10−41.8× 10^-4 0.015 0.10 12 7 32 CoLA 8×10−48× 10^-4 0.01 0.10 10 8 32 STS-B 5×10−45× 10^-4 0.01 0.05 20 5 32 MRPC 8×10−48× 10^-4 0.01 0.10 10 6 32 RTE 5×10−45× 10^-4 0.01 0.01 20 6 32 6.2 GLUE Benchmark Setup: RoBERTa-Large Table 8 details the experimental setup for the 355M parameter RoBERTa-large backbone. Table 8: Detailed hyperparameter configurations for SAPE (D=64,G=2D=64,G=2) fine-tuning on the RoBERTa-large backbone across all GLUE benchmark tasks. Across all experiments, the optimizer weight decay is held strictly constant at 0.010.01, the random seed is fixed to 4242, and learning rate scheduling follows a cosine annealing curve. Task Learning Rate Weight Decay Warmup Ratio Max Epochs Best Epoch Batch Size MNLI 4×10−44× 10^-4 0.01 0.10 7 7 32 QQP 5×10−45× 10^-4 0.01 0.10 5 5 32 QNLI 4×10−44× 10^-4 0.01 0.06 5 5 32 SST-2 3×10−43× 10^-4 0.01 0.10 12 2 32 CoLA 1×10−31× 10^-3 0.01 0.30 15 6 32 STS-B 8×10−48× 10^-4 0.01 0.20 15 10 32 MRPC 6×10−46× 10^-4 0.01 0.15 20 14 32 RTE 8×10−48× 10^-4 0.01 0.10 10 10 32 6.3 Generative & Reasoning Benchmarks: Llama-3.2-3B Table 9 documents the optimization hyperparameters and inference decoding parameters applied to our causal language model experiments. Table 9: Comprehensive hyperparameter, optimization, and decoding configurations for SAPE and baselines fine-tuning on the Llama-3.2-3B backbone across mathematical reasoning (GSM8K), commonsense reasoning (CSQA), and persona-grounded dialogue (ConvAI2). Asterisks (*) denote implicit framework training defaults. Hyperparameter GSM8K CommonsenseQA ConvAI2 (PersonaChat) Optimization & Training Setup Batch Size 2 2 2 Max Epochs 3 3 3 Learning Rate 5×10−55× 10^-5 * 5×10−55× 10^-5 * 5×10−55× 10^-5 * Weight Decay 0.0 * 0.0 * 0.0 * LR Scheduler Linear * Linear * Linear * Inference & Decoding Dynamics Primary Target Metric Exact Match Multiple Choice EM BLEU / ROUGE / METEOR / BERTScore Decoding Strategy Greedy (do_sample=False) Greedy (do_sample=False) Nucleus Sampling (top_p=0.9) Sampling Temperature – – 0.6 Repetition Penalty 1.0 1.0 1.1 Max Generated Tokens 256 2 64 Evaluation Monte Carlo Seeds 42\42\ 42\42\ 42,123,2026\42,123,2026\ (Mean ± SD) 6.4 Topology Ablation Study Setup Table 10: Hyperparameter and architectural configurations for the SAPE topology ablation study on DeBERTa V3-base. All four routing strategies (Regular, Top-Slice, Bottom-Slice, and Sandwich) were trained under identical optimization conditions to ensure a fair parameter budget. Asterisks (*) denote implicit framework defaults. Hyperparameter Value Batch Size (Train / Eval) 32 Learning Rate 1×10−41× 10^-4 Optimizer AdamW * LR Scheduler Linear * Weight Decay 0.0 * Warmup Ratio 0.0 * Max Training Epochs 30 Early Stopping Patience 3 Epochs Evaluation Strategy Epoch-wise