Paper deep dive
Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models
Dayan Pan, Jingyuan Wang, Xie Yu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines.
Tags
Links
- Source: https://arxiv.org/abs/2610.07848v1
- Canonical: https://arxiv.org/abs/2610.07848v1
Trouble viewing inline? Open PDF directly →
Full Text
74,091 characters extracted from source content.
Expand or collapse full text
Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of KoreaProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of KoreaDOI: 10.1145/3770855.3817911ISBN: 979-8-4007-2259-2/2026/08CCS: Information systems Data miningCCS: Computing methodologies Artificial intelligence Dayan Pan Affiliation: School of Computer Science and Engineering,, MOE Engineering Research Center of Advanced Computer Application Technology, Beihang University, Beijing, China email: dayan@buaa.edu.cn , Jingyuan Wang Note: Corresponding authors. Affiliation: School of Computer Science and Engineering,, School of Economics and Management, Beihang University, Beijing, China email: jywang@buaa.edu.cn and Xie Yu Affiliation: School of Computer Science and Engineering,, MOE Engineering Research Center of Advanced Computer Application Technology, Beihang University, Beijing, China email: yuxie_scse@buaa.edu.cn © c Abstract. Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines. The code is available for reproducibility11 1 https://github.com/Beihang-BIGSCity/DyPAM . Keywords: Large Language Models, Parameter-Efficient Fine-Tuning, Positional Attention, Attention Modulation †c-license: by Resource Availability: The source code of this paper has been made publicly available at https://doi.org/10.5281/zenodo.20507186. 1. Introduction Large language models (LLMs) have been widely adopted across a broad range of real-world applications, including question answering, code generation, and mathematical reasoning (Wang et al., 2023b; Wu et al., 2026; Cheng et al., 2025; Ma et al., 2025). Despite their strong general-purpose capabilities, different application scenarios and downstream tasks often impose substantially different requirements on model behavior. Effectively adapting an LLM to diverse task-specific requirements is essential in practical deployment (Zhao et al., 2023b; Li et al., 2022; Li et al., 2023; Yu et al., 2025b). Full-parameter fine-tuning for a specific task is effective but computationally and storage intensive, which limits its practicality in many settings. Parameter-efficient fine-tuning (PEFT) addresses this issue by updating only a small subset of parameters while keeping the backbone model frozen, achieving a favorable balance between efficiency and performance (Mangrulkar et al., 2022; Han et al., 2024; Pan et al., 2025; Li et al., 2025). Representative PEFT methods include LoRA(Hu et al., 2021), which reparameterizes weight updates into a low-rank subspace, IA3 (Liu et al., 2022), which modulates intermediate activations via lightweight scaling vectors, BOFT (Liu et al., 2023), which applies orthogonality-constrained multiplicative transformations with butterfly factorization, and Bone (Kang and Yin, 2024), which introduces block-wise affine updates with shared parameters. Despite their strong empirical performance, most existing PEFT methods apply additional parameters in a largely uniform manner across layers, attention heads, and feature dimensions (i.e., individual dimensions of the hidden representations), and therefore lack mechanisms to account for the structured and heterogeneous roles of different model components in a fine-grained manner. Figure 1 visualizes attention activation patterns of Llama-3.2-3B, focusing on the query (Q) representations used in self-attention. Similar heterogeneous patterns are also observed for key (K) representations, while such patterns are not significant for value (V) representations. Activations vary substantially across query dimensions, attention heads, and layers. As shown in Figure 1(a), attention exhibits distinct activation patterns across dimensions at different layers, indicating that different dimensions contribute differently to attention. Figure 1(b) further shows clear head-wise variation within the same layer, indicating that different attention heads exhibit distinct activation patterns. Figures 1(c) and (d) show that activation patterns also depend on the input tokens. Tokens with different semantic roles induce systematically different activation distributions, both when aggregated across dimensions and when examined at the level of individual dimensions for a fixed head and layer. Together, these visualizations demonstrate that attention in pretrained LLMs operates in a structured and heterogeneous manner across dimensions, heads, layers, and input tokens. Such heterogeneous activation patterns are consistent with the fact that different components of LLMs play distinct functional roles (Zhang et al., 2023). Recent studies have shown that Transformer modules are not functionally uniform, with different layers and components specializing in different aspects (Geva et al., 2021). For example, feed-forward networks have been shown to primarily store factual information (Meng et al., 2022), while attention mechanisms are more closely associated with contextual interaction (Clark et al., 2019b). Within the attention module, architectural design choices further introduce structured differences in how information is processed. In particular, positional encoding mechanisms in Transformer-based LLMs introduce structured differences in internal representations, resulting in non-uniform usage of attention dimensions across layers and heads (Barbero et al., 2024; Jin et al., 2025). Beyond structural differences across model components, attention behavior also varies with the input, where different tokens can induce different activation patterns depending on their semantic roles and contextual requirements (Wang et al., 2023a; Ji et al., 2026). Taken together, these findings suggest that LLMs exhibit intrinsic functional heterogeneity across components and inputs, and effective adaptation methods should account for such fine-grained structural differences. These observations suggest three requirements for structure-aware adaptation. First, adaptation should distinguish attention dimensions, since different dimensions exhibit different positional and functional behaviors. Second, it should account for head-wise and layer-wise differences, because the same dimension can play different roles across attention heads and network depths. Third, adaptation should remain input-conditioned, as different tokens and contexts can induce different attention activation patterns. This provides a direct way to adapt dependency modeling while keeping the original attention architecture unchanged. Motivated by these observations, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention. DyPAM addresses attention heterogeneity from two complementary perspectives. It combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation to adapt attention behavior. In contrast to prior PEFT methods that apply static adaptation shared across inputs, DyPAM models heterogeneity jointly at the token, dimension, head, and layer levels, and aligns the modulation with the RoPE-induced dimension pairing. Extensive experiments across multiple models and tasks demonstrate the effectiveness of this design. We summarize the main contributions of this work as follows: • We propose DyPAM, the first parameter-efficient fine-tuning method that explicitly adapts large language models by modulating positional attention in a fine-grained and structured manner. • DyPAM introduces input-conditioned, dimension-wise modulation, enabling different attention dimensions to be dynamically adjusted according to the input context. • DyPAM further incorporates head-wise and layer-wise structural modulation, allowing different attention heads and network layers to maintain distinct positional preferences. • Extensive experiments across multiple backbone models and downstream tasks demonstrate the effectiveness and robustness of DyPAM compared to existing PEFT methods. Figure 1. Activation heterogeneity in a pretrained Llama-3.2-3B model. The x-axis in (a), (b), and (d) indexes query dimensions of attention mechanism. Activation patterns vary across layers (a), heads (b), and input token types (c, d), indicating that attention operates heterogeneously across dimensions, heads, layers, and tokens. 2. Preliminaries We briefly review the attention mechanism, Rotary Position Embedding (RoPE), and parameter-efficient fine-tuning (PEFT), which provide the necessary background for describing our method. 2.1. Attention in Large Language Models Large language models are typically built upon the Transformer architecture (Vaswani et al., 2017), where self-attention is the core mechanism for modeling interactions among tokens. Given an input sequence, each layer ℓ of the model produces a sequence of hidden states (ℓ)=[1(ℓ),2(ℓ),…,T(ℓ)]H^( )=[h^( )_1,h^( )_2,…,h^( )_T], where t(ℓ)∈ℝdh^( )_t ^d denotes the hidden representation of the t-th token. In each attention layer, the hidden states are linearly projected into query, key, and value representations, (1) (ℓ)=(ℓ)Q(ℓ),(ℓ)=(ℓ)K(ℓ),(ℓ)=(ℓ)V(ℓ),Q^( )=H^( )W_Q^( ), ^( )=H^( )W_K^( ), ^( )=H^( )W_V^( ), where Q(ℓ),K(ℓ),V(ℓ)∈ℝd×dW_Q^( ),W_K^( ),W_V^( ) ^d× d are learned projection matrices. The projected representations are then reshaped into H attention heads. For each head h∈1,…,Hh∈\1,…,H\, we denote the per-head matrices as (ℓ,h),(ℓ,h),(ℓ,h)∈ℝT×dheadQ^( ,h),K^( ,h),V^( ,h) ^T× d_head, where dhead=d/Hd_head=d/H. For a given token position t, we denote by t(ℓ,h)∈ℝdheadq^( ,h)_t ^d_head and t(ℓ,h)∈ℝdheadk^( ,h)_t ^d_head the query and key vectors corresponding to the t-th token. We further denote by qt,j(ℓ,h)q^( ,h)_t,j the j-th feature dimension of t(ℓ,h)q^( ,h)_t. Self-attention is computed independently for each head. For head h at layer ℓ , the attention output is given by (2) Attn((ℓ,h),(ℓ,h),(ℓ,h))=softmax((ℓ,h)(ℓ,h)⊤dhead)(ℓ,h).Attn\! (Q^( ,h),K^( ,h),V^( ,h) )=softmax\! ( Q^( ,h)K^( ,h) d_head )V^( ,h). Notably, the attention operation itself is permutation-invariant and does not encode positional order (Press et al., 2021). As a result, positional information must be explicitly incorporated into the attention computation. Modern LLMs achieve this by applying position-dependent transformations to the query and key representations. Figure 2. Position-dependent responses across attention dimensions induced by RoPE. (a) Different dimensions respond differently to relative positional distances. (b) Heatmap of all dimensions, showing non-uniform positional sensitivity. Figure 3. The architecture of DyPAM framework. DyPAM applies input-conditioned, dimension-wise modulation together with head-wise and layer-wise structural biases to the query and key representations before RoPE, enabling fine-grained adaptation of positional attention within the PEFT paradigm. 2.2. Rotary Position Embedding Rotary Position Embedding (RoPE) (Su et al., 2024) incorporates positional information into attention by applying position-dependent transformations to the query and key representations. For each attention head, RoPE views the per-head vector as two halves of equal size: a “real” part and an “imaginary” part, each of dimension dhead/2d_head/2. RoPE then applies a 2D rotation to each index i by jointly rotating the paired components (zireal,ziimag) (z^real_i,z^imag_i ). As a consequence, the two halves share the same rotation at each index, and thus exhibit closely related positional behavior across corresponding dimensions. The rotations vary across dimensions, resulting in different positional behaviors. Figure 2 visualizes this effect, showing that different attention dimensions respond differently to relative positional distances in a non-uniform and multi-scale manner. As a result, different attention dimensions play different roles in encoding positional information, which motivates treating them differently when adapting positional attention. 2.3. Parameter-Efficient Fine-Tuning Parameter-efficient fine-tuning (PEFT) adapts LLMs by introducing lightweight transformations, while keeping the pretrained backbone fixed. We describe PEFT by applying a constrained transformation to an intermediate representation z in the model. Formally, the adapted representation can be written as (3) ′=(,),z =T (z; θ ), where (⋅,)T(·; θ) denotes a parameter-efficient transformation with a small number of trainable parameters θ. Many existing PEFT methods adopt an additive update, (4) ′=+Δ,Δ=(,),z =z+ , =A (z; θ ), where the adaptation module (⋅)A(·) is typically applied in a static and largely uniform manner across model components. While effective, this formulation typically applies the same adaptation mechanism uniformly across layers, attention heads, feature dimensions, and input tokens. However, attention behavior in modern LLMs is highly heterogeneous, particularly in how positional information is encoded. This motivates PEFT approaches that perform fine-grained and structure-aware adaptation. In this work, we follow the PEFT paradigm and apply multiplicative modulation to attention representations: (5) ′=()⊙,()=ℳ(,),z =s(x) , (x)=M (x; θ ), where ()s(x) is an input-conditioned modulation signal and ⊙ denotes element-wise multiplication. In DyPAM, this modulation is applied to the query and key representations in attention and indexed by the internal structure of attention, including layers, heads, tokens, and feature dimensions. Both formulations follow the same parameter-efficient adaptation paradigm. The key difference is that DyPAM performs input-conditioned, structure-aware modulation, enabling fine-grained adaptation of positional attention. 3. Method In this section, we introduce DyPAM, a PEFT method for LLMs that explicitly modulates positional information in attention. We first present an overview of the DyPAM framework and its core design principles, and then describe each component in detail. 3.1. Framework Overview In RoPE-based LLMs, attention exhibits heterogeneous behavior across feature dimensions, layers, heads, and input tokens. However, most existing PEFT methods rely on uniform and static adaptation mechanisms, without accounting for such heterogeneity. To address this limitation, we propose Dynamic Positional Attention Modulation (DyPAM), a parameter-efficient fine-tuning method that adapts how positional information contributes to attention. DyPAM operates directly on the query and key representations and jointly models input-conditioned, dimension-wise modulation together with head-wise and layer-wise structural modulation, enabling fine-grained and structured adaptation of positional attention. As illustrated in Figure 3, DyPAM introduces modulation into attention representations in a manner aligned with the internal structure of attention and conditioned on the input. The two modulation sources play different roles. The input-conditioned component captures token-dependent variation by generating modulation features from hidden states. The structural bias terms capture persistent preferences associated with heads and layers. Their combination allows DyPAM to adapt to each input while maintaining stable structural differences across the attention hierarchy. In the following, we present DyPAM by detailing how the modulation is constructed and how it is applied to attention representations, and summarize the overall procedure in Algorithm 1. 3.2. Query–Key Representations and Modulation Features DyPAM operates on the query and key representations used in self-attention. At each Transformer layer ℓ , these representations are derived from the token-level hidden states (ℓ)∈ℝB×T×dH^( ) ^B× T× d, where B is the batch size, T is the sequence length, and d is the model dimension. Following the standard attention formulation described in Section 2.1, the hidden states are linearly projected to obtain the query and key matrices (ℓ)Q^( ) and (ℓ)K^( ), which are subsequently reshaped into per-head representations (ℓ,h)Q^( ,h) and (ℓ,h)K^( ,h) as defined in Eq. (1). To enable input-conditioned adaptation of attention behavior, DyPAM derives modulation features directly from the same hidden states (ℓ)H^( ). Since the hidden states encode token-specific contextual information, the resulting modulation features are token-dependent and differ across inputs, providing the basis for input-conditioned modulation. Concretely, we apply a lightweight low-rank projection to the hidden states, yielding modulation features: (6) (ℓ)=(ℓ)(ℓ)(ℓ),(ℓ)∈ℝB×T×(H⋅de),M^( )=H^( )A^( )B^( ), ^( ) ^B× T×(H· d_e), where (ℓ)∈ℝd×rA^( ) ^d× r and (ℓ)∈ℝr×(H⋅de)B^( ) ^r×(H· d_e) are learnable matrices with rank r≪dr d, H is the number of attention heads, and ded_e denotes the feature dimension per head. The projected features are reshaped into H head-specific components, yielding modulation features t,h(ℓ)∈ℝdem^( )_t,h ^d_e for each token position t and attention head h. These features encode contextual information associated with each token and attention head, capturing how the current input is represented at different heads within the layer. They serve as an intermediate representation that bridges the token-level hidden states and the dimension-wise modulation subsequently applied to the query and key representations. 3.3. Input-Conditioned Dimension-Wise Modulation Given the modulation features constructed from the hidden states, DyPAM maps them to dimension-wise modulation values that are aligned with the query and key representations in attention. This mapping determines how the contribution of each attention dimension is modulated in an input-conditioned manner, allowing the model to dynamically adjust the influence of each dimension. By conditioning on both the token and its context, DyPAM enables fine-grained control over how positional information is utilized across different attention dimensions. For each layer ℓ , DyPAM introduces learnable dimension embedding matrices that project modulation features to the attention dimension space. Concretely, for the query and key representations, we use separate embedding matrices (7) Q(ℓ)∈ℝdhead2×de,K(ℓ)∈ℝdhead2×de,E^( )_Q d_head2× d_e, ^( )_K d_head2× d_e, where each row corresponds to one pair of attention dimensions. This design reflects the structure induced by RoPE, where each pair of dimensions shares the same positional rotation and therefore exhibits similar positional behavior. By assigning a single modulation value to each dimension pair, DyPAM reduces parameter overhead while respecting the RoPE-induced structure. This pair-wise sharing also avoids assigning unrelated modulation values to two coordinates that jointly represent the same rotated subspace. In this way, DyPAM changes the strength of each RoPE-aligned channel without breaking the internal pairing used by the positional transformation. This also supports grouped-query attention (GQA) (Ainslie et al., 2023), where multiple attention heads share key and value projections. In such cases, the key-side modulation is shared across heads that use the same key representation, while the query-side modulation remains head-specific. Given the modulation feature t,h(ℓ)∈ℝdem^( )_t,h ^d_e for token position t and attention head h, the dimension-wise modulation scores for queries and keys are computed as (8) t,h,Q(ℓ)=Q(ℓ)t,h(ℓ),t,h,K(ℓ)=K(ℓ)t,h(ℓ),g^( )_t,h,Q=E^( )_Qm^( )_t,h, ^( )_t,h,K=E^( )_Km^( )_t,h, where t,h,Q(ℓ),t,h,K(ℓ)∈ℝdhead2g^( )_t,h,Q,g^( )_t,h,K d_head2 denote the modulation scores for query and key dimension pairs, respectively. Each entry corresponds to one RoPE dimension pair, and we use i to index dimension pair i. At this stage, the modulation scores provide input-conditioned, dimension-wise adjustments for the query and key representations. Discussion. Input-conditioned dimension-wise modulation enables DyPAM to adapt the contribution of individual attention dimensions based on the input context. By aligning modulation with RoPE-induced dimension pairing, DyPAM selectively adjusts how positional information influences attention, while maintaining parameter efficiency. This mechanism provides fine-grained control over positional attention that is sensitive to both token-level context and the structured organization of attention dimensions. 3.4. Head-Wise and Layer-Wise Structural Modulation While input-conditioned dimension-wise modulation captures token-dependent variation, attention behavior also exhibits differences across attention heads and network layers. To model such structure-level heterogeneity, DyPAM introduces head-wise and layer-wise structural modulation that is independent of the input. For each layer ℓ , DyPAM maintains layer-wise bias vectors (9) Q(ℓ),K(ℓ)∈ℝdhead2, β^( )_Q, β^( )_K d_head2, which capture layer-specific preferences over attention dimension pairs for the query and key representations. In addition, for each attention head h at layer ℓ , DyPAM introduces head-wise bias vectors (10) h,Q(ℓ),h,K(ℓ)∈ℝdhead2, β^( )_h,Q, β^( )_h,K d_head2, allowing different heads within the same layer to maintain distinct structural biases. Here the subscripts Q and K indicate that the query and key representations use separate bias vectors of the same shape. These bias terms are added to the dimension-wise modulation scores. For queries and keys, the structurally augmented modulation scores are given by (11) ~t,h,Q(ℓ)=t,h,Q(ℓ)+h,Q(ℓ)+Q(ℓ), g^( )_t,h,Q=g^( )_t,h,Q+ β^( )_h,Q+ β^( )_Q, (12) ~t,h,K(ℓ)=t,h,K(ℓ)+h,K(ℓ)+K(ℓ), g^( )_t,h,K=g^( )_t,h,K+ β^( )_h,K+ β^( )_K, where t,h,Q(ℓ)g^( )_t,h,Q and t,h,K(ℓ)g^( )_t,h,K are the input-conditioned dimension-wise scores from Section 3.3. The bias terms are shared across token positions and encode structural preferences that persist across inputs. At this stage, the modulation scores integrate input-conditioned, dimension-wise adjustments with head-wise and layer-wise structural biases, capturing both token-dependent variation and persistent structural preferences in attention. These scores are then transformed into bounded modulation factors and applied to the query and key representations. Algorithm 1 DyPAM: Dynamic Positional Attention Modulation 1: Input sequence =(x1,…,xT)x=(x_1,…,x_T), pretrained RoPE-based LLM, DyPAM parameters 2: Model output distribution and training loss ℒL 3: Obtain token embeddings from x 4: for each Transformer layer ℓ=1,…,L =1,…,L do 5: Compute hidden states (ℓ)H^( ) 6: Project hidden states to query and key representations 7: (ℓ),(ℓ)Q^( ),K^( ) according to Eq. (1) 8: Reshape (ℓ),(ℓ)Q^( ),K^( ) into per-head representations 9: (ℓ,h),(ℓ,h)Q^( ,h),K^( ,h) 10: Construct modulation features from hidden states 11: t,h(ℓ)m^( )_t,h according to Eq. (6) 12: Compute input-conditioned, dimension-wise modulation scores 13: t,h,Q(ℓ),t,h,K(ℓ)g^( )_t,h,Q,g^( )_t,h,K according to Eq. (8) 14: Add head-wise and layer-wise structural biases 15: ~t,h,Q(ℓ),~t,h,K(ℓ) g^( )_t,h,Q, g^( )_t,h,K according to Eq. (11) and (12) 16: Normalize modulation scores to obtain modulation factors 17: st,h,i(ℓ)s^( )_t,h,i according to Eq. (13) 18: Apply modulation to query and key representations 19: ^(ℓ,h),^(ℓ,h) Q^( ,h), K^( ,h) according to Eq. (14) 20: Apply RoPE to modulated query and key representations 21: Compute attention outputs using modulated queries and keys 22: end for 23: Compute model outputs and training loss ℒL using Eq. (15) 3.5. Applying Modulation in Attention The combined modulation scores obtained from the previous steps encode both input-conditioned and structural adjustments over attention dimensions. We next apply a normalization step that maps these scores to bounded modulation factors, ensuring stable and controlled adaptation. Here, g~t,h,i(ℓ) g^( )_t,h,i denotes the modulation score for dimension pair i in the combined score vector for the query or key representation at layer ℓ , head h, and token position t. For brevity, we omit the explicit Q/KQ/K subscript when the formulation applies identically to both. For each layer ℓ , token position t, attention head h, and dimension pair i, the normalized modulation factor is computed as (13) st,h,i(ℓ)=1+α⋅(σ(g~t,h,i(ℓ))−0.5),s^( )_t,h,i=1+α· (σ( g^( )_t,h,i)-0.5 ), where σ(⋅)σ(·) is the sigmoid function and α controls the modulation strength. This normalization maps the modulation factors to a bounded interval [1−α2, 1+α2] [1- α2,\,1+ α2 ], centering them around the original scale and preventing deviation from the pretrained representations. The modulation factors are applied to the query and key representations before positional encoding. Let t,i(ℓ,h)∈ℝ2q^( ,h)_t,i ^2 and t,i(ℓ,h)∈ℝ2k^( ,h)_t,i ^2 denote the paired dimensions of the query and key vectors corresponding to dimension pair i. Both dimensions within each pair are modulated using the same factor: (14) ^t,i(ℓ,h)=st,h,i(ℓ)⋅t,i(ℓ,h),^t,i(ℓ,h)=st,h,i(ℓ)⋅t,i(ℓ,h). q^( ,h)_t,i=s^( )_t,h,i·q^( ,h)_t,i, k^( ,h)_t,i=s^( )_t,h,i·k^( ,h)_t,i. This operation corresponds to the multiplicative PEFT formulation introduced in Eq. (5), where the pretrained representation z corresponds to the query or key vectors, and the modulation signal ()s(x) is indexed by layer, head, token position, and dimension pair. Following the convention above, st,h,i(ℓ)s^( )_t,h,i is computed independently for the query and key sides from their respective scores ~t,h,Q(ℓ) g^( )_t,h,Q and ~t,h,K(ℓ) g^( )_t,h,K, so the query and key representations are modulated by their own factors. As the two coordinates within each pair share the same factor, the modulation signal ()s(x) takes a common value over each pair and thus spans the full dheadd_head dimensions of the query and key vectors. The modulated query and key representations are then passed through the RoPE mechanism and used in the standard attention computation. By applying modulation prior to RoPE, DyPAM aligns adaptation with the RoPE-induced positional structure. More broadly, DyPAM addresses attention heterogeneity by jointly modeling input-conditioned, dimension-wise modulation and head-wise, layer-wise structural modulation. This design enables attention dimensions to adapt to input context, while allowing different heads and layers to maintain distinct positional preferences. Rather than introducing uniform parameter updates, DyPAM performs targeted modulation aligned with how positional information is encoded and utilized in attention. As a result, DyPAM provides a structured and fine-grained mechanism for adapting positional attention within the PEFT paradigm. 3.6. Training Details DyPAM is trained end-to-end using the standard cross-entropy loss for language modeling. Given an input sequence =(x1,…,xT)x=(x_1,…,x_T) and the corresponding target sequence =(y1,…,yT)y=(y_1,…,y_T), the training loss is defined as (15) ℒ=−∑t=1Tlogp(yt∣x≤t),L=- _t=1^T p(y_t x_≤ t), where p(yt∣x≤t)p(y_t x_≤ t) denotes the model output distribution at position t. The model is parameterized by the pretrained backbone together with the DyPAM parameters. Algorithm 1 summarizes the overall forward computation and training procedure. Table 1. Comparison of DyPAM with PEFT baselines on mathematical reasoning benchmarks across three backbone models. Micro-avg and macro-avg denote micro- and macro-averaged performance. Best results are highlighted in bold, and second-best results are underlined. ∗ indicates statistically significant improvements over the best baseline (two-sided t-test, p<0.05p<0.05). Backbone LLM Method Param(%) MultiArith GSM8K AddSub AQuA SingleEq SVAMP MAWPS micro-avg(%)↑ macro-avg(%)↑ LLaMA 3.2 3B LoRA 1.12 71.50 33.21 78.48 22.44 81.50 54.10 76.47 54.96 59.67 AdaLoRA 2.22 75.67 36.32 80.51 22.83 87.80 55.60 78.57 57.90 62.47 OFT 0.73 87.17 40.18 85.82 24.02 86.42 61.50 84.03 62.75 67.02 Bone 1.14 87.50 39.73 85.57 23.62 86.61 63.70 81.93 63.03 66.95 IA3 0.02 58.33 27.37 68.61 20.47 72.83 47.90 58.82 46.89 50.62 LN-Tuning 0.01 58.00 26.38 66.58 21.26 74.80 44.90 60.08 46.01 50.29 FourierFT 0.73 78.67 33.21 82.03 20.47 85.43 54.30 77.31 56.72 61.63 SHiRA 1.12 82.50 38.82 84.81 24.02 87.99 56.90 81.93 60.59 65.28 RoSA 0.54 84.33 37.91 82.78 22.83 87.01 52.50 78.99 59.02 63.77 DyPAM (ours) 0.92 88.50 39.88 86.33 25.20 88.78 63.00 84.03 63.58∗ 67.96* Qwen3 8B LoRA 0.79 97.67 74.91 89.87 35.83 90.55 84.70 89.08 82.04 80.37 AdaLoRA 1.57 95.17 73.01 90.63 37.01 92.32 84.80 91.60 81.62 80.65 OFT 0.51 95.67 73.46 90.38 33.07 94.09 84.90 91.60 81.80 80.45 Bone 0.81 98.00 72.25 91.65 33.46 93.90 83.80 90.34 81.55 80.49 IA3 0.02 92.50 72.18 84.81 35.04 86.61 80.90 86.55 78.49 76.94 LN-Tuning 0.00 91.67 68.69 85.32 39.76 87.40 78.00 85.71 77.01 76.65 FourierFT 0.37 94.50 70.05 87.34 31.50 86.81 82.70 81.09 78.28 76.28 SHiRA 0.79 94.83 75.36 90.13 37.01 93.90 85.70 90.34 82.57 81.04 RoSA 0.36 97.83 74.07 90.38 35.43 94.49 84.80 92.02 82.48 81.29 DyPAM (ours) 0.61 99.17 76.72 91.90 40.94 95.28 85.50 92.86 84.24∗ 83.20* Gemma 3 4B LoRA 1.33 86.00 51.25 72.41 25.98 75.59 62.20 75.21 63.26 64.09 AdaLoRA 2.62 82.67 51.86 66.33 31.50 73.82 62.30 73.95 62.49 63.20 OFT 0.75 85.83 54.28 72.91 32.28 75.59 63.80 76.47 65.02 65.88 Bone 1.41 86.17 45.87 71.39 30.31 72.64 55.10 73.11 59.69 62.08 IA3 0.03 42.67 38.89 40.51 27.17 40.75 37.20 37.39 38.62 37.80 LN-Tuning 0.01 32.67 30.63 45.06 23.62 56.69 40.80 37.82 37.64 38.18 FourierFT 1.10 60.83 31.24 65.32 28.35 66.73 46.30 65.97 47.89 52.10 SHiRA 1.33 72.67 42.08 73.16 31.50 76.57 61.30 75.63 58.92 61.84 RoSA 0.40 34.50 38.51 66.84 31.10 63.19 43.70 62.18 45.53 48.58 DyPAM (ours) 0.62 86.33 55.19 73.42 32.68 76.18 62.70 76.89 65.28∗ 66.20* 4. Experiments This section evaluates DyPAM across tasks, models, and experimental settings. Our experimental design is organized around a set of research questions that assess overall performance, scalability across model sizes, component contributions, and learned positional modulation behavior. Specifically, we investigate the following research questions: • RQ1: Does DyPAM outperform existing PEFT methods with comparable numbers of trainable parameters? • RQ2: Is DyPAM effective across different backbone model sizes? • RQ3: Which components of DyPAM contribute most to its performance? • RQ4: How sensitive is DyPAM to its hyperparameters? • RQ5: What positional modulation patterns does DyPAM learn? We first introduce the experimental setup, followed by a systematic evaluation of DyPAM with respect to each research question. 4.1. Experimental Setup Datasets. We evaluate DyPAM on mathematical and commonsense reasoning tasks with distinct reasoning patterns and positional sensitivity, and train on two datasets drawn from multiple existing benchmarks (Hu et al., 2023), emphasizing multi-step arithmetic and general commonsense reasoning, respectively. Mathematical reasoning performance is evaluated on GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), MAWPS (Koncel-Kedziorski et al., 2016), AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2016), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021), while commonsense reasoning is evaluated on BoolQ (Clark et al., 2019a), PIQA (Bisk et al., 2020), Social IQA (Sap et al., 2019), ARC-Easy and ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019), and WinoGrande (Sakaguchi et al., 2020). Across benchmarks, inputs are natural-language questions and outputs are either final numeric answers or discrete choices, and we report accuracy as the evaluation metric with further details provided in the Appendix. Backbone Models. Experiments are conducted on three widely used RoPE-based LLM families with different architectures and design choices, namely LLaMA 3.2, Qwen3, and Gemma 3 (Grattafiori et al., 2024; Team, 2025b; Team, 2025a), to evaluate the robustness and generality of DyPAM. Baseline Methods. We compare DyPAM with a diverse set of PEFT methods that adopt different adaptation strategies. Low-rank adaptation methods include LoRA (Hu et al., 2021) and AdaLoRA (Zhang et al., 2023), which parameterize weight updates in a low-dimensional subspace, with AdaLoRA further adjusting rank allocation across layers according to their relative importance. We also consider structured weight reparameterization approaches, including OFT (Qiu et al., 2023) and Bone (Kang and Yin, 2024), where OFT constrains updates to orthogonal transformations and Bone employs block-wise affine parameterization to capture structured correlations within weight matrices. In addition, we include lightweight modulation-based methods such as IA3 (Liu et al., 2022) and LNTuning (Zhao et al., 2023a), which adapt the model by rescaling internal activations or normalization parameters with minimal trainable parameters. We further compare against FourierFT (Gao et al., 2024), which performs adaptation in the frequency domain by learning a compact set of spectral coefficients for weight updates. Finally, we include SHiRA (Bhardwaj et al., 2024), which applies sparse high-rank adapters to update only a small subset of backbone weights, and RoSA (Pan et al., 2026), which performs RoPE-aware selective adaptation over attention dimensions and layers. Implementation Details. All experiments are conducted using DeepSpeed (Rasley et al., 2020) with bfloat16 precision on NVIDIA RTX 4090 GPUs. For DyPAM, we use a modulation embedding dimension de=64d_e=64, a low-rank projection rank r=128r=128, and a modulation strength α=0.3α=0.3. For baseline methods, except for extremely low-parameter approaches, we match the number of trainable parameters to a comparable scale. Additional implementation details are provided in the appendix, and all code and data are released to ease reproducibility22 2 https://github.com/Beihang-BIGSCity/DyPAM . Table 2. Macro-averaged accuracy on mathematical reasoning benchmarks across Qwen3 model scales. The comparison includes representative strong PEFT baselines from the main experiments. Baseline Qwen 3 0.6B Qwen 3 1.7B Qwen 3 4B Qwen 3 8B LoRA 64.06 66.64 75.60 80.37 OFT 65.96 67.81 75.54 80.45 SHiRA 63.95 64.65 73.33 81.04 RoSA 63.99 67.38 77.92 81.29 DyPAM (ours) 66.13 69.24 78.24 83.20 Table 3. Comparison of DyPAM with PEFT baselines on commonsense reasoning benchmarks across three backbone models. Micro-avg and macro-avg denote micro- and macro-averaged performance. Best results are highlighted in bold, and second-best results are underlined. ∗ indicates statistically significant improvements over the best baseline (two-sided t-test, p<0.05p<0.05). Backbone LLM Method Param(%) BoolQ PIQA SocialIQA ARC-C ARC-E OpenBookQA HellaSwag WinoGrande Micro Avg Macro Avg LLaMA 3.2 3B LoRA 1.12 63.61 79.71 66.94 69.45 84.05 67.00 73.94 55.56 71.94 70.03 AdaLoRA 2.22 63.52 78.94 67.09 68.94 85.14 70.20 78.11 56.35 73.95 71.04 OFT 0.73 65.63 79.54 70.37 70.39 85.06 71.80 83.15 66.38 77.52 74.04 Bone 1.14 64.56 75.68 69.34 64.42 79.76 70.20 75.92 65.75 72.77 70.70 IA3 0.02 62.32 77.09 59.67 57.94 77.10 57.40 50.48 52.25 58.66 61.78 LN Tuning 0.01 62.51 76.99 59.52 59.81 76.52 59.00 52.02 52.17 59.42 62.32 FourierFT 0.73 62.14 79.49 61.98 61.86 80.93 62.40 73.21 49.09 69.75 66.39 SHiRA 1.12 65.23 79.65 69.14 71.16 84.97 71.20 83.18 65.67 77.35 73.78 RoSA 0.54 64.53 79.65 69.86 69.28 84.43 70.80 83.12 63.54 77.00 73.15 DyPAM (ours) 0.92 65.93 79.76 70.88 70.39 85.19 71.80 83.71 65.35 77.83* 74.13* Qwen3 8B LoRA 0.79 70.49 86.34 77.18 90.19 96.51 87.60 89.50 72.85 85.19 83.83 AdaLoRA 1.57 70.73 86.51 76.71 90.36 96.55 87.20 88.92 72.38 84.91 83.67 OFT 0.51 69.97 86.83 76.56 89.93 96.97 88.00 89.17 76.48 85.20 84.24 Bone 0.81 69.02 85.31 75.64 88.91 95.58 87.60 89.30 76.56 84.71 83.49 IA3 0.02 69.51 86.34 76.71 90.27 96.09 84.40 85.12 66.77 82.59 81.90 LN Tuning 0.00 69.33 86.40 75.95 90.27 96.00 83.00 83.86 65.43 81.82 81.28 FourierFT 0.37 69.54 84.49 73.13 85.92 95.29 77.80 80.48 62.27 79.34 78.62 SHiRA 0.79 70.83 87.05 77.33 90.36 96.97 88.20 89.56 75.77 85.57 84.51 RoSA 0.36 68.96 86.94 75.33 89.85 96.38 88.20 89.43 76.16 84.99 83.91 DyPAM (ours) 0.61 70.89 87.11 77.33 90.53 97.05 88.80 89.53 76.80 85.66* 84.75* Gemma 3 4B LoRA 1.33 65.72 79.71 69.40 74.49 87.08 71.00 74.53 55.01 73.37 72.12 AdaLoRA 2.62 66.09 79.49 68.73 76.54 89.02 74.00 73.20 58.09 73.30 73.14 OFT 0.75 65.69 81.99 74.51 76.71 88.47 78.00 83.86 65.27 79.17 76.81 Bone 1.41 64.68 75.35 71.24 70.39 82.83 75.80 78.33 64.48 74.70 72.89 IA3 0.02 62.17 71.49 57.32 57.51 73.19 55.20 44.89 57.85 55.30 59.95 LN Tuning 0.00 62.60 66.70 49.85 49.91 63.59 45.20 47.29 60.46 53.90 55.70 FourierFT 0.37 63.94 75.57 67.14 67.32 76.05 57.80 71.81 59.35 69.76 67.37 SHiRA 0.79 65.57 82.25 74.53 76.19 89.71 78.20 83.19 64.48 78.94 76.77 RoSA 0.40 63.70 79.54 67.40 72.27 86.66 69.40 48.53 47.51 60.62 66.88 DyPAM (ours) 0.62 66.21 82.59 74.82 77.13 89.23 79.20 84.09 65.35 79.56* 77.33* 4.2. Overall Performance (RQ1) Across both mathematical and commonsense reasoning benchmarks, DyPAM consistently outperforms existing PEFT baselines under comparable parameters, providing a clear positive answer to RQ1. The improvements are observed across all evaluated backbone models and task categories, indicating that the effectiveness of DyPAM generalizes across different models and datasets. Specifically, on mathematical reasoning benchmarks as shown in Table 1, DyPAM demonstrates strong and stable gains across heterogeneous tasks that require multi-step computation, and numerical reasoning. The improvements are reflected in both micro- and macro-averaged metrics, suggesting that DyPAM enhances overall reasoning capability rather than favoring a small subset of benchmarks. In contrast, existing PEFT baselines exhibit more uneven behavior. Low-rank methods such as LoRA and AdaLoRA tend to yield limited gains, particularly on more challenging datasets. Frequency-domain approaches such as FourierFT show moderate performance but lack robustness across datasets. Methods with extremely small parameter budgets, including IA3 and LN-Tuning, generally underperform due to their limited adaptation capacity. Structured or orthogonal methods such as OFT and Bone often perform well on specific tasks but struggle to maintain consistent improvements across the full benchmark suite, whereas sparse adaptation methods like SHiRA, which selectively activate a subset of parameters during fine-tuning, achieve strong performance on multiple tasks, highlighting the potential of structured parameter updates. RoSA also benefits from structured adaptation by introducing rotational subspace updates, leading to competitive results on several benchmarks. However, RoSA applies a static RoPE-aware selection that is shared across all inputs, whereas DyPAM modulates query and key dimension pairs in an input-conditioned, token-dependent manner coordinated with head-wise and layer-wise structure. This fine-grained and input-conditioned modulation, rather than the use of positional structure alone, accounts for the consistent margin of DyPAM over RoSA across backbones. Compared with these baselines, DyPAM achieves more balanced improvements across tasks and models, indicating superior robustness and generalization. A similar pattern is observed on commonsense reasoning tasks as shown in Table 3. DyPAM yields balanced improvements across diverse datasets spanning factual verification and social commonsense. The consistent gains in macro-averaged performance indicate improved robustness at the task level, while the improvements in micro-averaged accuracy reflect stronger overall performance. Overall, these results show that DyPAM provides a more reliable and general-purpose adaptation mechanism than prior PEFT approaches. By delivering consistent gains across tasks, domains, and backbone models, DyPAM effectively addresses RQ1 and demonstrates clear advantages over existing PEFT baselines. 4.3. Scalability Analysis (RQ2) Table 2 evaluates the scalability of DyPAM across Qwen3 models from 0.6B to 8B using macro-averaged accuracy on mathematical reasoning benchmarks. As model size increases, DyPAM consistently achieves better performance than the strongest PEFT baselines at each scale. Moreover, the performance difference between DyPAM and baselines becomes larger as the backbone grows, indicating that DyPAM continues to benefit from increased capacity. Overall, these results demonstrate that DyPAM maintains strong scalability across model sizes and remains effective when applied to both small and large backbone models, directly addressing RQ2. 4.4. Ablation and Sensitivity Analysis (RQ3, 4) We further analyze DyPAM through ablation studies and parameter sensitivity experiments, aiming to understand the contribution of individual components and the robustness of key hyperparameters. The ablation results show that each core component of DyPAM plays a complementary role in overall performance. Removing any single component consistently leads to performance degradation, indicating that the gains of DyPAM arise from their joint design rather than isolated architectural choices. In addition, the sensitivity analysis on the modulation strength α demonstrates the effectiveness of this design choice. Proper modulation significantly improves performance compared to weak or overly strong modulation. Overall, these analyses confirm that the performance improvements of DyPAM are structurally grounded and robust. 4.5. Analysis of Modulation Patterns (RQ5) We analyze the learned modulation from two perspectives: the structural bias terms and the effective modulation ranges. Figure 5 visualizes the learned layer-wise query bias over attention dimensions on LLaMA-3.2-3B under commonsense and mathematical reasoning settings. Rather than exhibiting uniform shifts, the bias values vary across both layers and dimensions, indicating that DyPAM learns heterogeneous, dimension-specific adjustments. This structured non-uniformity suggests that different attention dimensions develop distinct positional preferences at different depths, aligning with the intuition that positional information is utilized differently across layers. In addition, faint horizontal band patterns can be observed, hinting that certain layers may exhibit consistent preferences over specific subsets of attention dimensions. Figure 6 summarizes the effective modulation range of query and key representations under the same two settings. The scale factors remain centered around 1, while the range varies across layers and training data. This restrained behavior preserves the pretrained attention structure while allowing flexibility. Overall, these visualizations demonstrate that DyPAM learns structured and stable positional modulation patterns. By combining dimension-wise modulation with restrained head-wise and layer-wise scaling, DyPAM adapts positional attention in a targeted manner, shaping how token dependencies are formed and thereby helping explain its consistent performance gains across models and tasks. Figure 4. Ablation and hyperparameter sensitivity of DyPAM. The ablation removes individual modulation components, while the sensitivity study varies the modulation strength α. Figure 5. Learned layer-wise query bias over attention dimensions on LLaMA-3.2-3B under commonsense and mathematical reasoning settings. The heatmaps show that the structural bias varies across layers and dimensions rather than following a uniform pattern. Figure 6. Layer-wise modulation range of query and key representations on LLaMA-3.2-3B under commonsense and mathematical reasoning settings. The mean scale remains close to 1, while the effective range changes across layers and training data. 5. Related Work 5.1. Parameter-Efficient Fine-Tuning Recent advancements in PEFT focus on reducing the number of trainable parameters while maintaining performance. Low-rank approaches such as LoRA (Hu et al., 2021) and AdaLoRA (Zhang et al., 2023) adapt pre-trained weights through additive low-rank updates, while scaling-based methods including IA3 (Liu et al., 2022) and LNTuning (Zhao et al., 2023a) employ lightweight gating to modulate activations under strict parameter constraints. Structured PEFT approaches, such as OFT (Qiu et al., 2023) and Bone (Kang and Yin, 2024), further constrain the update space by enforcing geometric properties, whereas spectral-domain methods like FourierFT (Gao et al., 2024) perform adaptation via frequency-based transformations of model representations. In addition, RoSA (Pan et al., 2026) is motivated by the structure of RoPE and proposes RoPE-aware selective adaptation by emphasizing low-frequency attention components. Compared with RoSA, which applies static RoPE-aware selection, DyPAM generates input-conditioned modulation over query and key dimension pairs. In contrast to these approaches, our method explicitly models fine-grained, input-conditioned modulation within attention, enabling structure-aware adaptation. 5.2. Structured Modeling in Transformer Transformer attention has been recognized for its structural heterogeneity, with different attention heads specializing in various semantic or syntactic aspects of the input (Raganato et al., 2020). Recent works have analyzed head-level specialization and layer-wise differences, demonstrating that attention heads should not be treated uniformly (Voita et al., 2019; Zhang et al., 2022). Beyond language modeling, attention- and frequency-aware designs have also been widely adopted in spatiotemporal modeling (Jiang et al., 2023; Ma et al., 2026) and financial prediction (Wang et al., 2019; Wang et al., 2021), further underscoring the value of structure-aware mechanisms. Positional encodings, particularly in RoPE and its variants, are not just external biases but integral to the attention computation, providing a way to model relative positional relationships more effectively (Gu et al., 2025). Newer methods, such as ComRoPE (Yu et al., 2025a), introduce dynamic and input-conditioned positional encodings, enhancing the flexibility of position representations in transformers. RoSA (Pan et al., 2026) is also motivated by the structure of RoPE and explores selective adaptation of attention components guided by frequency characteristics. While these studies highlight the structured and positional nature of attention, they largely rely on static or coarse-grained mechanisms, motivating the need for fine-grained and input-conditioned modulation within attention. 6. Conclusion In this work, we introduce DyPAM, a PEFT method that adapts LLMs by modulating positional attention in a fine-grained and structured manner. Motivated by heterogeneous attention behavior across dimensions, heads, layers, and input tokens, DyPAM operates directly on the query and key representations and aligns adaptation with the RoPE-induced positional structure. It jointly models input-conditioned, dimension-wise and head-wise, layer-wise structural modulation, providing a principled approach to adapting positional attention within the PEFT paradigm. By reshaping how positional information influences attention, DyPAM adjusts the way dependencies are formed across tokens, which is consistent with its gains on tasks that rely on multi-step and relational reasoning. Extensive experiments across mathematical and commonsense reasoning benchmarks demonstrate that DyPAM consistently outperforms strong PEFT baselines. Future work will further optimize the runtime implementation and extend DyPAM across diverse architectural mechanisms. Acknowledgements. This work is supported by the National Natural Science Foundation of China (No. 725B2004, 72242101, 72222022), and the Science and Technology Development Fund Macau SAR (0052/2023/RIA1). References Ainslie et al. (2023) J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai Gqa: training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245. Cited by: §3.3. Barbero et al. (2024) F. Barbero, A. Vitvitskyi, C. Perivolaropoulos, R. Pascanu, and P. Veličković Round and round we go! what makes rotary positional encodings useful?. arXiv preprint arXiv:2410.06205. Cited by: §1. Bhardwaj et al. (2024) K. Bhardwaj, N. P. Pandey, S. Priyadarshi, V. Ganapathy, S. Kadambi, R. Esteves, S. Borse, P. Whatmough, R. Garrepalli, M. Van Baalen, H. Teague, and M. Nagel Sparse high rank adapters. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §4.1. Bisk et al. (2020) Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, p. 7432–7439. Cited by: Appendix C, §4.1. Cheng et al. (2025) J. Cheng, J. Wang, Y. Zhang, J. Ji, Y. Zhu, Z. Zhang, and X. Zhao Poi-enhancer: an llm-based semantic enhancement framework for poi representation learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 39, p. 11509–11517. Cited by: §1. Clark et al. (2019a) C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova Boolq: exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. Cited by: Appendix C, §4.1. Clark et al. (2019b) K. Clark, U. Khandelwal, O. Levy, and C. D. Manning What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. Cited by: §1. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: Appendix C, §4.1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: Appendix C, §4.1. Gao et al. (2024) Z. Gao, Q. Wang, A. Chen, Z. Liu, B. Wu, L. Chen, and J. Li Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.03003. Cited by: §4.1, §5.1. Geva et al. (2021) M. Geva, R. Schuster, J. Berant, and O. Levy Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 5484–5495. Cited by: §1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.1. Gu et al. (2025) Z. Gu, H. Zhang, R. Chen, Y. Hu, and H. Zhang Unpacking positional encoding in transformers: a spectral analysis of content-position coupling. arXiv preprint arXiv:2505.13027. Cited by: §5.2. Han et al. (2024) Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang Parameter-efficient fine-tuning for large models: a comprehensive survey. arXiv preprint arXiv:2403.14608. Cited by: §1. Hosseini et al. (2014) M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), p. 523–533. Cited by: Appendix C, §4.1. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §1, §4.1, §5.1. Hu et al. (2023) Z. Hu, L. Wang, Y. Lan, W. Xu, E. Lim, L. Bing, X. Xu, S. Poria, and R. K. Lee Llm-adapters: an adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933. Cited by: Appendix C, §4.1. Ji et al. (2026) J. Ji, J. Wang, Y. Mou, C. Long, and J. Wu How to break it down for building it up? theory-guided graph decomposition learning for spatiotemporal traffic prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1. Jiang et al. (2023) J. Jiang, C. Han, W. X. Zhao, and J. Wang Pdformer: propagation delay-aware dynamic long-range transformer for traffic flow prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 4365–4373. Cited by: §5.2. Jin et al. (2025) M. Jin, K. Mei, W. Xu, M. Sun, R. Tang, M. Du, Z. Liu, and Y. Zhang Massive values in self-attention modules are the key to contextual knowledge understanding. arXiv preprint arXiv:2502.01563. Cited by: §1. Kang and Yin (2024) J. Kang and Q. Yin Balancing lora performance and efficiency with simple shard sharing. arXiv preprint arXiv:2409.15371. Cited by: §1, §4.1, §5.1. Koncel-Kedziorski et al. (2015) R. Koncel-Kedziorski, H. Hajishirzi, A. Sabharwal, O. Etzioni, and S. D. Ang Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics 3, p. 585–597. Cited by: Appendix C, §4.1. Koncel-Kedziorski et al. (2016) R. Koncel-Kedziorski, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi MAWPS: a math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, p. 1152–1157. Cited by: Appendix C, §4.1. Li et al. (2022) J. Li, T. Tang, Z. Gong, L. Yang, Z. Yu, Z. Chen, J. Wang, W. X. Zhao, and J. Wen Eliteplm: an empirical study on general language ability evaluation of pretrained language models. In Proceedings of the 2022 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, p. 3519–3539. Cited by: §1. Li et al. (2023) J. Li, T. Tang, W. X. Zhao, J. Wang, J. Nie, and J. Wen The web can be your oyster for improving language models. In Findings of the Association for Computational Linguistics: ACL 2023, p. 728–746. Cited by: §1. Li et al. (2025) Z. Li, L. Hongjun, J. Wang, and K. Tang Approximation to smooth functions by low-rank swish networks. In Forty-second International Conference on Machine Learning, Cited by: §1. Ling et al. (2017) W. Ling, D. Yogatama, C. Dyer, and P. Blunsom Program induction by rationale generation: learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. Cited by: Appendix C, §4.1. Liu et al. (2022) H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35, p. 1950–1965. Cited by: §1, §4.1, §5.1. Liu et al. (2023) W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243. Cited by: §1. Ma et al. (2026) J. Ma J. Wang et al. Hierarchical frequency-decomposition graph neural networks for road network representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 15510–15518. Cited by: §5.2. Ma et al. (2025) J. Ma, J. Wang, W. X. Zhao, G. Liu, and X. Wen Spatio-temporal data enhanced vision-language model for traffic scene understanding. IEEE Transactions on Intelligent Transportation Systems. Cited by: §1. Mangrulkar et al. (2022) S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan PEFT: state-of-the-art parameter-efficient fine-tuning methods. Note: https://github.com/huggingface/peft Cited by: §B.1, §1. Meng et al. (2022) K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229. Cited by: §1. Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. Cited by: Appendix C, §4.1. Pan et al. (2025) D. Pan, Z. Fu, J. Wang, X. Han, Y. Zhu, and X. Zhao Contextual attention modulation: towards efficient multi-task adaptation in large language models. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 2273–2283. Cited by: §1. Pan et al. (2026) D. Pan, J. Wang, Y. Zhou, J. Cheng, P. Jia, and X. Zhao RoSA: enhancing parameter-efficient fine-tuning via rope-aware selective adaptation in large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 15600–15608. Cited by: §4.1, §5.1, §5.2. Patel et al. (2021) A. Patel, S. Bhattamishra, and N. Goyal Are nlp models really able to solve simple math word problems?. arXiv preprint arXiv:2103.07191. Cited by: Appendix C, §4.1. Press et al. (2021) O. Press, N. A. Smith, and M. Lewis Train short, test long: attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. Cited by: §2.1. Qiu et al. (2023) Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems 36, p. 79320–79362. Cited by: §4.1, §5.1. Raganato et al. (2020) A. Raganato, Y. Scherrer, and J. Tiedemann Fixed encoder self-attention patterns in transformer-based machine translation. arXiv preprint arXiv:2002.10260. Cited by: §5.2. Rasley et al. (2020) J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He Deepspeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, p. 3505–3506. Cited by: Appendix B, §4.1. Roy and Roth (2016) S. Roy and D. Roth Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413. Cited by: Appendix C, §4.1. Sakaguchi et al. (2020) K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi Winogrande: an adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, p. 8732–8740. Cited by: Appendix C, §4.1. Sap et al. (2019) M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi Socialiqa: commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728. Cited by: Appendix C, §4.1. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §2.2. Team (2025a) G. Team Gemma 3. External Links: Link Cited by: §4.1. Team (2025b) Q. Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4.1. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin Attention is all you need. Advances in neural information processing systems 30. Cited by: §2.1. Voita et al. (2019) E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. Cited by: §5.2. Wang et al. (2021) J. Wang, K. Tang, K. Feng, X. Lin, W. Lv, K. Chen, and F. Wang Impact of temperature and relative humidity on the transmission of covid-19: a modelling study in china and the united states. BMJ open 11 (2), p. e043863. Cited by: §5.2. Wang et al. (2023a) J. Wang, C. Yang, X. Jiang, and J. Wu WHEN: a wavelet-dtw hybrid attention network for heterogeneous time series analysis. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, p. 2361–2373. Cited by: §1. Wang et al. (2019) J. Wang, Y. Zhang, K. Tang, J. Wu, and Z. Xiong Alphastock: a buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, p. 1900–1908. Cited by: §5.2. Wang et al. (2023b) X. Wang, X. Tang, X. Zhao, J. Wang, and J. Wen Rethinking the evaluation for conversational recommendation in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 10052–10065. Cited by: §1. Wu et al. (2026) Z. Wu, J. Wang, Z. Cheng, Y. Zhou, W. Wang, J. Pu, C. Li, and C. Ma Icad-llm: one-for-all anomaly detection via in-context learning with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 15986–15994. Cited by: §1. Yu et al. (2025a) H. Yu, T. Jiang, S. Jia, S. Yan, S. Liu, H. Qian, G. Li, S. Dong, and C. Yuan ComRoPE: scalable and robust rotary position embedding parameterized by trainable commuting angle matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4508–4517. Cited by: §5.2. Yu et al. (2025b) X. Yu, J. Wang, Y. Yang, Q. Huang, and K. Qu BIGCity: a universal spatiotemporal model for unified trajectory and traffic state data analysis. In 2025 IEEE 41st International Conference on Data Engineering (ICDE), p. 4455–4469. Cited by: §1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi Hellaswag: can a machine really finish your sentence?. arXiv preprint arXiv:1905.07830. Cited by: Appendix C, §4.1. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao Adalora: adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512. Cited by: §1, §4.1, §5.1. Zhang et al. (2022) X. Zhang, Y. Shen, Z. Huang, J. Zhou, W. Rong, and Z. Xiong Mixture of attention heads: selecting attention heads per token. arXiv preprint arXiv:2210.05144. Cited by: §5.2. Zhao et al. (2023a) B. Zhao, H. Tu, C. Wei, J. Mei, and C. Xie Tuning layernorm in attention: towards efficient multi-modal llm finetuning. arXiv preprint arXiv:2312.11420. Cited by: §4.1, §5.1. Zhao et al. (2023b) W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223. Cited by: §1. Appendix A Attention Activation Patterns A.1. Effect of Positional Encoding Schemes To verify that the heterogeneous attention behavior motivating DyPAM is tied to the positional encoding mechanism, we compare query activation patterns across models using different positional encodings, including RoPE, ALiBi, and learned positional embeddings. Specifically, we visualize query representations from LLaMA-3.2-3B, BLOOM-560M, and OPT-350M under a unified scheme covering all attention dimensions (Figure 7). RoPE-based models exhibit structured, dimension-dependent activation patterns, whereas models using ALiBi or learned positional embeddings display substantially more homogeneous behavior across dimensions, supporting our focus on RoPE-induced positional structure. Figure 7. Cross-model comparison of query activation patterns across layers and attention dimensions. RoPE-based LLaMA-3.2-3B shows structured, dimension-dependent patterns, while ALiBi-based BLOOM-560M and OPT-350M with learned embeddings are more homogeneous. A.2. Token-Dependent Activation Patterns We further examine token-dependent variation in attention activations. We visualize query activations conditioned on different token types and input contexts, extending the analysis of Fig. 1(c,d) to multiple models (Figure 8). In the RoPE-based model, different tokens induce systematically different activation patterns, while such token-dependent structure is less pronounced under alternative positional encoding schemes. These observations motivate the input-conditioned, token-level component of DyPAM. Figure 8. Token-type activation patterns across layers and attention dimensions for LLaMA-3.2-3B, BLOOM-560M, and OPT-350M. Token-dependent variation is most pronounced in the RoPE-based model. Appendix B Experimental and Implementation Details All experiments reported in the main paper use a primary random seed of 2333, with additional runs using seeds 1000, 2000, 3000, and 4000 to assess statistical significance and reproducibility. Training employs mixed-precision (BF16) and DeepSpeed (Rasley et al., 2020) with ZeRO Stage 1. A randomly selected validation set of 300 samples from the training data is used for checkpoint selection, and the checkpoint with the lowest validation loss is used for testing. The default hyperparameter settings are summarized below: • Optimizer: AdamW; learning rate 1×10−31× 10^-3 with cosine schedule; warmup ratio 0.05; weight decay 0. • Batch size: 2 (gradient accumulation steps of 2); max sequence length: 2048. • Modulation feature dimension de=64d_e=64; low-rank projection rank r=128r=128; modulation strength α=0.3α=0.3. B.1. Software and Environment The software environment uses PyTorch 2.7.0, DeepSpeed 0.18.4, NumPy 2.2.6, PEFT 0.18.1, Transformers 4.51.0, Tokenizers 0.21.4, and CUDA 12.8. Experiments run on Ubuntu 24.04 LTS with an Intel Xeon Gold 6330 CPU, an NVIDIA GeForce RTX 4090 GPU, and 512GB RAM. For baseline methods, except for extremely low-parameter approaches, we match the number of trainable parameters to a comparable scale; all other hyperparameters follow their default configurations from the PEFT library (Mangrulkar et al., 2022) or the original implementations. Detailed implementation and datasets can be found in our codebase33 3 https://github.com/Beihang-BIGSCity/DyPAM . Appendix C Dataset Details We use two instruction-tuning datasets from the LLM-Adapters benchmark suite (Hu et al., 2023): Math10K for math word problems with step-by-step solutions and Commonsense15K for commonsense reasoning questions normalized into a consistent instruction format. Statistics are summarized in Table 4. Table 4. Statistics of the training datasets for commonsense and mathematical reasoning tasks. Dataset Samples Total Tokens Avg. Tokens/Sample Math10K 9,919 2,273,016 229.16 Commonsense15K 15,119 1,778,782 117.65 We evaluate on well-established commonsense and mathematical reasoning benchmarks; detailed statistics are in Table 5 (Mathematical) and Table 6 (Commonsense). a) Mathematical Reasoning: MultiArith (Roy and Roth, 2016) (multi-step arithmetic), GSM8K (Cobbe et al., 2021) (grade-school multi-step reasoning), AddSub (Hosseini et al., 2014) (addition/subtraction), AQuA (Ling et al., 2017) (algebraic problems with rationales), SingleEq (Koncel-Kedziorski et al., 2015) (equation-tree parsing), SVAMP (Patel et al., 2021) (structural variations), and MAWPS (Koncel-Kedziorski et al., 2016) (unified word-problem benchmark). Table 5. Statistics of Mathematical Reasoning Test Datasets. Dataset Samples Answer Type MultiArith 600 Numeric GSM8K 1,319 Numeric AddSub 395 Numeric AQuA 254 Multiple Choice (A–E) SingleEq 508 Numeric SVAMP 1,000 Numeric MAWPS 238 Numeric b) Commonsense Reasoning: BoolQ (Clark et al., 2019a) (yes/no QA), PIQA (Bisk et al., 2020) (physical commonsense), SIQA (Sap et al., 2019) (social commonsense), ARC-Challenge/ARC-Easy (Clark et al., 2018) (science QA), OBQA (Mihaylov et al., 2018) (open-book science QA), HellaSwag (Zellers et al., 2019) (adversarial NLI), and WinoGrande (Sakaguchi et al., 2020) (pronoun resolution). Table 6. Statistics of Commonsense Reasoning Test Datasets. Dataset Samples Answer Format BoolQ 3,270 true / false PIQA 1,838 solution1 / solution2 SIQA 1,954 answer1 / answer2 / answer3 ARC-Challenge 1,172 answer1 / answer2 / answer3 / answer4 ARC-Easy 2,376 answer1 / answer2 / answer3 / answer4 OBQA 500 answer1 / answer2 / answer3 / answer4 HellaSwag 10,042 ending1 / ending2 / ending3 / ending4 WinoGrande 1,267 option1 / option2 Appendix D Evaluation Protocol and Metrics All model outputs are generated with greedy decoding (do_sample= False, max_new_tokens=256) via the generate() API in Hugging Face Transformers, using the unified instruction template below. <s>Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: instruction ### Response: Predictions are extracted with task-specific regular expressions: for mathematical reasoning, numerical answers (absolute tolerance 10−310^-3) or alphabetic choices (A–E) for AQuA; for commonsense reasoning, exact-match answers (true/false and option labels) against ground-truth labels. All extraction and accuracy computation scripts are provided in our codebase. Table 7. Inference overhead of DyPAM compared with the base model and representative PEFT baselines on LLaMA-3.2-3B. Latency is reported per token. The per-token value is lower at batch size 3232 because more tokens are processed in parallel. Method BS=1 (ms/tok) BS=32 (ms/tok) Mem (GB) FLOPs overhead Base 14.49 0.49 6.00 — LoRA (unmerged) 27.64 1.52 6.06 73.0M (+1.13%) RoSA 22.40 1.45 6.03 34.9M (+0.54%) DyPAM (ours) 24.98 1.49 6.06 66.1M (+1.03%) Appendix E Inference Overhead We report the inference overhead of DyPAM against the base model and representative PEFT baselines in Table 7, measured on LLaMA-3.2-3B. DyPAM adds 1.03%1.03\% FLOPs overhead, lower than unmerged LoRA (1.13%1.13\%), as it avoids low-rank updates to all linear layers and the per-token modulation introduces only a small cost. DyPAM is about 10%10\% faster than unmerged LoRA at batch size 11 and comparable at batch size 3232, with negligible memory overhead. Unlike LoRA, DyPAM cannot be merged into the backbone weights due to its input-conditioned design, but the overhead remains controlled and can be further reduced with optimizations such as KV-cache reuse and kernel fusion.