Paper deep dive

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

Haris Riaz, Hyungji Kim, Mihai Surdeanu

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 105

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/9/2026, 1:13:51 AM

Summary

The paper introduces Syntax-informed Positional Embeddings (SiPE), a method that injects syntactic priors derived from dependency parses into Transformer positional embeddings. SiPE learns lightweight syntactic tags (hexatags) during pretraining and injects them into absolute, relative, and rotary positional encoding families. For encoders, injection is best done via input embeddings; for autoregressive decoders with relative positional encodings, multiplicative coupling with the relative-position attention term yields the best results. SiPE improves SyntaxGym scores by up to 10.3% and GLUE scores by up to 8.2%, while reducing perplexity, establishing a Pareto frontier between syntactic supervision and inference cost by conditioning on a single parse.

Entities (13)

SiPE · method · 98%GLUE · benchmark · 95%SyntaxGym · benchmark · 95%Transformer · model-architecture · 95%Positional Embeddings · component · 92%Hexatagging · technique · 90%RoBERTa · model-architecture · 90%Transformer-XL · model-architecture · 90%Hexatagger · tool · 88%Absolute Positional Embeddings · pe-family · 85%Relative Positional Embeddings · pe-family · 85%Rotary Positional Embeddings · pe-family · 85%Pareto frontier · concept · 80%

Relation Signals (11)

SiPE → improves → GLUE

confidence 95% · SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2%

SiPE → improves → SyntaxGym

confidence 95% · We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3%

SiPE → uses → Hexatagging

confidence 92% · We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses... Hexatagging... assigns each token two types of discrete syntactic labels

SiPE → composeswith → Relative Positional Embeddings

confidence 90% · SiPE composes with all three dominant positional encoding families—... relative... and rotary

SiPE → composeswith → Absolute Positional Embeddings

confidence 90% · SiPE composes with all three dominant positional encoding families—absolute... without modifying self-attention

SiPE → composeswith → Rotary Positional Embeddings

confidence 90% · SiPE composes with all three dominant positional encoding families—absolute, relative, and rotary

Hexatagger → generates → Hexatags

confidence 90% · Hexatagger... assigns each token two types of discrete syntactic labels... terminal tags... and non-terminal tags

SiPE → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed \textbf{P}ositional \textbf{E}mbeddings (\textbf{SiPE}), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate \emph{where} and \emph{how} the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to $10.3\%$ while simultaneously reducing perplexity by $9.0\%$ over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to $8.2\%$ over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

104,002 characters extracted from source content.

Expand or collapse full text

Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers Haris Riaz Hyungji Kim Mihai Surdeanu Department of Computer Science University of Arizona Tucson, AZ, USA Code Abstract Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate where and how the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly—while for encoders it is best added directly to the input embeddings, composing with each encoder’s native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3%10.3\% while simultaneously reducing perplexity by 9.0%9.0\% over a base model with no syntactic supervision—a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2%8.2\% over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost. Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers Haris Riaz and Hyungji Kim and Mihai Surdeanu Department of Computer Science University of Arizona Tucson, AZ, USA Code Figure 1: Our Syntax-Informed Positional Embeddings (SiPE) move the Pareto frontier between syntactic supervision and inference cost. Each point plots a method’s SyntaxGym score (Table 2) against the number of parse trees it evaluates per sentence at inference. Joint syntactic LMs (TG Sartran et al. (2022), Pushdown LM Murty et al. (2023), PLM Qian et al. (2021)) recover p(x)p(x) by marginalizing over ≈ 300 candidate parses; parser-free methods (TreeReg Nandi et al. (2025), Tree-Planted Transformers Yoshida et al. (2024)) inject syntax only during training and discard the parser at runtime, making inference cheap but sacrificing some syntactic knowledge. SiPE conditions on a single parse, retaining most of the benefit of full marginalization at 1/3001/300th of its inference cost. Figure 2: Left: From bottom to top, the figure shows the correspondence between dependency arcs, binary head tree (BHT), and hexatags (left-shifted) for the sentence “she watches funny videos.” Right: Our prior-injection method for absolute positional embeddings (input-pathway injection). From bottom to top, each token (at its first subword; subsection A.1) selects one row from each of four lookup tables: the token embedding (|V|×D|V|× D; omitted from the figure for brevity), the non-terminal tag embedding νpNE^N_ _p (5×D5× D), the terminal tag embedding τpTE^T_ _p (2×D2× D), and the positional embedding (N×DN× D). The selected rows are combined via an injection operation ⊕ (addition by default; we ablate concatenation, weighted-addition and addition to skip connection in Appendix E) into the syntax-informed input px_p (Equation 2). The transformer maps these inputs to contextual states ih_i, and at each masked position i∈ℳi three prediction heads recover the token xix_i, terminal tag τi _i, and non-terminal tag νi _i, yielding ℒ=ℒMLM+ℒT+ℒNTL=L_MLM+L_T+L_NT; one masked position is shown, and head colors match the embedding rows they supervise. For decoders, next-token prediction replaces MLM: the token head at position i predicts token i+1i+1 at every position, while the tag heads are applied only at first-subword positions and predict the tags of the next tagged position (subsection A.1). 1 Introduction The transformer architecture Vaswani et al. (2017) forms the foundation of modern LLMs. While the impact of LLMs is substantial, several linguistic simplifications were introduced in the underlying architecture to improve scalability. One key choice was the use of positional embeddings (PE) to encode each token’s location within a sequence. While this approach preserved computational efficiency and scalability, it reduced positional information to a simple ordering mechanism—sufficient to indicate where tokens appear, but insufficient to capture the syntactic relations that form the scaffolding for downstream semantics. That is, without explicitly modeling syntactic compositionality, semantic frames Baker et al. (1998); Kingsbury and Palmer (2002) are more likely to be extracted incorrectly, increasing the risk of misinterpretation for LLM-based agents. Consider, for example, this hypothetical agent request: “Please move this large file to another folder.” To execute the requested action correctly, the move predicate must be associated with its arguments: the object to be moved (file) and the destination (folder). Absolute positional encodings Devlin et al. (2019); Liu et al. (2019), provide no signal to link this predicate to its arguments. Relative positional encodings, e.g., DeBERTa He et al. (2021) and Transformer-XL Dai et al. (2019), weaken this signal. For instance, the relative distance between move and folder is large (six words), even though a direct oblique syntactic dependency connects the verb to its modifier De Marneffe et al. (2021). Partially due to this shallower representation, language models (LMs) still exhibit weak compositional generalization Guo et al. (2020). Prior work injects hierarchical inductive bias by constraining self-attention using constituency or dependency structure, often via quadratic token-token interactions and hard-coded attention masks Sartran et al. (2022); Zhao et al. (2024); Murty et al. (2023); Qian et al. (2021); Xie et al. (2021). This is a complex process that requires considerable changes in the transformer architecture and which increases the attention mechanism’s computational overhead (e.g., Pushdown LMs Murty et al. (2023) maintain a recursively updated stack over the sequence that reshapes the attention pattern at every step). The main contributions of our paper are: (1) We introduce Syntax-informed Positional Embeddings (SiPE) that augment a model’s positional pathway with a lightweight prior derived from linearized dependency trees—coarse directional indicators relating each token to its syntactic governor (§3), learned during pre-training via an auxiliary indicator-prediction objective alongside the LM loss. (2) SiPE composes with all three dominant positional encoding families—absolute Liu et al. (2019), relative He et al. (2021); Dai et al. (2019), and rotary Warner et al. (2024)—without modifying self-attention or any other Transformer component. (3) For encoders, SiPE improves BLiMP Warstadt et al. (2023) syntactic generalization across all three families (up to +2.3%+2.3\% relative for ModernBERT), with gains compounding under continued pre-training on BLLIP-LG (+3%+3\% for DeBERTa-v3 on out-of-domain evaluation), indicating the prior generalizes beyond its pre-training distribution (Table 3). The same models also improve on GLUE Wang et al. (2019) at both small and large scale (tables Table 4, 5 and 6). (4) For autoregressive decoders like Transformer-XL, injecting through the positional pathway alone is the strongest recipe: it achieves the best SyntaxGym Hu et al. (2020) score of any configuration of our method (80.6080.60, +10.3+10.3 relative improvement over the no-syntax baseline) while cutting perplexity by 9.0%9.0\% (relative), and drives a significant +8.2%+8.2\% relative improvement on GLUE for the same model (Table 6). (5) As shown in Figure 1, prior syntactic LMs sit at two extremes: parsing at inference (expensive marginalization over many parse trees) or only during training (discarding syntax at runtime, with weaker syntactic generalization). SiPE moves the Pareto frontier between these by conditioning on a single parse at inference. The injection of syntactic information adds no asymptotic cost over self-attention,111Dependency tags are predicted in parallel by a lightweight DeBERTa-v3-base classifier whose overhead on top of the encoder’s is linear in input length; see Appendix B. Implemented in-house with a multitask-learning encoder, two per-token tag heads: https://clulab.org/processors/ and unlike most prior syntactic LMs which jointly model syntax and language, SiPE leaves downstream usage unchanged—models architecturally remain compatible with continual pre-training, supervised fine-tuning and reinforcement learning. Empirically we outperform parser-free approaches—Tree-Planted Transformers Yoshida et al. (2024) and TreeReg Nandi et al. (2025)—on both BLLIP-LG perplexity and SyntaxGym (Table 2). 2 Related Work, Background, and Notations Prior work shows that pretraining on formal languages or injecting tree-based signals can improve syntactic generalization and data efficiency Hu et al. (2025). However, most methods encode syntax by constraining self-attention via pairwise token interactions and hard-coded attention masks, adding parameters and training complexity Xie et al. (2021); Sartran et al. (2022); Qian et al. (2021); Murty et al. (2023). We instead sidestep this by encoding dependency structure as two lightweight prior vectors derived from Hexatagging, and apply these priors to the positional embedding pathway. Modern Transformers encode position through three dominant schemes: absolute embeddings added to the input residual Liu et al. (2019), relative embeddings indexed by offset i−ji-j inside attention He et al. (2023); Dai et al. (2019), and rotary embeddings that rotate queries and keys by position-dependent angles Su et al. (2023). We show that our method seamlessly composes with all three PE types (subsection 3.1). Hexatagging. In this work, we leverage Hexatagger Amini et al. (2023), a “parsing-as-tagging” dependency parser that assigns each token two types of discrete syntactic labels (analogous to position or subword IDs): (a) the position of its syntactic governor (which can further be decomposed into two subtypes) and (b) the dependency relation. This formulation can be mapped to simple embedding lookup tables that can be linearly combined with token embeddings without modifying self-attention. Hexatagger maps a projective dependency tree to a binary head tree (BHT, a special form of constituency tree) by binarizing the structure and labeling each internal node with L/RL/R to indicate whether the span head lies in the left or right subtree, then linearizes the BHT via an in-order (left-corner) traversal. Each visited node yields a tag encoding its attachment direction (left vs. right child), and for non-terminals, the head-direction (L vs. R). This yields a fixed projective tag inventory with terminal tags → [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [origin=c]45.0$→$ and → [rgb]0,0,0 [named]pgfstrokecolorrgb0,0,0 @color@gray@stroke0 @color@gray@fill0 [origin=c]135.0$→$, plus non-terminal tags ⇒l [origin=c]45.0$ $ [rgb]0.80078125,0.3984375,0.46484375 [named]pgfstrokecolorrgb0.80078125,0.3984375,0.46484375 -1.5pt l, ⇒r [origin=c]45.0$ $ [rgb]0.80078125,0.3984375,0.46484375 [named]pgfstrokecolorrgb0.80078125,0.3984375,0.46484375 -1.5pt r, ⇒l [origin=c]135.0$ $ [rgb]0.80078125,0.3984375,0.46484375 [named]pgfstrokecolorrgb0.80078125,0.3984375,0.46484375 -1.5pt l, and ⇒r [origin=c]135.0$ $ [rgb]0.80078125,0.3984375,0.46484375 [named]pgfstrokecolorrgb0.80078125,0.3984375,0.46484375 -1.5pt r (Figure 2, left). This avoids unbounded (length-dependent) tag sets whose cardinality grows with the input length; instead each token’s tag can be predicted independently with a linear classifier. In our implementation, we further append a dedicated EOS non-terminal (via a left shift of between-token non-terminals) to obtain one non-terminal label per token.1 In this work, we omit dependency relation labels, finding that terminal/non-terminal hexatags alone provide sufficient syntactic inductive bias to improve downstream language modeling performance. 3 Approach Both training and inference operate on a hexatagged sequence: the input is first tagged by our parser (subsection A.1). Encoders train with the standard MLM objective; decoder training is standard next-token prediction, as each token carries its own tags and truncating the sequence at any prefix leaves the retained tags intact. 3.1 Deriving a Positional Syntactic Bias An inductive bias can enter the PE layer in three ways (Figure 2): (a) entangled with the token embedding, applied to the input before any positional or attention information is introduced; (b) composed with the default PE scheme under some operation; or (c) disentangled, a separate additive term in the attention score that modulates each head. Linearizing the binary head tree into discrete tags (the hexatagging procedure of section 2) yields two embedding tables, one per tag type, composable with the PE mechanism under any of these operations. At position p they contribute: mp⋅(τpT+νpN)m_p· ( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p ) (1) where T∈ℝ||×d [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T ^|T|× d and N∈ℝ||×d [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N ^|N|× d are learned tables (||=2|T|=2 terminals, ||=5|N|=5 non-terminals), and mp∈0,1m_p∈\0,1\ is the first-subword mask, equal to 1 only at each word’s first subword so the hexatag is injected once per word (subsection A.1). Tables are randomly initialized and trained jointly with the LM objective. Throughout the paper, every occurrence of τpT+νpN [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p is implicitly multiplied by mpm_p; we omit the mask from later equations for readability, so continuation subwords contribute no tag term anywhere. 3.2 RoBERTa: Absolute Positional Embeddings We begin with the simplest case. Adopting strategy (a), hexatag priors are injected directly into the input embedding, which combines a learned absolute position vector pp_p with the token, segment, and tag terms: p=ptok+p+pseg+mp⋅(τpT+νpN)x_p=e^tok_p+p_p+e^seg_p+m_p· ( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p ) (2) The tag term sits at the same level as pp_p, so the model learns to attend to a richer composite input without architectural conflict222However, the input embedding is not the only place in the transformer where we can inject these priors; see appendix E and figure 9.. 3.3 Transformer-XL: Syntactic Priors in an Autoregressive Decoder with Relative PE To test prior injection on autoregressive decoders, we use Transformer-XL Dai et al. (2019) (hereon referred to as TXL), which is one of the earliest “LLM-like” architectures carrying a sinusoidal relative-position encoding directly in the attention score and is a common syntactic-LM baseline Sartran et al. (2022); Zhao et al. (2024). Following prior work, we disable cross-segment caching (so relative positions apply only within the current window) and replace the adaptive softmax with a tied linear projection, making TXL a purely causal decoder. For a query at position i and key at position j in head n of layer ℓ , the attention score splits into a content term and a position term: Ai,j,n(ℓ)=⟨i,n(ℓ)+n,j,n(ℓ)⟩⏟i,j,n(ℓ)(content)+⟨i,n(ℓ)+n,Ri−j,n⟩⏟i,j,n(ℓ)(position) A^( )_i,j,n\;=\; ^( )_i,n+u_n,\;k^( )_j,n _ ^( )_i,j,n\ (content)\;+\; ^( )_i,n+v_n,\;W_R\, [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_i-j,n _ [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD^( )_i,j,n\ (position) (3) where i−j,n [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_i-j,n is the sinusoidal vector for offset i−ji\!-\!j, RW_R projects it into head n’s dheadd_head-dimensional subspace, and n,nu_n,v_n are the learned content and position bias vectors. Notably, this content-conditioned relative-position term remains competitive at frontier scale: the concurrent 975B-parameter open-weights model Inkling Thinking Machines Lab (2026) abandons RoPE for a relative positional bias of the same query–offset-embedding form, citing better quality and length extrapolation. Because [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD depends only on i−ji\!-\!j, it carries no information about which tokens occupy positions i and j or their syntactic role. We supply that missing signal from the hexatag embeddings τjT+νjNE^T_ _j+E^N_ _j and study where in the score it should enter, across five injection sites treated in turn below. 3.3.1 Input-Side Injection (ADD) The simplest approach adds the tag embeddings to the token embedding, so the prior enters the input residual at layer 0 and reaches the score only indirectly, through the content projections Q,KW_Q,W_K that build the query and key: p(0)=ptok+mp(τpT+νpN)h^(0)_p\;=\;e^tok_p\;+\;m_p\,( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p) (4) with mp∈0,1m_p\!∈\!\0,1\ masking tag-free positions. The relative-position encoding itself is untouched; the prior entangles with lexical content rather than with position. 3.3.2 Position-Pathway Injection (P-SiPE) The approach which we find works best is to leave the input residual clean and route the tag straight into the position term. Using a layer-specific projection WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ), we map the tag at key j into head n’s subspace and take its inner product with the same effective query i,n(ℓ)+nq^( )_i,n+v_n that appears in the position term [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD (Eq. 3): ci,j,n(ℓ)=1dhead⟨i,n(ℓ)+n,[WE(ℓ)(τjT+νjN)]n⟩ [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c^( )_i,j,n\;=\; 1 d_head\, \,q^( )_i,n+v_n,\; [ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( )( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j) ]_n\, (5) where [⋅]n[\,·\,]_n selects head n’s dheadd_head-dimensional block and where τjT [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j and νjN [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j are the terminal and nonterminal hexatag embeddings at key j. Eqs. 3 and 5 are deliberately parallel: the one effective query i,n(ℓ)+nq^( )_i,n+v_n poses two questions — “how well do I align with this offset?” ( [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD) and “how well do I align with this tag?” (c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c). The second is therefore a syntactic counterpart of the first, on the same per-head scale. We couple it to the position term multiplicatively, leaving the content term AC untouched: A~i,j,n(ℓ)=i,j,n(ℓ)+(1+ci,j,n(ℓ))⋅i,j,n(ℓ) $ A^( )_i,j,n\;=\;AC^( )_i,j,n\;+\; (1+ [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c^( )_i,j,n )· [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD^( )_i,j,n$ (6) In our syntactic evaluation, we observe the multiplicative form (Eq. 6) to outperform all other approaches (refer to experiments in table 2 and ablations in table 7, appendix A.2): its correction term c⋅ [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c· [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD gates by offset alignment, applying the syntactic adjustment in proportion to how strongly the query already attends to that offset. Where the query-offset alignment is strong (||| [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD| large), the tag exerts a large pull; where it is near zero, the tag exerts almost none, so syntax modulates an existing positional preference rather than acting in isolation.333We initialize WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ) small, so c≈0 [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c\!≈\!0 and A~≈A A\!≈\!A at initialization: training starts from the vanilla Transformer-XL score and learns the syntactic correction into it. The offset embedding i−j,n [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_i-j,n itself is never modified. 3.3.3 Fully-Disentangled Injection A third option completely disentangles the tag from both pathways. Rather than multiplying c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c with [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD, we add it as an independent third component of the logit, alongside the content and position terms and on the standard attention scale: A~i,j,n(ℓ)=i,j,n(ℓ)+i,j,n(ℓ)+ci,j,n(ℓ). A^( )_i,j,n\;=\;AC^( )_i,j,n\;+\; [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD^( )_i,j,n\;+\; [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c^( )_i,j,n. (7) Now ci,j,n(ℓ) [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c^( )_i,j,n depends on the query and the tag but not on i,j,n(ℓ) [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD^( )_i,j,n, so its size is fixed once the tag is chosen: a query at i adds the same syntactic bias toward key j whether or not it already attends to the offset i−ji\!-\!j. This is the opposite of the multiplicative coupling (Eq. 6), where that same bias is scaled by [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD, and therefore drops to nearly zero when the query does not attend to the offset (i.e., when ≈0 [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD\!≈\!0). In our experiments, we find that this ungated form is weaker (refer to Table 2 and ablations in Table 8, appendix A.3) than the multiplicatively entangled bias, which indicates that the bias from SiPE helps most when it augments an attention preference the position pathway has already established, rather than when it acts on its own. 3.3.4 Injecting Syntactic Bias in Self-Attention Motivated by nearly all prior work which imposes syntactic constraints on attention weights as either a hard or soft bias, Strubell et al. (2018); Murty et al. (2023); Zhao et al. (2024); Omote et al. (2019); Xie et al. (2021); Yoshida et al. (2024), we propose our hexatag-derived bias applied to attention logits and the attention output, leaving the input residual, AC, and the positional pathway [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD untouched (p(0)=ptokh^(0)_p=e^tok_p). The mechanism follows Shaw et al. (2018), who augment attention with learned key-side and value-side vectors. We project the summed hexatag embedding at key j two ways: through Ktag,(ℓ)W^tag,( )_K to form a key bias that is added to the attention score A~i,j,n(ℓ) A^( )_i,j,n, and through Vtag,(ℓ)W^tag,( )_V to form a value bias that is added to the attended output i,n(ℓ)z^( )_i,n: A~i,j,n(ℓ)=Ai,j,n(ℓ)+1dhead⟨i,n(ℓ),[Ktag,(ℓ)(τjT+νjN)]n⟩ A^( )_i,j,n\;=\;A^( )_i,j,n\;+\; 1 d_head ^( )_i,n,\;[W^tag,( )_K( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j)]_n (8) i,n(ℓ)=∑jαi,j,n(ℓ)(j,n(ℓ)+[Vtag,(ℓ)(τjT+νjN)]n) ^( )_i,n\;=\; _jα^( )_i,j,n (v^( )_j,n\;+\;[W^tag,( )_V( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j)]_n ) (9) where αi,j,n(ℓ)α^( )_i,j,n are the softmax weights, and [⋅]n[\,·\,]_n selects multi-head attention head n’s block. Two things differ from the multiplicative alignment score ci,j,n(ℓ) [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c^( )_i,j,n of Eq. 5. First, that score was built from the effective query i,n(ℓ)+nq^( )_i,n+v_n and scaled the position term [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD, whereas this key bias uses the plain query i,n(ℓ)q^( )_i,n and is added directly to the logit. Second, it is keyed on the tag at the single key position j rather than on a relative pair (i,j)(i,j), which keeps the added cost linear (rather than quadratic) in sequence length. The projections may be learned per layer (the (ℓ)( ) superscript) or tied to a single pair shared across all layers; in our experiments we find the former (per-layer variant) outperforms the latter (shared variant) on syntactic generalization tasks, both on its own and when combined with the input-side injection of Eq. 4 (Appendix A.4). 3.3.5 Injecting Syntax into Position and Attention Simultaneously The final variant pairs our two strongest single-site injections: SiPE on the positional pathway (the multiplicative coupling, Eq. 6) and the per-layer attention-side biases. Combining the two, however, hurts: the joint variant scores below either injection strategy used on its own (Table 2, Appendix A.4). Injecting the same hexatag signal as both a positional and attentional bias is thus redundant rather than complementary; for Transformer-XL, the prior is best supplied once, through the positional pathway. Having established where and how in the score the prior should be combined, we next ask which layer it should enter from: at least for the decoder model in our experiments, syntax is best infused from layer 1 onward (section 5). 3.4 DeBERTa-v3: Injecting the Prior under Disentangled Relative Attention DeBERTa-v3 He et al. (2023) carries no input-level position embedding; positional information enters only at the attention score, through disentangled relative attention. For a query at i and key at j, the score splits into a content term and two relative-position terms: Sij=QicKjc⊤⏟Sijcon+QicKδ(i,j)r⊤+KjcQδ(i,j)r⊤⏟Sijpos,S_ij\;=\; Q^c_iK^c_j^\! _ S^con_ij\;+\; Q^c_iK^r_δ(i,j)^\! +K^c_jQ^r_δ(i,j)^\! _ S^pos_ij, (10) where δ(i,j)δ(i,j) is the bucketed relative offset and Qc,KcQ^c,K^c are the content query and key. As with the relative-PE models above, we consider two injection sites: the input residual, and the position term SposS^pos. 3.4.1 Input-Side Injection Because DeBERTa has no absolute position embedding, the tag prior becomes the only positionally-localized signal at the input layer: p(0)=ptok+pseg+mp⋅(τpT+νpN).h^(0)_p=e^tok_p+e^seg_p+m_p\!·\! ( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p ). (11) The prior propagates through QW_Q and KW_K into the content terms of the attention score, while the relative-position mechanisms (its bucketing, lookup table, and projection matrices) are left untouched. Tag priors and relative position thus interact only through the content stream, which is what makes the injection portable across positional-encoding families. More details, including how the prior propagates to all three score terms, is given in Appendix A.5. 3.4.2 Position-Pathway Injection (P-SiPE) Mirroring our Transformer-XL injection, we do not add the prior to the input embeddings (the token representations enter the network unchanged) and instead route it only into the relative-position term SijposS^pos_ij of the attention score, leaving the content term SijconS^con_ij untouched. A per-layer projection WE(ℓ)W_E^( ) maps the summed hexatag embedding at key j into head space, and we form a tag–query alignment against the content query: cij c_ij =1dh⟨Qic,WE(ℓ)(τjT+νjN)⟩, \;=\; 1 d_h\, Q^c_i,\;W_E^( )( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j) , (12) S~ij S_ij =Sijcon+(1+cij)Sijpos. \;=\;S^con_ij\;+\;(1+c_ij)\,S^pos_ij. (13) The coefficient cijc_ij scales the relative-position score in proportion to how strongly the query aligns with the key’s syntactic tag, so syntax sharpens an existing positional preference rather than acting on its own, exactly as in the multiplicative Transformer-XL variant (Eq. 6).444WE(ℓ)W_E^( ) is initialized small so that cij≈0c_ij\!≈\!0 and S~ij≈Sij S_ij\!≈\!S_ij at initialization; training starts from the unmodified DeBERTa score and learns the syntactic correction into it. 3.5 ModernBERT: Injecting the Prior into Rotary Position Encoding ModernBERT Warner et al. (2024) like many other Large language models Dubey et al. (2024); DeepSeek-AI (2024); Yang et al. (2025); Kamath et al. (2025) delivers positional information through rotary positional embeddings (RoPE) Su et al. (2023), which rotate the content query and key by position-dependent angles after the content projections, so that relative-position dependence emerges from the geometry of two absolute rotations. Each head dimension is split into frequency pairs indexed by t; pair t is rotated by θp,t=pωt _p,t=p\, _t at absolute position p, with frequency ωt _t and a 2×22×2 rotation R(⋅)R(·). Because RoPE carries no input-level position term, we again consider two injection sites: the input residual, and the rotation angle itself. 3.5.1 Input-Side Injection Since RoPE has no input-level position embedding, the tag prior is added at the input residual exactly as in DeBERTa (Eq. 11, without the segment term, which ModernBERT does not use), and propagates into p,pq_p,k_p before rotation; the rotary mechanism itself is untouched. 3.5.2 Position-Pathway Injection (P-SiPE) Mirroring the Transformer-XL and DeBERTa injections, we can also route the prior into the position pathway instead of the input embedding, which for RoPE is the rotation angle. Using the summed hexatag embedding at position p, a per-layer projection Wδ(ℓ)W_δ^( ) produces a per-frequency phase offset Δθp,t=(Wδ(ℓ)(τpT+νpN))t, _p,t\;=\; (W_δ^( )( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _p+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _p) )_t, (14) which is added to the rotary angle before rotation, for both the query at position m and the key at position j: q~m,t q_m,t =R(mωt+Δθm,t)qm,t, \;=\;R\! (m\, _t+ _m,t )\,q_m,t, (15) k~j,t k_j,t =R(jωt+Δθj,t)kj,t. \;=\;R\! (j\, _t+ _j,t )\,k_j,t. (16) The effective angle between query m and key j at frequency t is then (m−j)ωt+(Δθm,t−Δθj,t)(m-j)\, _t+( _m,t- _j,t), so the tag enters as a syntactic phase shift on top of the positional one: the rotation that already encodes relative distance is nudged by how the two positions’ hexatags relate. Since the tag enters only as a rotation angle, it leaves the query and key magnitudes unchanged and perturbs only their direction (Appendix A.6). BLIMP Example ✓ The keys to the cabinet are on the table. ✗ The keys to the cabinet is on the table. SyntaxGym Example ✓ The farmer near the clerks knows many people. ✗ The farmer near the clerks know many people. Table 1: Examples from the BLIMP dataset (top) and SyntaxGym (bottom). In both datasets, p(p(✓)>p()>p(✗)), but in BLIMP this probability is computed for the complete sentence, whereas in SyntaxGym it is computed only over the region of interest. Model BLiMP ↑ Δ BLiMP PPL ↓ Δ PPL Syntax Gym ↑ Δ SyntaxGym No Inductive Bias Transformer-XL (tokens) 75.30 – 18.63 – 73.09 – GPT-2 (tokens) Radford et al. (2019) 72.20 ↓ -4.12 21.60 ↓ -15.94 71.90 ↓ -1.63 Constituency Parsing PLM (GPT-2) Qian et al. (2021) 75.10 ↓ -0.27 29.80 ↓ -59.96 80.20 ↑ +9.73 Transformer Grammar (Transformer-XL) Sartran et al. (2022) 73.50 ↓ -2.39 18.40 ↑ +1.23 82.50 ↑ +12.87 Pushdown LM (GPT-2) Murty et al. (2023) 75.60 ↑ +0.40 19.90 ↓ -6.82 82.30 ↑ +12.60 TreeReg (GPT-2) Nandi et al. (2025) 74.80 ↓ -0.66 22.30 ↓ -19.70 80.00 ↑ +9.45 Tree-Planted Transformers (TPT) Yoshida et al. (2024) TPT [zero] — — 47.50 ↓ -154.97 71.70 ↓ -1.90 TPT [rand.] — — 47.40 ↓ -154.43 69.00 ↓ -5.60 TPT [seq.] — — 47.30 ↓ -153.89 70.10 ↓ -4.09 TPT [dep.] — — 47.70 ↓ -156.04 77.10 ↑ +5.49 TPT [cons.] — — 45.50 ↓ -144.23 75.80 ↑ +3.71 TPT [bin.] — — 45.60 ↓ -144.77 73.00 ↓ -0.12 Syntax-Informed Embeddings with Hexatagging (Ours) SiPE (positional pathway) Input-side Injection 73.82 ↓ -1.97 16.16 ↑ +13.26 76.97 ↑ +5.31 Position-pathway Injection 74.01 ↓ -1.71 16.95 ↑ +9.02 80.60 ↑ +10.29 Fully-disentangled Injection 74.72 ↓ -0.77 16.66 ↑ +10.57 78.72 ↑ +7.70 Attention Bias Projections shared across layers 73.60 ↓ -2.26 16.73 ↑ +10.20 76.35 ↑ +4.46 Projections Learned Per-Layer 74.66 ↓ -0.85 16.24 ↑ +12.83 78.10 ↑ +6.86 SiPE + Attention Bias Input-side Injection 74.16 ↓ -1.51 15.89 ↑ +14.71 78.47 ↑ +7.36 Position-pathway Injection 74.44 ↓ -1.14 16.52 ↑ +11.32 77.97 ↑ +6.68 Table 2: Language modeling and syntax evaluation of decoder LMs; all our SiPE variants (Ours) use Transformer-XL as the decoder model, and each baseline’s model is noted in parentheses. BLiMP and SyntaxGym are higher-is-better, while PPL (perplexity, computed on BLLIP-LG test split Charniak et al. (2000)) is lower-is-better. Δ columns report relative improvement or degradation against vanillaTransformer-XL (tokens). Best scores for each dataset are bolded. Model Acc. (%) Δ (%) Our models on WikiText RoBERTa-base 70.68 – RoBERTa + SiPE (Input pathway) 71.68 ↑ +1.41 DeBERTa-base 67.40 – DeBERTa + SiPE (Input pathway) 68.93 ↑ +2.27 ModernBERT-base 49.94 – ModernBERT + SiPE (Input pathway) 51.10 ↑ +2.32 Transformer-XL 53.56 – Transformer-XL + SiPE (Positional Pathway) 53.44 ↓ -0.22 OOD Eval: WikiText → BLLIP-LG RoBERTa-base 71.29 – RoBERTa + SiPE (Input pathway) 72.62 ↑ +1.87 DeBERTa-base 69.01 – DeBERTa-base + SiPE (Input pathway) 71.12 ↑ +3.06 ModernBERT-base 51.25 – ModernBERT-base + SiPE (Input pathway) 53.41 ↑ +4.21 Table 3: BLiMP accuracy results for models trained with SiPE. For each model family we report its most performant GLUE variant: input-pathway SiPE injection for the encoders and positional-pathway SiPE injection for the Transformer-XL decoder; the OOD experiments likewise use the most performant GLUE variant of each family. For encoders, we use PLL-based accuracy. Δ : relative change w.r.t. the corresponding base model. OOD section: models pre-trained on WikiText, then continuously pre-trained on BLLIP-LG Charniak et al. (2000). ↑ improvement, ↓ degradation. Bold: best relative improvement with our method. RoBERTa (base) DeBERTa (base) ModernBERT (base) Task Base +SiPE Base +SiPE +P-SiPE Base +SiPE +P-SiPE CoLA 31.41 [0.5pt](±2.02) 31.77 [0.5pt](±0.71) 27.22 [0.5pt](±2.37) 26.09 [0.5pt](±2.14) 24.97 [0.5pt](±2.02) 25.00 [0.5pt](±0.83) 24.45 [0.5pt](±1.85) 23.26 [0.5pt](±2.83) SST-2 87.23 [0.5pt](±0.79) 87.22 [0.5pt](±0.52) 87.31 [0.5pt](±0.30) 87.88 [0.5pt](±0.54) 87.50 [0.5pt](±0.19) 85.09 [0.5pt](±1.01) 85.86 [0.5pt](±0.79) 84.25 [0.5pt](±1.38) QQP 83.24 [0.5pt](±0.21) 83.36 [0.5pt](±0.11) 84.02 [0.5pt](±0.31) 84.95 [0.5pt](±0.15) 84.88 [0.5pt](±0.13) 83.53 [0.5pt](±0.13) 83.09 [0.5pt](±0.45) 82.45 [0.5pt](±0.94) QNLI 77.65 [0.5pt](±0.50) 78.24 [0.5pt](±0.27) 68.07 [0.5pt](±0.08) 68.88 [0.5pt](±0.24) 67.94 [0.5pt](±0.29) 62.51 [0.5pt](±1.01) 61.83 [0.5pt](±0.70) 65.79 [0.5pt](±0.39) MNLI 73.03 [0.5pt](±0.33) 74.05 [0.5pt](±0.17) 71.25 [0.5pt](±0.14) 71.85 [0.5pt](±0.14) 71.74 [0.5pt](±0.30) 71.29 [0.5pt](±0.03) 72.24 [0.5pt](±0.35) 71.15 [0.5pt](±0.11) RTE 65.76 [0.5pt](±0.74) 64.66 [0.5pt](±0.56) 62.94 [0.5pt](±1.23) 63.06 [0.5pt](±1.73) 64.14 [0.5pt](±1.68) 62.21 [0.5pt](±0.45) 63.06 [0.5pt](±2.13) 63.90 [0.5pt](±1.02) STSB 79.11 [0.5pt](±0.30) 81.41 [0.5pt](±0.13) 76.20 [0.5pt](±0.56) 76.98 [0.5pt](±0.24) 76.33 [0.5pt](±1.32) 79.67 [0.5pt](±0.30) 81.21 [0.5pt](±0.78) 79.49 [0.5pt](±0.06) MRPC 80.96 [0.5pt](±0.44) 82.02 [0.5pt](±0.85) 80.49 [0.5pt](±0.84) 81.11 [0.5pt](±0.76) 83.45 [0.5pt](±0.71) 79.74 [0.5pt](±1.43) 82.82 [0.5pt](±1.08) 81.56 [0.5pt](±1.07) Macro 72.30 [0.5pt](±0.33) 72.84 [0.5pt](±0.42) 69.69 [0.5pt](±0.25) 70.10 [0.5pt](±0.39) 70.12 [0.5pt](±0.22) 68.63 [0.5pt](±0.08) 69.32 [0.5pt](±0.11) 68.98 [0.5pt](±0.37) Table 4: GLUE evaluation of baseline encoders vs. SiPE augmentation, over 3 seeds (std in parentheses). Baselines span three PE schemes: RoBERTa-base (absolute), DeBERTa-v3-base (relative), ModernBERT-base (RoPE). SiPE is injected via two routes: input pathway (into token embeddings) and positional pathway (into positional information; relative/RoPE only). Best variant per task in bold. SiPE improves every architecture on the macro average—RoBERTa +0.75%+0.75\%, DeBERTa-v3 +0.59%+0.59\%, ModernBERT +1.0%+1.0\% relative. RoBERTa (large) ModernBERT (large) Task Base +SiPE Base +SiPE CoLA 30.71 [0.5pt](±0.51) 30.36 [0.5pt](±0.88) 25.06 [0.5pt](±1.21) 24.58 [0.5pt](±0.87) SST-2 88.95 [0.5pt](±0.35) 87.50 [0.5pt](±0.09) 84.44 [0.5pt](±0.52) 85.47 [0.5pt](±0.36) QQP 84.31 [0.5pt](±0.18) 85.10 [0.5pt](±0.24) 82.54 [0.5pt](±0.39) 82.55 [0.5pt](±0.55) QNLI 80.18 [0.5pt](±0.18) 82.68 [0.5pt](±0.33) 61.99 [0.5pt](±0.52) 62.11 [0.5pt](±0.41) MNLI 74.72 [0.5pt](±0.10) 74.43 [0.5pt](±0.31) 72.03 [0.5pt](±0.53) 72.36 [0.5pt](±0.22) RTE 65.94 [0.5pt](±0.45) 64.62 [0.5pt](±1.06) 63.54 [0.5pt](±0.29) 61.49 [0.5pt](±2.27) STSB 81.89 [0.5pt](±0.59) 82.34 [0.5pt](±0.38) 79.03 [0.5pt](±0.41) 80.05 [0.5pt](±0.35) MRPC 80.92 [0.5pt](±1.00) 83.70 [0.5pt](±1.15) 81.86 [0.5pt](±1.24) 82.00 [0.5pt](±0.76) Macro 73.45 [0.5pt](±0.16) 73.84 [0.5pt](±0.13) 68.81 [0.5pt](±0.13) 68.83 [0.5pt](±0.35) We did not pre-train DeBERTa-large with SiPE (input pathway/positional pathway), nor ModernBERT-large with SiPE (positional pathway), due to limited academic compute and budget constraints. Table 5: GLUE evaluation results for large encoder models. Transformer-XL Task Base +P-SiPE CoLA 13.80 [0.5pt](±0.26) 26.26 [0.5pt](±0.22) SST-2 84.14 [0.5pt](±0.11) 88.80 [0.5pt](±0.44) QQP 83.83 [0.5pt](±0.12) 85.64 [0.5pt](±0.21) QNLI 76.00 [0.5pt](±0.26) 82.28 [0.5pt](±0.29) MNLI 68.07 [0.5pt](±0.32) 75.22 [0.5pt](±0.14) RTE 63.42 [0.5pt](±1.70) 67.51 [0.5pt](±0.29) STSB 76.90 [0.5pt](±0.17) 81.80 [0.5pt](±0.12) MRPC 79.20 [0.5pt](±0.21) 82.75 [0.5pt](±0.26) Macro 68.17 [0.5pt](±0.17) 73.78 [0.5pt](±0.05) Table 6: GLUE results for the decoder-only Transformer-XL (TXL) baseline and its SiPE (positional pathway) variant (blue tint); best per task in bold. SiPE improves every task and gains +8.2% relative on the macro average (68.17→ 73.78), showing the positional pathway best augments TXL’s relative positional encoding. 4 Experimental Results 4.1 Experimental Settings and Datasets Prior work injecting syntax into Transformers focused mainly on intrinsic evaluation—directly testing syntactic understanding Sartran et al. (2022); Xie et al. (2021); Omote et al. (2019); Yoshida and Oseki (2022); Zhao et al. (2024); Murty et al. (2023). We argue these gains should also be assessed extrinsically, on downstream tasks where syntax is indirectly useful, and organize our experiments accordingly. Intrinsic evaluation: We test syntactic understanding on BLiMP Warstadt et al. (2023) and SyntaxGym Hu et al. (2020), both targeted minimal-pair benchmarks contrasting grammaticality (Table 1). BLiMP requires assigning higher probability to the grammatical sentence overall, whereas SyntaxGym compares probabilities only in the critical region where the ungrammaticality occurs. Following prior work Zhao et al. (2024); Sartran et al. (2022), we pretrain Transformer-XL on BLLIP-LG Charniak et al. (2000) with their hyperparameters, using the training splits of Hu et al. (2020).555We do not use the dataset’s syntactic annotations; we hexatag the sentences with an in-house parser (URL hidden for blind review). We report BLiMP accuracy via sentence log-likelihood, SyntaxGym via the standard suite-level voting protocol, and BLLIP-LG test perplexity to confirm language-modeling ability is preserved. For our models, test sentences are hexatagged once by the parser before scoring, so our perplexity is conditioned on this single fixed parse, p(x∣T^(x))p(x T(x)), whereas joint syntactic LMs report a marginal p(x)p(x) approximated by summing over many candidate parses.666This is the trade-off SiPE makes by design: a single parse at inference instead of expensive marginalization (Figure 1). In a second set of experiments, we pretrain three encoders: RoBERTa, DeBERTa, ModernBERT Liu et al. (2019); He et al. (2023); Warner et al. (2024) and Transformer-XL for 1M steps on a 50M-token, offline-hexatagged slice of WikiText-103 Merity et al. (2016).777Our goal is to study syntactic embeddings on real downstream tasks, so we pretrain on WikiText and then fine-tune on GLUE. For encoders, we also re-use the same checkpoints for syntactic evaluation as a separate BLLIP-LG pretraining run with its own hyperparameter tuning was beyond our academic budget. We evaluate these models on a hexatagged BLiMP Warstadt et al. (2023): using encoders, for each minimal pair we prefer the sentence with higher pseudo-log-likelihood (PLL) Salazar et al. (2020) under the masked-LM objective; for the Transformer-XL decoder we instead use sentence log-likelihood (the sum of next-token log-probabilities). Accuracy is scored against BLiMP’s gold labels. Extrinsic evaluation: To assess the downstream impact of injecting syntactic information directly in the transformer architecture, we carry out experiments on the GLUE benchmark Wang et al. (2019), which contains a suite of real-world NLP tasks. We finetune our three encoder models as well as Transformer-XL (which were all previously pre-trained with SiPE on Wikitext) on each task using standard GLUE hyperparameters for each model (refer to Tables 4, 5 and 6 for results) and table 11 in appendix D. 4.2 Key Takeaways Positional-pathway injection is the strongest recipe for a relative-PE decoder. On Transformer-XL, injecting the SiPE prior into the positional pathway gives our best decoder result—SyntaxGym 80.60 (+10.29+10.29 over the vanilla token baseline) with perplexity cut from 18.63 to 16.95 (Table 2). This carries through to GLUE, improving the TXL macro average from 68.17 to 73.78, a dramatic +8.2%+8.2\% relative gain (Table 6). Using only a single parse tree at inference, SiPE also surpasses parser-free approaches, TreeReg Nandi et al. (2025) and all Tree-Planted Transformer variants Yoshida et al. (2024), on both SyntaxGym and BLLIP-LG perplexity, cutting perplexity by roughly a quarter against the strongest such baseline (16.95 vs. 22.30 for TreeReg) and by more than half against the Tree-Planted Transformer variants (16.95 vs. 45.5–47.7) while matching or exceeding their syntactic generalization. For encoders, the simpler input-embedding injection is better, with gains that strengthen out of distribution. Across all three encoder families—absolute (RoBERTa), disentangled-relative (DeBERTa-v3), and rotary (ModernBERT), input-pathway SiPE consistently outperforms the base model on GLUE (Table 4); the positional-pathway route also helps for the relative and rotary schemes but yields slightly smaller gains, so for encoders mixing the prior directly with input embeddings is sufficient and most reliable. All three encoders likewise improve on BLiMP—+1.41%+1.41\% (RoBERTa), +2.27%+2.27\% (DeBERTa-v3), +2.32%+2.32\% (ModernBERT) (Table 3)—and these gains grow under continued pre-training on BLLIP-LG after WikiText, rising to +1.87%+1.87\%, +3.06%+3.06\%, and +4.21%+4.21\% respectively, indicating the prior generalizes beyond its original distribution rather than overfitting. 888The small BLiMP drops are expected: BLiMP scores the full sentence, so language-modeling ability dominates over the ungrammatical span alone Zhao et al. (2024). Most baselines in Table 2 (PLM, Transformer Grammars, TreeReg, most TPT variants) likewise fall below the vanilla token baseline on BLiMP. Figure 3: Layerwise SiPE injection sweep on Transformer-XL. Injecting from layer k=1k=1 (all layers) is best; skipping the first layer already drops SyntaxGym accuracy sharply, and later entry points degrade syntactic generalization further. 5 Where Should Syntax Enter? A Layerwise Injection Study To understand at which depth syntax should interact with semantics, we sweep the entry point of the positional pathway SiPE bias (P-SiPE) in Transformer-XL. For a model with N layers, we inject the bias from layer k onwards: layers 1,…,k−11,…,k-1 use the default positional information, while layers k,…,Nk,…,N receive the SiPE bias. We vary k∈2,4,6,8,10,12,14k∈\2,4,6,8,10,12,14\ and compare injecting syntactic information in every layer (from k=1k=1 onwards), measuring syntactic generalization (SyntaxGym) and word-level perplexity on BLLIPLG (Figure 3). Two findings emerge. First, syntax is most impactful at the very first layer: full injection (k=1k=1) is clearly best, and skipping just the first layer (k=2k=2) already causes a sharp drop in syntactic generalization: SyntaxGym falls from 80.680.6 to 73.573.5. Second, and more broadly, syntactic information matters most in the lower layers of a decoder like Transformer-XL: injecting it in later layers yields consistently weaker syntactic generalization. Together, these results indicate that for autoregressive models the positional pathway should carry syntactic structure from the earliest layer onward, rather than being introduced in later layers. Complementing this view of where syntax should enter, Appendix C examines how the injected prior manifests in attention on BLiMP object–verb agreement: the encoders redistribute verb→ attention to mirror syntactic adjacency, whereas Transformer-XL surfaces the prior only weakly in its attention maps and instead converts it into the largest downstream semantic (GLUE) gains of any model we train. 6 Conclusion We proposed a simple strategy to infuse syntactic information in any transformer architecture. Our proposed modification is lightweight, adding only O(1,000)O(1,000) parameters per model via an additive prior embedding, yet it yields reliable gains in both intrinsic and extrinsic evaluations and multiple encoder and decoder transformer variants. Limitations Our approach assumes access to hexatag annotations at inference time: each input sequence must be tagged by a dependency parser before being passed to the syntax-infused encoder. While this adds a preprocessing step, the tagger is lightweight and fast in practice (we use DeBERTa-v3-base as the hexatagger, 184M parameters) and runs efficiently relative to encoder pre-training and downstream finetuning. The approach also inherits the hexatagger’s errors, so its benefit may be smaller in domains where the parser is less accurate. We consider only coarse-grained syntactic priors derived from directional terminal and non-terminal tags. Preliminary pre-training experiments incorporating full dependency-relation labels (deprel) did not yield meaningful gains on GLUE (Appendix E, Figures 7 and 8), but we did not investigate this further. Our study of where to inject the prior is broad but not exhaustive. We study the main injection sites and coupling choices for each positional scheme, but under a fixed academic compute budget we could not pre-train every conceivable combination, particularly for the relative and rotary attention-side variants, where the design space is large. The configurations we report are therefore the strongest we found rather than a guaranteed global optimum, and our per-dataset results with RoBERTa (Figure 8) show that no single setting dominates across all tasks. Perhaps the biggest limitation, which we plan to address in future work, is fast autoregressive text generation with the decoder. Because the injected prior is conditioned on hexatags, generating each new token requires re-tagging the sequence produced so far. Our hexatagger is fast, so this is feasible in principle, but it is not compatible with standard KV-caching: as the sequence grows, the parser may revise the tags of earlier tokens, changing their injected representations and invalidating the cached keys and values. Efficient incremental decoding under a per-step syntactic prior is therefore an open research-engineering problem, and the direction we consider most important for future work. Finally, our experiments are limited to small models, English text, and the pre-training budgets used here. Whether the gains hold at larger scale, in other languages, or under substantially longer pre-training, especially with billion parameter LLMs remains to be seen; its unclear how the relative benefit of an explicit syntactic prior will scale as model and data scale increase. References Amini et al. (2023) Afra Amini, Tianyu Liu, and Ryan Cotterell. 2023. Hexatagging: Projective dependency parsing as tagging. Preprint, arXiv:2306.05477. Baker et al. (1998) Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics. Charniak et al. (2000) Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJ corpus release 1. LDC2000T43. Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. Preprint, arXiv:1901.02860. De Marneffe et al. (2021) Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255–308. DeepSeek-AI (2024) DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. The llama 3 herd of models. Guo et al. (2020) Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020. Hierarchical poset decoding for compositional generalization in language. Advances in Neural Information Processing Systems, 33:6913–6924. He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543. He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. Preprint, arXiv:2006.03654. Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. 2020. A systematic assessment of syntactic generalization in neural language models. Preprint, arXiv:2005.03692. Hu et al. (2025) Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases. Preprint, arXiv:2502.19249. Kamath et al. (2025) Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram’e, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gael Liu, and 191 others. 2025. Gemma 3 technical report. ArXiv, abs/2503.19786. Kingsbury and Palmer (2002) Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1989–1993. Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Murty et al. (2023) Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. 2023. Pushdown layers: Encoding recursive structure in transformer language models. Preprint, arXiv:2310.19089. Nandi et al. (2025) Ananjan Nandi, Christopher D. Manning, and Shikhar Murty. 2025. Sneaking syntax into transformer language models with tree regularization. Preprint, arXiv:2411.18885. Omote et al. (2019) Yutaro Omote, Akihiro Tamura, and Takashi Ninomiya. 2019. Dependency-based relative positional encoding for transformer nmt. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 854–861, Varna, Bulgaria. INCOMA Ltd. Qian et al. (2021) Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3735–3745, Online. Association for Computational Linguistics. Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. Sartran et al. (2022) Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022. Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale. Transactions of the Association for Computational Linguistics, 10:1423–1439. Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. Preprint, arXiv:1803.02155. Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038, Brussels, Belgium. Association for Computational Linguistics. Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864. Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. Thinking Machines Lab (2026) Thinking Machines Lab. 2026. Inkling: Our open-weights model. https://thinkingmachines.ai/news/introducing-inkling/. Blog post; model weights at https://huggingface.co. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint, arXiv:1804.07461. Warner et al. (2024) Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. Preprint, arXiv:2412.13663. Warstadt et al. (2023) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2023. Blimp: The benchmark of linguistic minimal pairs for english. Preprint, arXiv:1912.00582. Xie et al. (2021) Yikuan Xie, Wenyong Wang, Mingqian Du, and Qing He. 2021. Transformer with syntactic position encoding for machine translation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1536–1544, Held Online. INCOMA Ltd. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Yoshida and Oseki (2022) Ryo Yoshida and Yohei Oseki. 2022. Composition, attention, or both? In Findings of the Association for Computational Linguistics: EMNLP 2022, page 5822–5834. Association for Computational Linguistics. Yoshida et al. (2024) Ryo Yoshida, Taiga Someya, and Yohei Oseki. 2024. Tree-planted transformers: Unidirectional transformer language models with implicit syntactic supervision. In Findings of the Association for Computational Linguistics ACL 2024, page 5120–5134. Association for Computational Linguistics. Zhao et al. (2024) Yida Zhao, Chao Lou, and Kewei Tu. 2024. Dependency transformer grammars: Integrating dependency structures into transformer language models. Preprint, arXiv:2407.17406. Appendix A Preliminaries A.1 Subword-Level Tag Injection Hexatag annotations are produced at the word level: every word wiw_i in a sentence carries a terminal tag tit_i and a nonterminal tag nin_i (drawn from small vocabularies T and N, respectively, where |T|=2|T|=2 and |N|=5|N|=5). The language model, however, operates on subword tokens: the tokenizer may split wiw_i into one or more subwords si,1,…,si,kis_i,1,…,s_i,k_i. We must therefore specify how a word-level hexatag is associated with its corresponding subword positions. We adopt a assign hexatag to first-subword only strategy: the tag pair (ti,ni)(t_i,n_i) is attached to the position of the first subword si,1s_i,1, and the remaining subwords si,2,…,si,kis_i,2,…,s_i,k_i receive no tag (in our implementation this is equivalent to receiving a tag ID of −100-100). Concretely, for each subword position p in the flattened input sequence, we define the following mask: mp=1if p is the first subwordof some word wi,0otherwise.m_p= cases1&if p is the first subword\\ &of some word w_i,\\ 0&otherwise. cases (17) At subword positions with mp=1m_p=1, the prior contributes a learned tag embedding to augment the subword embedding; at positions with mp=0m_p=0 (continuation subwords), no prior is added, and the subword embedding is used as-is. Tag information still reaches continuation subwords, but only indirectly, through self-attention in subsequent layers. This design has two practical benefits: (i) The total tag embedding signal injected per word is invariant to its subword length kik_i, avoiding over-amplification for words that break into many subwords (i) it gives a clean one-to-one correspondence between words and tagged subword positions, which is required by the auxiliary tag-prediction objective: this objective is a per-position classification head that predicts the (terminal, nonterminal) tag pair, and it only contributes loss at positions where mp=1m_p=1. The assign hexatag to first-subword only is also the simpler choice here. The natural alternative would be to assign the same hexatag to all kik_i subwords of a word, which would require supervising all pieces, contributing kik_i loss terms per word and thus over-weighting words that fragment into many subwords. We leave this variant to future work. The indicator mpm_p thus controls both prior injection and supervision, giving each word exactly one tag signal at the input and one prediction at the output. The auxiliary objective takes one of two concrete forms in our experiments. In autoregressive models, the head at position p predicts the tags (τp+1,νp+1)( _p+1, _p+1) of the next tagged position, teacher-forced from gold left context. In masked language models, the head at position p predicts (τp,νp)( _p, _p) at masked positions only; wherever a token is masked, its injected tag embedding is masked along with it (the mpm_p term is zeroed), so the auxiliary head must infer the tag from surrounding context rather than copy it from its own input. The remainder of this section details the design choices and trade-offs for each injection strategy we study. These fall into a few families: adding the syntactic bias to the input embeddings; entangling it directly with the positional encoding; keeping it disentangled and adding it as a separate term to the attention score; injecting it as a separate attention-side bias; and combining any of these. We examine these strategies across four model architectures that span the major positional-encoding schemes: RoBERTa (encoder, absolute positional embeddings), DeBERTa-v3 (encoder, relative positional embeddings), ModernBERT (encoder, rotary positional embeddings), and Transformer-XL (autoregressive decoder, relative positional embeddings). A.2 SiPE Position-Pathway Design Choices for Transformer-XL The multiplicative position-pathway injection of section 3.3 has two design axes. Layer sharing controls whether a single projection WE [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E is reused across all layers or each layer learns its own WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ). Projection target controls the dimension WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ) projects the tag into before its inner product with the query forms the alignment score c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c: either a single attention head’s dheadd_head-dimensional subspace (a small map that places the tag in the same space as that head’s query, so c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c is formed by the same query that produces [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD and lands on the same per-head scale; the one projected vector is then shared across all heads), or the full model dimension dmodeld_model (a larger map whose output is split into nheadn_head blocks, giving each head its own projected tag). Table 7 reports all four combinations. Sharing Projection BLiMP ↑ SG ↑ Shared dheadd_head 73.95 78.80 Per-layer dheadd_head 74.15 75.84 Shared dmodeld_model 73.95 75.09 Per-layer dmodeld_model 74.01 80.60 Table 7: Multiplicative position-pathway injection across the two design axes: layer sharing (shared vs. per-layer WE [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E) and projection target (dheadd_head head subspace vs. dmodeld_model full dimension). Per-layer projection into the full model dimension gives the best average across BLiMP and SyntaxGym (SG); this is the configuration we adopt as SiPE (positional pathway) for Transformer-XL. For Transformer-XL, the best configuration we find is the per-layer projection into the full model dimension (74.01 BLiMP, 80.60 SG). Neither axis dominates alone: the gain comes from their pairing. Per-layer projections matter because the tag correction is scaled by each layer’s own [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD, so a separate WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ) lets each layer specialize how syntax modulates its positional preference. The full-dimension target matters because it gives the tag a richer map than a single head’s subspace allows. Notably, the better projection target flips with the coupling: the multiplicative coupling here works better with dmodeld_model, whereas the fully-disentangled coupling of Appendix A.3 works better in the dheadd_head head subspace. The best target therefore depends on how the prior is coupled to the position pathway, not on the projection axis in isolation. Beyond these two axes, the form of the coupling itself matters: multiplicative coupling scales the position term by the tag alignment (Eq. 6), while the disentangled form adds the alignment as a standalone logit term (section 3.3.3, Eq. 7). The multiplicative variants are the strongest overall, but the ordering is informative: disentangled injection already outperforms input-side injection (Eq. 4), and multiplicative coupling outperforms both. Entangling the prior with position thus helps progressively more as the coupling becomes gated by the query’s existing offset alignment, so that syntax sharpens a positional preference the model has already formed rather than adding an unconditional bias. A.3 Fully-Disentangled Syntax Injection Design Choices for Transformer-XL The fully-disentangled variant of section 3.3.3 adds the tag–query alignment c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c to the attention logit as a standalone term (Eq. 7), leaving both the content term AC and the position term [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD exactly as in standard Transformer-XL. It is governed by the same two design axes (layer sharing and projection target) as the multiplicative position-pathway variant of Appendix A.2, the only difference being that there c [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125c multiplies [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125BD whereas here it is added as a standalone logit term: WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( ) may map the tag into the full model dimension (then reshaped into per-head blocks) or directly into a single head’s dheadd_head-dimensional subspace, and it may be shared across all layers or learned per layer. Since neither axis has an obvious right answer a priori, we ran all four combinations and report them in Table 8. Sharing Projection BLiMP ↑ SG ↑ Shared dheadd_head 74.17 77.72 Shared dmodeld_model 75.03 76.10 Per-layer dmodeld_model 73.51 77.20 Per-layer dheadd_head 74.72 78.72 Table 8: Fully-disentangled injection across the two projection axes: layer sharing (a single WE [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E shared across all layers vs. a per-layer WE(ℓ) [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375W_E^( )) and projection target (mapping the tag into a single head’s dheadd_head-dimensional subspace vs. the full model dimension dmodeld_model, reshaped into per-head blocks). Per-layer projection into the head subspace gives the best average across BLiMP and SyntaxGym (SG); this is the configuration reported in the main results. The best disentangled configuration is per-layer projection into the head subspace (74.72 BLiMP, 78.72 SG; Table 8). In terms of syntactic generalization performance, it outperforms input-side injection (73.82 BLiMP, 76.97 SG) but lags behind the multiplicative coupling (§3.3.2). We thus find that the least to most performant design choices for entangling the syntactic prior are: input-side << fully disentangled << multiplicative. A.4 Attention-Side and Combined Injection Design Choices for Transformer-XL Relation to the Shaw formulation: The key and value biases of Eqs. 8–9 adapt Shaw et al. (2018), who add learned per-pair vectors ijKa^K_ij and ijVa^V_ij to the key-side logit and the value-side aggregation, eij+=i⊤ijKe_ij +=q_i a^K_ij and i+=∑jαijijVz_i += _j _ija^V_ij. We change two things. First, our bias is indexed by the key position j alone rather than by the relative pair (i,j)(i,j), so the tag at j contributes a single per-key vector that every query sees; this is what keeps the added cost linear in sequence length rather than quadratic. Second, the bias is conditioned on hexatag identity rather than relative offset, and is learned jointly with the language model. The terminal and nonterminal tag embeddings are projected separately and summed, and a position-validity mask mjm_j zeroes the contribution wherever the tag label is -100. The tag bias is formed from the unmodified query i(ℓ)q^( )_i, not i(ℓ)+nq^( )_i+u_n as in the content term, so it bypasses the global content bias nu_n of Dai et al. (2019). And since we set mem_len=0 mem\_len=0, it spans the same keys as intra-segment attention, so no padding is needed for memory positions. This places our design between the dependency-distance bias of Omote et al. Omote et al. (2019) and the root-to-node path embedding of Xie et al. Xie et al. (2021). Both put syntactic structure inside attention rather than in the residual stream, but our bias is keyed on hexatag identity rather than a tree-distance proxy, is a single per-key term rather than a pairwise one, and is applied in a causally masked unidirectional language model rather than a bidirectional encoder. Sharing across layers. The one design consideration here is whether KtagW^tag_K and VtagW^tag_V are shared across all layers or learned per layer. In the shared form a single pair is reused at every layer; in the per-layer form each of the L layers owns its own pair, giving L independent key and value maps and letting each layer specialize its tag-to-attention routing, in the same way the content projections Q,K,VW_Q,W_K,W_V already specialize per layer. The per-layer form is the stronger of the two on syntactic evaluations, both for the attention-side injection on its own and for its combination with the input-side injection of Eq. 4, so we use it wherever the attention-side biases appear. A.5 DeBERTa-v3: Disentangled Relative Positional Embeddings and Tag Injection DeBERTa-v3 He et al. (2023) does not use absolute positional embeddings at the input level. Instead it delivers positional information through disentangled relative attention: each pair of positions (i,j)(i,j) contributes three distinct terms to the attention score, only one of which depends on token content alone. This raises a natural question for our setting: where do tag priors enter when there is no input-level position embedding to sit alongside? Background: disentangled attention. For a query at position i and a key at position j, the score decomposes into a content-to-content (c2c), a content-to-position (c2p), and a position-to-content (p2c) term, Aij= A_ij\;=\; i⊤j⏟c2c+i⊤c2pδ(i,j)⏟c2p q_i k_j_c2c\;+\; q_i W_c2p\, [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_δ(i,j)_c2p +j⊤p2cδ(j,i)⏟p2c, \;+\; k_j W_p2c\, [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_δ(j,i)_p2c, (18) where i=Qi(0)q_i=W_Qh^(0)_i and j=Kj(0)k_j=W_Kh^(0)_j. The relative-position embeddings δ(⋅,⋅)r_δ(·,·) are pulled from a learned table indexed by bucketed signed offsets, and c2p,p2cW_c2p,W_p2c project content vectors into this relative-position subspace.999The buckets δ collapse pairs of positions with similar offsets into shared rows of the relative-position table, following Shaw et al. (2018): small offsets −3,−2,−1,0,1,2,3\-3,-2,-1,0,1,2,3\ each receive their own bucket, while larger offsets are progressively merged — e.g. +9,…,+15\+9,…,+15\ might collapse into a single bucket — so the table stays small while preserving fine-grained discrimination near the diagonal. Input-side injection: where priors enter, and where they do not. Under input-side injection (Eq. 11) each input vector carries the summed tag prior, so the queries and keys carry it into the score through the content stream: i= _i\;=\; Q(itok+iseg+mi⋅(τiT+νiN)), _Q\! (e^tok_i+e^seg_i+m_i\!·\! ( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _i+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _i )\! ), (19) j= _j\;=\; K(jtok+jseg+mj⋅(τjT+νjN)). _K\! (e^tok_j+e^seg_j+m_j\!·\! ( [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T_ _j+ [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N_ _j )\! ). (20) Reading Eqs. 18–20 together makes the design visible. The tag priors sit inside i(0)h^(0)_i and j(0)h^(0)_j and therefore propagate through Q,KW_Q,W_K into all three score terms via the content projections. The relative-position table, by contrast, appears only in c2p and p2c, and is purely a function of the offset. The two streams enter through entirely separate pathways and meet only when their score terms are summed to form the attention logit AijA_ij; the relative-position components themselves — the bucketing, the δ [rgb]0.1796875,0.48828125,0.1953125 [named]pgfstrokecolorrgb0.1796875,0.48828125,0.1953125r_δ table, and the c2p,p2cW_c2p,W_p2c projections — are left entirely unmodified. Tag priors interact with relative position through the content stream alone, which is what makes the design portable across positional schemes. The tag tables T,N [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^T, [rgb]0.70703125,0.37109375,0.0234375 [named]pgfstrokecolorrgb0.70703125,0.37109375,0.0234375E^N are constructed similarly to RoBERTa: uniformly initialized, masked by the first-subword indicator mpm_p, and added inside the input layer norm. Position-pathway injection: The position-pathway variant of section 3.4.2 leaves the input clean and instead multiplies the two relative-position terms (c2p and p2c, grouped as SposS^pos in Eq. 10) by the tag–query alignment cijc_ij of Eq. 12, leaving the content-to-content term untouched. Here cijc_ij is formed from the content query QicQ^c_i and the per-layer projection WE(ℓ)W_E^( ) of the summed hexatag embedding, and the relative-position table and the c2p,p2cW_c2p,W_p2c projections are left unmodified. This is the direct DeBERTa analogue of the multiplicative Transformer-XL injection: the prior scales the position pathway and leaves the content pathway untouched with the only difference being that DeBERTa’s position pathway carries two terms (c2p and p2c) rather than Transformer-XL’s single BD. A.6 ModernBERT: Rotary Injection Details Portability of input-side injection: The unifying observation across the three encoders is that input-level injection commutes with whatever positional mechanism the model already uses, whether absolute (RoBERTa), relative (DeBERTa), or rotary (ModernBERT). Identity at initialization. With Wδ(ℓ)→W_δ^( )\!→\!0 we have Δθ→0 θ\!→\!0 in Eq. 14, so Eqs. 15–16 reduce to plain RoPE: training starts from an unmodified ModernBERT, exactly as in the Transformer-XL and DeBERTa variants. Appendix B Time Complexity SiPE adds two small embedding tables for terminal and non-terminal tags, T∈ℝ2×dE^T ^2× d and N∈ℝ5×dE^N ^5× d, totaling 7d7d parameters per pathway (∼ 5K at d=768d=768, ∼ 7K at d=1024d=1024), well under 0.01%0.01\% of the LM’s total parameters101010For comparison, a single attention layer in RoBERTa-base contains ∼ 2.4M parameters; the priors add less than 0.3%0.3\% of a single layer’s parameters.. Looking up and adding these embeddings costs (Ld)O(Ld) per sequence, which is lower-order than self-attention’s (L2d)O(L^2d) cost. Position-pathway variants introduce tag–query interactions with (L2d)O(L^2d) complexity, matching the asymptotic cost of self-attention. SiPE therefore preserves the underlying Transformer’s asymptotic complexity. At inference, a single auxiliary parser pass supplies the hexatags, while two lightweight token-level heads add only linear overhead beyond the parser encoder. Appendix C Analyzing Attention Patterns on Object–Verb Agreement Tasks A natural desideratum for SiPE-pretrained models is that they should up-weight attention between tokens that are linearly distant but syntactically adjacent (distance 11 in the dependency tree). We probe this on the BLiMP Causative split for three model architectures spanning the three positional encoding families: RoBERTa-base (absolute), ModernBERT-base (rotary), and Transformer-XL (relative), each against its SiPE counterpart. Verb→ attention is averaged across all heads; the layers we average over differ by architecture. For Transformer-XL, our layerwise injection study (section5) shows that syntax matters at every layer, so we average over all of its layers. We have not run the corresponding sweep for the encoders, so we instead follow Tenney et al. (2019), who find that “BERT’s intermediate layers encode a rich hierarchy of linguistic information, with surface features at the bottom, syntactic features in the middle and semantic features at the top,” and therefore average over each encoder’s middle layer band.111111Layers 5–8 for the 12-layer RoBERTa-base and layers 9–12 for the 22-layer ModernBERT-base; all layers 1–16 for the Transformer-XL we pre-train. Figure 4: Verb→ attention on BLiMP Warstadt et al. (2023) for RoBERTa-base vs. RoBERTa-base+SiPE (input pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On the subset of Causative examples where SiPE assigns higher PLL to the grammatical sentence but the base model does not, verb→ attention is higher under SiPE in 90% of cases (Appendix C). Figure 5: Verb→ attention on BLiMP for ModernBERT-base vs. ModernBERT-base+SiPE (input pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1). On Causative examples where SiPE is correct and the base model is not, the SiPE model assigns higher verb→ attention in 17/20 (85%) cases, mirroring the RoBERTa pattern despite ModernBERT’s rotary positional encoding. Figure 6: Verb→ attention on BLiMP for Transformer-XL vs. Transformer-XL+SiPE (positional pathway injection), both pre-trained on the 50M-token WikiText slice (subsection 4.1), averaged over all 1616 layers. On Causative examples where SiPE assigns higher sentence log-likelihood to the grammatical sentence but the base model does not, SiPE places higher verb→ attention in 11/2011/20 (55%55\%) cases: the same direction as the encoders (Figs. 4 and 5), but a much weaker majority. The green dashed line (right axis) marks the verb→ surface distance. Across all three families, SiPE attains higher overall BLiMP accuracy than the corresponding base model. To understand how this syntactic information influences attention, we restrict the analysis to the subset of minimal pairs where the SiPE model is correct and the base model is not. We manually inspect 20 examples from this subset. We find that RoBERTa+SiPE places higher verb→ attention than its base model in 90%90\% of cases (Figure 4) and ModernBERT+SiPE does so in 85%85\% of cases (Figure 5). Transformer-XL moves in the same direction but far more weakly: only a bare majority (11/2011/20 (55%55\%)) of such cases shift attention toward the object (Figure 6), where the encoders push it almost every time. Moreover, in over 80%80\% of the subset of 20 examples we examine, the object is nonadjacent to the verb, indicating a redirection of attention weights to the syntactically relevant object, even when it is linearly distant from the verb. This suggests that SiPE effectively leverages syntactic information to guide attention in a way that captures long-range dependencies, which is crucial for understanding complex sentence structures. The weak Transformer-XL effect poses an open question. The encoders appear to spend most of the syntactic prior directly on attention, redistributing mass to mirror syntactic adjacency. Transformer-XL barely does so in its attention maps, yet it converts the same signal into the largest downstream gain of any model we train (+8.2%+8.2\% relative on GLUE; Table 6). This suggests the decoder routes more of the enhanced syntactic knowledge beyond attention, towards semantic processing. Figure 7: Average GLUE performance across prior-injection variants for RoBERTa-base, shown as the improvement over the no-prior baseline (68.64±0.6168.64± 0.61, shaded band at zero); bar-end labels give absolute scores and whiskers denote one standard deviation over 3 seeds. The strongest downstream performance is achieved by directly adding terminal and non-terminal prior embeddings to the absolute positional embedding model (ADD T+NT), outperforming concatenation, weighted addition, and variants using full dependency labels (the residual-injection variant is evaluated separately in Table 9). In these experiments, we set α=0.5α=0.5. Figure 8: GLUE performance averaged across 3 random seeds per dataset under various prior-injection variants for RoBERTa-base. The strongest downstream performance is achieved by directly adding terminal and non-terminal prior embeddings to the absolute positional embedding model, outperforming concatenation, weighted addition, and variants using full dependency labels (the residual-injection variant is evaluated separately in Table 9). Task Baseline ADD_T_NT CONCAT_T_NT ADD_T_NT (Residual Connection) CoLA 31.41 (±2.02) 31.77 (±0.71) ↑ 26.85 (±1.06) ↓ 32.35 (±0.79) ↑ SST-2 87.23 (±0.79) 87.22 (±0.52) ↓ 87.27 (±0.35) ↑ 87.42 (±0.19) ↑ QQP 83.24 (±0.21) 83.36 (±0.11) ↑ 82.60 (±0.22) ↓ 83.13 (±0.15) ↓ QNLI 77.65 (±0.50) 78.24 (±0.27) ↑ 70.79 (±0.70) ↓ 75.47 (±0.54) ↓ MNLI 73.03 (±0.33) 74.05 (±0.17) ↑ 72.89 (±0.31) ↓ 73.90 (±0.14) ↑ RTE 65.76 (±0.74) 64.66 (±0.56) ↓ 64.74 (±1.45) ↓ 65.46 (±0.45) ↓ STSB 79.11 (±0.30) 81.41 (±0.13) ↑ 79.36 (±0.26) ↑ 80.41 (±0.47) ↑ MRPC 80.96 (±0.44) 82.02 (±0.85) ↑ 82.76 (±0.72) ↑ 81.06 (±0.96) ↑ Average 72.30 72.84 ↑ 70.91 ↓ 72.40 ↑ Table 9: RoBERTa performance on GLUE under alternative strategies for combining syntactic priors. ADD_T_NT adds the terminal prior it_i and non-terminal prior in_i directly to the token representation (i.e., additive fusion at the embedding layer). CONCAT_T_NT concatenates the original embedding (including standard positional information) with it_i and in_i, then applies a learned linear down-projection to the model dimension before feeding the encoder. ADD_T_NT (Residual connection) injects (i+i)(t_i+n_i) into the residual stream before the multi-head self-attention sublayer. Results are averaged over 3 random seeds. Green arrows indicate improvements over the baseline and red arrows indicate degradations. Overall, ADD_T_NT yields the strongest macro-average (72.84%), while concatenation is the weakest configuration (70.91%), with particularly large drops on QNLI and CoLA. Task Baseline α = 0.1 α = 0.5 α = 0.9 CoLA 31.41 (±2.02) 29.40 (±1.62) ↓ 29.89 (±0.71) ↓ 30.37 (±0.54) ↓ SST-2 87.23 (±0.79) 86.93 (±0.34) ↓ 86.81 (±0.29) ↓ 88.13 (±0.06) ↑ QQP 83.24 (±0.21) 83.04 (±0.08) ↓ 83.01 (±0.11) ↓ 83.66 (±0.48) ↑ QNLI 77.65 (±0.50) 77.08 (±0.73) ↓ 73.72 (±0.22) ↓ 76.45 (±2.00) ↓ MNLI 73.03 (±0.33) 72.58 (±0.02) ↓ 73.50 (±0.37) ↑ 73.07 (±0.05) ↑ RTE 65.76 (±0.74) 66.25 (±0.18) ↑ 64.98 (±0.62) ↓ 66.07 (±1.44) ↑ STSB 79.11 (±0.30) 78.38 (±0.03) ↓ 79.64 (±0.51) ↑ 78.25 (±0.14) ↓ MRPC 80.96 (±0.44) 82.77 (±0.09) ↑ 81.28 (±1.51) ↑ 80.42 (±0.04) ↓ Average 72.30 72.05 ↓ 71.60 ↓ 72.05 ↓ Table 10: Effect of initializing the interpolation gate α for RoBERTa under the ADD_T_NT setting. We initialize α∈0.1,0.5,0.9α∈\0.1,0.5,0.9\ and allow it to be updated during pretraining via backpropagation; all results report downstream GLUE performance after pretraining. Despite task-level variability (notably on QNLI), none of the α initializations improves upon the baseline on average, and all are weaker than the simple additive fusion. Appendix D Hyperparameters Table 11 reports the task-specific fine-tuning hyperparameters used for each backbone. Unless noted otherwise, all models use AdamW with (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999) and ϵ=10−6ε=10^-6. LR WD RoBERTa-base/large & DeBERTa-v3-base/large all tasks 1e−51e-5 0.10.1 ModernBERT-base/large (per task) CoLA 8e−58e-5 1e−61e-6 SST-2 8e−58e-5 1e−51e-5 MRPC 5e−55e-5 5e−65e-6 STS-B 8e−58e-5 5e−65e-6 QQP 5e−55e-5 5e−65e-6 MNLI 5e−55e-5 5e−65e-6 QNLI 8e−58e-5 5e−65e-6 RTE 5e−55e-5 1e−51e-5 Table 11: GLUE fine-tuning hyperparameters. Only ModernBERT’s LR/WD vary by task; RoBERTa and DeBERTa-v3 use LR 1e−51e-5, WD 0.10.1 throughout. All runs: 2020 epochs, AdamW, ϵ=10−6ε=10^-6. Appendix E Alternative Embedding Combination Strategies Throughout the paper, each token carries just two hexatag priors: a terminal tag i=τiTt_i=E^T_ _i and a non-terminal tag i=νiNn_i=E^N_ _i, read from small learned tables (2×D2× D for the terminals, 5×D5× D for the non-terminals). But is this the richest syntactic signal we could inject? A natural alternative is the token’s full dependency-relation label (deprel; 40 labels such as nsubj and dobj), which would come from a much larger 40×D40× D table, giving a per-token embedding ideprel_i. This opens two questions we study together: whether the richer deprel prior helps, and, once it is combined with the terminal and non-terminal priors, how all of these vectors should be fused with the token and positional embeddings, by addition, concatenation, a learned interpolation weight, or injection deeper into the network. We explore both questions as a small neural-architecture search over injection strategies. To keep it tractable we run the search on the simplest positional scheme, absolute positional embeddings, using RoBERTa-base as the encoder (the case illustrated in Figure 9). The strategies we explore are collected in Table 12 and split into two studies at different pretraining scales. Strategy Injected input representation i′x _i Description Fusion mechanism (terminal + non-terminal tags) ADD_T_NT i′=i+i+ix _i=x_i+t_i+n_i Add both priors to the input embedding (default; no extra parameters). CONCAT_T_NT i′=[i;i;i]x _i=W\,[\,x_i;\,t_i;\,n_i\,] Concatenate with the input embedding, then down-project to D. ADD_T_NT (Residual Connection) i(1)=LN(i+i+i+Attn()i)h^(1)_i=LN (x_i+t_i+n_i+Attn(x)_i ) Add the priors to the skip connection of the first attention sublayer. ADD_T_NT (α) i′=αi+(1−α)(i+i)x _i=α\,x_i+(1-α)(t_i+n_i) Linearly interpolate the input embedding with the (terminal + non-terminal) prior; learned α initialized in 0.1,0.5,0.9\0.1,0.5,0.9\. Adding dependency relations (ideprel_i, 40 labels) ADD_T_NT_DR i′=i+i+i+ix _i=x_i+t_i+n_i+deprel_i Add all three priors to the input embedding. CONCAT_POS_NT_T_DR i′=[i;i;i;i]x _i=W\,[\,x_i;\,n_i;\,t_i;\,deprel_i\,] Concatenate all priors with the input embedding, then down-project. ADD_DNT i′=i+i+ix _i=x_i+deprel_i+n_i Swap the terminal tag for the deprel. ADD_DNT (α) i′=αi+(1−α)(i+i)x _i=α\,x_i+(1-α)(deprel_i+n_i) Interpolated counterpart of ADD_DNT; learned α initialized in 0.1,0.5,0.9\0.1,0.5,0.9\. CONCAT_POS_DNT i′=[i;i;i]x _i=W\,[\,x_i;\,deprel_i;\,n_i\,] Concatenate (deprel replacing the terminal), then down-project. Table 12: Injection strategies we explore for absolute positional embeddings on RoBERTa-base. i=i+ix_i=e_i+p_i is the standard input embedding (token ie_i plus positional ip_i), i′x _i the representation actually fed to the encoder, and W a learned projection back to the model dimension D. it_i/in_i are the terminal/non-terminal hexatag priors and ideprel_i the dependency-relation prior. For the residual variant we write the first attention sublayer explicitly, (1)=LN(+Attn())h^(1)=LN(x+Attn(x)): the priors are added on the skip branch, so unlike ADD_T_NT they do not enter the self-attention computation itself. The top block fuses only the terminal and non-terminal tags; the bottom block additionally brings in deprels. Fusion mechanism (1M steps). We first fix the priors to it_i and in_i and vary only how they enter the model, pretraining RoBERTa-base for 1M steps (top block of Table 12, Table 9). The simplest option wins: ADD_T_NT, which just adds the two priors to the input embedding with no extra parameters, attains the best macro-average. CONCAT_T_NT is the weakest (with large drops on QNLI and CoLA), and injection into the model’s residual (skip) connection (ADD_T_NT (Residual Connection)) yields only modest gains. Replacing the plain addition with a learned linear interpolation between the input embedding and the prior, ADD_T_NT (α): i′=αi+(1−α)(i+i)x _i=α\,x_i+(1-α)(t_i+n_i), never improves over ADD_T_NT at any α initialization (Table 10). Adding capacity or depth to the fusion, therefore, does not help; direct addition at the embedding layer is both the cheapest and the strongest. Adding dependency relations (500k steps). We then enrich the prior itself: we embed each token’s deprel label and combine it with the terminal and non-terminal priors through the bottom-block strategies of Table 12. Because this study is more expensive, every variant, including a re-run baseline and ADD_T_NT, is pretrained for 500k steps for an apples-to-apples comparison (Figures 7, 8). Every prior-injection variant improves over the no-prior baseline on macro-average, yet ADD_T_NT still attains the highest mean. Layering deprels on top of i+it_i+n_i (ADD_T_NT_DR, CONCAT_POS_NT_T_DR) does not close the gap, and swapping the terminal tag for the deprel is no better, whether added directly (ADD_DNT), concatenated (CONCAT_POS_DNT), or interpolated through a learned weight, ADD_DNT (α): i′=αi+(1−α)(i+i)x _i=α\,x_i+(1-α)(deprel_i+n_i). Richer syntactic signal, at least in this form, does not translate into stronger downstream performance. Summary and scope. Taken together, these experiments identify ADD_T_NT, direct addition of the terminal and non-terminal priors at the input embedding, as the strongest injection strategy for absolute positional embeddings on the RoBERTa encoder. Two considerations bound the generality of this finding. First, the per-dataset results (Figure 8) show that no single variant is optimal on every task; the relative ordering of strategies varies across GLUE tasks, and ADD_T_NT is strongest in the macro-average rather than uniformly. Second, our search covers only the absolute-PE encoder; the optimal injection strategy for relative (DeBERTa-v3, Transformer-XL) and rotary (ModernBERT) positional schemes, and under larger pretraining budgets, remains open. We therefore adopt ADD_T_NT as a simple and robust default, and leave a systematic study of injection strategies across positional schemes to future work. Figure 9: Injection points for syntactic priors in a Transformer encoder. We study where and how to inject hexatag-derived priors (terminal it_i, non-terminal in_i, and optionally dependency-relation labels ideprel_i) into RoBERTa-base. Three architectural locations are considered: (i) at the input embedding, alongside the token, segment, and positional embeddings; (i) at the residual (skip) connection entering the first self-attention sublayer; and (i) implicitly in the attention computation, by injecting at the input and propagating through Q,KW_Q,W_K.

Instant research discovery