Paper deep dive
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing
Kuzey Torlak, Hüseyin Arda Arslan, Anıl Dervişoğlu, Beyza Nur Deniz, Onur Boyar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/20/2026, 7:48:17 AM
Summary
The paper proposes the Parallel Hybrid Architecture (PHA), a novel approach to long-context modeling that combines Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs) as independent parallel branches. Unlike serial hybrid models, PHA uses a learnable static mixing mechanism to fuse these branches, allowing each to specialize in either global context or selective retrieval. Experimental results on WikiText-103 and OpenWebText demonstrate that PHA achieves competitive or superior perplexity compared to standard Transformers and existing hybrid models (like H3 and Hedgehog) while offering significantly higher throughput and lower memory usage.
Entities (8)
Relation Signals (5)
Parallel Hybrid Architecture → comprises → Gated State Spaces
confidence 100% · PHA, which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs)
Parallel Hybrid Architecture → comprises → Grouped Query Attention
confidence 100% · PHA, which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs)
Parallel Hybrid Architecture → uses → Learnable Static Mixing
confidence 100% · fused by a learnable mixing mechanism
Parallel Hybrid Architecture → outperforms → H3
confidence 90% · outperforming Hedgehog (16.70) and H3-125M (23.70)
Parallel Hybrid Architecture → outperforms → Hedgehog
confidence 90% · outperforming Hedgehog (16.70)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Modeling long-range dependencies remains a central challenge in natural language processing. Transformer architectures achieve strong performance via self-attention but scale quadratically ($O(N^2)$) with sequence length, while State Space Models (SSMs) scale linearly ($O(N)$) but suffer from a selective recall bottleneck, struggling to retrieve precise information from compressed states. This creates a fundamental tradeoff between efficiency and perplexity. To tackle these challenges, we propose the \textit{Parallel Hybrid Architecture (PHA)}, which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed-Forward Networks (FFNs) as independent parallel branches fused by a learnable mixing mechanism. Instead of forcing SSMs to approximate attention or serializing the two paradigms, PHA allows each branch to specialize: GSS captures global context, while attention performs selective retrieval, with FFN providing complementary processing. On WikiText-103, PHA achieves 16.51 PPL at 125M parameters, outperforming Hedgehog (16.70) and H3-125M (23.70). Scaling to 180M parameters yields 16.42 PPL, which gives comparable results with the pure attention baseline while delivering 24\% higher throughput and up to 40\% lower memory usage at long contexts. On OpenWebText, our 125M model achieves 19.72 PPL, outperforming standard Transformers (20.60) and GSS hybrid baselines (19.80). These results demonstrate that separating sequence modeling paradigms into parallel specialists enables Transformer-level perplexity with substantially improved efficiency for long-context language modeling.
Tags
Links
- Source: https://arxiv.org/abs/2606.16093v1
- Canonical: https://arxiv.org/abs/2606.16093v1
Trouble viewing inline? Open PDF directly →
Full Text
38,526 characters extracted from source content.
Expand or collapse full text
Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing Kuzey Torlak 1 , H ̈useyin Arda Arslan 2 , Anıl Dervi ̧so ̆glu 3 , Beyza Nur Deniz 4 , and Onur Boyar 5 1 Kadık ̈oy Anadolu High School 2 Politecnico di Torino 3 Istanbul Technical University 4 Bo ̆gazi ̧ci University 5 IBM Research - Tokyo Abstract. Modeling long-range dependencies remains a central chal- lenge in natural language processing. Transformer architectures achieve strong performance via self-attention but scale quadratically (O(N 2 )) with sequence length, while State Space Models (SSMs) scale linearly (O(N )) but suffer from a selective recall bottleneck, struggling to re- trieve precise information from compressed states. This creates a funda- mental tradeoff between efficiency and perplexity. To tackle these chal- lenges, we propose the Parallel Hybrid Architecture (PHA), which runs Gated State Spaces (GSS), Grouped Query Attention (GQA), and Feed- Forward Networks (FFNs) as independent parallel branches fused by a learnable mixing mechanism. Instead of forcing SSMs to approximate attention or serializing the two paradigms, PHA allows each branch to specialize: GSS captures global context, while attention performs selective retrieval, with FFN providing complementary processing. On WikiText-103, PHA achieves 16.51 PPL at 125M parameters, outper- forming Hedgehog (16.70) and H3-125M (23.70). Scaling to 180M pa- rameters yields 16.42 PPL, which gives comparable results with the pure attention baseline while delivering 24% higher throughput and up to 40% lower memory usage at long contexts. On OpenWebText, our 125M model achieves 19.72 PPL, outperforming standard Transformers (20.60) and GSS hybrid baselines (19.80). These results demonstrate that sep- arating sequence modeling paradigms into parallel specialists enables Transformer-level perplexity with substantially improved efficiency for long-context language modeling. Keywords: Natural Language Processing· State Space Models· Trans- formers· Hybrid Architectures· Efficient Inference· Long-Context Mod- eling 1 Introduction Most Large Language Models (LLMs) are built on the Transformer architec- ture [1]. Its core mechanism, Self-Attention, allows for modeling dense depen- arXiv:2606.16093v1 [cs.CL] 15 Jun 2026 2K. Torlak et al. dencies between any two tokens in a sequence, regardless of distance. This prop- erty makes Transformers what we define as Retrieval Experts, which possess the capability to look back and remember specific information (e.g., a name or a number) from the distant past. However, this capability comes at a quadratic cost (O(N 2 )) in both compute and memory, creating a severe bottleneck for applications requiring long context windows, such as document summarization, genomic modeling, or extended dialogue [2]. To circumvent these limitations, Structured State Space Models (SSMs) like S4 [17] and Mamba [3] have emerged. By compressing history into a fixed-size recurrent latent state, SSMs achieve linear scaling (O(N )) during training via parallel scans and constant-time inference (O(1)) per token. Despite these effi- ciency gains, pure SSMs often exhibit a recall bottleneck. Since the entire history must be compressed into a state vector of fixed dimension, information loss is inevitable as the sequence length grows. Consequently, SSMs often struggle with needle-in-a-haystack tasks, e.g. retrieving a specific token from thousands of steps ago, where attention-based models thrive. This trade-off has driven the development of Hybrid Architectures. A promi- nent example is H3 [9], which attempts to mitigate the recall problem by heav- ily engineering the SSM layer itself to approximate the projection operations of Linear Attention. While effective, this approach introduces an architectural complexity, complicating the layer design to mimic a mechanism that standard Attention already performs naturally. Other approaches, like Jamba [14], em- ploy serial stacking (interleaving SSM and Transformer layers), which can still create information bottlenecks where the SSM layers compress the signal before it reaches the attention layers. To address these challenges, we propose an architecture that combines an SSM and an attention mechanism without forcing either component to mimic the other. Instead of arranging them sequentially, the two modules operate in parallel, allowing each to specialize: the SSM maintains long-range syntactic coherence through efficient state propagation, while attention performs sparse, high-fidelity retrieval of relevant tokens. We instantiate this idea in the Par- allel Hybrid Architecture (PHA) (Fig. 1), which integrates Gated State Spaces (GSS) [4], Grouped Query Attention (GQA) [27], and a Feed-Forward Network (FFN) through a Learnable Static Mixing layer that determines the relative con- tribution of each branch to the block output. We primarily validate our approach on the WikiText-103 benchmark and further extend the evaluation to the large- scale OpenWebText corpus to assess its scalability and performance relative to pure SSMs, standard Transformers, and other contemporary hybrid architec- tures. Using these two datasets enables evaluation of the proposed architecture under different levels of difficulty and data scale. Our results show that the proposed method consistently outperforms or ranks among the top-performing approaches across both datasets. Long-Context Hybrid Modeling3 2 Background and Related Work In this section, we first introduce the evaluation metric and task setting used in our study to provide context for the subsequent discussion. We then organize prior work into three categories: Efficient Transformers, State Space Models, and Hybrid Approaches, and discuss the baselines used in our experimental comparisons. 2.1 Evaluation Metric The fundamental task of Causal Language Modeling is to predict the probability of the next token x t+1 given the entire history x 0 ,...,x t . The standard metric for evaluating this capability is Perplexity (PPL). Perplexity is defined as the exponentiated average negative log-likelihood of a sequence: PPL(X) = exp − 1 T T X t=1 logP (x t | x <t ) ! (1) Intuitively, PPL measures how surprised the model is by the text; a lower score indicates better prediction and stronger grasp of context. To rigorously test long-range dependency modeling, we utilize the WikiText-103 benchmark [20]. Unlike sentence-level datasets, WikiText-103 consists of full Wikipedia articles, requiring the model to track dependencies across paragraphs and thousands of tokens to achieve a low perplexity score. 2.2 Architectural Approaches Standard Transformers excel as retrieval experts but suffer from the O(N 2 ) bottleneck. To mitigate this, recurrent variants were introduced. Transformer- XL [15] employs segment-level recurrence, allowing information to propagate beyond fixed windows (achieving ∼18.3 PPL in WikiText-103 dataset) 1 . Com- pressive Transformers [16] improve this by compressing past activations into a memory bank (17.1 PPL). Other optimizations like ALiBi [22] and Adaptive In- put Representations [24] focus on positional encoding and embedding efficiency, yet they do not fundamentally alter the quadratic attention cost. Another major line of research attempts to approximate the Softmax attention map to achieve linear complexity. Linear Transformers [13] use kernel feature maps to express attention as an RNN. While fast, they often suffer from poor precision (31.10 PPL). CosFormer [21] attempts to fix this with cosine-based reweighting (23.10 PPL). Recently, Hedgehog [11] introduced a learnable linear attention that sig- nificantly narrows the gap with standard attention (16.70 PPL), representing the current state-of-the-art in this category. Sparse approaches propose an al- ternative to linear attention by limiting the number of attended tokens. The 1 All PPL values shown in parentheses in this section are reported results on the WikiText-103 dataset from the respective papers. 4K. Torlak et al. Routing Transformer [12] uses k-means clustering to dynamically select relevant tokens, achieving excellent results (15.80 PPL). However, such methods often require complex, hardware-unfriendly implementations (e.g., gather/scatter op- erations) compared to dense matrix multiplications. SSMs, on the other hand, treat sequence modeling as a continuous signal processing problem. S4 [17] uses the Hippo matrix to handle long dependencies efficiently. Although S4 is effec- tive for modeling long-range dependencies, pure S4 (20.95 PPL) struggles with token-level copying behavior that is important for language modeling, where Transformers typically perform better. Other non-attention baselines like AWD- QRNN [26] (33.00 PPL) and convolutional models like Hyena [18] (18.60 PPL) offer alternative O(N logN ) paths but typically trail behind Transformers in retrieval-heavy benchmarks. Recognizing that neither pure Attention nor pure SSMs are perfect, hybrid models have gained traction. Early approaches like H3 [9] attempted to solve the SSM recall weakness by engineering the layer to mimic linear attention (23.7 PPL at 125M). More recently, Hymba [10] proposed a parallel architecture similar to ours, employing concurrent SSM and Attention branches. However, Hymba relies on complex, specialized hybrid heads to manage feature mixing. In the following section, we show that our approach simplifies this process by using learnable scalar weights. 3 Methodology We propose a modular Parallel Hybrid Architecture (PHA) that processes infor- mation through multiple complementary mechanisms operating in parallel. An overview of the architecture is shown in Fig. 1. The model consists of a stack of L identical layers. Given an input sequence x ∈R T×D , we first embed tokens into a D-dimensional space with tied input–output embeddings (i.e., the lan- guage model head shares weights with the embedding layer) and scale by √ D, apply RMSNorm [28], and feed the normalized representation into three parallel branches. To capture long-range dependencies while maintaining stable state updates, we use a GSS layer. Compared with earlier SSM variants such as S4 or Mamba- style models, GSS provides improved training stability in our parallel setting through its gating mechanism, which regulates how much information is incor- porated into the recurrent state. The recurrence is defined as h t = σ(W g x t )⊙ (Ah t−1 + Bx t ),(2) where W g ∈R H×D is the gate projection matrix and H denotes the state dimension. The matrix A represents the structured state transition operator, typ- ically initialized using HiPPO-based methods [29] to capture long-range memory, while B is the input projection matrix. The sigmoid gate σ(·) modulates the state update, allowing the model to control how much new information is incorporated into the hidden state h t . In practice, this mechanism helps maintain a smooth Long-Context Hybrid Modeling5 global context representation, while finer token-level interactions are handled by the attention branch. To capture precise token-level dependencies, we include an attention branch implemented with GQA. The query-to-key/value ratio varies by model scale (see Table 1): the 90M and 125M variants use a 4:1 ratio (H q = 8, H kv = 2), while the 180M variant uses a 3:1 ratio (H q = 12, H kv = 4). In the larger OpenWebText experiments, we use a 4:1 ratio. Compared with standard Multi-Head Attention, this configuration reduces the KV-cache memory footprint by approximately 3× while preserving sufficient representational capacity for complex retrieval patterns. Positional information is encoded using Rotary Positional Embeddings (RoPE) [7] with base frequency θ = 10,000, enabling robust relative position modeling. To further improve numerical stability in deep hybrid stacks, we replace stan- dard dot-product attention with Scaled-Cosine Attention [30], which bounds the attention logits and stabilizes training. Finally, each layer includes a FFN with SwiGLU activation [6]. To balance parameter efficiency and expressiveness, we adopt a Dual-FFN design consist- ing of a primary processing branch with an expansion factor of 2.5× (or 2.0× for larger datasets) and a secondary auxiliary branch with a smaller expansion factor. 3.1 Fusion Strategy: Learnable Static Mixing The presence of multiple parallel branches requires a mechanism to combine their outputs into a single representation. A straightforward solution would be fixed averaging, but this implicitly assumes that all branches contribute equally at every layer. More complex routing strategies, such as Mixture-of-Experts [31], introduce sparsity and load-balancing challenges that are unnecessary in our setting. Instead, we introduce Learnable Static Mixing. We define a set of learn- able scalar parameters p ∈R 3 corresponding to the Attention, GSS, and FFN branches. The normalized mixing weights w are computed using Softplus to ensure positivity: w i = Softplus(p i ) P j∈attn,ssm,ffn Softplus(p j ) .(3) Before mixing, each branch output is independently normalized via RM- SNorm to ensure comparable scales across branches. The fused output y mix is then obtained as the weighted combination of the normalized branch outputs: y mix = X k∈attn,ssm,ffn w k · RMSNorm(z k ).(4) This formulation allows the model to learn the relative importance of each branch through gradient descent, enabling it to emphasize GSS for global context modeling and attention for selective retrieval. 6K. Torlak et al. Input x t RMSNorm Attention Branch GQA (4:1) + RoPE Scaled-Cosine GSS Branch Proj → Gated Recurrence Feature Filter Aux FFN Branch SwiGLU (Expansion 1.25x) Σ Learnable Weights w w gss w attn w ffn + DeepNorm Residual Path RMSNorm Main FFN (Final) SwiGLU (Expansion 2.5x) Output y t Fig. 1. In PHA, the input is processed by three parallel branches: GSS, Attention, and an Auxiliary FFN. These are fused via Learnable Mixing (Σ). The aggregated signal is then processed by a Main FFN at the end of the block. This structure balances parallel feature extraction with sequential depth processing. 3.2 Stabilization: DeepNorm Residual Scaling Training hybrid models can be unstable due to the interaction between re- current state updates and attention dynamics. To stabilize optimization, we adopt DeepNorm-style residual scaling [5], which scales residual connections by α = (2L) 0.25 to bound the magnitude of layer updates. This design enables sta- ble training of deep networks without relying on complex warm-up schedules or gradient clipping, which are often required in SSM-based models. 4 Experimental Setup To validate the efficacy of our Parallel Hybrid architecture, we conducted rigor- ous experiments across two datasets, three model scales, and multiple random seeds. 4.1 Datasets and Tokenization In our first sets of experiments, we evaluate our models on the WikiText-103 benchmark [20]. Text is tokenized using the standard GPT-2 Byte-Pair Encoding Long-Context Hybrid Modeling7 (BPE) tokenizer [8] with a vocabulary of V = 50,257. The corpus is segmented into non-overlapping sequences with a context length of T = 1024. In addition, to assess scalability in a larger and more diverse setting, we addi- tionally train on the OpenWebText corpus [23], which contains web documents collected from URLs linked to Reddit. Following the H3 protocol, we perform document-level splits with 0.5% reserved for validation and 0.5% for testing. 4.2 Model Configurations We evaluate PHA at three parameter scales to study scaling behavior. Table 1 summarizes the configurations. In this table, FFN Expansion denotes the factor by which the hidden layer expands the input dimension. Table 1. Model Configurations. Architectural details for the three PHA variants evaluated. VariantParams d model Layers H q H kv FFN Expansion PHA-90M89.7M48014822.0× / 0.67× PHA-125M 125M60812822.0× / 0.67× PHA-180M 180.5M7681012 42.5× / 1.25× 4.3 Training Protocol All models were trained on a single NVIDIA A100 GPU using PyTorch [33] with full precision (fp32) via the Hugging Face Accelerate library [34]. WikiText-103. Models were trained for 12 epochs using Cross-Entropy loss with the AdamW optimizer. Initial learning rates were set per scale: 4× 10 −4 for PHA-180M, 4.5× 10 −4 for PHA-125M, and 5× 10 −4 for PHA-90M. We used a ReduceLROnPlateau scheduler (factor 0.75, patience 3). The global batch size was 24. Regularization included dropout 0.1, weight decay 0.1, and gradient clipping at norm 1.0. We applied stochastic weight averaging (SWA) [25] by tracking the top-5 checkpoints by validation perplexity and averaging the best three at the end of training. All results are reported as mean ± std across 5 random seeds with deterministic CuDNN settings. OpenWebText. For larger-scale experiments, we trained a ∼125M parameter model for 1 epoch (d model = 608, n layers = 12, n heads = 8, GQA 4:1). FFN expansion factors were reduced to 2× and 0.67×. We used a cosine learning rate schedule with 2,000 warmup steps, peak LR 4.5× 10 −4 , and decay to 10% of the maximum. The batch size was 24. SWA was applied using the Top-5 checkpoint group. 8K. Torlak et al. PHA-90M (5 seeds) PHA-125M (3 seeds) PHA-180M (2 seeds) Model Size 15.5 16.0 16.5 17.0 17.5 18.0 SWA Perplexity (WikiText-103) Hedgehog (125M): 16.70 Routing Trans. (257M): 15.80 17.16 16.49 16.36 Individual seeds Mean ± 1 std Hedgehog baseline Routing Trans. baseline Fig. 2. Individual seed results (circles), means (horizontal bars), and±1 std error bars for each PHA model variants. The 125M and 180M variants consistently outperform Hedgehog (125M), approaching Routing Transformer (257M) quality at a fraction of the parameter count. 5 Results and Analysis In this section, we present our experimental results on WikiText-103 and Open- WebText. We then perform an extensive set of ablation studies to evaluate the contribution of individual components of the proposed parallel hybrid architec- ture. 5.1 WikiText-103 Experiments We benchmark our models against the suite of architectures discussed in Section 2. For our models, we report the mean± std across seeds. Table 2 benchmarks our 125M model. From this table, we observe that our methodology provides better results than various benchmark studies, including another hybrid methodology H3-Hybrid [9] by a large margin. Long-Context Hybrid Modeling9 Table 2. Comparison of ∼125M Parameter Models on WikiText-103. Our PHA-125M outperforms all direct competitors. ModelParams Context Tokenizer Test PPL (↓)Arch. Type Ours (PHA-125M) † 125M1024 GPT-2 BPE 16.51 ± 0.12 Hybrid (GSS+Attn) Hedgehog [11]125M1024GPT-2 BPE16.70Linear Attn Hyena-3-slim [18]125M2048GPT-2 BPE18.50Convolutional Hyena-3 [18]125M2048GPT-2 BPE18.60Convolutional H3-Hybrid (125M) [9]125M2048GPT-2 BPE23.70Hybrid SSM Linear Transformer [13]90M384Word-level*31.10Linear Attn AWD-QRNN [26]185M-Word-level*33.00RNN GPT-2 Small [19]117M1024BPE37.50Transformer BERT-CAS [32]140M512WordPiece39.85Encoder-Decoder We also compared our approach against models with more than 200M param- eters. Table 3 compares our 180M model against significantly larger baselines. Notably, many of these baselines use Word-level tokenizers, which typically yield numerically lower perplexity than BPE (as the vocabulary is smaller). Despite this disadvantage, our BPE-based model remains highly competitive. Table 3. Comparison with Larger Baselines (200M+). Despite having fewer parameters (180M), our model outperforms major baselines like S4, Transformer-XL, and Compressive Transformer. ModelParams Context Tokenizer Test PPL (↓) Arch. Type Routing Transformer [12]257M4096Word-level*15.80Sparse Attn Ours (PHA-180M) † 180M1024 GPT-2 BPE 16.42 ± 0.09Hybrid H3-355M [9]355M2048GPT-2 BPE16.90Hybrid SSM Compressive Transformer [16] 257M2048Word-level*17.10Recurrent Transformer-XL [15]257M1600Word-level*18.30Recurrent ALiBi Transformer [22]247M3072Word-level*18.30Transformer Adaptive Input Rep. [24]247M2560Word-level*18.70Transformer BERT-Large-CAS [32]395M512WordPiece20.42Transformer S4 [17]249M8192GPT-2 BPE20.95Pure SSM CosFormer [21]200M+512Word-level*23.10Linear Attn Standard Softmax Trans. [21] 477M512Word-level*24.92Transformer In addition to above experiments, to demonstrate the model’s scaling capa- bility across different model sizes, along with 125M and 180M parameter models, we also experimented with 90M parameter model and tested them in WikiText- 103 dataset. Figure 2 shows the scaling behaviour of the proposed PHA and benchmarks it against 125M Hedgehog model, while Table 4 presents PPL val- ues along with the accuracy. Results show that PHA performance increases as the number of parameters increase. In addition, Fig. 3 compares perplexity on WikiText-103 with model size across several approaches in the literature. The results show that PHA achieves competitive performance while using fewer pa- rameters than many existing methods. 10K. Torlak et al. Table 4. Scaling Behavior of PHA. Perplexity improves consistently from 90M to 180M. The 125M→180M jump (+55M params) reduces PPL by only 0.09, suggesting diminishing returns and that the 125M variant offers the best quality-per-parameter ratio at this training budget. ModelParams Config SWA PPL (↓) Accuracy (%) PHA-90M † 89.7M 480d × 14L 17.16 ± 0.2445.95 ± 0.21 PHA-125M † 125M 608d × 12L 16.51 ± 0.1246.62 ± 0.12 PHA-180M † 180.5M 768d × 10L 16.42 ± 0.0946.78 ± 0.13 100150200250300350400450500 Parameters (M) 15 20 25 30 35 40 Perplexity (WikiText-103) PHA-90M PHA-125M PHA-180M Linear Trans. GPT-2 Small Hedgehog Hyena-3-slim Hyena-3 H3-125M BERT-CAS AWD-QRNN CosFormer S4 Adaptive Input ALiBi Trans. Transformer-XL Compressive Trans. Routing Trans. H3-355M BERT-Large-CAS Std. Softmax Trans. Parameter Efficiency: Perplexity vs. Model Size Transformer SSM Linear Attn RNN Ours Fig. 3. Parameter Efficiency: Perplexity vs. Model Size. All PHA variants (stars) lie on the Pareto frontier of best-performing models. The dashed line traces the best models across scales. PHA-125M matches Hedgehog at 1.3× fewer perplexity points while PHA-180M approaches Routing Transformer quality at 70% of the pa- rameter count. 5.2 OpenWebText Experiments To validate our architecture in a large-data regime, we evaluated the 125M vari- ant on the OpenWebText corpus following the H3 methodology [9]. All baseline values are sourced from [9], where every model (Transformer, GSS, and H3 vari- ants) was trained for 100K steps with a batch size of 512 and sequence length 1024, accumulating approximately 50 billion tokens. Our model, by contrast, was trained for a single epoch with a batch size of 24, seeing approximately 12 billion tokens in total, roughly 23% of the baseline token budget. Long-Context Hybrid Modeling11 Despite this significantly reduced training budget, PHA-125M achieves a per- plexity of 19.72 using SWA over the Top-5 checkpoints, outperforming the stan- dard Transformer (20.60) and the GSS Hybrid (19.80), while remaining com- petitive with the H3 Hybrid (19.60). Table 5 summarizes these results. The fact that our architecture reaches competitive performance with less than a quarter of the training tokens suggests that the parallel hybrid design offers improved sample efficiency relative to both pure and serial hybrid baselines. Table 5. OpenWebText Evaluation (125M Class). All baselines were trained on ∼50B tokens; our model was trained on∼12B tokens (1 epoch). Despite seeing roughly 4× fewer tokens, PHA-125M outperforms pure SSMs and the standard Transformer. ModelParamsTokens SeenTest PPL (↓) GSS [4]∼125M ∼50B24.00 H3 [9]∼125M ∼50B21.00 Transformer∼125M ∼50B20.60 GSS Hybrid [4] ∼125M ∼50B19.80 H3 Hybrid (125M) [9] ∼125M ∼50B19.60 Ours (PHA-125M) 125M ∼12B19.72 Overall, results on WikiText-103 and OpenWebText demonstrate that PHA achieves competitive or superior performance across both benchmarks. On Open- WebText in particular, the model’s strong showing at a fraction of the training compute suggests that the parallel hybrid design extracts more useful signal per token than conventional architectures, making it a promising direction for compute-constrained settings. 5.3 Ablation Studies We conduct a set of controlled ablations on WikiText-103 to isolate three ques- tions: (i) whether GSS and Attention should be composed in parallel or se- quentially, (i) how much each branch contributes to language modeling perfor- mance, and (i) why the full hybrid remains preferable even when Attention- Only achieves similar perplexity. Unless otherwise noted, all variants are trained with matched hyperparameters at the ∼180M scale. Composition Strategy: Parallel vs. Sequential A central design choice in PHA is to run GSS and Attention as independent parallel branches with learn- able mixing, rather than stacking them in a fixed serial order. To test whether this composition itself matters, we construct a Sequential Hybrid baseline using the same components—GSS, GQA with the same head configuration, SwiGLU FFN, DeepNorm, RoPE, and Scaled-Cosine Attention—but arranged in a strict serial pipeline: x Norm −→ Attn +residual −→ Norm GSS −→ +residual −→ Norm FFN −→ +residual −→ x ′ . (5) 12K. Torlak et al. In this design, GSS only sees attention-transformed representations, FFN only sees GSS-transformed features, and the relative contribution of branches is fixed by the ordering. To match the parameter budget, the sequential variant uses 12 layers (vs. 10 for PHA), totaling 185.8M parameters. Table 6 shows that the parallel design performs slightly better despite being smaller: PHA-180M achieves lower SWA perplexity (16.42 vs. 16.46) with 5.3M fewer parameters, while matching the sequential model in best-checkpoint per- plexity. Although the gap is modest, it is notable because the sequential baseline is deeper, which typically favors serial architectures. Table 6. Parallel vs. Sequential Composition on WikiText-103. Both models use identical components (GSS, GQA, SwiGLU, DeepNorm) at a comparable param- eter budget. The parallel design achieves lower perplexity with fewer parameters, sup- porting the use of architectural independence. † Mean ± std across seeds. CompositionParams Config SWA PPL (↓) Best-Ckpt PPL Acc (%) Parallel (PHA-180M) † 180.5M 768d × 10L 16.42 ± 0.0916.9546.78 Sequential185.8M 768d × 12L16.4616.9546.93 The result suggests that the benefit comes not just from the component set, but from how the components interact. In PHA, all branches receive the same normalized input and can specialize independently, while the learned mixer adapts their relative importance by layer. In contrast, the sequential design con- strains information flow through a fixed pipeline. Branch Contribution Having established that parallel composition is benefi- cial, we next ask which branches are most important within that parallel struc- ture. Starting from the full PHA-180M model, we train variants that remove one or more branches while keeping the overall budget comparable. We also include an Attention-Only baseline as the natural endpoint obtained by removing both GSS and the auxiliary FFN. Table 7. Branch Removal Ablation (PHA-180M on WikiText-103). Each row removes one or more branches from the full architecture while maintaining a compa- rable parameter budget. ∆ PPL denotes the gap relative to the full model. Removing Attention causes by far the largest degradation, while removing GSS or the auxiliary FFN yields more modest losses. † Mean ± std across seeds. ConfigurationBranches KeptSWA PPL (↓) ∆ PPL Removed Attention-Only (Transformer) Attention16.32–0.10 GSS + Aux FFN Full PHA-180M † GSS + Attention + FFN 16.42 ± 0.09— Attention + FFN (–GSS)Attention + Aux FFN16.82+0.40 GSS Attention + GSS (–FFN)Attention + GSS17.12+0.70 Aux FFN GSS + FFN (–Attention)GSS + Aux FFN20.69+4.27 Attention GSS Only (–Attn –FFN)GSS21.30+4.88 Attention + Aux FFN Long-Context Hybrid Modeling13 These extensive analyses and ablation studies clarify the role of each design choice in PHA. Parallel composition is preferable to sequential stacking even when using identical components, indicating that architectural independence it- self is beneficial. Attention provides the critical retrieval capability required for competitive perplexity, while GSS and the auxiliary FFN improve the computa- tional tradeoff around this capability. At the same time, the small gap between the full model and Attention-Only (16.42 vs. 16.32) suggests an important nuance: at this scale, Attention alone is sufficient for near-optimal perplexity. This naturally raises the next question: if Attention-Only is so strong, why use the hybrid at all? The answer is efficiency. Although PHA and the Attention-Only baseline achieve near-identical perplexity, the hybrid is substantially more efficient at longer contexts because it offloads a large fraction of sequence processing to the linear-time GSS branch. Table 8 shows that the gap widens with sequence length: at 8192 tokens, PHA is 23.9% faster, while also using 22.0% less memory. The memory benefit is largest at shorter contexts, reaching 39.6% at 2048 tokens. Table 8. Efficiency: PHA-180M vs. Attention-Only Transformer. Throughput and memory comparison across sequence lengths. Both models achieve near-identical perplexity, but PHA is consistently more efficient, especially at longer contexts. Seq Length Throughput (tok/s) ↑Memory (GB) ↓Hybrid Gain Attn-Only HybridAttn-Only HybridSpeed Mem. 204832,62034,4092.121.28+5.5% –39.6% 409631,38235,2762.701.86+12.4% –31.1% 819226,68333,0583.873.02+23.9% –22.0% 16384N/AN/A6.195.34N/A–13.7% This computational advantage is consistent with the learned mixing behavior in Table 9. The mixing weights show a stable “Sandwich” pattern: GSS domi- nates near the input and output layers, while Attention becomes more promi- nent in the middle layers. Thus, branch utilization and branch criticality are not the same. Attention is the indispensable retrieval expert, but GSS is the main workhorse that carries much of the routine sequence processing when both are available. This explains why the full hybrid can match Transformer-level perplexity while being more efficient. Learnable Mixing Beyond removing individual components, it is also infor- mative to examine how the model allocates computation across branches during training. A natural question is whether the mixer collapses to a single dominant branch or instead distributes weight across branches in a structured way. To investigate this, we analyze the learned mixing weights of PHA-180M across lay- ers and multiple checkpoints (Steps 48k–58k). The distribution remains stable throughout training, indicating that the mixing mechanism does not collapse to a single branch. 14K. Torlak et al. 5121K2K4K8K Sequence Length 17k 20k 22k 25k 27k 30k 32k 35k Throughput (tokens/sec) (a) Throughput Parallel Hybrid (Ours) Sequential Hybrid Attention-Only 5121K2K4K8K16K Sequence Length 1 2 3 4 5 6 Peak Memory (GB) (b) Peak Memory Parallel Hybrid (Ours) Sequential Hybrid Attention-Only Inference Efficiency: Throughput and Memory vs. Sequence Length Fig. 4. Inference Efficiency: Throughput and Memory vs. Sequence Length. The hybrid model maintains higher throughput and lower memory usage than the Attention-Only baseline, with the throughput advantage increasing at longer contexts. Table 9 reveals a clear layerwise specialization pattern. GSS receives the largest weight near the input and output layers, while Attention becomes most prominent in the middle layers. This forms a functional “Sandwich” structure: GSS dominates the boundaries where global context is established and consoli- dated, while Attention becomes more active in intermediate layers where selec- tive retrieval from the context is most useful. The auxiliary FFN consistently receives a smaller but non-negligible share, suggesting a supporting processing role rather than serving as the primary information pathway. Table 9. Layer-wise Branch Weight Analysis (PHA-180M). The learned mixer discovers a stable “Sandwich” specialization: GSS dominates near the boundaries to anchor global context, while Attention becomes more prominent in the middle layers for selective retrieval. Average across layers: GSS 46.9%, Attention 31.2%, FFN 21.9%. LayerGSS (Context) Attention (Retrieval) FFN (Processing) L054.0%29.1%17.0% L153.2%20.0%26.8% L252.4%22.7%24.9% L343.9%33.1%23.1% L439.9%41.4%18.7% L542.5%37.7%19.9% L639.1%41.9%19.1% L741.8%37.9%20.3% L847.6%29.7%22.7% L954.6%18.9%26.5% Average46.9%31.2%21.9% This analysis also helps interpret the branch-removal results. Although GSS receives the largest average weight, utilization and necessity are distinct prop- Long-Context Hybrid Modeling15 erties: Attention provides the indispensable retrieval capability, while GSS acts as the primary computational workhorse when Attention is present. Taken together, the learned mixing behavior supports the central design intu- ition of PHA: the branches specialize across depth and cooperate in a structured way, rather than operating redundantly. Parallel composition is preferable to se- quential stacking even with identical components, Attention provides the critical retrieval capability, and GSS improves the computational tradeoff around it. As a result, PHA achieves Transformer-level perplexity while replacing a substantial portion of quadratic computation with linear-time processing. 6 Conclusion We introduced the PHA, which combines Gated State Spaces and Grouped Query Attention through a learnable static mixing mechanism to address long- context modeling. Experiments on WikiText-103 across multiple model scales show that PHA achieves strong performance in both quality and efficiency. Our 125M model surpasses the H3-125M baseline by over 7 perplexity points, while the 180M model outperforms H3-355M despite using roughly half the parameters. At the same time, PHA matches the quality of pure attention models while delivering higher throughput and lower memory usage at long context lengths. Controlled ablations confirm that the parallel design is superior to sequential hybridization, improving both perplexity and runtime efficiency. Further analysis reveals complementary roles between the branches: attention provides critical token-level retrieval, while the state-space component carries the majority of the computation for global context modeling. Finally, consistent performance across multiple random seeds and model scales demonstrates that the proposed architecture is robust and scalable. Addi- tional experiments on OpenWebText further validate these findings, where the 125M model achieves competitive perplexity while maintaining the efficiency ad- vantages of the hybrid design. Overall, our results suggest that combining spe- cialized sequence modeling mechanisms in parallel offers a practical and scalable approach for efficient long-context language modeling. References 1. Vaswani, A., et al.: Attention is All You Need. NeurIPS (2017) 2. Tay, Y., et al.: Efficient Transformers: A Survey. arXiv:2009.06732 (2020) 3. Gu, A., Dao, T.: Mamba: Linear-Time Sequence Modeling with Selective SSMs. arXiv:2312.00752 (2023) 4. Nguyen, T., et al.: Gated State Space Models. arXiv (2024) 5. Wang, H., et al.: DeepNet: Scaling Transformers to 1,000 Layers. arXiv:2203.00555 (2022) 6. Shazeer, N.: GLU Variants Improve Transformer. arXiv:2002.05202 (2020) 7. Su, J., et al.: RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864 (2021) 16K. Torlak et al. 8. Sennrich, R., et al.: Neural Machine Translation of Rare Words with Subword Units. ACL (2016) 9. Fu, D., et al.: Hungry Hungry Hippos: Towards Language Modeling with SSMs. arXiv:2212.14052 (2023) 10. Dong, B., et al.: Hymba: A Hybrid-Head Architecture. arXiv:2411.00772 (2024) 11. Zhang, M., et al.: The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. arXiv:2402.04347 (2024) 12. Roy, A., et al.: Efficient Content-Based Sparse Attention with Routing Transform- ers. TACL (2020) 13. Schlag, I., et al.: Linear Transformers Are Secretly Fast Weight Programmers. ICML (2021) 14. Lieber, O., et al.: Jamba: A Hybrid Transformer-Mamba Model. arXiv:2403.19887 (2024) 15. Dai, Z., et al.: Transformer-XL: Attentive Language Models. arXiv:1901.02860 (2019) 16. Rae, J.W., et al.: Compressive Transformers. arXiv:1911.05507 (2019) 17. Gu, A., et al.: Efficiently Modeling Long Sequences with SSMs. arXiv:2111.00396 (2021) 18. Poli, M., et al.: Hyena Hierarchy: Towards Larger Convolutional Language Models. arXiv:2302.10866 (2023) 19. Radford, A., et al.: Language Models are Unsupervised Multitask Learners. Ope- nAI (2019) 20. Merity, S., et al.: Pointer Sentinel Mixture Models. arXiv:1609.07843 (2016) 21. Qin, Z., et al.: CosFormer: Rethinking Softmax in Attention. ICLR (2022) 22. Press, O., et al.: Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR (2022) 23. Gokaslan, A., Cohen, V.: OpenWebText Corpus. http://Skylion007.github.io/ OpenWebTextCorpus (2019) 24. Baevski, A., Auli, M.: Adaptive Input Representations for Neural Language Mod- eling. ICLR (2019) 25. Izmailov, P., et al.: Averaging Weights Leads to Wider Optima and Better Gener- alization. UAI (2018) 26. Merity, S., et al.: An Analysis of Neural Language Modeling at Multiple Scales. arXiv:1803.08240 (2018) 27. Ainslie, J., et al.: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP (2023) 28. Zhang, B., Sennrich, R.: Root Mean Square Layer Normalization. NeurIPS (2019) 29. Gu, A., et al.: HiPPO: Recurrent Memory with Optimal Polynomial Projections. NeurIPS (2020) 30. Liu, Z., et al.: Swin Transformer V2: Scaling Up Capacity and Resolution. CVPR (2022) 31. Shazeer, N., et al.: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR (2017) 32. Wang, A., et al.: BERT Has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. arXiv:1902.04094 (2019) 33. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 34. Hugging Face. Accelerate: Training and inference at scale. https://github.com/ huggingface/accelerate, 2022.