Paper deep dive
ELiTeFormer: An Efficient Transformer for FPGAs
Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/7/2026, 9:01:56 AM
Summary
ELiTeFormer is a novel Transformer architecture co-designed for FPGA deployment, combining hybrid linear attention with ultra-low-precision ternary linear projections. It utilizes a custom processing element micro-architecture that eliminates multiplications via bitmasking, completely avoiding DSP blocks. Evaluated against LLaMA 3, it achieves significant weight and KV cache compression, competitive MMLU accuracy, and superior latency and energy efficiency on Xilinx VCK5000 Versal FPGAs using high-level synthesis.
Entities (9)
Relation Signals (10)
ELiTeFormer → iscomparedto → LLaMA 3
confidence 97% · compared to LLaMA 3, while maintaining competitive accuracy
ELiTeFormer → isdeployedon → Xilinx VCK5000 Versal
confidence 96% · deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board
LLaMA 3 → runson → NVIDIA A100
confidence 95% · than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU)
ELiTeFormer → uses → Hybrid Linear Attention
confidence 95% · unifies hybrid linear attention with ultra-low-precision (ternary) linear projections
ELiTeFormer → isevaluatedon → MMLU
confidence 94% · 31.9% on the MMLU benchmark
ELiTeFormer → uses → Ternary Quantization
confidence 94% · leverages BitNet b1.58-style ternary-quantized linear projections
ELiTeFormer → isimplementedusing → High-Level Synthesis (HLS)
confidence 93% · using high-level synthesis (HLS) flows
ELiTeFormer → achievesbetterlatencythan → LLaMA 3
confidence 92% · achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3
Processing Element (PE) → eliminates →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.
Tags
Links
- Source: https://arxiv.org/abs/2607.03652v1
- Canonical: https://arxiv.org/abs/2607.03652v1
Trouble viewing inline? Open PDF directly →
Full Text
77,978 characters extracted from source content.
Expand or collapse full text
ELiTeFormer: An Efficient Transformer for FPGAs Victor Agostinelli Oregon State UniversityCorvallisOregonUSA Pacific Northwest National LaboratoryRichlandWashingtonUSA agostiniv@oregonstate.edu , Nicolas Bohm Agostini Pacific Northwest National LaboratoryRichlandWashingtonUSA nicolas.agostini@pnnl.gov and Antonino Tumeo Pacific Northwest National LaboratoryRichlandWashingtonUSA antonino.tumeo@pnnl.gov Abstract. Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10×10× model weight compression and 12.8×12.8× key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6×9.6× speedup for FFN operations and 4.4×4.4× speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9×3.9× lower latency and 3.2×3.2× better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration. †copyright: none†conference: ; ; 1. Introduction Transformer-based large language models (LLMs) have seen explosive growth in both industry and academia, with rapid progress in model capabilities and widespread adoption across domains. LLM use spans a diverse range of applications, including question/answering and general conversational artificial intelligence (AI) (OpenAI, 2024), assisting in image generation (Liu et al., 2024), code generation and analysis (Jiang et al., 2024), basic sentiment analysis (Zhang et al., 2024c), and more. As these models scale in size and complexity, delivering efficient compute and memory solutions has become a significant challenge. This is particularly true in scenarios demanding low latency and energy efficiency, such as real-time inference and edge deployment. Even in datacenter settings, reducing the cost and energy footprint of inference at scale remains critical. On the algorithmic side, researchers have proposed several alternatives to address inefficiencies. These efforts typically focus on optimizing individual components such as attention or feed-forward network (FFN) blocks. Efficient attention mechanisms aim to subquadratic scaling with context length, including sparse attention (Child et al., 2019; Tay et al., 2020; Zaheer et al., 2020) and linear attention (Katharopoulos et al., 2020; Agostinelli and Chen, 2023; Yang et al., 2024a; Agostinelli et al., 2024). FFN efficiency has been pursued via reduced-precision arithmetic (Dettmers et al., 2022; Wang et al., 2023; Ma et al., 2024) and fine-grained mixture-of-experts approaches (Chen et al., 2022). However, these optimizations are often explored in isolation. To the best of our knowledge, no prior work has jointly optimized both the attention and FFN blocks with co-designed hardware architectures tailored to exploit both algorithmic innovations simultaneously. Recent work explored hardware acceleration for Transformer models and their lightweight variants in custom silicon or with reconfigurable devices (Ham et al., 2020; Wang et al., 2021; Fan et al., 2022; Zeng et al., 2024). Field-programmable gate arrays (FPGAs) provide the ability to implement highly customized and performant accelerators for these Transformer-based models. Application-specific integrated circuits (ASICs) generally achieve even higher performance and superior energy efficiency. However, deep learning models evolve quickly, often rendering fixed-function accelerators obsolete. The reconfigurability of FPGAs provides the flexibility needed to track these rapid shifts, rendering them attractive despite their performance gap relative to ASICs. Prior work (Zeng et al., 2024; Chen et al., 2024) has shown that, especially for generative inference, FPGAs can outperform GPUs by exploiting fine-grained pipelining and dataflow parallelism. However, these prior FPGA-based approaches inherit inefficiencies from general-purpose Transformer designs. Notably, they attempt to support both prefilling and generative stages on a unified architecture, which leads to increased complexity and poor key-value (KV) cache management. Furthermore, they accelerate only standard attention mechanisms without leveraging algorithmic innovations in efficient attention or ultra-low-precision FFNs. These architectural and algorithmic mismatches hinder the ability to fully realize the performance and energy benefits of FPGA-based deployment. To address these limitations, we propose a hardware-software (HW/SW) co-design approach centered around a custom, Transformer-based model optimized for both algorithmic and hardware efficiency. We introduce ELiTeFormer—the Efficient Linear Ternary Transformer—a novel architecture designed to overcome the bottlenecks observed in existing FPGA-targeted LLM deployments. For the first time, ELiTeFormer jointly optimizes the attention and FFN blocks by combining recent advances in efficient attention and extremely low-precision weights, and co-designs specialized hardware to exploit these algorithmic innovations. It employs a hybrid-linear attention mechanism with a sliding window, trained via LoLCATs (Zhang et al., 2024a) for efficient long-context modeling via distillation. Simultaneously, it leverages BitNet b1.58-style ternary-quantized linear projections (Ma et al., 2024) in both the attention and FFN components. These low-precision operations are well-suited for FPGAs, which can natively support arbitrary bit-width computation and flexible datapaths. To exploit this, we develop a custom processing element (PE) micro-architecture that eliminates multipliers entirely, replacing them with efficient bitmasking operations. The design is synthesized using high-level synthesis (HLS) flows (Ferrandi et al., 2021; Inc., 2025), simulated for performance analysis, and deployed on Xilinx FPGAs. This work makes the following key contributions: • We propose ELiTeFormer, to the best of our knowledge, the first Transformer-based model that jointly combines hybrid-linear attention and ultra-low-precision linear projections. This enables 10× model weight compression and 12.8× or more KV cache compression in long-context scenarios compared to LLaMA 3 (Grattafiori et al., 2024), a state-of-the-art open-weight LLM. We accomplish this while incurring only a 3.0% accuracy drop on the MMLU benchmark (Gao et al., 2024) • We present ELiTeFormer PE (ELTF PE), a novel PE micro-architecture that accelerates multiply-and-accumulate (MAC) operations for BitNet b1.58-styled linear projections. ELTF PE avoids ternary multiplication via bit-masking, eliminating the usage of dedicated FPGA DSP blocks for linear projections, and is the first ternary PE applied towards Transformer workloads. • We synthesize, simulate, and deploy an accelerator platform specialized for ELiTeFormer using HLS flows (Ferrandi et al., 2021; Inc., 2025) on Xilinx FPGA-based data-center accelerator boards. To the best of our knowledge, this is the first custom hardware design and FPGA realization that includes both an efficient linear-attention and BitNet b1.58-styled linear projections. • We demonstrate that ELiTeFormer achieves significant latency improvements in simulation and deployment. We achieve a 9.6×9.6× speedup on the linear projections and a 4.4×4.4× speedup on the attention block when compared to quadratic references in simulation. On hardware, when deploying a scaled-down ELiTeFormer on a Xilinx VCK5000 board, we achieve up to a 3.9×3.9× latency speedup with long context lengths and 3.2×3.2× better average energy efficiency than a similarly scaled version of a LLaMA 3 LLM running on an NVIDIA A100 GPU. 2. Background and Motivation In this section, we review the Transformer model design (Vaswani et al., 2017), including a brief overview of the attention mechanism and key concepts in efficient, alternative operators. We also discuss a major challenge of accelerating attention on FPGAs: unbounded generation. 2.1. Transformer Basics Transformers blocks are formed, fundamentally, by symmetrical layers containing attention blocks, an FFN, and normalization blocks. We focus on decoder-only implementations (Radford et al., 2018) in this example. Equation 1 describes the output y as a function of input x and a single layer of the original Transformer architecture in a more formal manner, where the self-attention block is denoted as a, the FFN block is denoted as FFNFFN, and wrapped by normalization blocks (typically LayerNorm) denoted as LNLN. (1) x¯(x)=LN(x+a(x))y(x)=LN(x¯+FFN(x¯)) split x(x)=LN(x+a(x))\\ y(x)=LN( x+FFN( x)) split The attention and FFN blocks are computationally demanding and are commonly the focus of efficiency optimizations. Decoder-only Transformers typically have two phases during inference: prefilling and decoding (hereafter called generative inference or generation). Prefilling takes a series of contextual vectors, analogous to a prompt for an LLM, and generates embedded or hidden representations that are saved in the KV cache. This context is then used for attention during generative inference, where the model generates a single token at each time-step and adds that token’s hidden representations to the KV cache as context for the next time-step. 2.2. Attention and Efficient Variants Here, we cover the basics of classical attention as introduced by Vaswani et. al (Vaswani et al., 2017) in addition to a range of efficient variants for this operator, including sparse attention (Child et al., 2019) and linear attention (Katharopoulos et al., 2020). 2.2.1. Classical Attention The attention mechanism, as originally introduced in Transformers (Vaswani et al., 2017) can be expressed in a simplified form for self-attention as seen in Equation 2 on a per-attention head and time-step basis (before concatenation). In this example, we define qh,t∈ℝ1×dkq_h,t ^1× d_k, Kh∈ℝt×dkK_h ^t× d_k, and Vh∈ℝt×dvV_h ^t× d_v. The head dimensionality is referred to as dkd_k and dvd_v in this example and is called d when unified for qh,tq_h,t, Kh,tK_h,t, and Vh,tV_h,t for the sake of simplicity. t refers to a given time-step during generation and also corresponds to the sequence length of Kh,tK_h,t and Vh,tV_h,t, which contain cached tokens from previous time-steps up to t. h corresponds to some h∈Hh∈ H where H is the set of available attention heads. (2) ah,t=softmax(qh,tKh,tTd)Vh,ta_h,t=softmax ( q_h,tK^T_h,t d )V_h,t Classical attention has quadratic O(n2)O(n^2) cost in both time and memory footprint for long sequences, driven by the qh,tKh,tTq_h,tK^T_h,t interaction. This has led to strong interest in more efficient alternatives. 2.2.2. Sparse Attention Sparse attention leverages sparsity in the attention activations to reduce the computational complexity of attention operations down to a constant cost. Popular sparse attention variations take many forms (Child et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020), most approaches rely on fixed, structured sparsity patterns that map well to general-purpose hardware and reduce memory traffic. More dynamic sparsity patterns are possible, but usually require custom hardware to materialize their theoretical efficiency gains. 2.2.3. Linear Attention Another efficient attention alternative, linear attention (Katharopoulos et al., 2020) attempts to reformulate the enforced order of computation in Equation 2 with a lossy approximation by replacing the expexp in the softmax(…)softmax(...) operator with some similarity-measuring function or kernel that is separable. After this substitution, the attention operator’s calculation can be reordered to form a linear operator with respect to the length of the sequence in terms of runtime and memory footprint. This can be observed in Equation 3, where kh,tk_h,t represents a vector at row t in Kh,tK_h,t and x~ x represents an approximation of vector or matrix x. We drop d d scaling for simplicity. (3) ah,t≈a~h,t=q~h,t(K~h,tTVh,t)q~h,t∑j=0tk~h,jTa_h,t≈ a_h,t= q_h,t( K^T_h,tV_h,t) q_h,tΣ^t_j=0 k^T_h,j During casual training and generation, Equation 3 can be modeled as a recurrent function with a constant-sized state. Hybrid attention variants featuring linear attention alongside sparse attention (Zhu et al., 2021; Arora et al., 2024; Dong et al., 2024) have gained popularity recently and represent promising prospects in terms of model quality. 2.3. Low-Precision FFNs FFNs are a common component in many neural network architectures, and their optimization has been widely studied. In particular, low-precision FFNs (Wang et al., 2023; Ma et al., 2024; Dettmers and Zettlemoyer, 2023) offer a promising and efficient alternative. These variants reduce memory footprints through quantized representations. When low-precision computation is supported by hardware, these compression schemes can provide significant theoretical speedups. However, these gains are rarely realized on general-purpose hardware. One recent implementation, BitNet b1.58 (Ma et al., 2024), proposes replacing the weights of large scale models with purely ternary representations (i.e., discretized to [−1,0,1][-1,0,1]). While this extreme quantization results in some model quality loss, models at scale are often able to overcome even significant approximation error due to redundant logical pathways during inference (similar to the intuition behind why model pruning works well (Sun et al., 2024)) especially with additional training. 2.4. A Challenge: Unbounded Generation and FPGAs FPGA-based Transformer accelerators often target vision tasks (Rahman et al., 2016; Wang et al., 2022), partly due to their popularity and partly to avoid a key engineering challenge: static compilation on FPGAs complicates support for dynamic workloads such as unbounded generation with a growing context window (Zeng et al., 2024). One common approach for general acceleration is to tile input data using a host processor during the prefilling stage (Chen et al., 2024; Li et al., 2020; Khan et al., 2021). However, this still requires provisioning large buffers to handle a range of possible key-value matrix sizes during attention, which can be costly (Chen et al., 2024; Li et al., 2020). Another explored solution is to design a custom instruction set architecture (ISA) that supports different datapath configurations. While flexible, this approach can introduce significant overhead. For example, FlightLLM (Zeng et al., 2024) requires 3.25 GB of DRAM just to support its ISA. In general, existing techniques for prefilling in Transformer-based LLMs on FPGAs adequately address concerns with high utilization and middling waste. Other works even propose employing heterogeneous systems for prefilling (Chen et al., 2024), focusing on aligning architectural strengths of different accelerators with different workload phases. Unfortunately, existing designs that attempt to unify prefilling and generation within the same hardware solution are necessarily inefficient during generation, as generation imposes additional overhead to support different runtime configurations. Furthermore, adapting these approaches is non-trivial, as many are handcrafted and tightly coupled to low-level implementations, making them difficult to retarget or generalize with HLS flows that leverage high-level algorithmic descriptions. Ultimately, we argue that current efforts to accelerate attention during generation rely on algorithms that are mismatched to FPGA architectures, overlooking alternatives that align more naturally with the strengths of fine-grained reconfigurable hardware. This understanding should underscore the HW/SW co-design philosophy and efforts advocated for in recent work (Zeng et al., 2024; Chen et al., 2024). Figure 1. Depiction of ELiTeFormer’s attention block during generative time-steps. Hedgehog Linear Attention (Zhang et al., 2024b) + Sliding Window Sparse Attention (Child et al., 2019) are executed in parallel for each attention head. 3. ELiTeFormer Model and FPGA Implementation As mentioned previously, accelerating efficient Transformer variants in hardware is not new and is sometimes considered critical for the viability of a proposed solution. For example, fairly dynamic attention sparsity can be better exploited in ASIC and FPGA-based solutions than when an architecture is deployed on general-purpose hardware (Fan et al., 2022; Zeng et al., 2024; Shen et al., 2022). However, prior works have not necessarily modified algorithms and/or model architectures enough to guarantee suitability for a given custom hardware platform. Moreover, such works often attempt to leverage FPGAs for prefilling when GPUs are already well-suited for the task. As such, we focus only on generation moving forward and assume ELiTeFormer is deployed in a heterogeneous environment that features separate acceleration platforms for prefilling and generation. 3.1. An Algorithmic Fit for FPGAs When it comes to FPGAs, two factors are critical for a given Transformer algorithm variant to be easily adaptable and viable. First, a given variant should be computationally efficient in that it should avoid abusing runtime and memory footprints. At large deployment scales, more efficient attention and FFN variants can iterate more quickly, process multiple queries simultaneously via large batches, and potentially save on energy costs. Alternatively, at smaller scales or on the edge, hard runtime and memory constraints may be required to be met. Additionally, a given variant should be near or completely static in its memory footprint. This eliminates any possible waste from length-adaptive overhead required to support unbounded generation. From this perspective, linear attention and specific sparse/linear hybrid architectures represent interesting prospects. As referenced in Section 2.2.1, the computational profile of linear attention during generation is determined purely by model hyperparameters. It is thus calculable a priori for static compilation. Some sparse/linear hybrid attention variants are similarly determined (Arora et al., 2024; Dong et al., 2024). These architectures are useful for their efficiency and are demonstrably static. As such, we believe that they should be more widely used on FPGAs for generative workloads. Unfortunately, after replacing an attention operator with an efficient alternative, further efficiency gains are usually limited without also optimizing or substituting the basic Transformer FFN block. Thus, we observe that optimizations here are somewhat symbiotic. Efficiency-focused substitutions in one block encourage efficiency-focused substitutions in the other, resulting in additional speed-ups and reduced memory footprints. For this reason, we also propose applying BitNet b1.58 (Ma et al., 2024) to the FFN block, representing the first time a Transformer-based model has featured both an efficient attention mechanism and a ternary FFN and, to the best of our knowledge, the first time that linear attention or BitNet b1.58 has been synthesized and deployed on an FPGA. 3.2. Pushing the Pareto Frontier of Efficient Models at Scale When combining attention and FFN architectural optimizations, we can push the Pareto frontier of efficiency. We propose a novel, ultra-efficient generative model architecture, which we call ELiTeFormer. Our Transformer-based model combines the data-center scalability and wasteless structure of linear attention during generative inference with the vastly increased speed and reduced model memory footprint of low-precision FFNs. For the baseline model, we employ LLaMA 3 8B for several key reasons: it is a state-of-the-art open-source model architecture, widely adopted across industrial and academic settings, and there is a BitNet b1.58 checkpoint111The BitNet b1.58 checkpoint we utilize was pre-trained on 100B tokens. Based on the reported training efficiency of LLaMA 3 8B, this requires roughly 8,700 GPU hours to produce the parameter-efficient model. based on LLaMa 3 8B that we adapt for ELiTeFormer. An ELiTeFormer attention block is depicted in Figure 1, which outlines a hybrid linear attention and sliding window architecture that has demonstrated success in prior work (Zhang et al., 2024a; Arora et al., 2024), but with the typical attention head projections replaced with low-precision alternatives based on BitNet b1.58 (Ma et al., 2024) and a classical sliding window implementation versus a “terraced” alternative (Zhang et al., 2024a). As part of our hybrid attention, additional learned feature maps are included (Zhang et al., 2024b) in full precision to maintain model stability, adding approximately 1% additional parameters to the base parameters for ELiTeFormer. Low-rank adapters (LoRAs) (Hu et al., 2022) are also typically included in these types of adapted attention blocks. We choose not to include adapters in the case of ELiTeFormer, as we found them ineffective (elaborated upon in Section 4.1). This enables ELiTeFormer to maintain a static inference profile for all context lengths, a significant improvement in terms of memory footprint, particularly at long contexts. The ELiTeFormer FFN block and attention linear projections (see BitLinear blocks in Figure 1) are taken directly from BitNet b1.58, employing the same low-precision structure and ternary quantization scheme. This is particularly interesting for FPGA deployment, as arithmetic operations featuring extremely low-precision values are usually poorly supported on traditional architectures. Moreover, ternary weights offer the opportunity to completely eliminate multiplication operations and only accumulate during linear projections, which is almost impossible to exploit in general-purpose hardware. Figure 2. An example illustration of the timing diagram for the ELiTeFormer top-level dataflow, pipelining the generative inference stages across two batches. Stages are load balanced to have similar latency. 3.3. Defining ELiTeFormer Inference on FPGAs To design ELiTeFormer for efficient inference on FPGAs, we draw inspiration from the analytical performance models provided by Chen et al. (Chen et al., 2024), which we adapt for our customized model architecture and deployment environment. The top-level design is pipelined per-layer into the following discrete stages, where each stage operates on a batch of samples. We define query/key/value projections with positional embeddings and the half-precision per-head linear projections required for linear attention as QKVQKV. Jointly, we define the operations implementing linear attention, sliding window attention, and the attention output projection as attnattn. As is necessary for LLaMA 3-styled models, we also define the first parallel halves (colloquially called up and gate projections) of the FFN as FFN1,2FFN_1,2, and the second half of the FFN as FFN3FFN_3. This pipeline is depicted in Figure 2. Notably, the proposed dataflow architecture differs slightly from the one proposed by Chen et al. (Chen et al., 2024), not just in how stages are constructed, but also by another critical factor: all stages have synchronization boundaries, and computation cannot be overlapped. In such settings, commonly used normalization layers introduce synchronization points because the full token output is required to compute statistical properties, such as the mean or variance. Since the BitNet b1.58 projections (Ma et al., 2024) are prepended with a RMSNorm block in order to improve model accuracy, full token-level computation is required before a given stage featuring such a projection can begin. 3.4. FPGA Design and Ternary Precision PE μ-arch ELiTeFormer and its accelerator platform was carefully co-designed to target the Vitis HLS and Vivado design flow. Here, we review the design choices employed in the proposed ELiTeFormer’s FPGA implementation, with a focus on the memory interface specification and the ELiTeFormer’s low-precision PE micro-architecture, which we refer to as ELTF PEs for BitNet b1.58-styled linear projections. Figure 3. An illustration of the proposed ELiTeFormer linear projection architecture (left) and PE-specific architecture (right) in hardware. On the left, the input, output, and weight matrices are partially partitioned to enable parallel accesses. The input and output are also split into mini-batches mBmB and enumerated as mBxmBx, with the example in this figure setting mB=4mB=4. Data transfers related to the output are color-coded on the left, represented by orange, indicating the wide availability of the output buffer to the PEs. On the right, a bitwise mask-based replacement for multiplication is employed for efficiency, where unpacked 2-bit weights index the masks. Each weight dataframe is compressed column-wise, so a reduction tree is employed to modify the associated output value. 3.4.1. Memory Interface Specification and General Notes Regarding our memory interface, we employed typical memory-mapped AXI interfaces for the model weights. This type of interface supports bus widths of up to 512 bits and can burst-read up to 4 kB at a time. Most weights are packed into 8-bit dataframes, supporting 4 weights per dataframe (8x compression), which could be further improved with more complex look-up table (LUT) decoding, such as a 5/3 scheme using 30/32 bits in a 32-bit dataframe. Consequently, we read 256 weights in a single read before bursting. Some smaller weight matrices (e.g., normalization weight vectors) that need to remain in 32-bit floating point are loaded along similarly sized memory buses. The ELiTeFormer attention module accesses both its linear-attention state and sliding-window cache through similar interfaces. Although full parallelization across all attention heads is possible, it depends on platform resources. After applying our optimizations, the operator runs efficiently even when heads are processed sequentially. Achieving full head-level parallelism requires adequate memory banking and compute capacity. Moreover, the attnattn stage already consumes a large share of system resources (elaborated later in Section 4.3). For this reason, aggressive parallelization can be impractical. In our later experiments, we instead partially parallelize the query heads in accordance with a single key-value head in grouped-query attention (GQA) (slight variant of attention in LLaMa 3 models where Hq>Hk,vH_q>H_k,v). For example, when four query heads map to a single key-value attention head, we compute all four query attention heads in parallel. This approach increases compute usage but does not add pressure to the platform’s off-device memory interface. In general, all computational blocks followed a dataflow structure, leveraging co-execution of different stages when multiple iterations are in-flight and dependencies are met. Every major block follows the same pattern: (1) load relevant data from off-chip memory in 512-bit frames, (2) decode the 512-bit frames into their intended datatypes, (3) engage in potentially multi-stage computation, and sometimes (4) encode and write back to off-chip memory in 512-bit frames. The final write-back stage only occurs for linear attention recurrent state modifications and in the top-level dataflow. In contrast, the output of every other stage can be stored in (on-chip) block RAM (BRAM) to avoid off-chip accesses. 3.4.2. ELiTeFormer Linear Projections and PE μ-arch Our proposed architecture for ELiTeFormer linear projections can be observed in Figure 3, where the left half of the figure articulates the roughly systolic structure of the PE array and its memory interfaces, and the right half defines the specialized ELiTeFormer PE micro-architecture, which is a novel PE micro-architecture. A mini-batch dimension, which we denote as mBmB, improves parallelism in the input and output matrices of these linear projections. These projections are stored in BRAM blocks of size mB×dmodelmB× d_model each and are then partitioned to enable parallel access. This additional mini-batch dimension significantly improves weight reuse at the expense of increased hardware resources, thereby proportionally reducing latency and increasing throughput in multi-batch inference scenarios. In reality, the increased resource cost is also not particularly constraining, as this optimization is only really applicable to non-attention operations (since attention caches scale with mBmB whereas linear projections do not). This will be elaborated upon in Section 4.3. The right half of Figure 3 elaborates on the novel micro-architecture of the proposed ELTF PEs that efficiently accelerate BitNet b1.58-styled linear projections. The ELTF PEs first read the broadcasted input values for the implemented compression scheme, which represent 4 input values in this example222We employ a simplified 8x compression scheme. A more aggressive scheme (up to 10x theoretical compression) would require more input values to be read and a larger weight dataframe.. These input values are held stationary in the PEs over time until all relevant weights have been used. After the input values are loaded (in parallel to the corresponding output value being loaded), the 8-bit weight dataframe is loaded from the partitioned weight buffers and bit-shifted into four indices, which are used to select specific masks for our proposed mask-based multiplication emulation. Since we are employing ternary weights (i.e., in the range [-1, 0, 1]), the ELTF PEs only need to nullify a value, flip its sign, or allow it to pass through for accumulation to correctly emulate multiplication, which is accomplished via the application of the indexed masks in corresponding bit-wise operators. Finally, the results of these bitwise operations are accumulated via a reduction tree, combined with the output value, and then written back to the appropriately partitioned output buffer, which eventually writes to BRAM. 3.4.3. When Does ELiTeFormer Perform Best? The unique inference profile of ELiTeFormer naturally lends itself to environments that require high energy efficiency and low latency, especially in long-context scenarios. Its extremely compressed memory footprint for both the model weights and the attention cache renders it ideal for constrained systems. However, ELiTeFormer is not limited to those environments. At long contexts, cutting-edge LLM models still bottleneck their throughput on general-purpose hardware due to large attention caches taking up significant space in DRAM. This can limit the size of the inference batch on the hardware in question, lowering throughput even if the hardware is capable of parallelizing batches of that size on-chip. Using LLaMA 3 8B as an example, with a batch size of 128 at 4096 tokens of context, the attention cache takes up nearly 64 GB of DRAM. That attention cache is a little more than four times the size of the model itself. ELiTeFormer algorithmic changes to attention not only reduce its latency, but also drastically reduce the size of the attention cache. Using the same example inference parameters, ELiTeFormer’s attention cache would require only 5 GB of DRAM, representing a 12.8×12.8× improvement. Naturally, the above observation presents another opportunity for ELiTeFormer: high-throughput inference at extremely large batch sizes. Whereas LLaMA 3 8B on an NVIDIA A100 80 GB would likely be limited to batch sizes of around 128, ELiTeFormer would theoretically be capable of supporting batches of over 16k, assuming similarly sized DRAM. This operation mode is possible due to algorithmic changes to the attention block, so any model leveraging hybrid attention, such as ELiTeFormer, would be capable of it. However, coupling it with ternary precision linear projections enables ELiTeFormer to achieve lower latency than those alternatives. 4. Results We evaluate ELiTeFormer along two axes: model quality under aggressive efficiency constraints, and the extent to which its expected hardware advantages appear in practice. For training and evaluation, we fine-tune a BitNet b1.58 8B checkpoint (Ma et al., 2024), adapting the attention-distillation workflow from LoLCATs (Zhang et al., 2024a) in PyTorch to produce an ELiTeFormer-8B model. During training and evaluation, we used a single NVIDIA A100 GPU per run. Training employed a standard 50k slice of samples from the LoLCATs version of the Alpaca dataset333https://huggingface.co/datasets/yahma/alpaca-cleaned (Taori et al., 2023). Models were trained with batch sizes of 8 and sequence lengths of 1024 tokens. Learning rates started at 2e-5 before decaying via an inverse square scheduler to a minimum of 1e-7. 4000 weight updates of training (around 2.5 epochs) were required across the 50k slice of samples before convergence was observed. For evaluation in hardware, including simulations, we provide results for comparisons between baseline Transformer computational blocks and our proposed ELiTeFormer blocks in simulation via Bambu HLS (Ferrandi et al., 2021), an open-source HLS flow. This functions as a limited ablation study. We also provide platform-wide simulations and deploy a scaled-down version of our proposed platform to an FPGA via Xilinx’s Vitis HLS and Vivado flows (Inc., 2025). 4.1. ELiTeFormer Attention Distillation and Model Quality Evaluation Training extremely low-precision neural networks, especially at scale, is a non-trivial task (Wang et al., 2023; Ma et al., 2024). Furthermore, it is not immediately apparent how to adapt such a low-precision model to employ an efficient attention block. Towards this end, we utilize the LoLCATs framework (Zhang et al., 2024a) for attention distillation, incorporating some custom adjustments. Table 1. Perplexity values (lower is better) on the development set of the Alpaca dataset slice, comparing checkpoints trained and evaluated with varying sliding window sizes. For efficient notation, T64 refers to a checkpoint trained with a sliding window of 64 tokens, and E64 refers to a checkpoint being evaluated with a sliding window of 64 tokens, where the two values can differ. Best result is bolded. Experiment Setting Dev. Set PPL ELiTeFormer T256 & E64 14.84 ELiTeFormer T256 & E128 11.94 ELiTeFormer T256 & E256 09.15 ELiTeFormer T64 & E64 14.52 ELiTeFormer T64 & E128 11.62 ELiTeFormer T64 & E256 08.97 Table 2. Perplexity values (lower is better) on the development set of the 50k sample slice from the cleaned Alpaca dataset comparing checkpoints trained with and without the secondary low-rank fine-tuning step. Best result is bolded. Epochs of Fine-tuning Dev. Set PPL ELiTeFormer @0.5 FT epochs 73.80 ELiTeFormer @0.1 FT epochs 09.24 ELiTeFormer @0.0 FT epochs 08.97 Table 3. Various LLM architectures compared on common language modeling benchmarks with scores provided out of 100. All are based on the LLaMA 3 model with non-ternary model weights and activations being in bfloat16 precision. Where possible, scores have been reused from existing publications (Ma et al., 2024; Zhang et al., 2024a; Grattafiori et al., 2024), although some scores have been regenerated if experimental differences or unavailable data was observed. Attention cache scaling during generative inference is defined via model and inference hyperparameters: B is the batch size, N is the context length, h is the number of attention heads, w is the size of the local sliding window, and d is the attention head dimensionality where usually N>>wN>>w and N>>dN>>d. Model Architecture Efficient Model Cache PiQA ARC-e ARC-c HellaSw. Wino- MMLU Avg. Avg. and Quant. Method Attention Size Scaling (norm) (norm) grande (5-shot) w/o MMLU Llama 3 8B ✗ 14.9 GB 2BNhd2BNhd 79.9 80.1 53.3 79.1 73.1 66.6 72.0 73.1 Llama 3 8B Hedgehog ✓ 14.9 GB Bhd2Bhd^2 77.4 71.1 40.6 66.5 54.3 24.2 55.7 62.0 Llama 3 8B Hedgeh. w/ SW ✓ 14.9 GB Bh(2wd+d2)Bh(2wd+d^2) 80.9 81.6 55.1 79.3 72.7 50.7 70.0 73.9 BitNet b1.58 8B ✗ 01.5 GB 2BNhd2BNhd 75.8 74.2 43.7 68.6 58.0 34.9 59.2 64.1 ELiTeFormer 8B (Ours) ✓ 01.5 GB Bh(2wd+d2)Bh(2wd+d^2) 72.5 70.4 37.0 58.0 54.6 31.9 54.0 58.5 4.1.1. Attention Distillation for ELiTeFormer Fundamentally, LoLCATs functions via two subsequent training phases. In the first phase, a given architecture has its attention block replaced with an efficient attention block and has all of its weights frozen except for those involved directly in the newly replaced blockThen, the architecture is trained block-by-block with a goal of minimizing output reconstruction error, as described by Equation 4. This is done on a per-attention head basis, where m is a given layer in the model, a∈ℝ1×da ^1× d is the reference attention output vector on a per time-step basis from the original model functioning as a teacher, and a~∈ℝ1×d a ^1× d is the approximated output from the student model whose attention block has been replaced with an efficient alternative. The final, model-wide loss is averaged from the per-head losses (Zhang et al., 2024a). (4) lMSEh,m=1d∑i=1d(ai,h,m−ai,h,m~)2l^h,m_MSE= 1d _i=1^d(a_i,h,m- a_i,h,m)^2 While uncompressed model architectures appeared to show no significant degradation down to a window size of 64 tokens in sliding-window attention, we found that the ELiTeFormer model quality with a sliding window of this size was too poor to be useful. This is demonstrated by the perplexity values observed in Table 1. However, increasing the size of the sliding window to 256 tokens significantly improved results. Furthermore, training with a smaller sliding window than that used during evaluation also significantly improved the resulting model, seemingly acting as a form of regularization. For the second phase, the model being trained will typically have low-rank adapters, or LoRAs (Hu et al., 2022), added to the input and output projections of the attention blocks before the model undergoes some light fine-tuning to enhance inter-layer error robustness. While this works well for most distillation flows, ELiTeFormer checkpoints were particularly sensitive to low-rank adjustments, as observed in Table 2, where further fine-tuning after attention distillation only deteriorated model quality. A range of experiments modifying hyperparameters yielded no improvements. We posit that this occurs because the model has more or less converged during attention distillation, and that low-rank adjustments to linear projections are likely too coarse-grained for such an extremely compressed model, whose weights need to be carefully tuned. 51210242048409601,0001,0002,0002,0003,0003,0004,0004,000Model Dimensionality for Linear ProjectionsCycles (k)Basic Linear ProjectionELiTeFormer Linear Projection 51210242048409601,0001,0002,0002,0003,0003,0004,0004,000Context Lengths for Attention BlocksCycles (k)Llama 3 Attention CalculationELiTeFormer Attention Calculation Figure 4. A simulated comparison of latency in cycles (k) (lower is better) between a typical linear projection block and ELiTeFormer’s low-precision alternative for a single forward pass during generative inference (left) in addition to a comparison of per-head latency in the attnattn stage (with the output projection removed) of Llama 3 and its ELiTeFormer alternative (right). 4.1.2. Evaluation of ELiTeFormer Model Quality Beyond development set perplexity, we evaluate the final ELiTeFormer 8B checkpoint on a range of popular language modeling benchmarks, reusing the framework provided by EleutherAI for plug-and-play LLM evaluation (Gao et al., 2024). These results can be observed in Table 3, where we compare ELiTeFormer against similar model architectures. All models in Table 3 are derived from the Llama 3 8B checkpoint. Generally, ELiTeFormer performs reasonably well on easier tasks in this suite of benchmarks while achieving both 10x compression like BitNet b1.58 8B (Ma et al., 2024) and the fast, compressed attention of Hedgehog (Zhang et al., 2024b), utilizing a 256-token sliding window. On more challenging tasks, particularly HellaSwag and ARC-c, there is noticeable degradation compared to less efficient alternatives. For HellaSwag in particular, this may indicate a weakness in adversarial reasoning tasks. However, it is important to note that the MMLU results (where MMLU is commonly considered the most difficult of these benchmarks) were very close to those produced by ELiTeFormer’s initial checkpoint, BitNet b1.58 8B, indicating that our modified distillation method is performing well. It is worth mentioning that the necessary pretraining approaches for these models were different and favored the Hedgehog checkpoints. The explicit LLaMA 3 8B models and the efficient attention alternatives were pre-trained on proprietary data by Meta (Grattafiori et al., 2024). In contrast, our initial BitNet b1.58 8B checkpoint, based on LLaMA 3 8B, was further fine-tuned on an additional 100B tokens from FineWeb-edu (Penedo et al., 2024). Subsequently, the Hedgehog and Hedgehog with sliding window models were trained on 200M tokens (Zhang et al., 2024b, a). Since extremely low-precision weights currently need significant additional pretraining (approximately 8,700 GPU hours for BitNet b1.58 8B) across all model weights, we had no choice but to begin from the available BitNet b1.58 8B checkpoint before attention distillation, creating what is likely a “quality of data moat” between our resulting ELiTeFormer checkpoint and the baseline LLaMA 3 models. 4.2. ELiTeFormer Block-wise FPGA Simulations We test our proposed accelerator platform for ELiTeFormer via HLS in a few environments. First, we examine the characteristics of ELiTeFormer blocks when generating register transfer-level (RTL) code and simulating via the open-source Bambu HLS 2021 (Ferrandi et al., 2021) and Verilator, targeting the timings of the programmable logic on the Xilinx VCK5000 board444The VCK5000 board hosts a XCVC1902 Versal device that includes FPGA programmable logic (PL) and the adaptable intelligent engines. We only use the reconfigurable logic in our experiments. Second, we present simulation results for ELiTeFormer, utilizing the same version of Vitis HLS (version 2021.2) (Inc., 2025) for RTL generation, and targeting the same board. Third, we examine the performance of ELiTeFormer when deployed to an actual VCK5000, employing Xilinx’s Vivado flow for placement, routing, and deployment to the board. This section will cover the block-wise simulations, functioning as a limited ablation study. To acquire a more granular understanding of how individual ELiTeFormer blocks perform, it is worthwhile to execute iterative simulations. To this end, we build an implementation in C, augmented with limited pragmas and directives supported by Bambu HLS 2021 (Ferrandi et al., 2021). We test two blocks that provide insight into the efficiency capabilities of ELiTeFormer. First, we synthesize and test the BitNet b1.58-styled linear projections that ELiTeFormer features against a typical full-precision linear projection. Second, we compare the attention blocks of ELiTeFormer and Llama 3 against one another, measuring the efficiency gains in throughput and cache compression of ELiTeFormer’s efficient attention block. Results are provided for both simulations in Figure 4, where ELiTeFormer’s BitNet b1.58-styled projections perform exceptionally well in terms of accelerating the block, on average achieving a 9.6×9.6× in clock cycles. Similar results are observed for accelerating attnattn (without the output linear projection, only core attention calculations), where the efficient attention block of ELiTeFormer is very competitive at longer context lengths. This results in up to a 4.4×4.4× speedup in cycles compared to a typical LLaMA 3 attention block, while also compressing the KV cache by about 12.8×12.8×. Collectively, these simulations confirm the high efficiency potential of ELiTeFormer in custom hardware. 4.3. Simulating and Deploying ELiTeFormer on FPGAs Beyond just designing FPGA implementations of specific blocks, we also simulate and deploy a platform to accelerate ELiTeFormer as a whole, targeting a VCK5000 to test its efficacy. To this end, we compare our accelerator architecture to runtime observations on a LLaMA 3 model running on an NVIDIA A100 GPU and an optimized BitNet b1.58 model running on an A100 GPU and an Intel Xeon Platinum 8168 CPU with 20 cores. In latency and throughput-related experiments, we controlled for overhead and latency common to all platforms (such as data movement) but did not control for platform-specific latency. This applies mostly to GPU baselines, which can include kernel launch overhead and any just-in-time (JIT) compilation overhead in their latency. Our ELiTeFormer accelerator is written in C and employs Vitis HLS to generate RTL before being fed into the Vivado toolchain for synthesis and FPGA deployment. Evaluation results are gathered via the Xilinx Runtime library xrt for the VCK5000 and typical profiling in Python for the A100. Where appropriate, we employ both naive and optimized inference frameworks including vLLM (Kwon et al., 2023) for optimized LLaMA 3 baselines and bitnet.cpp555https://github.com/microsoft/BitNet for BitNet b1.58 baselines. Our simulations are at 8B parameter scales, but our deployed implementation scales down model size (dFFN=8096,L=6)d_FFN=8096,L=6) to better reflect the capabilities of the VCK5000, which only features DDR memory, and emulates a low-latency serving scenario (we also scale down models profiled on the A100 for an apples-to-apples comparison). This results in an end-model size of around 1B parameters and a memory footprint of 0.19 GB from the model weights. 4.3.1. ELiTeFormer 8B FPGA Simulation Results We provide some brief simulation results regarding latency and throughput for the ELiTeFormer accelerator at the 8B parameter scale, at 2142^14 tokens of context, and with a mBmB of 2, targeting a VCK5000 board with programmable logic at 275 MHz, based on timing closure estimations from Vitis HLS. Table 4 showcases the comparison between a simulated ELiTeFormer 8B to LLaMA 3 8B running on an NVIDIA A100 and using vLLM for inference, as well as a fairly optimized BitNet b1.58 8B checkpoint using bitnet.cpp as a backend. Not only does the simulated ELiTeFormer accelerator improve on LLaMA 3 run via vLLM latency by 4.5×4.5× at long contexts, but it also improves on the baseline’s throughput by 2.2×2.2×. Improvements upon BitNet b1.58 with a bitnet.cpp backend are more modest, but we still achieve a latency improvement of 2.7×2.7×. The throughput of the simulated platform is worse than that of BitNet b1.58. However, when normalizing performance by scaling the memory-bound design from the VCK5000’s DDR bandwidth to the Xilinx U280’s HBM bandwidth, we anticipate seeing ELiTeFormer achieve a 5.8×5.8× improvement in throughput over BitNet b1.58. These results speak further to the potential performance of ELiTeFormer at larger scales. Table 4. Brief latency and throughput results based on the ELiTeFormer 8B accelerator simulations derived from system estimations from Vitis HLS targeting a Xilinx VCK5000. Latency is based on a single forward pass during generation for a single sample batch, isolating for latency related to data movement but not for GPU-specific overhead. Throughput is based on the average of 5 tokens for a batch size of 32. Normalized throughput is provided by scaling the bandwidth of the simulated device for ELiTeFormer to Xilinx U280 HBM bandwidth (the VCK5000 is DDR only). All measurements are for a context length of 2142^14. Best values are bolded. Arch. and Device Latency Thrpt. Norm. Thrpt. (ms) (tok/s) HBM (tok/s) Llama 3 8B, A100 (vLLM) 544 05.71 05.71 BitNet b1.58 8B, A100 (bitnet.cpp) 319 14.12 14.12 ELiTeFormer 8B (Ours) 120 12.52 82.27 2112^112122^122132^132142^1410210^210310^3Context Length (tokens)Latency (ms)Llama 3 1B on A100 (Naive)Llama 3 1B on A100 (vLLM)BitNet b1.58 1B on CPU (bitnet.cpp)BitNet b1.58 1B on A100 (bitnet.cpp)ELiTeFormer 1B on VCK5000 (Ours) Figure 5. Observed latency (ms) (lower is better) on a single forward pass during generation for our ELiTeFormer 1B accelerator deployed to a VCK5000 and LLaMa 3 1B running on an A100 for varying context lengths. Latency is post initialization and isolates for data movement, but does include GPU-specific overhead. Results are the average of five runs. 4.3.2. ELiTeFormer 1B Latency Profile As previously discussed, we provide deployment results for our proposed platform with a scaled-down configuration, which better reflects the capabilities of the VCK5000 and emulates a low-latency serving scenario. Regarding observed latency, Figure 5 shows that our proposed, accelerated solution, running at a clock frequency of 275 MHz, achieves a 3.9×3.9× speedup in terms of latency at long context lengths compared to a basic LLaMA 3 implementation executing on an A100. Even when employing a more optimized implementation in vLLM (Kwon et al., 2023), a remarkably efficient, general framework for inference with popular LLMs, we achieve a 2.6×2.6× improvement in latency. Indeed, while LLaMA 3 executed via vLLM on an A100 improves significantly on the original model baseline at shorter context lengths, the gap closes at longer context lengths. Regarding the BitNet b1.58 comparisons, the CPU implementation is competitive with a naive LLaMa 3 implementation at shorter contexts, but is not competitive with more optimized frameworks. On the other hand, the highly optimized GPU implementation is very competitive, despite still operating at a higher precision (i.e., ternary precision is not fully utilized in general-purpose hardware).. Even so, we still achieve a 1.3×1.3× improvement in latency over the optimized BitNet b1.58 GPU baseline at long contexts. 4.3.3. ELiTeFormer 1B Energy Efficiency Table 5. Energy efficiency results for a single forward pass during generation for our ELiTeFormer deployed to a VCK5000 and LLaMA 3 running on an A100 at 2142^14 tokens of context. Throughput in tokens/sec is divided by watts to form tokens/J (higher is better), serving as a composite efficiency metric. Best values are bolded. Arch. and Device Power (W) Tokens/J Llama 3 1B on A100 (vLLM) 83 0.03 BitNet b1.58 1B on A100 (bitnet.cpp) 80 0.07 ELiTeFormer 1B on VCK5000 (Ours) 26 0.30 Table 6. Utilization values (in %) for accelerated ELiTeFormer on a VCK5000 for the whole design relative to the platform (left) and then LUT and BRAM utilization broken down by stage relative to the design (right, stages defined in Section 3.3). This is based off of Vitis HLS system estimations. VCK5000 Resource Resource Utilization BRAM 41.5 DSP 19.5 F 30.2 LUT 44.7 ELiTeFormer LUT BRAM Stage Utiliz. Utiliz. QKVQKV 05.4 17.0 attnattn 60.5 80.4 FFN1,2FFN_1,2 22.0 03.1 FFN3FFN_3 11.2 01.5 Table 7. High-level comparison of various works that propose LLM inference acceleration in FPGAs. Headings marked with an asterisk (*) indicate that some adaptation may be necessary for the accelerator to be applicable, such as varying degrees of fine-tuning, pruning, and sparsification. For numerical values, higher is better. Normalized throughput is calculated based on scaling these accelerators assuming they are memory bound (unless otherwise stated in the work) to U280 HBM bandwidth (if the card was not used) and to 8B parameters in a proportional manner. Accelerator Platform Accelerates Accelerates Accelerates Specialized for Model Cache Normalized Full LLM Arch. Generic LLM* Ternary LLM* Long Context Compr. Ratio Compr. Ratio Thrpt. (tok/s) TransFRU666TransFRU does not have its throughput scaled by parameter count due to only accelerating attention. Instead, it is scaled by embedding dimension and layer count. (Wang et al., 2024) ✗ ✓ ✗ ✗ 04.0× 04.0× 04.83 FlightLLM (Zeng et al., 2024) ✓ ✓ ✗ ✗ 04.6× 02.0× 48.12 ELiTeFormer (Ours) ✓ ✗ ✓ ✓ 10.1× 12.8× 82.27 We compare average energy efficiency for both runtime environments in Table 5, which shows a 3.2×3.2× improvement in average power consumption for our ELiTeFormer accelerator on a VCK5000 compared to LLaMA 3 on an A100. This data was gathered by profiling during runtime with xbutil and nvidia-smi for VCK5000 and GPU runs (which report average power consumption during an execution), respectively. Moreover, we also include a composite metric for energy efficiency (ignoring device area) in tokens/J, where throughput is defined as tokens/sec for the models being tested, and we measure the device average power while running those layers. As observed in Table 5, our ELiTeFormer accelerator achieves a 5.6×5.6× gain in the proposed composite metric, underscoring just how drastically more efficient it is than the LLaMA 3 on an A100 baseline. 4.3.4. ELiTeFormer 1B Resource Utilization Breakdown Resource utilization results are presented in Table 6, demonstrating that the implemented design efficiently utilizes resources. The utilization of LUTs and BRAMs is also broken down by stage in Table 6, which illustrates differing per-stage resource consumption profiles. The attnattn stage is particularly resource-intensive compared to the other stages because it is responsible for a series of complex floating-point computations, even if the computations themselves are relatively low-latency. The FFN1,2FFN_1,2 stage exhibits approximately double the resource utilization of the FFN3FFN_3 stage, which is expected, as FFN1,2FFN_1,2 accounts for two parallel linear projections, versus only a single linear projection in FFN3FFN_3. 5. Related Work There is a range of prior solutions for customized Transformer acceleration. We review important related work in this section, outlining their core contributions and advantages. 5.1. Comparisons to Recent FPGA Transformer Accelerators Two contemporary works stand out as competitive regarding accelerating LLM-related workloads via FPGAs: TransFRU (Wang et al., 2024) and FlightLLM (Zeng et al., 2024). TransFRU focuses on accelerating only the classical attention operator, leveraging mixed precision and attention activation sparsity via sorting. It rightfully notes that attention is a bottleneck in long-context workloads for models like BERT (Devlin et al., 2019) and around its size. Their proposed acceleration is possibly synergistic with the sliding window component of our attention mechanism, but we would also emphasize that their analysis of attention being a bottleneck occurs more often for the smaller models they tested, especially on bidirectional models like BERT. We deploy at larger scales, only on generative workloads, and accelerate attention first from an algorithmic perspective with hybrid attention. On the other hand, FlightLLM is the current state-of-the-art for FPGA-based LLM acceleration. Unlike this work, they propose accelerating both prefilling and generation with a unified architecture. In comparison, we propose leveraging a heterogeneous inference environment where another device performs resource-intensive prefilling and our FPGA-based accelerator performs generation, providing a better overall fit for FPGAs as a device. That being said, FlightLLM does achieve impressive results, although there is some overhead to accommodate their custom ISA (around 3.25 GB of DRAM). When deployed on a Xilinx Alveo U280, they observe a peak throughput of 55 tok/sec for multi-batch workloads at low context lengths. We compare our proposed ELiTeFormer and accelerator architecture to these works in Table 7, using our 8B parameter simulation results for throughput and proportionally normalizing to U280 bandwidth for our memory-bound design. While the platforms above benefit from being applicable to more general LLM architectures, they are unable to properly accelerate ternary architectures, do not specialize in long-context modeling and acceleration, and exhibit subpar model and KV cache compression ratios. Moreover, when normalizing the performance of these platforms in accordance with U280 HBM bandwidth (we use ELiTeFormer 8B’s high-throughput simulation results for this data point), we demonstrate superior throughput as well. As a brief note, throughput comparisons here do not normalize for differing context lengths because prior works often report results at short contexts or omit the context length entirely. This omission is unfavorable to ELiTeFormer, which achieves its strongest performance at long contexts. We also do not account for differences in resource usage (e.g., DSP consumption), a factor that would advantage ELiTeFormer because its ternary blocks require no DSPs. 5.2. Other Transformer Acceleration Efforts While specific accelerators were previously discussed in Section 5.1, we cover this topic more generally here. When it comes to accelerating Transformers on FPGAs, some existing work has explored challenges and opportunities in this area. For example, some works have investigated the potential for FPGAs to exploit complex pruning behavior (Peng et al., 2021; Shi et al., 2024; Wang and Zhang, 2024). Other works have investigated holistic optimizations, most of which were only rendered possible via fine-grained optimizations provided in FPGAs (Wang et al., 2022; Sarkar et al., 2023). Similar to investigations rooted in ASIC designs, several works have explored leveraging more dynamic sparsity in FPGAs for attention acceleration (Chen et al., 2023; Fang et al., 2022a, b). Some prior work has examined the viability of constructing Transformer-based accelerators via HLS (Plagwitz et al., 2022). Finally, one prior work did attempt to accelerate linear attention in an efficient ASIC design (Dass et al., 2023). 5.3. Efficient Attention Alternatives A range of alternative attention operators focused on computational efficiency have been proposed to replace attention. Sparsity-focused attention alternatives were among the first to be proposed (Child et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020), focusing on ignoring certain attention activations to improve computational efficiency. Other common sub-quadratic options tend to feature low-rank approximations or bucketizing in some fashion (Wang et al., 2020; Kitaev et al., 2020; Peng et al., 2022). Linear attention, as opposed to simply sub-quadratic options, generally replaces the softmax operator in typical attention with separable kernel functions such that computation can be reordered into an RNN-like hidden state-based structure, resulting in linear runtime and memory complexity (Katharopoulos et al., 2020; Agostinelli and Chen, 2023; Yang et al., 2024a; Agostinelli et al., 2024; Zhang et al., 2024b; Xiong et al., 2021; Choromanski et al., 2020; Yang et al., 2024b; Kacham et al., 2024). 6. Conclusion In this work, we introduce ELiTeFormer, a pioneering efficient Transformer-based LLM model that integrates hybrid linear attention with ultra-low-precision linear projections, achieving unprecedented model weight and key-value cache compression, making it suitable for LLMs deployed in constrained or large-batch environments. ELiTeFormer achieves only 3.0% MMLU degradation compared to BitNet b1.58 while maintaining 10×10× model and achieving 12.8×12.8× cache compression. We propose a novel PE for low-precision linear projection computations that efficiently exploits the ternary weights of our model. Additionally, we synthesize, simulate, and deploy a co-designed, highly customized accelerator for ELiTeFormer on a Xilinx VCK5000. We observe, on average, a 9.6×9.6× speedup in our FFN blocks and a 4.4×4.4× speedup in our attention blocks compared to LLaMA 3. When deploying our proposed accelerator, we achieve a 3.9×3.9× speedup in terms of latency and a 3.2×3.2× improvement in energy efficiency compared to LLaMA 3 running on an NVIDIA A100 GPU. These accomplishments underscore the potential of reconfigurable hardware and ELiTeFormer for accelerated, efficient LLMs. Acknowledgements.This work was supported by the US Department of Energy, Office of Science, Office of Advanced Scientific Computing Research’s Computer Science Competitive Portfolios program and the Advanced Memory to support Artificial Intelligence for Science (AMAIS) at Pacific Northwest National Laboratory (PNNL). References V. Agostinelli and L. Chen (2023) Improving autoregressive nlp tasks via modular linearized attention. In Machine Learning and Knowledge Discovery in Databases: Research Track, D. Koutra, C. Plant, M. Gomez Rodriguez, E. Baralis, and F. Bonchi (Eds.), Cham, p. 90–106. External Links: ISBN 978-3-031-43421-1 Cited by: §1, §5.3. V. Agostinelli, S. Hong, and L. Chen (2024) LeaPformer: enabling linear transformers for autoregressive and simultaneous tasks via learned proportions. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: §1, §5.3. S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré (2024) Simple linear attention language models balance the recall-throughput tradeoff. External Links: 2402.18668, Link Cited by: §2.2.3, §3.1, §3.2. I. Beltagy, M. E. Peters, and A. Cohan (2020) Longformer: the long-document transformer. arXiv. External Links: Document, Link Cited by: §2.2.2, §5.3. H. Chen, J. Zhang, Y. Du, S. Xiang, Z. Yue, N. Zhang, Y. Cai, and Z. Zhang (2024) Understanding the potential of fpga-based spatial acceleration for large language model inference. ACM Trans. Reconfigurable Technol. Syst.. Note: Just Accepted External Links: ISSN 1936-7406, Link, Document Cited by: §1, §2.4, §2.4, §2.4, §3.3, §3.3. Z. Chen, Z. Qu, Y. Quan, L. Liu, Y. Ding, and Y. Xie (2023) Dynamic n:m fine-grained structured sparse attention mechanism. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP ’23, New York, NY, USA, p. 369–379. External Links: ISBN 9798400700156, Link, Document Cited by: §5.2. Z. Chen, Y. Deng, Y. Wu, Q. Gu, and Y. Li (2022) Towards understanding the mixture-of-experts layer in deep learning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 23049–23062. External Links: Link Cited by: §1. R. Child, S. Gray, A. Radford, and I. Sutskever (2019) Generating long sequences with sparse transformers. arXiv. External Links: Document, Link Cited by: §1, Figure 1, Figure 1, §2.2.2, §2.2, §5.3. K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser, D. Belanger, L. Colwell, and A. Weller (2020) Rethinking attention with performers. arXiv. External Links: Document, Link Cited by: §5.3. J. Dass, S. Wu, H. Shi, C. Li, Z. Ye, Z. Wang, and Y. Lin (2023) ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), p. 415–428. Note: ISSN: 2378-203X External Links: Link, Document Cited by: §5.2. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer (2022) LLM.int8(): 8-bit matrix multiplication for transformers at scale. External Links: 2208.07339, Link Cited by: §1. T. Dettmers and L. Zettlemoyer (2023) The case for 4-bit precision: k-bit inference scaling laws. ICML’23. Cited by: §2.3. J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. External Links: 1810.04805, Link Cited by: §5.1. H. Dong, X. Yang, Z. Zhang, Z. Wang, Y. Chi, and B. Chen (2024) Get more with LESS: synthesizing recurrence with KV cache compression for efficient LLM inference. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 11437–11452. External Links: Link Cited by: §2.2.3, §3.1. H. Fan, T. Chau, S. I. Venieris, R. Lee, A. Kouris, W. Luk, N. D. Lane, and M. S. Abdelfattah (2022) Adaptable butterfly accelerator for attention-based nns via hardware and algorithm co-design. In MICRO-55: 55th Annual IEEE/ACM International Symposium on Microarchitecture, Cited by: §1, §3. C. Fang, S. Guo, W. Wu, J. Lin, Z. Wang, M. K. Hsu, and L. Liu (2022a) An efficient hardware accelerator for sparse transformer neural networks. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), Vol. , p. 2670–2674. External Links: Document Cited by: §5.2. C. Fang, A. Zhou, and Z. Wang (2022b) An algorithm–hardware co-optimized framework for accelerating n:m sparse transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 30 (11), p. 1573–1586. External Links: Document Cited by: §5.2. F. Ferrandi, V. G. Castellana, S. Curzel, P. Fezzardi, M. Fiorito, M. Lattuada, M. Minutoli, C. Pilato, and A. Tumeo (2021) Invited: bambu: an open-source research framework for the high-level synthesis of complex applications. In 2021 58th ACM/IEEE Design Automation Conference (DAC), p. 1327–1330. External Links: Document, ISSN 0738-100X Cited by: 3rd item, §1, §4.2, §4.2, §4. L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2024) A framework for few-shot language model evaluation. Zenodo. External Links: Document, Link Cited by: 1st item, §4.1.2. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, and al et. (2024) The llama 3 herd of models. External Links: 2407.21783, Link Cited by: 1st item, §4.1.2, Table 3, Table 3. T. J. Ham, S. Jung, S. Kim, Y. H. Oh, Y. Park, Y. Song, J. Park, S. Lee, K. Park, J. W. Lee, and D. Jeong (2020) A3A^3: Accelerating attention mechanisms in neural networks with approximation. 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), p. 328–341. External Links: Link Cited by: §1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §3.2, §4.1.1. A. Inc. (2025) AMD Vitis Unified Software Platform. Note: https://w.amd.com/en/products/software/adaptive-socs-and-fpgas/vitis.htmlAccessed: 2025-01-17 Cited by: 3rd item, §1, §4.2, §4. J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim (2024) A survey on large language models for code generation. External Links: 2406.00515, Link Cited by: §1. P. Kacham, V. Mirrokni, and P. Zhong (2024) PolySketchFormer: fast transformers via sketching polynomial kernels. External Links: 2310.01655, Link Cited by: §5.3. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret (2020) Transformers are rnns: fast autoregressive transformers with linear attention. arXiv. External Links: Document, Link Cited by: §1, §2.2.3, §2.2, §5.3. H. Khan, A. Khan, Z. Khan, L. B. Huang, K. Wang, and L. He (2021) NPE: an fpga-based overlay processor for natural language processing. In The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA ’21, New York, NY, USA, p. 227. External Links: ISBN 9781450382182, Link, Document Cited by: §2.4. N. Kitaev, L. Kaiser, and A. Levskaya (2020) Reformer: the efficient transformer. arXiv. External Links: Document, Link Cited by: §5.3. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: §4.3.2, §4.3. B. Li, S. Pandey, H. Fang, Y. Lyv, J. Li, J. Chen, M. Xie, L. Wan, H. Liu, and C. Ding (2020) FTRANS: energy-efficient acceleration of transformers using fpga. In Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design, ISLPED ’20, New York, NY, USA, p. 175–180. External Links: ISBN 9781450370530, Link, Document Cited by: §2.4. M. Liu, Y. Ma, Y. Zhen, J. Dan, Y. Yu, Z. Zhao, Z. Hu, B. Liu, and C. Fan (2024) LLM4GEN: leveraging semantic representation of llms for text-to-image generation. External Links: 2407.00737, Link Cited by: §1. S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei (2024) The era of 1-bit llms: all large language models are in 1.58 bits. External Links: 2402.17764, Link Cited by: §1, §1, §2.3, §2.3, §3.1, §3.2, §3.3, §4.1.2, §4.1, Table 3, Table 3, §4. OpenAI (2024) GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1. G. Penedo, H. Kydlíček, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V. Werra, and T. Wolf (2024) The fineweb datasets: decanting the web for the finest text data at scale. External Links: 2406.17557, Link Cited by: §4.1.2. H. Peng, J. Kasai, N. Pappas, D. Yogatama, Z. Wu, L. Kong, R. Schwartz, and N. A. Smith (2022) ABC: attention with bounded-memory control. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 7469–7483. External Links: Link, Document Cited by: §5.3. H. Peng, S. Huang, T. Geng, A. Li, W. Jiang, H. Liu, S. Wang, and C. Ding (2021) Accelerating transformer-based deep learning models on fpgas using column balanced block pruning. In 2021 22nd International Symposium on Quality Electronic Design (ISQED), Vol. , p. 142–148. External Links: Document Cited by: §5.2. P. Plagwitz, F. Hannig, and J. Teich (2022) TRAC: compilation-based design of transformer accelerators for fpgas. In 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL), Vol. , p. 17–23. External Links: Document Cited by: §5.2. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018) Improving language understanding with unsupervised learning. Cited by: §2.1. A. Rahman, J. Lee, and K. Choi (2016) Efficient fpga acceleration of convolutional neural networks using logical-3d compute array. In ’2016 Design, Automation & Test in Europe Conference & Exhibition (DATE)’, Vol. , p. 1393–1398. External Links: Document Cited by: §2.4. R. Sarkar, H. Liang, Z. Fan, Z. Wang, and C. Hao (2023) Edge-moe: memory-efficient multi-task vision transformer architecture with task-level sparsity via mixture-of-experts. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), Vol. , p. 01–09. External Links: Document Cited by: §5.2. G. Shen, J. Zhao, Q. Chen, J. Leng, C. Li, and M. Guo (2022) SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences. arXiv. Note: arXiv:2206.14550 External Links: Link, Document Cited by: §3. K. Shi, M. Zhang, K. Zhao, X. Wu, Y. Liu, J. Yu, and K. Wang (2024) Fitop-trans: maximizing transformer pipeline efficiency through fixed-length token pruning on fpga. In 2024 34th International Conference on Field-Programmable Logic and Applications (FPL), Vol. , p. 243–249. External Links: Document Cited by: §5.2. M. Sun, Z. Liu, A. Bair, and J. Z. Kolter (2024) A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.3. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023) Stanford alpaca: an instruction-following llama model. GitHub. Note: https://github.com/tatsu-lab/stanford_alpaca Cited by: §4. Y. Tay, D. Bahri, L. Yang, D. Metzler, and D. Juan (2020) Sparse sinkhorn attention. External Links: Link Cited by: §1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . External Links: Link Cited by: §2.2.1, §2.2, §2. H. Wang, Z. Zhang, and S. Han (2021) SpAtten: efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Vol. , p. 97–110. External Links: Document Cited by: §1. H. Wang, Y. Bai, J. Yu, and K. Wang (2024) TransFRU: Efficient Deployment of Transformers on FPGA with Full Resource Utilization. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), p. 521–526. Note: ISSN: 2153-697X External Links: Link, Document Cited by: Table 7, §5.1. H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei (2023) BitNet: scaling 1-bit transformers for large language models. External Links: 2310.11453, Link Cited by: §1, §2.3, §4.1. S. Wang and H. Zhang (2024) Efficient fpga-based transformer accelerator using in-block balanced pruning. In 2024 13th International Conference on Communications, Circuits and Systems (ICCCAS), Vol. , p. 18–23. External Links: Document Cited by: §5.2. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma (2020) Linformer: self-attention with linear complexity. arXiv. External Links: Document, Link Cited by: §5.3. T. Wang, L. Gong, C. Wang, Y. Yang, Y. Gao, X. Zhou, and H. Chen (2022) ViA: a novel vision-transformer accelerator based on fpga. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 41 (11), p. 4088–4099. External Links: Document Cited by: §2.4, §5.2. Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung, Y. Li, and V. Singh (2021) Nyströmformer: a nyström-based algorithm for approximating self-attention. Cited by: §5.3. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim (2024a) Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 56501–56523. External Links: Link Cited by: §1, §5.3. S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim (2024b) Parallelizing linear transformers with the delta rule over sequence length. External Links: 2406.06484, Link Cited by: §5.3. M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed (2020) Big bird: transformers for longer sequences. External Links: Document, Link Cited by: §1, §2.2.2, §5.3. S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang, Y. Dai, J. Li, Z. Wang, R. Zhang, K. Wen, X. Ning, and Y. Wang (2024) FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. arXiv. Note: arXiv:2401.03868 External Links: Link Cited by: §1, §1, §2.4, §2.4, §3, Table 7, §5.1. M. Zhang, S. Arora, R. Chalamala, A. Wu, B. Spector, A. Singhal, K. Ramesh, and C. Ré (2024a) LoLCATs: on low-rank linearizing of large language models. External Links: 2410.10254, Link Cited by: §1, §3.2, §4.1.1, §4.1.2, §4.1, Table 3, Table 3, §4. M. Zhang, K. Bhatia, H. Kumbong, and C. Re (2024b) The hedgehog & the porcupine: expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Figure 1, Figure 1, §3.2, §4.1.2, §4.1.2, §5.3. W. Zhang, Y. Deng, B. Liu, S. Pan, and L. Bing (2024c) Sentiment analysis in the era of large language models: a reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 3881–3906. External Links: Link, Document Cited by: §1. C. Zhu, W. Ping, C. Xiao, M. Shoeybi, T. Goldstein, A. Anandkumar, and B. Catanzaro (2021) Long-short transformer: efficient transformers for language and vision. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 17723–17736. External Links: Link Cited by: §2.2.3.