Paper deep dive

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

Hanzhi Zhang, Qiao Zhang, Qinglei Cao, Heng Fan, Yan Huang, Kewei Sha, Yunhe Feng

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 121

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 8/19/2026, 4:48:21 AM

Summary

The paper introduces TileMix, a tile-centric mixed-precision attention kernel designed to accelerate Large Language Model (LLM) inference during the prefill phase. TileMix partitions the attention matrix into hardware-aligned score tiles and uses compact bitmasks to route each tile group through either FP16 or INT8 computation paths within a fused dense attention kernel. This approach preserves dense token connectivity while improving prefill throughput over standard FP16 and recovering quality lost under uniform INT8 quantization. The method supports grouped-query attention, variable-length batches, and INT8 key/value caches, demonstrating effectiveness across LLaMA, Qwen, and Vicuna models on benchmarks like LongEval and LV-Eval.

Entities (14)

TileMix · method · 100%FP16 · dataformat · 95%Hanzhi Zhang · person · 95%INT8 · dataformat · 95%University of North Texas · organization · 95%A100 · hardware · 90%Llama · model · 90%LLM Inference Acceleration · task · 90%LongEval · benchmark · 90%LV-Eval · benchmark · 90%Qwen · model · 90%Vicuna · model · 90%FlashAttention · method · 85%Grouped Query Attention · technique · 85%

Relation Signals (10)

Hanzhi Zhang → affiliatedwith → University of North Texas

confidence 95% · Hanzhi Zhang Affiliation: LLaVi Lab, Computer Science & Engineering, University of North Texas

TileMix → usesprecisionformat → INT8

confidence 95% · dispatches each tile group through FP16 or INT8 score computation

TileMix → usesprecisionformat → FP16

confidence 95% · dispatches each tile group through FP16 or INT8 score computation

TileMix → evaluatedonbenchmark → LongEval

confidence 90% · Across LongEval, LV-Eval, and A100 prefill benchmarks

TileMix → evaluatedonbenchmark → LV-Eval

confidence 90% · Across LongEval, LV-Eval, and A100 prefill benchmarks

TileMix → evaluatedonmodel → Vicuna

confidence 90% · benchmarks on LLaMA, Qwen, and Vicuna

TileMix → evaluatedonmodel → Llama

confidence 90% · benchmarks on LLaMA, Qwen, and Vicuna

TileMix → evaluatedonmodel → Qwen

confidence 90% · benchmarks on LLaMA, Qwen, and Vicuna

TileMix → isbasedon →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or select token interactions, leaving spatial precision routing over hardware-aligned score tiles outside fused dense attention. We introduce TileMix, a tile-centric precision-routing kernel that makes numerical precision an executable spatial decision over score-tile groups within fused dense attention. TileMix partitions the attention matrix into hardware-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online-softmax state. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware-aligned compute tiles and compact metadata at long contexts. By routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped-query attention, variable-length batches, and INT8 key/value caches. Across LongEval, LV-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy-efficiency frontier across model families. The implementation is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

118,153 characters extracted from source content.

Expand or collapse full text

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration Hanzhi Zhang Affiliation: LLaVi Lab, Computer Science & Engineering, University of North Texas Qiao Zhang Qinglei Cao Heng Fan Affiliation: LLaVi Lab, Computer Science & Engineering, University of North Texas Yan Huang Affiliation: LLaVi Lab, Computer Science & Engineering, University of North Texas Kewei Sha Affiliation: Computer Science, Saint Louis University; Data Science, University of North Texashanzhi.zhang,heng.fan,yan.huang,kewei.sha,yunhe.feng@unt.eduqiao.zhang,qinglei.cao@slu.edu Yunhe Feng Affiliation: LLaVi Lab, Computer Science & Engineering, University of North Texas Abstract Long-context prefill in large language models (LLMs) incurs substantial computation and memory traffic because dense self-attention computes quadratic query-key scores. Existing methods either use a uniform low-precision path or select token interactions, leaving spatial precision routing over hardware-aligned score tiles outside fused dense attention. We introduce TileMix, a tile-centric precision-routing kernel that makes numerical precision an executable spatial decision over score-tile groups within fused dense attention. TileMix partitions the attention matrix into hardware-aligned score tiles, packs routing decisions into compact bitmasks, and dispatches each tile group through FP16 or INT8 score computation while both paths update a shared online-softmax state. Scalable precision grouping lets each routing bit govern multiple adjacent key tiles, preserving hardware-aligned compute tiles and compact metadata at long contexts. By routing all legal tile groups, TileMix preserves dense token connectivity, requires no training, and supports grouped-query attention, variable-length batches, and INT8 key/value caches. Across LongEval, LV-Eval, and A100 prefill benchmarks on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality lost under uniform INT8 and improves prefill throughput over FP16, yielding a controllable accuracy-efficiency frontier across model families. The implementation is available at https://github.com/HanzhiZhang-Ulrica/TileMix. 1 Introduction Transformer models increasingly rely on long contexts for document summarization 7, multi-page question answering 30, and retrieval-augmented generation 20, making efficient long-sequence processing central to practical LLM inference. During prefill, dense self-attention computes interactions between all query and key tokens, producing O⁡(L2)O(L^2) score computation for sequence length L. This quadratic computation makes attention a primary execution bottleneck for long documents and other context-intensive workloads. Figure 1: Comparison of attention efficiency strategies. Colors denote execution states: blue for high precision, gray for low precision, and white for removed interactions. Quantization reduces weight/activation memory but often keeps attention softmax and accumulation in higher precision. Sparse/block attention executes a selected subset of token interactions through structured or dynamic patterns. TileMix preserves all legal token interactions and routes score-tile groups through FP16 or INT8 paths inside one fused attention kernel. Existing acceleration methods mainly optimize numerical format, token connectivity, or IO scheduling, represented in Figure 1 by quantization, sparsity, and IO-aware fused attention. (1) Low-precision quantization such as INT8 33; 38 improves arithmetic and memory efficiency across model operators including weight, activation, and attention. Quantized attention kernels commonly use one arithmetic path per invocation or stage, leaving spatial precision routing over the L×L× L score tiles outside the streaming loop. (2) Sparsity-based methods 35; 37; 3 reduce computation by selecting active token interactions. (3) IO-aware fused attention 11 partitions attention into hardware-aligned tiles and fuses score computation, online softmax, and value aggregation. These kernels use tiles for data movement and work partitioning, while retaining a uniform score-computation path. Together, these directions suggest using hardware-aligned score tiles as the spatial unit of precision. A fused kernel can preserve the complete attention graph while routing score-tile groups through multiple arithmetic paths in one streaming computation. Modern fused-attention kernels achieve high utilization through regular Tensor Core tiling and coordinated work partitioning, online softmax, and data movement across the GPU memory hierarchy 12; 25. Tile-group precision routing must reconcile distinct FP16 and INT8 Tensor Core paths, including INT8 rescaling, before both paths update the shared row-wise maximum, normalizer, and output accumulator 6; 4. Precision dispatch therefore falls inside the latency-critical inner loop, making compact tile-aligned routing essential for regular long-context execution and creating a kernel-design problem beyond invocation-level precision selection. To address these challenges, we introduce TileMix, a tile-centric precision-routing kernel that integrates heterogeneous score arithmetic into a single FlashAttention-style execution. For each query-tile row, TileMix loads a packed routing word, decodes each key-tile-group decision with constant-time bit operations, and dispatches QK computation to FP16 or INT8 Tensor Core paths. After rescaling, both paths update the shared online-softmax state, preserving dense streaming execution. Scalable precision grouping lets each routing bit govern adjacent key tiles while retaining the underlying hardware-aligned compute tiles and compact metadata as context length grows. The resulting kernel combines the dense connectivity of full attention with the arithmetic flexibility of mixed precision under training-free deployment. It also supports grouped-query attention, variable-length batching, and INT8 key/value caches, and exposes a controllable accuracy-efficiency frontier between FP16 and uniform INT8 attention. Our contributions are as follows: 1. Tile-Centric Precision Routing for Dense Attention: We introduce tile-group precision as a spatial execution abstraction for fused dense attention, enabling fine-grained FP16/INT8 allocation across all legal token interactions. 2. Shared-State Heterogeneous Score Execution: We design a fused kernel that aligns FP16 and INT8 score paths to a common score domain and integrates them through one online-softmax recurrence. 3. Compact and Scalable Kernel-Native Routing: We develop packed bitmask routing with constant-time inner-loop lookup and ⁡(HkTm)O(H_kT_m) metadata, together with precision grouping that scales routing to long contexts while retaining hardware-aligned compute tiles. 4. Practical Long-Context Inference and Evaluation: We implement grouped-query attention, variable-length batching, and INT8 key/value cache support, and validate TileMix through long-context retrieval, question answering, prefill efficiency, and numerical analyses across LLaMA, Qwen, and Vicuna models. 2 Related Work Transformer acceleration primarily follows three directions: low-precision quantization, IO-aware tiled attention, and structural sparsity. These approaches reduce long-context inference cost through numerical compression, data-movement optimization, or selective token connectivity. Low-precision quantization reduces memory and arithmetic costs by representing weights and activations in formats such as INT8 31; 27 and INT4 40. Quantization-aware and post-training methods improve robustness through calibration, activation transformation, outlier handling, and blockwise scaling 34; 33; 24. Recent quantized attention kernels integrate low-precision score computation, value aggregation, and numerical approximation into fused execution 4; 16; 38; 26. These designs typically assign formats at the tensor, operator, attention-stage, or quantization-block level, while two-dimensional score-tile groups follow one arithmetic path within the streaming loop 14; 31; 18. IO-aware attention kernels 11; 12; 13 process attention in SRAM-resident tiles and fuse score computation, online softmax, and value aggregation, avoiding full attention-matrix materialization in HBM. These tiles govern data movement and work partitioning, while arithmetic precision is commonly fixed per kernel invocation or attention stage 25; 4. Sparse attention methods reduce computation by selecting token interactions through sliding-window, strided, dilated, or mixed local-global patterns 3; 5; 37; 2. Other systems use content-based selection and clustering 23; 32; 17, positional mechanisms 10; 39, adaptive architectures 8, or dynamically constructed active block sets 15; 19; 35. These methods use spatial structure to select executed token interactions, changing attention connectivity while keeping precision outside the selection decision. 3 Preliminaries We briefly review (i) FlashAttention-style tiled attention with online softmax, and (i) blockwise quantization for low-precision matrix multiplication. Their interaction defines the central kernel challenge addressed by TileMix: FP16 and INT8 score tiles follow different arithmetic paths but contribute to one shared streaming softmax state. Appendix Table 4 summarizes the notation used throughout the paper. 3.1 Tiled Attention with Online Softmax Consider one self-attention head with Q,K,V∈ℝL×dQ,K,V ^L× d, where L is the sequence length and d is the head dimension. Attention computes the score matrix S∈ℝL×LS ^L× L, attention weights P∈ℝL×LP ^L× L, and output O∈ℝL×dO ^L× d as S=QK⊤d,P=softmax⁡(S),O=PV.S= QK d,P=softmax(S),O=PV. Materializing S and P in GPU high-bandwidth memory (HBM) requires an ⁡(L2)O(L^2) intermediate-memory footprint and substantial HBM read/write traffic at long context lengths. FlashAttention streams key/value tiles while keeping score and probability tiles on chip and writing only normalized outputs to HBM, providing the execution substrate for TileMix’s tile-group precision routing. The kernel partitions Q into Tm=⌈L/bq⌉T_m= L/b_q tiles Qmm=1Tm\Q_m\_m=1^T_m and (K,V)(K,V) into Tn=⌈L/bkv⌉T_n= L/b_kv tiles (Kn,Vn)n=1Tn\(K_n,V_n)\_n=1^T_n. For each QmQ_m, it streams over n=1,…,Tnn=1,…,T_n while maintaining the shared online-softmax state (m~mn,z~mn,O~mn) ( m_m^\,n, z_m^\,n, O_m^\,n ), comprising the row-wise maximum and normalizer m~mn,z~mn∈ℝbq m_m^\,n, z_m^\,n ^b_q and unnormalized accumulator O~mn∈ℝbq×d O_m^\,n ^b_q× d. Initialized by m~m 0=−∞ m_m^\,0=-∞, z~m 0=0 z_m^\,0=0, and O~m 0=0 O_m^\,0=0, the state updates as m~mn m_m^\,n =max⁡m~mn−1,rowmax⁡(Smn), = \! \ m_m^\,n-1,rowmax(S_m^\,n) \, z~mn z_m^\,n =em~mn−1−m~mnz~mn−1+rowsum⁡(eSmn−m~mn), =e m_m^\,n-1- m_m^\,n z_m^\,n-1+rowsum\! (e^S_m^\,n- m_m^\,n ), O~mn O_m^\,n =em~mn−1−m~mnO~mn−1+eSmn−m~mnVn, =e m_m^\,n-1- m_m^\,n O_m^\,n-1+e^S_m^\,n- m_m^\,nV_n, Smn S_m^\,n =QmKn⊤d. = Q_mK_n d. Here, rowmax⁡(⋅)rowmax(·) and rowsum⁡(⋅)rowsum(·) reduce across the bkvb_kv columns of Smn∈ℝbq×bkvS_m^\,n ^b_q× b_kv. After tile TnT_n, row-wise division of O~mTn O_m^\,T_n by z~mTn z_m^\,T_n yields the output for QmQ_m. 3.2 Blockwise Quantization For a matrix product C=ABC=AB, blockwise quantization applies ψ⁡(⋅)ψ(·) independently to operand blocks, producing low-precision representations and their scale factors: (A^,δA)=ψ⁡(A),(B^,δB)=ψ⁡(B),C≈δAδB(A^B^).( A, _A)=ψ(A),\;( B, _B)=ψ(B),\;C≈ _A _B( A B). The representations A A and B B may use INT8, FP8, or other low-precision formats. We instantiate ψ⁡(⋅)ψ(·) with INT8 because NVIDIA A100 GPUs provide optimized INT8 Tensor Core primitives for high-throughput matrix multiplication. These Tensor Cores execute MMA (matrix multiply-accumulate) instructions that multiply INT8 fragments and accumulate partial sums in INT32 registers. Aligning the quantization blocks with attention tiles allows the corresponding scales δA _A and δB _B to be applied efficiently during fused execution. FP16 and INT8 score tiles exhibit different rounding, accumulation, and rescaling behavior. Because every score tile contributes to the shared state (m~mn,z~mn,O~mn)( m_m^\,n, z_m^\,n, O_m^\,n), path-specific numerical effects propagate through running-maximum tracking, normalization, and output accumulation. Mixed-precision attention is therefore a shared-state kernel problem: heterogeneous score paths must enter a common score domain before updating the same running maximum, normalizer, and output accumulator. Figure 2: TileMix fused attention with tile-group precision routing. Left: The Lq×LkL_q× L_k attention matrix is processed in BLOCKM×BLOCKNBLOCK_M×BLOCK_N compute tiles. For each query-tile row m (outer loop), the kernel streams key/value tiles n (inner loop) from HBM to on-chip SRAM and registers, dispatches the score tile Smn=QmKn⊤/dS_m^\,n=Q_mK_n / d to either FP16 matmul or INT8 Tensor Core MMA with INT32 accumulation and rescaling, and updates a shared FP16 online-softmax state to produce OmO_m without materializing the full score matrix. Right: A binary routing map is grouped along the key-tile dimension with width BLOCKNmask=gBLOCKNBLOCK_N^mask=g\,BLOCK_N and packed per (hk,m)(h_k,m) into a 64-bit mask bhk,mb_h_k,m. The bit at position gj=⌊n/g⌋g_j= n/g selects the arithmetic path for the corresponding key-tile group (1: INT8, 0: FP16). Kernel-side legality masks, including causality and boundary conditions, are enforced independently of the routing map. 4 TileMix: Tile-Group Precision Routing TileMix realizes tile-group precision routing without retraining by combining: (i) a precision policy that assigns each legal score-tile group to FP16 or INT8 score computation, (i) packed bitmasks that convert the policy into constant-time inner-loop dispatch, and (i) a FlashAttention-style fused kernel in which both score paths update one shared online-softmax state. For each KV head and query-tile row, the inner loop reads one packed routing word and extracts the arithmetic-path decision for each key-tile group. The fused kernel natively supports grouped-query attention (GQA) and variable-length batching through prefix-sum metadata, while the implementation provides an INT8 key/value cache interface for decode. 4.1 TileMix Overview TileMix partitions the Lq×LkL_q× L_k attention into hardware-aligned two-dimensional score tiles of size BLOCKM×BLOCKNBLOCK_M×BLOCK_N (Figure 2), with BLOCKM≡bqBLOCK_M≡ b_q and BLOCKN≡bkvBLOCK_N≡ b_kv. The compute tile remains the execution unit, while one routing group may govern multiple adjacent compute tiles along the key dimension. The fused kernel follows the two-level structure of IO-aware attention: Outer loop (over query tiles). The kernel iterates over query tiles Qmm=1Tm\Q_m\_m=1^T_m, where Tm=⌈Lq/BLOCKM⌉T_m= L_q/BLOCK_M . Each Triton program instance owns one query-tile row m. Inner loop (streaming key/value tiles). For each QmQ_m, the kernel streams key/value tiles (Kn,Vn)n=1Tn\(K_n,V_n)\_n=1^T_n from HBM into SRAM and registers, where Tn=⌈Lk/BLOCKN⌉T_n= L_k/BLOCK_N . Across this inner loop, the kernel updates one online-softmax state and retains the output accumulator on chip until normalization after the final tile. For each compute tile (m,n)(m,n), the routing group containing key-tile index n selects the FP16 or INT8 score-computation path. Kernel-side causal and boundary masks determine legal interactions, while the routing policy determines the arithmetic path of each legal tile group. TileMix supports grouped-query attention (GQA) and variable-length batching through one routing interface. Let HqH_q and HkH_k denote the numbers of query and KV heads. Each query head hq∈0,…,Hq−1h_q∈\0,…,H_q-1\ maps to a KV head hk=⌊hqHq/Hk⌋,h_k= h_qH_q/H_k , which indexes routing lookup. Query heads mapped to the same KV head share tile-group routing decisions. For variable-length batching, flattened inputs use cu_seqlenscu\_seqlens and prefix-sum metadata for padding-free routing and execution. 4.2 Routing Policy and Bitmask Encoding Fused attention kernels derive tile execution from pointers, strides, legality masks, and a shared arithmetic configuration. TileMix introduces binary routing metadata that selects the score-computation path inside the inner loop. Policy definition. TileMix accepts a binary tile-group routing map R indexed by KV head hkh_k, query-tile row m, and key-tile group gjg_j. In our evaluation, static, data-free structured templates instantiate R, distributing FP16-routed groups spatially under configurable INT8 coverage budgets. The decision Rhk,m,gj=1R_h_k,m,g_j=1 dispatches the corresponding group to INT8, while Rhk,m,gj=0R_h_k,m,g_j=0 dispatches it to FP16. (See Appendix B for the full list of precision layouts and their definitions.) Figure 3: Key-tile grouping and bitmask encoding for one query-tile row m. Adjacent compute tiles along the key dimension form routing groups gjg_j, each controlled by one bit in the packed mask bhk,mb_h_k,m (1: INT8, 0: FP16). Why grouping is needed. TileMix targets constant-time routing lookup inside the attention inner loop. As Figure3 shows, we pack up to 64 key-tile-group decisions for each query-tile row into a single 64-bit word. To extend the same routing word across longer key sequences while retaining hardware-aligned compute tiles, TileMix defines the routing-group width BLOCKNmaskBLOCK_N^mask through a grouping factor g∈ℤ+g ^+: BLOCKNmask=g⋅BLOCKN,Tmask=⌈LkBLOCKNmask⌉≤64.BLOCK_N^mask=g·BLOCK_N,\;T_mask= L_kBLOCK_N^mask ≤ 64. When g>1g>1, one routing bit controls a contiguous group of g adjacent key tiles (Figure 3). Each decision therefore spans g⋅BLOCKNg·BLOCK_N key tokens. This grouping extends one routing word across long key sequences while preserving the hardware-aligned compute-tile structure. Figure 4: Constant-time shift-and-mask routing lookup. Shifting bit gjg_j to the least-significant position yields the arithmetic-path decision Rhk,m,gjR_h_k,m,g_j for the corresponding key-tile group. Bitmask packing. After grouping, the routing map R is indexed by KV head hkh_k, query-tile row m, and key-tile group gj∈0,…,Tmask−1g_j∈\0,…,T_mask-1\. For each (hk,m)(h_k,m), we pack the group decisions into a 64-bit integer, or packed bitmask word, bhk,m=∑gj=0Tmask−1Rhk,m,gj 2gj,Rhk,m,gj∈0,1.b_h_k,m= _g_j=0^T_mask-1R_h_k,m,g_j\,2^g_j,\;R_h_k,m,g_j∈\0,1\. Constant-time lookup. Inside the kernel, one shift-and-mask operation retrieves the routing decision for group gjg_j (Figure 4): Rhk,m,gj=((bhk,m≫gj)& 1).R_h_k,m,g_j= ((b_h_k,m g_j)\;\&\;1 ). The general representation requires one 64-bit word per (hk,m)(h_k,m), totaling ⁡(HkTm)O(H_kT_m) routing metadata. Packed bitmasks therefore provide compact kernel-native routing metadata consumed directly by the streaming inner loop. 4.3 Tile-Group Mixed-Precision Attention TileMix preserves the FlashAttention-style online-softmax recurrence defined in Section 3; routing changes only the arithmetic path used to construct each score tile SmnS_m^\,n. The FP16 and INT8 paths contribute distinct rounding, accumulation, and rescaling behavior. The FP16 and INT8 paths use different multiplication, accumulation, and scale-restoration procedures. After INT8 scale restoration, score tiles from both paths enter a common floating-point domain before updating the shared running maximum, normalizer, and output accumulator. TileMix maintains this shared online-softmax state in FP16, while the routing map R assigns FP16 or INT8 score computation to each legal tile group. Both arithmetic paths apply the same 1/d1/ d scaling and exponentiation implementation before contributing to the shared normalization. Tile-group dispatch. For query-tile row m and key-tile index n, the routing-group index is gj=⌊n/g⌋g_j= n/g , with BLOCKNmask=gBLOCKNBLOCK_N^mask=g\,BLOCK_N. The corresponding routing bit defines the routed score tile SmnS_m^\,n: Smn=Rescalem,n⁡(Q8,mK8,n⊤)/d,Rhk,m,gj=1,QmKn⊤/d,Rhk,m,gj=0.S_m^\,n= casesRescale_m,n\! (Q_8,mK_8,n )/ d,&R_h_k,m,g_j=1,\\[2.0pt] Q_mK_n / d,&R_h_k,m,g_j=0. cases INT8 path and scale alignment. For the primary evaluated score-routing path, TileMix quantizes Q and K blockwise to produce Q8Q_8 and K8K_8 together with per-block scales, while V and the PVPV computation remain in FP16. The quantize-once mode prepares Q8Q_8, K8K_8, and their scales once per attention call and reuses them across all INT8-routed tile groups. Figure 5: Line-level retrieval accuracy on the LongEval benchmark for LLaMA 3.2 3B under different tile-group routing layouts, evaluated across prompt lengths from 3.1k to 38.7k tokens. Line plots (left y-axis) report exact-match retrieval accuracy, while bar plots (right y-axis, Δ accuracy vs. FP16) show differences relative to the FP16 attention baseline. Each panel corresponds to a routing layout with 25%, 50%, or 75% of tile groups routed to INT8. Colors indicate routing settings: green denotes the original FP16 attention baseline, red denotes One (all legal score-tile groups routed to INT8), and blue denotes mixed configurations with partially INT8-routed tile groups. The INT8 path computes Q8,mK8,n⊤Q_8,mK_8,n with INT8 MMA and INT32 accumulation, then applies block scales and 1/d1/ d to produce SmnS_m^\,n. FP16-routed tiles compute QmKn⊤/dQ_mK_n / d through the FP16 path and update the same online-softmax state. Variable-length execution stores compact per-block scales indexed through the fused kernel’s prefix-sum metadata. Appendix D distinguishes this cache interface from the prefill path and details the tensor, scale, temporary-buffer, and data-movement layouts. Finally, tile-group policies become inner-loop dispatch through constant-time bitmask lookup. The selected FP16 and INT8 score paths update a shared FP16 online-softmax state. Together, compact metadata, constant-time lookup, and shared-state execution integrate heterogeneous score arithmetic into one dense fused attention computation. 5 Experiments 5.1 Experimental Setup We evaluate TileMix from four perspectives: (i) long-context retrieval, (i) long-context question answering, (i) prefill efficiency, and (iv) numerical behavior under routed precision. The main evaluation uses LLaMA 3.2 3B 1. Appendix E extends quality evaluation to Vicuna-7B, Qwen-2-7B 28, and Qwen-2.5-7B 29, while Appendix F reports efficiency across the same model families. Dataset and Metrics. We evaluate long-context question answering on LV-Eval 36, which covers 11 English and Chinese datasets at context lengths from 16k to 64k tokens. We use LongEval 21 to evaluate line-level retrieval with exact-match accuracy. Each LongEval example embeds a uniquely labeled target line in a sequence of up to 54k tokens and queries the model for its associated content. Baselines. We compare against (i) dense FP16 attention as the full-precision reference, (i) One, which routes all legal score-tile groups to INT8 using the same TileMix kernel substrate, (i) FlashAttention as the IO-aware FP16 execution baseline, (iv) MInference 15 and FlexPrefill 19 as sparse long-context baselines, and (v) SageAttention 38 as a representative INT8 attention kernel. Hardware and Measurement. All experiments run on NVIDIA A100 40GB GPUs. TileMix’s offline autotuner selects an A100 kernel configuration that remains fixed throughout evaluation. Throughput evaluation uses batch size 8, three warmup iterations, and five timed iterations with the same hardware and model wrapper for all methods. Timings include quantization, scale restoration, routing, memory staging, and kernel-scheduling costs. We therefore report end-to-end prefill throughput rather than isolated MMA throughput. Dataset | Len | FP16 | One | SpTrans25 SpTrans50 SpTrans75 | BigBird25 BigBird50 BigBird75 | MInference | FlexPrefill | SageAttn 1 | 16k | 32.04 | 28.78 | 31.75 32.18 30.25 | 31.27 29.64 29.75 | 26.93 | 27.11 | 29.79 | 32k | 15.08 | 11.62 | 15.56 14.77 14.37 | 15.22 14.02 13.55 | 11.55 | 11.74 | 13.50 | 64k | 7.75 | 5.42 | 8.01 7.44 6.83 | 8.01 6.47 5.72 | 5.31 | 5.39 | 5.77 2 | 16k | 18.49 | 15.62 | 18.64 18.08 16.78 | 18.61 16.98 16.62 | 14.61 | 14.92 | 16.67 | 32k | 15.12 | 12.02 | 15.27 14.69 14.18 | 15.14 13.53 13.28 | 12.01 | 11.64 | 14.59 | 64k | 11.84 | 7.79 | 12.06 11.34 11.04 | 12.04 10.28 10.03 | 7.96 | 6.81 | 11.22 3 | 16k | 6.72 | 4.45 | 21.04 20.53 21.65 | 6.29 6.97 4.57 | 5.41 | 5.01 | 5.88 | 32k | 3.78 | 1.95 | 3.35 2.78 2.68 | 2.35 1.81 1.20 | 2.75 | 2.51 | 3.15 | 64k | 3.38 | 1.42 | 3.24 2.67 2.47 | 3.24 1.70 1.37 | 2.13 | 1.91 | 2.73 4 | 16k | 6.00 | 2.27 | 6.00 5.43 5.04 | 6.00 4.46 4.49 | 4.77 | 4.53 | 5.19 | 32k | 18.00 | 13.98 | 18.00 17.43 17.54 | 18.50 16.46 15.99 | 12.99 | 11.99 | 15.08 | 64k | 14.50 | 10.07 | 15.00 14.43 14.04 | 15.00 13.46 12.49 | 9.03 | 8.31 | 11.55 5 | 16k | 17.51 | 12.86 | 17.66 17.09 16.88 | 17.66 15.17 14.84 | 13.97 | 13.16 | 15.27 | 32k | 10.80 | 8.38 | 11.01 10.44 10.06 | 10.61 9.07 9.00 | 7.81 | 7.20 | 9.11 | 64k | 6.82 | 5.32 | 7.63 6.25 5.97 | 7.63 6.09 5.62 | 4.26 | 3.86 | 5.50 6 | 16k | 12.33 | 8.51 | 12.31 11.74 11.03 | 12.38 10.40 10.30 | 9.82 | 9.28 | 10.69 | 32k | 6.73 | 3.22 | 7.08 6.51 5.95 | 6.52 5.18 4.45 | 4.88 | 4.48 | 5.69 | 64k | 1.84 | 0.74 | 1.97 1.37 1.22 | 1.91 1.35 1.13 | 1.12 | 1.07 | 1.44 7 | 16k | 21.75 | 15.69 | 22.07 21.16 21.05 | 21.96 20.81 20.09 | 15.59 | 14.05 | 19.94 | 32k | 19.89 | 12.11 | 19.91 19.37 18.56 | 20.17 18.41 17.79 | 12.23 | 10.95 | 15.92 | 64k | 14.18 | 6.93 | 14.23 13.66 13.34 | 14.09 12.37 11.97 | 7.85 | 6.67 | 11.24 8 | 16k | 18.24 | 12.72 | 17.53 16.99 17.46 | 17.57 16.45 15.96 | 12.32 | 11.39 | 13.93 | 32k | 14.25 | 9.12 | 13.63 13.18 13.00 | 13.73 12.73 12.29 | 9.26 | 7.40 | 10.54 | 64k | 11.08 | 7.54 | 11.31 10.65 10.38 | 11.05 9.60 9.04 | 7.24 | 5.08 | 9.45 9 | 16k | 45.68 | 34.70 | 45.77 44.60 44.88 | 45.13 43.66 43.86 | 38.16 | 35.24 | 42.12 | 32k | 26.84 | 18.17 | 26.05 25.94 25.11 | 26.47 24.94 23.78 | 20.09 | 21.40 | 22.50 | 64k | 16.36 | 9.19 | 16.03 15.25 14.71 | 15.72 14.18 14.04 | 10.24 | 10.69 | 13.21 10 | 16k | 28.88 | 22.37 | 29.10 28.31 27.23 | 28.95 27.57 26.33 | 23.28 | 23.43 | 25.18 | 32k | 18.33 | 13.66 | 18.26 17.65 17.41 | 18.17 16.70 16.88 | 12.70 | 10.28 | 13.94 | 64k | 15.75 | 12.81 | 15.94 15.32 15.39 | 15.93 14.22 13.88 | 9.42 | 7.68 | 13.37 11 | 16k | 22.44 | 16.97 | 22.94 22.37 21.74 | 22.44 20.24 20.76 | 15.99 | 12.64 | 18.19 | 32k | 14.85 | 12.02 | 15.61 15.04 15.06 | 16.11 14.57 13.34 | 11.33 | 10.25 | 14.57 | 64k | 9.26 | 3.41 | 8.92 8.35 7.40 | 8.92 8.38 7.91 | 3.10 | 4.02 | 5.79 Table 1: LV-Eval long-context question answering accuracy for LLaMA 3.2 3B. We compare FP16 attention, One (100% INT8), representative TileMix routing layouts, sparse long-context baselines (MInference and FlexPrefill), and an INT8 attention baseline (SageAttention). Sparse baselines reduce computation by pruning token interactions, whereas TileMix preserves dense legal connectivity and routes legal tile groups to different score-computation paths. Cells are color-coded by comparison to One and FP16: red indicates performance below One, blue indicates performance between One and FP16, and green indicates performance above FP16. 5.2 Model Performance Long-context Retrieval. LongEval evaluates exact retrieval from sequences of labeled lines. Each input contains entries such as “line wacky-cob: CONTENT”, and the query specifies a label. The model returns the associated content, with exact-match accuracy measuring retrieval success. Figure 5 reports LLaMA 3.2 3B retrieval accuracy from 3.1k to 38.7k tokens. FP16 and One provide the full-precision and matched uniform-INT8 references, respectively. Each panel shows one spatial routing layout at 25%, 50%, and 75% INT8 tile-group coverage. Retrieval quality depends on FP16 placement in addition to nominal INT8 coverage. row_rand and sptrans retain stronger accuracy as INT8 coverage increases, while align_sparse, band, and global retain more quality under conservative coverage. Together, routing layout determines where FP16 computation is retained, while coverage controls INT8 execution. Long-context Question Answering. LV-Eval covers 11 English and Chinese long-context QA datasets under confusing-fact insertion (CFI), keyword and phrase replacement (KPR), and keyword-recall (AK) settings. Datasets 1 cmrc_mixup and 2 dureader_mixup cover Chinese machine reading comprehension and question answering, and datasets 3 factrecall_en and 4 factrecall_zh cover bilingual factual recall. Dataset 5 hotpotwikiqa_mixup evaluates multi-hop retrieval, dataset 6 lic_mixup evaluates precise information localization, and datasets 7 loogle_CR_mixup, 8 loogle_MIR_mixup, and 9 loogle_SD_mixup evaluate content recall, multi-information retrieval, and sequential dependency retrieval. Datasets 10 multifieldqa_en_mixup and 11 multifieldqa_zh_mixup evaluate bilingual multi-field question answering. Len | Metric | Torch | FlashAttn | One | SpTrans25 | SpTrans50 | SpTrans75 | BigBird25 | BigBird50 | BigBird75 | MInference | FlexPrefill | SageAttn 1k | Thpt | 11.14 | 17.45 | 32.27 | 27.66 | 32.11 | 33.50 | 26.92 | 31.48 | 32.37 | 1.96 | 8.01 | 19.91 | TOPS | 41.70 | 65.31 | 120.80 | 103.50 | 120.20 | 125.38 | 100.78 | 117.85 | 121.17 | 7.33 | 29.98 | 74.54 2k | Thpt | 7.78 | 16.48 | 32.06 | 27.11 | 31.46 | 33.92 | 26.57 | 28.83 | 32.71 | 2.79 | 12.32 | 20.20 | TOPS | 30.50 | 64.64 | 125.70 | 106.32 | 123.38 | 132.99 | 104.23 | 113.05 | 128.31 | 10.93 | 48.30 | 79.19 4k | Thpt | OOM | 14.33 | 29.80 | 27.14 | 30.59 | 31.80 | 24.69 | 26.84 | 27.48 | 4.70 | 15.45 | 19.91 | TOPS | OOM | 61.31 | 127.48 | 116.09 | 130.85 | 136.03 | 105.63 | 114.83 | 117.56 | 20.13 | 66.11 | 85.19 8k | Thpt | OOM | OOM | 27.41 | 22.81 | 25.69 | 26.61 | 23.02 | 25.17 | 25.84 | 7.24 | 15.80 | 18.79 | TOPS | OOM | OOM | 136.76 | 113.83 | 128.21 | 132.84 | 114.92 | 125.58 | 128.90 | 36.13 | 78.84 | 93.79 Table 2: Implementation-level prefill throughput (Thpt, K tokens/s) and TOPS for LLaMA 3.2 3B-Instruct. All methods use the same A100 40GB hardware, batch size 8, three warmup iterations, five timed iterations, and model wrapper; timings include method-specific quantization, scale restoration, routing, memory staging, and scheduling costs. TOPS uses a common dense-attention operation count divided by measured end-to-end time. Colors compare each row with FlashAttention and One: red indicates performance below FlashAttention, blue indicates performance between FlashAttention and One, and green indicates performance above One; OOM entries are gray. Table 1 compares FP16, the uniform-INT8 reference One, representative TileMix configurations, sparse methods, and SageAttention. One often trails FP16, showing that uniform INT8 can be too coarse for long-context QA. TileMix narrows this gap by assigning FP16 to selected score-tile groups and INT8 to the rest while preserving all legal interactions. The strong sptrans result at 16k factual recall recurs across LLaMA and Qwen models at all coverage ratios, indicating a consistent layout–task interaction. Appendix E reports complete results across models and routing layouts. Seq Len | 0% 5% 10% 1k | 7.27×10−57.27×10^-5 7.47×10−47.47×10^-4 1.19×10−31.19×10^-3 2k | 5.89×10−55.89×10^-5 4.29×10−44.29×10^-4 1.16×10−31.16×10^-3 4k | 4.87×10−54.87×10^-5 6.71×10−46.71×10^-4 1.10×10−31.10×10^-3 8k | 6.84×10−56.84×10^-5 8.34×10−48.34×10^-4 6.32×10−36.32×10^-3 Seq Len | 15% 20% 25% 1k | 1.56×10−31.56×10^-3 1.67×10−31.67×10^-3 2.03×10−32.03×10^-3 2k | 1.49×10−31.49×10^-3 1.66×10−31.66×10^-3 1.84×10−31.84×10^-3 4k | 1.41×10−31.41×10^-3 1.59×10−31.59×10^-3 1.78×10−31.78×10^-3 8k | 6.59×10−36.59×10^-3 6.71×10−36.71×10^-3 6.84×10−36.84×10^-3 Table 3: Mean absolute output deviation from the fixed Torch FP16 reference under different score-tile-group INT8 coverage ratios; lower values indicate closer agreement with the reference implementation. 5.3 Efficiency Table 2 reports prefill throughput and TOPS for LLaMA 3.2 3B-Instruct from 1k to 8k tokens. Under this protocol, TileMix improves throughput over FlashAttention by routing score-tile groups to INT8 Tensor Cores. At 4k tokens, SpTrans75 reaches 31.80 K tokens/s, compared with 14.33 K tokens/s for FlashAttention and 29.80 K tokens/s for One. TileMix preserves dense legal connectivity while changing selected tile-group arithmetic paths. End-to-end throughput includes quantization, scale restoration, routing, memory staging, and kernel scheduling. The ordering between One and high-coverage mixed configurations reflects the complete pipeline, including layout-dependent dispatch and memory behavior. Appendix F reports complete throughput and TOPS across models. 5.4 Attention-Kernel Numerical Behavior We measure mean absolute deviation from a fixed Torch FP16 reference on randomized attention inputs. Differences reflect quantization, scale restoration, accumulation, reduction order, and rounding across routed FP16 and INT8 score paths. Table 3 reports deviation across INT8 coverage ratios and sequence lengths. The 0% configuration remains close to FP16, while deviation generally increases with INT8 coverage. At 8k tokens, deviation increases markedly between 5% and 10% coverage and remains at a similar scale through 25%, showing that coverage provides a practical numerical-control knob. Appendix G extends the analysis with model-depth and sequence-length studies, fused-kernel comparisons, FP16-versus-FP32 accumulation controls, larger-model checks, and heavy-hitter exposure under static routing. 6 Conclusion TileMix establishes score-tile-group precision routing as a kernel-native abstraction for long-context attention. Its kernel assigns FP16 and INT8 paths across legal tile groups, preserving the attention graph while varying precision within one computation. Scalable grouping lets each routing decision span adjacent key tiles while retaining hardware-aligned compute tiles and compact metadata. Packed bitmasks and constant-time lookup provide inner-loop control, while both paths update a shared online-softmax state. It supports training-free deployment, grouped-query attention, variable-length batching, and INT8 KV caches. Across LongEval, LV-Eval, prefill benchmarks, and numerical analyses on LLaMA, Qwen, and Vicuna, TileMix recovers long-context quality from uniform INT8 while improving prefill throughput over FP16. This establishes a controllable accuracy-efficiency frontier and demonstrates spatial precision control as a practical dimension for dense attention kernels. Limitations TileMix targets forward inference during long-context prefill, the deployment setting evaluated throughout this work. The current implementation instantiates tile-group routing with FP16 and INT8 Tensor Core paths on NVIDIA A100 GPUs. Other numerical formats require format-specific scale handling and kernel scheduling. Static routing templates provide deterministic policy construction, compact metadata, and constant-time kernel dispatch. The kernel interface can consume alternative static or adaptive routing policies. Ethical Considerations TileMix is a systems method for improving the efficiency of long-context LLM inference and does not introduce new training data, human-subject data, or model capabilities. Its primary broader impact is computational. Improved attention efficiency may reduce GPU time and energy consumption per supported workload, while lower inference cost may also increase aggregate deployment and total compute demand. References AI (2024) M. AI Llama 3.2 model card. External Links: Link Cited by: §5.1. Ainslie et al. (2020) J. Ainslie, S. Ontanon, C. Alberti, V. Cvicek, Z. Fisher, P. Pham, A. Ravula, S. Sanghai, Q. Wang, and L. Yang ETC: encoding long and structured inputs in transformers. External Links: 2004.08483, Link Cited by: §2. Beltagy et al. (2020) I. Beltagy, M. E. Peters, and A. Cohan Longformer: the long-document transformer. arXiv preprint arXiv:2004.05150. Cited by: Appendix B, Appendix B, §1, §2. Chen et al. (2024) S. Chen, Z. Liu, Z. Wu, C. Zheng, P. Cong, Z. Jiang, Y. Wu, L. Su, and T. Yang Int-flashattention: enabling flash attention for int8 quantization. arXiv preprint arXiv:2409.16997. Cited by: §1, §2, §2. Child et al. (2019) R. Child, S. Gray, A. Radford, and I. Sutskever Generating long sequences with sparse transformers. External Links: 1904.10509, Link Cited by: Appendix B, §2. Choquette et al. (2021) J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky Nvidia a100 tensor core gpu: performance and innovation. IEEE Micro 41 (2), p. 29–35. Cited by: §1. Cohan et al. (2018) A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, W. Chang, and N. Goharian A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), p. 615–621. Cited by: §1. Correia et al. (2019) G. M. Correia, V. Niculae, and A. F. T. Martins Adaptively sparse transformers. External Links: 1909.00015, Link Cited by: §2. Dai et al. (2019a) Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov Transformer-xl: attentive language models beyond a fixed-length context. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 2978–2988. Cited by: Appendix B. Dai et al. (2019b) Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov Transformer-xl: attentive language models beyond a fixed-length context. External Links: 1901.02860, Link Cited by: Appendix B, §2. Dao et al. (2022) T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré FlashAttention: fast and memory-efficient exact attention with io-awareness. External Links: 2205.14135, Link Cited by: §1, §2. Dao (2023) T. Dao Flashattention-2: faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. Cited by: §1, §2. Dege et al. (2025) P. Dege, Q. Luo, R. Mao, and C. Kong FlashMLA-etap: efficient transpose attention pipeline for accelerating mla inference on nvidia h20 gpus. In International Conference on Neural Information Processing, p. 3–17. Cited by: §2. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems 35, p. 30318–30332. Cited by: §2. Jiang et al. (2024) H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C. Lin, et al. Minference 1.0: accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems 37, p. 52481–52515. Cited by: §2, §5.1. Kang et al. (2024) H. Kang, S. Bharadwaj, J. Hensman, T. Krishna, V. Ruhle, and S. Rajmohan TurboAttention: efficient attention approximation for high throughputs llms. External Links: 2412.08585, Link Cited by: §2. Kitaev et al. (2020) N. Kitaev, Ł. Kaiser, and A. Levskaya Reformer: the efficient transformer. External Links: 2001.04451, Link Cited by: §2. Kluska et al. (2024) P. Kluska, A. Castelló, F. Scheidegger, A. C. I. Malossi, and E. S. Quintana-Ortí Qattn: efficient gpu kernels for mixed-precision vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 3648–3657. Cited by: §2. Lai et al. (2025) X. Lai, J. Lu, Y. Luo, Y. Ma, and X. Zhou Flexprefill: a context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766. Cited by: §2, §5.1. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §1. Li et al. (2023) D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, and H. Zhang How long can context length of open-source llms truly promise?. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, Cited by: §5.1. Press et al. (2021) O. Press, N. A. Smith, and M. Lewis Train short, test long: attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. Cited by: Appendix B. Roy et al. (2020) A. Roy, M. Saffar, A. Vaswani, and D. Grangier Efficient content-based sparse attention with routing transformers. External Links: 2003.05997, Link Cited by: §2. Saxena et al. (2024) U. Saxena, S. Sharify, K. Roy, and X. Wang Resq: mixed-precision quantization of large language models with low-rank residuals. arXiv preprint arXiv:2412.14363. Cited by: §2. Shah et al. (2024) J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao Flashattention-3: fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems 37, p. 68658–68685. Cited by: §1, §2. Sharratt (2026) J. Sharratt ThriftAttention: selective mixed precision for long-context fp4 attention. arXiv preprint arXiv:2605.23081. Cited by: §2. Srinivasa Kumar (2025) P. K. Srinivasa Kumar Evaluating full int8 quantization and inference techniques for causal language model. Master’s Thesis, University of Twente. Cited by: §2. Team et al. (2024) Q. Team et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2 (3). Cited by: §5.1. Team (2024) Q. Team Qwen2.5: a party of foundation models. External Links: Link Cited by: §5.1. Tito et al. (2023) R. Tito, D. Karatzas, and E. Valveny Hierarchical multimodal transformers for multipage docvqa. Pattern Recognition 144, p. 109834. Cited by: §1. Van Baalen et al. (2023) M. Van Baalen, A. Kuzmin, S. S. Nair, Y. Ren, E. Mahurin, C. Patel, S. Subramanian, S. Lee, M. Nagel, J. Soriaga, et al. FP8 versus int8 for efficient deep learning inference. arXiv preprint arXiv:2303.17951. Cited by: §2. Wang et al. (2021) S. Wang, L. Zhou, Z. Gan, Y. Chen, Y. Fang, S. Sun, Y. Cheng, and J. Liu Cluster-former: clustering-based sparse transformer for long-range dependency encoding. External Links: 2009.06097, Link Cited by: §2. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han Smoothquant: accurate and efficient post-training quantization for large language models. In International conference on machine learning, p. 38087–38099. Cited by: §1, §2. Yao et al. (2022) Z. Yao, R. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. Z. He Efficient and affordable post-training quantization for large-scale transformers, 2022. URL https://arxiv. org/abs/2206.01861. Cited by: §2. Yuan et al. (2026) J. Yuan, C. Shinn, K. Xu, J. Cui, G. Klimiashvili, G. Xiao, P. Zheng, B. Li, Y. Zhou, Z. Ye, et al. BLASST: dynamic blocked attention sparsity via softmax thresholding. Proceedings of Machine Learning and Systems 8, p. 843–859. Cited by: §1, §2. Yuan et al. (2024) T. Yuan, X. Ning, D. Zhou, Z. Yang, S. Li, M. Zhuang, Z. Tan, Z. Yao, D. Lin, B. Li, et al. Lv-eval: a balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136. Cited by: §5.1. Zaheer et al. (2021) M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed Big bird: transformers for longer sequences. External Links: 2007.14062, Link Cited by: Appendix B, Appendix B, Appendix B, §1, §2. Zhang et al. (2025) J. Zhang, J. Wei, H. Huang, P. Zhang, J. Zhu, and J. Chen SageAttention: accurate 8-bit attention for plug-and-play inference acceleration. External Links: 2410.02367, Link Cited by: §1, §2, §5.1. Zhang et al. (2023) X. Zhang, Z. Lv, and Q. Yang Adaptive attention for sparse-based long-sequence transformer. In Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 8602–8610. External Links: Link, Document Cited by: §2. Zhao et al. (2024) Y. Zhao, C. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci Atom: low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems 6, p. 196–209. Cited by: §2. Appendix A Notations For clarity and ease of reference, we group notation in Table 4 by functional role rather than by order of appearance. This organization reflects the structure of the attention computation and kernel design: core attention definitions, tiling and indexing used by IO-aware kernels, online-softmax state variables, blockwise quantization primitives, head mappings for grouped-query attention, and tile-group routing metadata. Grouping symbols in this way allows readers to quickly locate related quantities when following the kernel execution flow and routing logic described in Sections 3 and 4. Category Symbol Description Attention Q,K,VQ,K,V Query, key, and value matrices: Q∈ℝLq×dQ ^L_q× d, K∈ℝLk×dK ^L_k× d, V∈ℝLk×dV ^L_k× d Lq,LkL_q,L_k Query and key lengths; Lq=Lk=L_q=L_k=L for self-attention d Query, key, and value head dimension S Scaled score matrix: S=QK⊤/d∈ℝLq×LkS=QK / d ^L_q× L_k P Attention weights: P=softmax⁡(S)∈ℝLq×LkP=softmax(S) ^L_q× L_k O Attention output: O=PV∈ℝLq×dO=PV ^L_q× d Tiling bq,bkvb_q,b_kv Query and key/value tile sizes BLOCKM,BLOCKNBLOCK_M,BLOCK_N Hardware-aligned compute-tile sizes; BLOCKM≡bqBLOCK_M≡ b_q, BLOCKN≡bkvBLOCK_N≡ b_kv TmT_m Query-tile count: Tm=⌈Lq/BLOCKM⌉T_m= L_q/BLOCK_M TnT_n Key/value-tile count: Tn=⌈Lk/BLOCKN⌉T_n= L_k/BLOCK_N i,ji,j Query and key token indices m,nm,n Query and key/value tile indices Online Softmax m~mn m_m^\,n Row-wise maximum for query tile m after tile n z~mn z_m^\,n Row-wise normalizer for query tile m after tile n O~mn O_m^\,n Unnormalized output accumulator after tile n Quantization ψ⁡(⋅)ψ(·) Blockwise quantization operator A^,B A, B Low-precision forms of operands A and B δA,δB _A, _B Block scales for A A and B B Q8,K8Q_8,K_8 INT8 query and key representations Heads Hq,HkH_q,H_k Numbers of query and KV heads hq,hkh_q,h_k Query-head and mapped KV-head indices Routing Policy BLOCKNmaskBLOCK_N^mask Routing-group width along the key dimension g Key-tile grouping factor: BLOCKNmask=gBLOCKNBLOCK_N^mask=g\,BLOCK_N TmaskT_mask Groups per query-tile row: Tmask=⌈Lk/BLOCKNmask⌉≤64T_mask= L_k/BLOCK_N^mask ≤ 64 gjg_j Group index for key tile n: gj=⌊n/g⌋g_j= n/g Rhk,m,gjR_h_k,m,g_j Binary route: 1 for INT8 and 0 for FP16 bhk,mb_h_k,m Packed 64-bit routing word ρINT8 _INT8 Fraction of legal groups routed to INT8 Table 4: Notation for attention, tiling, online softmax, quantization, head mapping, and routing. Appendix B Precision Layouts Figure 6 visualizes attention maps across layers and heads for LLaMA 3.2 3B on the Multi-News dataset. The maps provide qualitative motivation for structured tile-group precision allocation by showing that attention values are distributed non-uniformly across the query–key plane. The evaluated routing templates are statically constructed and do not use benchmark outputs for policy selection. In TileMix, these layouts do not remove interactions; they only determine whether each legal tile group is routed to FP16 or INT8. These observations motivate a precision-routing view of attention acceleration. Figure 6: Visualization of attention layouts across layers and heads. The figure compares the attention values from a LLaMA 3.2 3B Instruct model on the Multi-News dataset. TileMix supports block-level routing layouts that specify whether each legal tile group is computed through the FP16 or INT8 score path. These layouts reuse spatial structures studied in prior sparse-attention work as precision-routing templates. The resulting policies preserve dense token connectivity while assigning selected spatial regions to FP16 and the remaining legal regions to INT8. Policy Construction and Sharing. For each sequence length and block geometry, a layout is instantiated as a two-dimensional routing template R¯m,gj∈0,1, R_m,g_j∈\0,1\, where m is the query-tile row and gjg_j is the key-tile-group index. The convention is R¯m,gj=1,INT8 score path,0,FP16 score path. R_m,g_j= cases1,&INT8 score path,\\ 0,&FP16 score path. cases Causal and sequence-boundary masks determine whether token interactions are legal independently of this precision decision. The same template is reused across transformer layers and batch examples and is broadcast across KV heads: Rhk,m,gj=R¯m,gj,hk∈0,…,Hk−1.R_h_k,m,g_j= R_m,g_j, h_k∈\0,…,H_k-1\. Query heads mapped to the same KV head and read the same packed routing word. The layer and batch indices are omitted from R because the evaluated routing values are shared along these dimensions. To keep one routing word sufficient at long sequence lengths, the routing-group width is selected as a multiple of the key compute-tile width: b64=⌈Lk64⌉,BLOCKNmask=BLOCKN⌈max⁡(b64,BLOCKN)BLOCKN⌉.b_64= L_k64 ,\;BLOCK_N^mask=BLOCK_N \! (b_64,BLOCK_N )BLOCK_N . Consequently, g=BLOCKNmaskBLOCKN,Tmask=⌈LkBLOCKNmask⌉≤64,g= BLOCK_N^maskBLOCK_N,\;T_mask= L_kBLOCK_N^mask ≤ 64, and key compute tile n uses routing-group index gj=⌊ng⌋.g_j= ng . For each query-tile row, the decisions are packed into bhk,m=∑gj=0Tmask−1Rhk,m,gj 2gj.b_h_k,m= _g_j=0^T_mask-1R_h_k,m,g_j\,2^g_j. The template is regenerated when the sequence length or selected block geometry changes and is cached otherwise. Coverage Accounting. Let =(hk,m,gj):ℐm×gjG= \(h_k,m,g_j):I_m×J_g_j \ denote the set of legal routing groups. The realized INT8 coverage is ρ^INT8=∑(hk,m,gj)∈Rhk,m,gj||. ρ_INT8= _(h_k,m,g_j) R_h_k,m,g_j|G|. The reported 25%, 50%, and 75% settings denote legal score-tile-group INT8 coverage. They do not denote the fraction of total attention FLOPs executed in INT8. For each layout, its parameters are selected at tile-group granularity to match the requested coverage as closely as possible. Minor differences between requested and realized coverage may arise from discrete routing groups and sequence boundaries. For query-tile row m, let m=gj:(hk,m,gj)∈G_m= \g_j:(h_k,m,g_j) \ denote its legal key-tile groups, and let ℱm=gj∈m:Rhk,m,gj=0F_m= \g_j _m:R_h_k,m,g_j=0 \ denote its FP16-routed groups. The layouts below differ in the spatial construction of ℱmF_m. One (All INT8). All legal tile groups are routed to INT8: ℱmOne=∅,Rhk,m,gj=1.F_m^One= ,\;R_h_k,m,g_j=1. This pattern corresponds to uniform INT8 score computation and serves as the matched low-precision reference using the same TileMix execution substrate. Zero (All FP16). All legal tile groups are routed to FP16: ℱmZero=m,Rhk,m,gj=0.F_m^Zero=G_m,\;R_h_k,m,g_j=0. This pattern corresponds to all-FP16 of the TileMix kernel and serves as its full-precision reference. Band (Local Template). 3; 9 Band retains a contiguous FP16 region around the query–key diagonal. Let γ⁡(m)=⌊mBLOCKMBLOCKNmask⌋γ(m)= m\,BLOCK_MBLOCK_N^mask denote the key-group index aligned with the beginning of query-tile row m. For a half-width w, the FP16 set is ℱmBand(w)=gj∈m:|gj−γ⁡(m)|≤w.F_m^Band(w)= \g_j _m:|g_j-γ(m)|≤ w \. Groups inside the band are routed to FP16, while the remaining legal groups are routed to INT8. The width w is selected to match the requested INT8 coverage at tile-group granularity. This template transfers the locality bias of sliding-window attention into precision allocation while preserving the complete attention. Global. 37; 3 Let global⊆0,…,Tmask−1G_global \0,…,T_mask-1\ denote a fixed set of key-tile groups associated with designated global positions. Global uses ℱmGlobal=global∩m.F_m^Global=G_global _m. The selected global groups remain in FP16 for every query-tile row, while the remaining legal groups are routed to INT8. The number of global groups is selected to match the requested INT8 coverage. This template follows the global-token bias used in BigBird- and Longformer-style mechanisms. Row-Random. 37 Each query-tile row independently selects a fixed-size subset of legal key-tile groups uniformly at random for FP16 routing: ℱmRowRand∼UniformSubset⁡(m,km),F_m^RowRand (G_m,k_m ), where km=round⁡((1−ρINT8)|m|).k_m=round ((1- _INT8)|G_m| ). Sampling is performed without replacement using a fixed random seed, and the resulting routing template is reused throughout evaluation. The remaining groups are routed to INT8. This template follows the random component of BigBird and distributes FP16 across nearby and long-range positions. Aligned Sparse. 10; 22 Aligned Sparse assigns a right-aligned region of legal key-tile groups to FP16. Let a⁡(m)=min⁡(|m|,⌈α+βm⌉)a(m)= (|G_m|, α+β m ) denote the number of FP16 groups assigned to query-tile row m, where α controls the initial width and β controls its growth. If gmmax=max⁡m,g_m = _m, then ℱmAlign=gj∈m:gj≥gmmax−a⁡(m)+1.F_m^Align= \g_j _m:g_j≥ g_m -a(m)+1 \. The number of FP16-routed groups therefore increases monotonically with the query index, producing a right-aligned template with an expanding FP16 region. The parameters α and β are selected to match the requested INT8 coverage. BigBird. 37 This template combines (i) local-band routing, (i) global-token routing, and (i) row-wise random routing. Its FP16 set is ℱmBigBird=ℱmBand∪ℱmGlobal∪ℱmRowRand.F_m^BigBird=F_m^Band _m^Global _m^RowRand. A legal tile group selected by any component remains on the FP16 path, while the remaining legal groups use INT8. The local, global, and random component sizes are selected jointly to match the requested tile-group INT8 coverage. TileMix therefore interprets the corresponding spatial topology as a composite precision-routing template rather than a structural attention mask. Sparse Transformer (SpTrans). 5 SpTrans combines (i) the local stride containing the current query position and (i) the final c positions of each preceding stride. For stride length s and query position i, let u⁡(i)=⌊is⌋u(i)= is denote its current stride. The selected FP16 key positions are iSpTrans=[u⁡(i)s,i]∪⋃r=0u⁡(i)−1[max⁡(rs,(r+1)s−c),(r+1)s−1].K_i^SpTrans= [u(i)s,\,i ]∪ _r=0^u(i)-1 [ \! (rs,(r+1)s-c ),\,(r+1)s-1 ]. For query-tile row m, the corresponding FP16 routing set is ℱmSpTrans=gj∈m:∃i∈ℐm such that gj∩iSpTrans≠∅.F_m^SpTrans= \g_j _m:∃ i _m such that J_g_j _i^SpTrans≠ \. These selected tile groups are routed to FP16, while the remaining legal tile groups are routed to INT8. The stride length s and tail width c determine the spatial structure, and the resulting tile-group allocation is configured for the requested INT8 coverage. Layouts Reported in the Main Evaluation. The main LV-Eval table reports BigBird and SpTrans as representative structured layouts with complementary spatial organizations and empirical quality–efficiency behavior. Complete results for Band, Global, Row-Random, Aligned Sparse, BigBird, and SpTrans are reported in this appendix across models, context lengths, and INT8 coverage levels. The supported routing layouts do not introduce sparsity into the attention computation. Instead, TileMix translates established spatial structures into hardware-aligned, block-level precision-routing decisions. This decouples which interactions are legal from which arithmetic path computes them, enabling training-free heterogeneous-precision execution with dense token connectivity. Appendix C Quantization Configuration This section specifies the numerical configuration of the primary score-routing implementation. Blockwise INT8 quantization is applied to Q and K, while V and the PVPV computation remain in FP16. Both score paths enter one common floating-point score domain before updating the shared online-softmax state. Blockwise INT8 Quantization. For a valid floating-point block X(r)X^(r), TileMix uses signed symmetric INT8 quantization with absmax scale δX(r)=maxx∈X(r)⁡|x|127. _X^(r)= _x∈ X^(r)|x|127. The quantized representation is obtained by nearest-integer conversion: X^(r)=round⁡(X(r)δX(r)),X^(r)∈−127,…,127. X^(r)=round ( X^(r) _X^(r) ),\; X^(r)∈\-127,…,127\. Because the scale is determined by the block maximum, finite normalized values lie within the stated signed range. The quantization kernel converts block values to FP32 when computing the scale and quantized representation. The resulting scale tensors are stored in the operand dtype by the production wrapper; the FP16 experiments therefore use FP16 scale tensors. Quantization Granularity. The evaluated score-routing path uses BLKQ=128,BLKK=64.BLK_Q=128, _K=64. Each query block has shape 128×d,128× d, and each key block has shape 64×d,64× d, with one scale per block and attention head. For variable-length batches, the scale tensors have logical shapes [∑b⌈Lq,b128⌉,Hq]and[∑b⌈Lk,b64⌉,Hk]. [ _b L_q,b128 ,H_q ] [ _b L_k,b64 ,H_k ]. Prefix-sum metadata maps each sequence and token block to its scale entry. INT8 Score Computation. For query block r and key block s, an INT8-routed score tile first computes Cr,sINT32=Q8(r)(K8(s))⊤C_r,s^INT32=Q_8^(r) (K_8^(s) ) using INT8 Tensor Core multiplication with INT32 accumulation. The accumulator is converted to floating point and rescaled as Sr,sINT8=δQ(r)δK(s)Cr,sINT32d.S_r,s^INT8= _Q^(r) _K^(s)C_r,s^INT32 d. An FP16-routed tile computes Sr,sFP16=Q(r)(K(s))⊤d.S_r,s^FP16= Q^(r) (K^(s) ) d. Both paths enter FP16 score domain before exponentiation and update shared online-softmax state. The running maximum, normalizer, and output accumulator are stored in FP16. Tile maxima, exponentiation, and row reductions use FP32 intermediates before their results are incorporated into the shared FP16 state. The resulting probability tile and the PVPV update use FP16. Quantize-Once Execution. The implementation-level prefill path generates Q8Q_8, K8K_8, and their compact block scales once before the attention launch. These operands are reused by all INT8-routed score-tile groups, while FP16-routed groups read the original FP16 Q and K tensors. The all-FP16 configuration skips this quantization step. Reported implementation-level timings include operand quantization, scale preparation, routing, rescaling, memory staging, and fused attention. Additional Execution Interfaces. The implementation also provides an on-the-fly operand-generation mode and an INT8 key/value cache interface. The cache interface stores K8K_8 and V8V_8 together with compact per-block scales. During decode, cached K8K_8 participates in the routed INT8 score path; cached V8V_8 is converted to FP16 on chip, with its block scale folded into the probability tile before the PVPV dot product. This interface is separate from the primary prefill configuration, which quantizes the routed Q/KQ/K score operands and retains FP16 V for every PVPV update. Appendix D Storage and Execution Layout Building on the numerical configuration in Appendix C, TileMix keeps the dense FP16 Q,K,VQ,K,V tensors in the standard contiguous HBM layout and stores routing metadata separately as one packed 64-bit word for each (hk,m)(h_k,m), where hkh_k is a KV-head index and m is a query-tile row. The fused inner loop reads this word to select the FP16 or INT8 score path for each legal tile group. Tensor addressing, token connectivity, and tile legality retain the dense FlashAttention-style execution structure. TileMix supports two operand-preparation modes, as illustrated in Figure 7. The quantize-once mode is used by the reported quality and implementation-level efficiency evaluations. It generates temporary INT8 Q/KQ/K tensors and their block scales once per attention call and reuses them across all INT8-routed groups. The original FP16 Q/KQ/K tensors remain available to FP16-routed groups, while V and PVPV computation remain FP16 throughout the primary evaluated path. TileMix quantize-once TileMix fused on-the-fly ------------------------------------- ------------------------------------- [HBM] FP16 Q/K/V [HBM] FP16 Q/K/V | | |-- pre-quantization pass |-- no separate pre-quantization pass | [L2] stream BLK x d tiles into quant kernel | | [On-SM] compute scales, quantize to INT8 | | [HBM] write temporary QINT8/KINT8 + scales | v v [HBM] Q/K/V + QINT8/KINT8 + scales + bitmask [HBM] Q/K/V + bitmask | | v v [L2 / On-SM] [L2 / On-SM] | | | load Q or QINT8 once per query row | load FP Q once per query row | stream K/V tiles | stream FP K/V tiles | online-softmax in registers | compute scales on chip | | quantize only INT8-dispatched groups | | online-softmax in registers | | v v [Tensor Cores] [Tensor Cores] | | | INT8 QK: qint8 @ kint8 -> int32 MMA -> rescale | INT8 QK: qint8 @ kint8 -> int32 MMA -> rescale | FP QK: q @ k -> fp16/fp32 MMA | FP QK: q @ k -> fp16/fp32 MMA | FP PV: p @ v | FP PV: p @ v | FP PV: p @ v | FP PV: p @ v | | v v [HBM] Out [HBM] Out Figure 7: Storage and execution layout of the two TileMix operand-preparation modes for the primary prefill path. Quantize-once materializes temporary INT8 Q/KQ/K tensors and their scales once per attention call and reuses them across INT8-routed score-tile groups. Fused on-the-fly generates INT8 Q/KQ/K fragments inside the kernel only for INT8-routed groups. Both modes retain V and the PVPV computation in FP16. The separate INT8 key/value cache interface is described in the surrounding text. The fused on-the-fly mode is an additional implementation mode that loads the FP16 operands and generates INT8 fragments only for groups selected by the routing map. This mode removes the attention-call-local Q8/K8Q_8/K_8 scratch tensors from HBM and performs scale computation and quantization on-chip. Both modes use the same packed routing policy, routed score computation, and shared online-softmax update. The kernel interface additionally supports INT8 key/value caches with per-block scale metadata. In this cache path, cached K8K_8 supplies the routed INT8 score computation, while cached V8V_8 is converted to FP16 on chip and used in the floating-point PVPV update. This cache interface is described separately from the primary prefill path illustrated in Figure 7. The primary evaluation uses routed FP16/INT8 QK⊤QK score computation with FP16 V/PV/PV, as specified in Appendix C. For clarity, Table 5 reports attention-call-local memory and data-movement accounting under one representative configuration: Hq=32H_q=32, Hk=8H_k=8, Lq=Lk=8192L_q=L_k=8192, d=128d=128, BLKQ=128BLK_Q=128, and BLKK=64BLK_K=64. Here, HqH_q and HkH_k are the query-head and KV-head counts, LqL_q and LkL_k are the query and key sequence lengths, d is the per-head dimension, and BLKQBLK_Q and BLKKBLK_K are the query and key quantization-block sizes. The accounting distinguishes: (i) tensors resident in HBM during one attention call; (i) global/L2 operand reads for one routed QK or optional PV tile event; and (i) the simultaneously live on-SM working state for one FP16 tile event. The INT8 QK column corresponds to the primary score-routing path, while the V8V_8-backed PV column separately characterizes value-cache traffic in the INT8 key/value cache interface. Resident HBM for the TileMix quantize-once row reports the primary score-only prefill configuration. These quantities describe operator-local storage and traffic, distinct from full-model peak GPU memory. Route Resident HBM INT8 QK tile global/L2 read V8V_8-backed PV tile global/L2 read FP QK tile on-SM live set FP PV tile on-SM live set FP QK/PV Tensor Core accumulator FlashAttention 96 MB — — 128 KB 128 KB 64 KB TileMix quantize-once 136.01 MiB 16.25 KB 16.25 KB 96 KB 96 KB 32 KB TileMix fused on-the-fly 96 MB 32 KB 32 KB 96 KB 96 KB 32 KB Table 5: Memory and data-movement accounting for FlashAttention and TileMix under Hq=32H_q=32, Hk=8H_k=8, Lq=Lk=8192L_q=L_k=8192, d=128d=128, BLKQ=128BLK_Q=128, and BLKK=64BLK_K=64. Resident HBM counts persistent tensors and route-specific temporary scratch tensors during the attention call. Global/L2 reads are reported for one INT8-routed QK or PV tile event. The on-SM live set reports the simultaneously live on-chip working state for the corresponding FP16-routed tile event. Under this representative configuration, quantize-once uses 136.01 MiB of resident HBM because it materializes reusable INT8 scratch tensors and their scales for the duration of the attention call. This preparation reduces the operand read for one INT8-routed QK tile event from 32 KB in fused on-the-fly execution to 16.25 KB. The same traffic relation applies to V8V_8-backed PV reads in the optional INT8 key/value cache interface. Fused on-the-fly retains the same 96 MB resident-HBM footprint as the FlashAttention configuration in this accounting and generates INT8 operands from FP16 tiles on chip. For FP16-routed QK and PV events, the TileMix tile path uses a 96 KB on-SM live set and a 32 KB Tensor Core accumulator, compared with 128 KB and 64 KB, respectively, under the reported FlashAttention accounting. These measurements characterize the local storage and data-movement behavior of the attention implementations; full-model OOM behavior is considered separately from this operator-level accounting. Appendix E Model Performance Long-context Retrieval. Figure 8 and Figure 9 report LongEval results for Qwen 2.5 7B and Qwen 2 7B across extended context lengths. Vicuna 7B, however, supports a maximum context length of 16k tokens. To avoid sparsely populated plots and to present results at all supported lengths clearly, we report Vicuna 7B performance in tabular form (Table 6) instead of figures. The table follows the same evaluation protocol, precision layouts, and INT8 ratios as the figure-based results, allowing direct comparison across models within their respective context limits. Figure 8: Line-level retrieval accuracy on the LongEval benchmark for Qwen 2.5 7B under different tile-group routing layouts, evaluated across prompt lengths from 3.1k to 38.7k tokens. Line plots (left y-axis) report exact-match retrieval accuracy, while bar plots (right y-axis, Δ accuracy vs. FP16) show differences relative to the FP16 attention baseline. Each panel corresponds to a routing layout with 25%, 50%, or 75% of tile groups routed to INT8. Colors indicate routing settings: green denotes the original FP16 attention baseline, red denotes One (all legal score-tile groups routed to INT8), and blue denotes mixed configurations with partially INT8-routed tile groups. Figure 9: Line-level retrieval accuracy on the LongEval benchmark for Qwen 2 7B under different precision policy layouts, evaluated across prompt lengths from 3.1k to 38.7k tokens. Line | Len | FP16 | INT8 | AlignSparse Band BigBird Global RowRand SpTrans | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 200 | 4776.33 | 0.99 | 0.97 | 0.99 0.99 1.00 | 0.99 0.99 1.00 | 1.00 1.00 1.00 | 0.99 1.00 1.00 | 1.00 1.00 1.00 | 1.00 1.00 1.00 300 | 7081.42 | 1.00 | 0.99 | 1.00 1.00 1.00 | 1.00 1.00 1.00 | 1.00 1.00 1.00 | 0.99 1.00 1.00 | 1.00 1.00 1.00 | 1.00 1.00 1.00 400 | 9384.62 | 0.99 | 0.98 | 0.98 0.99 0.99 | 0.98 0.99 0.99 | 0.99 0.99 0.99 | 0.98 0.99 0.99 | 0.99 0.99 0.99 | 0.99 0.99 0.99 500 | 11691.21 | 0.85 | 0.76 | 0.82 0.80 0.78 | 0.82 0.81 0.82 | 0.82 0.82 0.83 | 0.82 0.82 0.82 | 0.82 0.82 0.82 | 0.81 0.82 0.82 600 | 13991.58 | 0.73 | 0.63 | 0.69 0.65 0.63 | 0.69 0.69 0.68 | 0.69 0.70 0.70 | 0.69 0.69 0.70 | 0.69 0.69 0.69 | 0.69 0.69 0.67 700 | 16294.20 | 0.52 | 0.44 | 0.48 0.47 0.45 | 0.49 0.48 0.46 | 0.49 0.49 0.50 | 0.49 0.49 0.50 | 0.49 0.49 0.48 | 0.49 0.48 0.47 Table 6: Line-level retrieval accuracy on the LongEval benchmark for Vicuna 7B under different precision layouts inspired by sparse-attention patterns. The task requires the model to scan long input sequences and exactly retrieve the content associated with a queried line identifier. Accuracy is measured as exact-match retrieval rate, reported across increasing input lengths. Results compare full FP16 attention, pure INT8 attention, and mixed-precision configurations with varying precision layouts and INT8 ratios, illustrating how different precision-allocation patterns affect robustness under low-precision execution. Together, Figure 8, Figure 9, and Table 6 show similar qualitative trends across Qwen 2.5 7B, Qwen 2 7B, and Vicuna 7B. Across models, uniform INT8 attention (One) often degrades retrieval accuracy as prompt length increases, showing that a single low-precision arithmetic path can be too aggressive for long-context retrieval. In contrast, mixed-routing configurations generally recover accuracy by keeping selected tile groups on the FP16 path while routing the remaining groups to INT8. Across the three model families, mixed-routing configurations recover substantial retrieval quality relative to One, while sensitivity to layout and INT8 coverage varies with the model and prompt length. bigbird, row_rand, and sptrans provide the most consistent recovery across the evaluated models, particularly at higher INT8 coverage. Overall, these results suggest that tile-group precision routing provides a useful precision-budget mechanism across model families, while the best routing layout and INT8 ratio remain model- and task-dependent. Long-context Question Answering. Table 7 isolates the comparison against sparse long-context and INT8 attention baselines on Qwen 2 7B. Across the reported LV-Eval subsets, SpTrans mixed-precision configurations close much of the gap to FP16 while outperforming or matching sparse baselines in most settings. This supports the central distinction of TileMix: it recovers long-context quality through precision routing while preserving dense token connectivity. Tables 8, 9, and 10 report LV-Eval accuracy for LLaMA 3.2 3B, Qwen 2 7B, and Qwen 2.5 7B across context lengths from 16k to 64k under FP16 attention (Baseline), One (100% INT8 tiles), and mixed-precision configurations with different precision layouts and INT8 ratios (25/50/75%). Dataset | Len | One | SpTrans25 | SpTrans50 | SpTrans75 | MInference FlexPrefill SageAttn 1 | 16k | 31.60 | 36.00 | 34.30 | 33.40 | 29.00 26.13 32.60 | 32k | 14.70 | 20.00 | 19.00 | 18.50 | 15.00 11.89 16.90 | 64k | 6.60 | 9.70 | 8.80 | 8.20 | 6.40 8.98 9.10 2 | 16k | 13.90 | 16.70 | 16.20 | 15.10 | 14.10 15.74 14.20 | 32k | 11.40 | 15.10 | 14.60 | 14.00 | 11.50 15.24 13.50 | 64k | 8.70 | 13.20 | 12.50 | 12.10 | 9.50 12.65 10.80 7 | 16k | 12.20 | 17.10 | 16.50 | 16.30 | 14.90 17.09 15.80 | 32k | 9.30 | 15.30 | 14.90 | 14.40 | 12.60 16.08 13.80 | 64k | 5.80 | 11.70 | 11.10 | 10.80 | 9.00 10.29 11.40 8 | 16k | 12.10 | 16.90 | 16.30 | 16.70 | 15.20 13.65 15.10 | 32k | 8.40 | 12.60 | 12.20 | 12.00 | 10.80 12.26 10.80 | 64k | 7.30 | 10.80 | 10.10 | 9.90 | 8.10 9.46 9.50 9 | 16k | 25.70 | 33.80 | 33.10 | 33.30 | 28.50 29.07 31.40 | 32k | 15.70 | 22.90 | 22.60 | 22.10 | 18.10 20.71 21.00 | 64k | 9.00 | 15.90 | 15.50 | 15.00 | 11.40 14.53 13.70 10 | 16k | 19.30 | 25.10 | 24.60 | 24.00 | 21.20 23.89 22.60 | 32k | 14.20 | 19.30 | 18.60 | 18.20 | 15.30 17.67 17.90 | 64k | 14.00 | 17.50 | 16.90 | 17.00 | 12.80 15.95 15.90 11 | 16k | 25.30 | 34.30 | 33.50 | 32.30 | 28.70 22.74 31.40 | 32k | 20.10 | 25.30 | 24.60 | 24.50 | 19.30 15.98 23.50 | 64k | 6.90 | 18.40 | 17.20 | 15.30 | 13.00 15.22 16.10 Table 7: LV-Eval long-context question answering results on Qwen2-7B. We compare TileMix pure INT8 (One), TileMix SpTrans mixed-precision variants, and long-context attention baselines. SpTrans25, SpTrans50, and SpTrans75 denote configurations with 25%, 50%, and 75% INT8-routed tile groups. Datasets are denoted by the same indices defined in the experimental setup. Dataset | Len | FP16 | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1 | 16k | 32.04 | 28.78 | 32.91 32.77 32.78 | 32.52 30.76 30.86 | 31.27 29.64 29.75 | 31.75 30.18 29.87 | 31.89 30.94 30.64 | 31.75 32.18 30.25 | 32k | 15.08 | 11.62 | 15.75 13.38 12.42 | 12.28 13.18 13.61 | 15.22 14.02 13.55 | 15.56 14.24 13.86 | 15.26 13.23 12.61 | 15.56 14.77 14.37 | 64k | 7.75 | 5.42 | 7.81 6.86 5.86 | 7.81 5.84 5.81 | 8.01 6.47 5.72 | 8.01 7.03 6.25 | 8.01 6.32 6.08 | 8.01 7.44 6.83 2 | 16k | 18.49 | 15.62 | 18.95 17.08 16.83 | 19.16 16.57 17.23 | 18.61 16.98 16.62 | 18.64 17.62 16.88 | 18.39 16.58 16.56 | 18.64 18.08 16.78 | 32k | 15.12 | 12.02 | 15.18 14.34 13.16 | 15.48 13.59 13.09 | 15.14 13.53 13.28 | 15.27 14.78 13.59 | 15.58 14.15 13.09 | 15.27 14.69 14.18 | 64k | 11.84 | 7.79 | 12.15 10.00 9.12 | 12.30 10.17 10.10 | 12.04 10.28 10.03 | 12.06 10.96 10.28 | 12.19 10.39 10.13 | 12.06 11.34 11.04 3 | 16k | 6.72 | 4.45 | 6.46 5.62 4.93 | 6.52 24.37 4.37 | 6.29 6.97 4.57 | 6.29 5.31 4.50 | 6.57 4.72 4.61 | 21.04 20.53 21.65 | 32k | 3.78 | 1.95 | 3.77 3.07 2.92 | 3.64 1.35 1.49 | 2.35 1.81 1.20 | 3.35 2.37 1.70 | 5.96 2.04 2.65 | 3.35 2.78 2.68 | 64k | 3.38 | 1.42 | 3.43 2.34 1.54 | 3.40 1.42 1.41 | 3.24 1.70 1.37 | 3.24 2.26 1.45 | 3.40 1.51 1.43 | 3.24 2.67 2.47 4 | 16k | 6.00 | 2.27 | 7.50 6.00 5.50 | 5.50 4.50 4.20 | 6.00 4.46 4.49 | 6.00 5.02 4.21 | 6.00 4.76 5.02 | 6.00 5.43 5.04 | 32k | 18.00 | 13.98 | 18.50 17.10 14.20 | 18.50 16.50 17.00 | 18.50 16.46 15.99 | 18.00 17.02 16.71 | 18.50 16.26 16.02 | 18.00 17.43 17.54 | 64k | 14.50 | 10.07 | 15.00 14.20 12.50 | 14.00 12.00 12.00 | 15.00 13.46 12.49 | 15.00 14.02 12.71 | 15.50 13.26 13.02 | 15.00 14.43 14.04 5 | 16k | 17.51 | 12.86 | 17.21 17.21 16.19 | 17.69 15.51 15.25 | 17.66 15.17 14.84 | 17.66 15.87 15.87 | 17.66 15.92 16.49 | 17.66 17.09 16.88 | 32k | 10.80 | 8.38 | 11.01 10.15 9.01 | 10.85 8.79 8.49 | 10.61 9.07 9.00 | 11.01 10.03 8.75 | 10.80 9.28 9.21 | 11.01 10.44 10.06 | 64k | 6.82 | 5.32 | 7.70 6.39 5.70 | 6.89 5.97 5.39 | 7.63 6.09 5.62 | 7.63 6.65 5.84 | 7.70 5.96 4.91 | 7.63 6.25 5.97 6 | 16k | 12.33 | 8.51 | 12.43 11.77 9.79 | 11.66 9.70 6.91 | 12.38 10.40 10.30 | 12.31 11.33 10.52 | 12.02 10.27 10.53 | 12.31 11.74 11.03 | 32k | 6.73 | 3.22 | 6.12 5.76 3.88 | 7.04 4.84 4.90 | 6.52 5.18 4.45 | 7.08 6.10 4.73 | 6.66 5.19 4.56 | 7.08 6.51 5.95 | 64k | 1.84 | 0.74 | 1.99 1.87 1.02 | 0.84 0.86 0.86 | 1.91 1.35 1.13 | 1.87 1.59 1.05 | 1.99 1.89 1.24 | 1.97 1.37 1.22 7 | 16k | 21.75 | 15.69 | 21.88 22.36 20.47 | 27.84 19.86 19.03 | 21.96 20.81 20.09 | 22.07 21.08 20.23 | 22.41 20.10 20.08 | 22.07 21.16 21.05 | 32k | 19.89 | 12.11 | 19.16 17.35 14.23 | 19.39 17.36 16.63 | 20.17 18.41 17.79 | 19.91 19.07 17.99 | 19.88 17.96 17.47 | 19.91 19.37 18.56 | 64k | 14.18 | 6.93 | 14.13 13.98 10.91 | 14.13 12.45 11.25 | 14.09 12.37 11.97 | 14.23 13.31 12.23 | 14.14 12.27 11.79 | 14.23 13.66 13.34 8 | 16k | 18.24 | 12.72 | 18.16 16.12 13.13 | 18.50 16.65 15.53 | 17.57 16.45 15.96 | 17.53 16.97 15.75 | 18.12 16.27 16.28 | 17.53 16.99 17.46 | 32k | 14.25 | 9.12 | 13.98 12.13 11.43 | 14.44 12.64 11.40 | 13.73 12.73 12.29 | 13.63 13.03 12.21 | 14.19 12.35 12.17 | 13.63 13.18 13.00 | 64k | 11.08 | 7.54 | 11.06 9.11 8.28 | 11.22 9.23 8.07 | 11.05 9.60 9.04 | 11.31 10.09 9.56 | 11.08 9.18 8.73 | 11.31 10.65 10.38 9 | 16k | 45.68 | 34.70 | 46.02 43.19 39.69 | 45.91 43.60 42.61 | 45.13 43.66 43.86 | 45.77 44.79 43.52 | 45.52 43.93 44.04 | 45.77 44.60 44.88 | 32k | 26.84 | 18.17 | 25.76 23.20 21.69 | 26.08 24.36 23.41 | 26.47 24.94 23.78 | 26.05 25.49 24.71 | 26.90 25.05 24.58 | 26.05 25.94 25.11 | 64k | 16.36 | 9.19 | 15.60 14.58 11.96 | 16.30 14.21 12.45 | 15.72 14.18 14.04 | 16.03 14.85 14.24 | 16.14 14.26 13.78 | 16.03 15.25 14.71 10 | 16k | 28.88 | 22.37 | 28.26 26.21 25.80 | 28.80 27.06 25.79 | 28.95 27.57 26.33 | 29.10 28.13 27.15 | 28.17 26.31 26.42 | 29.10 28.31 27.23 | 32k | 18.33 | 13.66 | 18.03 15.99 15.98 | 17.84 16.20 15.13 | 18.17 16.70 16.88 | 18.26 17.35 16.44 | 18.25 16.55 16.51 | 18.26 17.65 17.41 | 64k | 15.75 | 12.81 | 16.08 15.91 14.98 | 16.18 14.13 13.18 | 15.93 14.22 13.88 | 15.94 14.94 14.13 | 15.85 14.77 14.41 | 15.94 15.32 15.39 11 | 16k | 22.44 | 16.97 | 23.56 22.64 20.75 | 22.36 21.03 19.89 | 22.44 20.24 20.76 | 20.29 21.46 26.06 | 21.74 20.34 21.62 | 22.94 22.37 21.74 | 32k | 14.85 | 12.02 | 15.53 15.28 14.49 | 15.28 13.28 14.18 | 16.11 14.57 13.34 | 15.61 14.63 13.56 | 15.81 13.57 13.54 | 15.61 15.04 15.06 | 64k | 9.26 | 3.41 | 8.36 7.36 5.57 | 8.36 6.36 5.36 | 8.92 8.38 7.91 | 8.92 7.94 7.13 | 8.92 7.12 6.38 | 8.92 8.35 7.40 Table 8: LV-Eval long-context question answering for LLaMA 3.2 3B. Cells are color-coded by comparison to One and Baseline: red indicates performance below One, blue indicates performance between One and the Baseline, and green indicates performance above the Baseline. Across datasets and lengths, One generally underperforms the FP16 baseline, indicating that uniform INT8 attention can be too coarse for long-context QA. Mixed-routing configurations usually narrow this gap, especially at moderate INT8 ratios. The effect of increasing INT8 coverage is layout dependent: align_sparse, band, and global often benefit from more conservative INT8 ratios, whereas bigbird, row_rand, and sptrans often tolerate higher INT8 coverage. A consistent layout–task interaction appears on the 16k factrecall_en setting. Across SpTrans25, SpTrans50, and SpTrans75, the scores reach 21.04/20.53/21.65 on LLaMA 3.2 3B, 50.6/49.7/52.0 on Qwen 2 7B, and 31.4/30.7/32.2 on Qwen 2.5 7B, compared with the corresponding FP16 scores of 6.72, 16.39, and 10.22. The recurrence across model families and all three routing ratios shows a cross-model consistent layout–task interaction at this setting, while the corresponding SpTrans results at 32k and 64k return to the usual quality range. Overall, these tables support the use of tile-group routing as a controllable precision-budget mechanism for long-context QA, while the best routing layout depends on the model, task, and context length. Appendix F Efficiency Tables 11, 12, 13, and 14 report implementation-level prefill efficiency across LLaMA 3.2 3B, Qwen 2.5 7B, Qwen 2 7B, and Vicuna 7B using throughput (Thpt, K tokens/s) and TOPS from 1k to 8k where executable, with model-dependent maximum sequence lengths (batch size 8; 3 warmup iterations; 5 measurement iterations). We compare the standard Torch implementation, FlashAttention as an IO-aware FP16 baseline, One (uniform all legal score-tile groups routed to INT8) as an efficiency-oriented reference, and TileMix mixed-routing variants under different routing layouts and INT8 ratios (25/50/75%). Dataset | Len | FP16 | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1 | 16k | 34.97 | 31.6 | 36.0 35.6 35.5 | 35.1 33.6 33.8 | 34.4 32.7 32.9 | 35.0 33.6 33.1 | 35.8 34.7 34.1 | 36.0 34.3 33.4 | 32k | 19.14 | 14.7 | 20.0 17.0 15.9 | 15.6 16.7 17.0 | 19.4 17.8 17.3 | 19.8 18.1 17.6 | 19.5 17.0 16.4 | 20.0 19.0 18.5 | 64k | 9.34 | 6.6 | 9.6 8.4 7.3 | 9.5 7.2 7.1 | 9.7 7.8 7.1 | 9.6 8.5 7.6 | 9.8 7.7 7.4 | 9.7 8.8 8.2 2 | 16k | 16.39 | 13.9 | 16.8 15.4 15.1 | 17.1 15.0 15.5 | 16.4 15.0 14.8 | 16.6 15.7 15.0 | 16.4 14.9 14.9 | 16.7 16.2 15.1 | 32k | 14.42 | 11.4 | 14.3 13.4 12.6 | 14.6 12.9 12.4 | 14.5 13.0 12.6 | 14.6 14.0 12.9 | 14.8 13.4 12.7 | 14.6 14.0 13.5 | 64k | 12.97 | 8.7 | 13.3 11.0 10.3 | 13.4 11.2 11.0 | 13.2 11.4 10.9 | 13.3 12.0 11.2 | 13.5 11.6 11.2 | 13.2 12.5 12.1 3 | 16k | 16.39 | 10.7 | 15.8 13.9 12.2 | 15.9 5.9 10.9 | 15.4 16.6 11.0 | 15.4 13.2 11.4 | 16.1 11.5 11.2 | 50.6 49.7 52.0 | 32k | 10.12 | 5.2 | 9.9 8.0 7.6 | 9.6 3.5 3.9 | 6.2 4.8 3.1 | 9.2 6.5 4.7 | 15.7 5.4 6.8 | 9.1 7.4 7.1 | 64k | 5.33 | 2.2 | 5.4 3.9 2.6 | 5.4 2.4 2.4 | 5.2 2.7 2.3 | 5.2 3.5 2.4 | 5.5 2.5 2.4 | 5.3 4.3 4.0 5 | 16k | 14.21 | 10.6 | 14.2 14.1 13.2 | 14.4 12.8 12.5 | 14.4 12.4 12.1 | 14.4 12.9 12.8 | 14.3 12.7 13.5 | 14.5 14.0 13.8 | 32k | 8.16 | 6.3 | 8.3 7.6 6.7 | 8.1 6.6 6.4 | 8.0 6.9 6.8 | 8.3 7.6 6.7 | 8.2 7.1 7.0 | 8.3 7.8 7.5 | 64k | 6.81 | 5.3 | 7.6 6.3 5.6 | 7.4 6.4 5.7 | 7.7 6.6 6.1 | 7.7 6.7 6.0 | 7.8 6.1 5.1 | 7.7 6.3 6.0 6 | 16k | 23.28 | 16.0 | 23.3 22.1 18.6 | 21.9 18.2 13.3 | 23.2 19.7 19.4 | 23.0 21.2 19.8 | 22.6 19.3 19.9 | 23.2 22.1 21.3 | 32k | 13.73 | 6.6 | 13.1 12.3 8.3 | 14.4 9.9 10.0 | 13.4 10.7 9.1 | 14.6 12.6 9.7 | 13.8 10.7 9.4 | 14.6 13.5 12.3 | 64k | 6.91 | 2.9 | 7.4 7.0 3.8 | 3.2 3.3 3.3 | 7.2 5.1 4.3 | 7.0 6.0 3.9 | 7.4 7.0 4.7 | 7.3 5.2 4.6 7 | 16k | 16.96 | 12.2 | 17.1 17.3 15.9 | 21.1 15.1 14.6 | 17.0 15.9 15.4 | 17.2 16.4 15.8 | 17.3 15.6 15.5 | 17.1 16.5 16.3 | 32k | 15.09 | 9.3 | 14.7 13.6 11.1 | 14.9 13.3 12.7 | 15.4 14.1 13.5 | 15.2 14.6 13.7 | 15.1 13.8 13.4 | 15.3 14.9 14.4 | 64k | 11.62 | 5.8 | 11.6 11.2 8.7 | 11.5 10.1 9.2 | 11.6 10.2 9.8 | 11.6 10.9 10.2 | 11.6 10.1 9.6 | 11.7 11.1 10.8 8 | 16k | 17.36 | 12.1 | 17.2 15.3 12.5 | 17.7 16.0 14.9 | 16.8 15.7 15.2 | 16.6 16.1 15.1 | 17.2 15.6 15.7 | 16.9 16.3 16.7 | 32k | 13.03 | 8.4 | 12.7 11.0 10.2 | 13.3 11.6 10.5 | 12.6 11.6 11.2 | 12.4 11.8 11.1 | 13.1 11.5 11.3 | 12.6 12.2 12.0 | 64k | 10.73 | 7.3 | 10.8 9.1 8.2 | 11.0 9.3 8.2 | 10.7 9.2 8.7 | 10.9 9.8 9.3 | 10.7 9.3 8.9 | 10.8 10.1 9.9 9 | 16k | 33.66 | 25.7 | 33.7 31.7 29.1 | 33.8 32.2 31.5 | 33.4 32.2 32.4 | 33.7 33.0 32.1 | 33.6 32.4 32.6 | 33.8 33.1 33.3 | 32k | 23.30 | 15.7 | 22.6 20.5 19.0 | 22.8 21.3 20.6 | 23.2 21.9 20.9 | 22.9 22.4 21.6 | 23.6 22.0 21.5 | 22.9 22.6 22.1 | 64k | 15.95 | 9.0 | 16.0 14.8 12.1 | 15.9 13.9 12.5 | 15.7 14.2 14.1 | 16.0 14.9 14.3 | 16.1 14.4 13.9 | 15.9 15.5 15.0 10 | 16k | 24.88 | 19.3 | 24.4 22.9 22.5 | 25.0 23.3 22.2 | 25.2 23.9 23.0 | 25.3 24.5 23.7 | 24.7 23.1 23.3 | 25.1 24.6 24.0 | 32k | 19.22 | 14.2 | 18.8 16.7 16.5 | 18.8 17.1 16.1 | 19.1 17.6 17.3 | 19.3 18.4 17.5 | 19.2 17.5 17.3 | 19.3 18.6 18.2 | 64k | 17.20 | 14.0 | 17.7 17.2 16.3 | 17.6 15.4 14.6 | 17.4 15.7 15.4 | 17.5 16.4 15.6 | 17.3 16.2 15.9 | 17.5 16.9 17.0 11 | 16k | 33.70 | 25.3 | 35.3 33.9 31.2 | 33.5 31.5 29.8 | 33.7 30.3 31.0 | 33.9 32.4 31.2 | 33.6 31.0 32.1 | 34.3 33.5 32.3 | 32k | 24.82 | 20.1 | 25.6 25.0 23.7 | 25.2 22.0 23.6 | 26.2 23.7 21.8 | 25.4 23.8 22.1 | 25.9 22.3 22.2 | 25.3 24.6 24.5 | 64k | 18.51 | 6.9 | 17.1 15.3 11.6 | 17.1 13.1 11.1 | 18.3 17.2 16.3 | 18.3 16.4 14.9 | 18.2 14.6 13.1 | 18.4 17.2 15.3 Table 9: LV-Eval for Qwen 2 7B. Cells are color-coded by comparison to One and Baseline: red indicates performance below One, blue indicates performance between One and the Baseline, and green indicates performance above the Baseline. Len | Metric | Torch | Flash | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1k | Thpt | 11.14 | 17.45 | 32.27 | 29.22 32.02 33.48 | 27.06 29.30 34.56 | 26.92 31.48 32.37 | 26.71 29.59 32.03 | 28.66 32.09 34.95 | 27.66 32.11 33.50 | TOPS | 41.70 | 65.31 | 120.80 | 109.38 119.86 125.29 | 101.31 109.71 129.42 | 100.78 117.85 121.17 | 99.98 110.75 119.92 | 107.25 120.12 130.80 | 103.50 120.20 125.38 2k | Thpt | 7.78 | 16.48 | 32.06 | 29.50 32.41 33.84 | 26.77 28.89 34.30 | 26.57 28.83 32.71 | 26.45 29.29 31.74 | 26.34 31.35 34.88 | 27.11 31.46 33.92 | TOPS | 30.50 | 64.64 | 125.70 | 115.70 127.08 132.67 | 104.99 113.30 134.48 | 104.23 113.05 128.31 | 103.69 114.83 124.46 | 103.28 122.91 136.77 | 106.32 123.38 132.99 4k | Thpt | OOM | 14.33 | 29.80 | 27.71 30.35 31.62 | 25.65 28.59 32.81 | 24.69 26.84 27.48 | 24.50 27.19 29.35 | 26.97 30.06 32.89 | 27.14 30.59 31.80 | TOPS | OOM | 61.31 | 127.48 | 118.53 129.85 135.26 | 109.76 122.32 140.39 | 105.63 114.83 117.56 | 104.82 116.35 125.56 | 115.37 128.62 140.73 | 116.09 130.85 136.03 8k | Thpt | OOM | OOM | 27.41 | 23.17 25.39 26.44 | 23.23 25.03 28.19 | 23.02 25.17 25.84 | 22.84 25.39 27.47 | 22.78 25.33 27.74 | 22.81 25.69 26.61 | TOPS | OOM | OOM | 136.76 | 115.64 126.72 131.96 | 115.89 124.89 140.73 | 114.92 125.58 128.90 | 113.98 126.74 137.09 | 113.72 126.35 138.42 | 113.83 128.21 132.84 Table 11: Throughput (Thpt, K tokens/s) and TOPS on LLaMA 3.2 3B-Instruct across sequence lengths. All methods are evaluated on the same A100 40GB hardware with batch size 8, 3 warmup iterations, and 5 measurement iterations under the same model wrapper. One denotes all legal score-tile groups routed to INT8. OOM entries are shown in gray and excluded from relative color comparisons. Len | Metric | Torch | Flash | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1k | Thpt | 6.98 | 7.98 | 19.27 | 15.10 16.12 18.06 | 14.28 16.98 17.59 | 15.03 16.47 18.18 | 14.74 16.53 17.93 | 14.90 15.87 17.96 | 15.09 16.05 17.76 | TOPS | 55.11 | 63.01 | 152.10 | 119.20 127.24 142.63 | 112.75 134.02 138.79 | 118.71 130.01 143.58 | 116.37 130.48 141.49 | 117.63 125.32 141.87 | 119.14 126.73 140.17 2k | Thpt | 5.21 | 7.72 | 19.13 | 14.95 15.93 17.84 | 14.18 16.80 17.41 | 14.88 16.23 17.94 | 14.63 16.35 17.76 | 14.71 15.71 17.76 | 14.89 15.84 17.52 | TOPS | 42.23 | 62.54 | 155.00 | 121.07 129.05 144.61 | 114.88 136.17 140.99 | 120.58 131.59 145.36 | 118.51 132.50 143.85 | 119.23 127.32 143.91 | 120.61 128.32 141.98 4k | Thpt | OOM | 7.09 | 18.03 | 14.09 14.95 16.74 | 13.50 15.98 16.43 | 14.12 15.37 16.84 | 13.89 15.43 16.65 | 13.89 14.75 16.65 | 14.03 14.97 16.49 | TOPS | OOM | 60.41 | 153.54 | 120.09 127.37 142.70 | 115.00 136.10 139.94 | 120.31 130.89 143.35 | 118.25 131.40 141.80 | 118.37 125.64 141.89 | 119.51 127.56 140.44 Table 12: Throughput (Thpt, K tokens/s) and TOPS on Qwen 2.5 7B across sequence lengths. Dataset | Len | FP16 | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1 | 16k | 33.88 | 28.9 | 34.7 34.4 34.5 | 33.9 32.0 32.1 | 33.2 31.1 31.3 | 33.7 32.0 31.7 | 34.8 33.7 33.2 | 34.9 33.1 32.2 | 32k | 17.61 | 13.6 | 18.3 15.6 14.6 | 14.4 15.3 15.7 | 17.8 16.4 15.9 | 18.2 16.6 16.1 | 17.8 15.5 14.9 | 18.3 17.4 16.9 | 64k | 9.22 | 6.5 | 9.3 8.1 7.0 | 9.3 7.0 6.9 | 9.5 7.6 6.8 | 9.4 8.3 7.4 | 9.5 7.4 7.1 | 9.4 8.5 7.9 2 | 16k | 17.37 | 14.8 | 17.8 16.2 15.9 | 18.1 15.8 16.4 | 17.4 15.9 15.6 | 17.6 16.7 15.9 | 17.4 15.8 15.7 | 17.6 17.2 16.0 | 32k | 15.89 | 12.6 | 15.8 14.8 13.9 | 16.0 14.1 13.6 | 15.9 14.2 13.8 | 16.0 15.4 14.2 | 16.2 14.7 13.9 | 16.0 15.3 14.8 | 64k | 12.12 | 8.1 | 12.4 10.2 9.5 | 12.5 10.4 10.2 | 12.3 10.6 10.1 | 12.4 11.2 10.5 | 12.6 10.8 10.4 | 12.3 11.6 11.2 3 | 16k | 10.22 | 6.7 | 9.8 8.6 7.6 | 9.9 3.7 6.8 | 9.6 10.4 6.9 | 9.6 8.2 7.0 | 10.1 7.2 6.9 | 31.4 30.7 32.2 | 32k | 6.03 | 3.1 | 6.0 4.8 4.6 | 5.7 2.1 2.3 | 3.7 2.9 1.9 | 5.5 3.9 2.8 | 9.4 3.3 4.1 | 5.4 4.4 4.2 | 64k | 3.17 | 1.3 | 3.2 2.3 1.5 | 3.2 1.4 1.4 | 3.1 1.6 1.3 | 3.1 2.1 1.4 | 3.3 1.5 1.4 | 3.2 2.6 2.4 4 | 16k | 4.64 | 1.7 | 5.8 4.6 4.3 | 4.3 3.6 3.3 | 4.6 3.5 3.5 | 4.6 3.9 3.3 | 4.7 3.8 4.0 | 4.6 4.1 3.8 | 32k | 3.58 | 2.8 | 3.7 3.4 2.8 | 3.7 3.3 3.4 | 3.7 3.3 3.2 | 3.6 3.4 3.3 | 3.7 3.3 3.2 | 3.6 3.5 3.6 | 64k | 1.74 | 1.2 | 1.8 1.7 1.5 | 1.7 1.5 1.4 | 1.8 1.6 1.5 | 1.8 1.6 1.5 | 1.9 1.6 1.5 | 1.8 1.7 1.6 5 | 16k | 19.13 | 14.2 | 18.9 18.8 17.9 | 19.2 17.1 16.7 | 19.2 16.7 16.4 | 19.2 17.3 17.1 | 19.1 17.0 18.0 | 19.3 18.7 18.2 | 32k | 8.72 | 6.8 | 8.9 8.1 7.3 | 8.7 7.1 6.9 | 8.6 7.4 7.3 | 8.9 8.0 7.1 | 8.8 7.6 7.5 | 8.9 8.3 8.0 | 64k | 8.09 | 6.3 | 9.0 7.5 6.7 | 8.8 7.7 6.9 | 9.1 7.8 7.2 | 9.1 7.9 7.1 | 9.2 7.2 6.0 | 9.0 7.4 7.1 6 | 16k | 23.19 | 16.0 | 23.2 22.0 18.5 | 21.8 18.1 13.1 | 23.1 19.6 19.3 | 22.9 21.1 19.7 | 22.5 19.2 19.8 | 23.1 22.0 21.2 | 32k | 16.05 | 7.7 | 15.3 14.4 9.7 | 16.9 11.6 11.8 | 15.7 12.5 10.7 | 17.1 14.8 11.4 | 16.1 12.5 11.0 | 17.1 15.8 14.4 | 64k | 5.58 | 2.3 | 6.0 5.7 3.1 | 2.6 2.7 2.7 | 5.8 4.1 3.4 | 5.7 4.9 3.2 | 6.0 5.7 3.8 | 5.9 4.2 3.7 7 | 16k | 21.60 | 15.5 | 21.8 22.1 20.3 | 26.8 19.1 18.5 | 21.7 20.4 19.8 | 21.9 20.9 20.1 | 22.0 20.2 20.0 | 21.8 21.0 20.9 | 32k | 18.85 | 11.6 | 18.3 16.9 13.8 | 18.7 16.7 16.0 | 19.4 17.8 17.0 | 19.2 18.4 17.3 | 19.1 17.4 16.9 | 19.3 18.8 18.1 | 64k | 12.67 | 6.3 | 12.6 12.1 9.4 | 12.5 11.0 10.0 | 12.6 11.1 10.7 | 12.7 11.9 11.1 | 12.6 11.0 10.5 | 12.7 12.1 11.8 8 | 16k | 21.27 | 14.8 | 21.1 18.7 15.2 | 21.6 19.5 18.2 | 20.5 19.2 18.6 | 20.3 19.7 18.4 | 21.1 19.1 19.2 | 20.6 19.8 20.3 | 32k | 16.56 | 10.7 | 16.2 14.1 13.1 | 16.9 14.8 13.4 | 16.1 14.9 14.4 | 15.9 15.2 14.3 | 16.7 14.6 14.4 | 16.1 15.6 15.4 | 64k | 12.90 | 8.8 | 12.9 10.9 9.8 | 13.2 11.1 9.8 | 12.8 11.0 10.4 | 13.1 11.8 11.2 | 12.9 11.2 10.7 | 13.0 12.2 11.9 9 | 16k | 41.68 | 31.8 | 41.9 39.3 36.1 | 41.8 39.9 39.0 | 41.3 39.9 40.1 | 41.7 40.9 39.7 | 41.6 40.1 40.3 | 41.8 41.0 41.2 | 32k | 26.25 | 17.7 | 25.4 23.0 21.3 | 25.7 24.0 23.2 | 26.1 24.6 23.5 | 25.8 25.2 24.3 | 26.6 24.8 24.2 | 25.9 25.6 25.0 | 64k | 14.08 | 7.9 | 14.1 13.0 10.6 | 14.0 12.2 11.0 | 13.8 12.5 12.4 | 14.1 13.1 12.6 | 14.2 12.7 12.2 | 14.0 13.6 13.1 10 | 16k | 26.81 | 20.8 | 26.3 24.6 24.2 | 26.9 25.1 23.9 | 27.1 25.7 24.7 | 27.2 26.3 25.5 | 26.6 24.9 25.1 | 27.0 26.4 25.8 | 32k | 18.63 | 13.8 | 18.2 16.2 16.0 | 18.2 16.6 15.6 | 18.5 17.1 16.8 | 18.7 17.8 16.9 | 18.6 16.9 16.7 | 18.7 18.0 17.6 | 64k | 16.42 | 13.4 | 16.9 16.4 15.5 | 16.8 14.7 13.9 | 16.6 15.0 14.7 | 16.7 15.7 14.9 | 16.5 15.4 15.1 | 16.7 16.1 16.2 11 | 16k | 33.47 | 25.1 | 35.1 33.7 31.0 | 33.3 31.3 29.6 | 33.5 30.1 30.8 | 33.7 32.2 31.0 | 33.4 30.8 31.9 | 34.1 33.3 32.1 | 32k | 22.36 | 18.1 | 23.1 22.6 21.4 | 22.7 19.8 21.2 | 23.6 21.3 19.6 | 22.9 21.4 19.9 | 23.3 20.1 20.0 | 22.8 22.1 22.0 | 64k | 14.06 | 5.2 | 12.9 11.5 8.8 | 13.0 9.9 8.4 | 13.9 13.0 12.3 | 13.9 12.4 11.2 | 13.8 11.0 9.9 | 14.0 13.1 11.6 Table 10: LV-Eval long-context question answering for Qwen 2.5 7B under different precision layouts and INT8 ratios (25/50/75%). Results are reported for FP16 attention (Baseline), One (all legal score-tile groups routed to INT8), and mixed-precision configurations. Cells are color-coded by comparison to One and Baseline: red indicates performance below One, blue indicates performance between One and the Baseline, and green indicates performance above the Baseline. Len | Metric | Torch | Flash | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1k | Thpt | 7.16 | 8.21 | 19.78 | 16.20 19.82 18.04 | 16.12 17.93 17.95 | 16.23 18.09 18.07 | 16.20 18.04 20.56 | 16.25 18.02 17.98 | 16.25 20.26 20.14 | TOPS | 56.51 | 64.81 | 156.18 | 127.94 156.43 142.38 | 127.26 141.56 141.74 | 128.19 142.87 142.69 | 127.95 142.38 162.36 | 128.23 142.28 141.99 | 128.34 159.91 159.07 2k | Thpt | 5.35 | 7.89 | 19.56 | 15.99 19.56 17.77 | 15.89 18.52 19.29 | 16.04 17.86 17.78 | 16.02 17.78 20.29 | 16.01 17.77 17.77 | 16.01 19.86 19.76 | TOPS | 43.34 | 63.93 | 158.52 | 129.50 158.47 143.92 | 128.76 150.08 156.33 | 129.92 144.62 144.07 | 129.79 144.14 164.38 | 129.65 144.00 144.00 | 129.76 160.98 160.15 4k | Thpt | OOM | 7.22 | 18.49 | 15.17 16.80 16.80 | 15.17 17.62 18.31 | 15.26 16.91 16.80 | 15.25 16.91 19.23 | 15.22 16.85 16.82 | 15.07 18.76 18.66 | TOPS | OOM | 61.48 | 157.51 | 129.19 143.03 143.12 | 129.21 150.04 156.04 | 130.00 144.09 143.19 | 129.94 144.05 163.77 | 129.58 143.53 143.21 | 128.32 159.69 158.85 Table 13: Throughput (Thpt, K tokens/s) and TOPS on Qwen 2 7B across sequence lengths. Len | Metric | Torch | Flash | One | AlignSparse Band BigBird Global RowRand SpTrans | | | | | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 | 25 50 75 1k | Thpt | 6.52 | 8.40 | 18.13 | 15.54 16.71 18.02 | 15.38 16.71 18.02 | 15.32 16.48 17.88 | 15.23 16.38 18.36 | 15.08 16.54 17.83 | 15.01 16.15 17.52 | TOPS | 44.86 | 66.30 | 124.65 | 106.85 114.97 123.98 | 105.80 114.97 123.98 | 105.39 113.33 122.93 | 104.72 112.67 126.36 | 103.68 113.78 122.71 | 103.28 111.07 120.47 2k | Thpt | 4.60 | 8.07 | 17.35 | 14.91 16.05 17.30 | 14.76 16.05 17.30 | 14.66 15.88 17.11 | 14.61 15.73 17.64 | 14.46 15.89 17.13 | 14.37 15.56 16.77 | TOPS | 32.88 | 65.40 | 124.02 | 106.64 114.82 123.67 | 105.58 114.82 123.67 | 104.81 113.53 122.37 | 104.51 112.53 126.05 | 103.47 113.64 122.40 | 102.72 111.26 119.93 4k | Thpt | OOM | 7.39 | 15.80 | 13.78 14.80 15.92 | 13.64 14.80 15.92 | 13.51 14.67 15.66 | 13.51 14.51 16.22 | 13.37 14.65 15.75 | 13.24 14.38 15.35 | TOPS | OOM | 62.89 | 121.58 | 106.03 113.93 122.50 | 104.98 113.93 122.50 | 103.93 112.87 120.48 | 103.91 111.65 124.85 | 102.88 112.76 121.24 | 101.85 110.62 118.07 Table 14: Throughput (Thpt, K tokens/s) and TOPS on Vicuna 7B across sequence lengths. Across models and lengths, Torch provides the lowest throughput and exhibits earlier OOM behavior, while FlashAttention serves as a strong baseline when it fits in memory. One provides the uniform INT8 reference, while mixed-routing layouts form nearby empirical operating points whose ordering depends on complete-pipeline execution. Across models, higher INT8 coverage generally increases throughput and TOPS, and several mixed-routing configurations match or exceed One. Differences among layouts reflect complete-pipeline dispatch, rescaling, memory-access, and scheduling behavior under different spatial routing arrangements; all layouts preserve the same dense legal connectivity. Overall, the results suggest that TileMix provides a practical accuracy-efficiency knob: higher INT8 ratios increase low-precision execution, while more conservative ratios preserve more FP16-routed tile groups for layouts or tasks that are more accuracy-sensitive. Seq Len | Torch Value | Flash | 100% INT8 | 75% INT8 | 50% INT8 | 25% INT8 1024 | ×1002.85\!×\!10^0 | ×10−21.57\!×\!10^-2 | ×10−25.40\!×\!10^-2 | ×10−23.54\!×\!10^-2 | ×10−21.68\!×\!10^-2 | ×10−31.95\!×\!10^-3 2048 | ×1002.82\!×\!10^0 | ×10−21.42\!×\!10^-2 | ×10−26.49\!×\!10^-2 | ×10−24.25\!×\!10^-2 | ×10−22.20\!×\!10^-2 | ×10−31.95\!×\!10^-3 4096 | ×1002.98\!×\!10^0 | ×10−21.12\!×\!10^-2 | ×10−25.86\!×\!10^-2 | ×10−24.00\!×\!10^-2 | ×10−22.00\!×\!10^-2 | ×10−31.95\!×\!10^-3 8192 | ×1003.46\!×\!10^0 | ×10−21.22\!×\!10^-2 | ×10−26.25\!×\!10^-2 | ×10−24.10\!×\!10^-2 | ×10−22.10\!×\!10^-2 | ×10−31.95\!×\!10^-3 Table 15: Single-layer model on random inputs. Torch Value reports the maximum absolute logit magnitude under the fixed Torch FP16 reference; all other columns report maximum absolute deviation from this reference. Seq Len | Torch Value | Flash | 100% INT8 | 75% INT8 | 50% INT8 | 25% INT8 1024 | ×1002.91\!×\!10^0 | ×10−11.48\!×\!10^-1 | ×10−12.03\!×\!10^-1 | ×10−11.34\!×\!10^-1 | ×10−26.93\!×\!10^-2 | ×10−33.91\!×\!10^-3 2048 | ×1002.79\!×\!10^0 | ×10−11.40\!×\!10^-1 | ×10−11.75\!×\!10^-1 | ×10−11.23\!×\!10^-1 | ×10−26.40\!×\!10^-2 | ×10−33.17\!×\!10^-3 4096 | ×1003.05\!×\!10^0 | ×10−11.47\!×\!10^-1 | ×10−12.19\!×\!10^-1 | ×10−11.46\!×\!10^-1 | ×10−27.42\!×\!10^-2 | ×10−33.42\!×\!10^-3 8192 | ×1003.22\!×\!10^0 | ×10−11.52\!×\!10^-1 | ×10−12.22\!×\!10^-1 | ×10−11.48\!×\!10^-1 | ×10−27.25\!×\!10^-2 | ×10−33.91\!×\!10^-3 Table 16: Numerical behavior of a 12-layer attention model on random inputs. Torch Value reports the maximum absolute logit magnitude under the selected Torch FP16 reference; all other columns report maximum absolute deviation from this reference. Seq Len | Torch Value | FlashAttn | 100% INT8 | 75% INT8 | 50% INT8 | 25% INT8 1024 | ×1003.02\!×\!10^0 | ×10−13.55\!×\!10^-1 | ×10−13.56\!×\!10^-1 | ×10−12.22\!×\!10^-1 | ×10−11.14\!×\!10^-1 | ×10−35.86\!×\!10^-3 2048 | ×1002.90\!×\!10^0 | ×10−13.18\!×\!10^-1 | ×10−14.48\!×\!10^-1 | ×10−12.96\!×\!10^-1 | ×10−11.37\!×\!10^-1 | ×10−35.00\!×\!10^-3 4096 | ×1003.13\!×\!10^0 | ×10−13.79\!×\!10^-1 | ×10−13.69\!×\!10^-1 | ×10−12.51\!×\!10^-1 | ×10−11.28\!×\!10^-1 | ×10−35.86\!×\!10^-3 8192 | ×1003.21\!×\!10^0 | ×10−13.31\!×\!10^-1 | ×10−13.98\!×\!10^-1 | ×10−12.60\!×\!10^-1 | ×10−11.34\!×\!10^-1 | ×10−35.86\!×\!10^-3 Table 17: Numerical behavior of a 32-layer attention model on random inputs. Torch Value reports the maximum absolute logit magnitude under the selected Torch FP16 reference; all other columns report maximum absolute deviation from this reference. Config MaxDiff MeanDiff | Config MaxDiff MeanDiff Zero (0% INT8) 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 | BigBird75 1.96×10−41.96×10^-4 8.45×10−68.45×10^-6 One (100% INT8) 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 | Global0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 AlignSparse0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 | Global25 1.96×10−41.96×10^-4 6.75×10−66.75×10^-6 AlignSparse25 1.96×10−41.96×10^-4 7.01×10−67.01×10^-6 | Global50 1.96×10−41.96×10^-4 8.01×10−68.01×10^-6 AlignSparse50 1.96×10−41.96×10^-4 7.23×10−67.23×10^-6 | Global75 1.96×10−41.96×10^-4 8.41×10−68.41×10^-6 AlignSparse75 1.96×10−41.96×10^-4 7.93×10−67.93×10^-6 | RowRand0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 Band0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 | RowRand25 1.96×10−41.96×10^-4 6.62×10−66.62×10^-6 Band25 1.96×10−41.96×10^-4 6.52×10−66.52×10^-6 | RowRand50 1.96×10−41.96×10^-4 8.07×10−68.07×10^-6 Band50 1.96×10−41.96×10^-4 7.67×10−67.67×10^-6 | RowRand75 1.96×10−41.96×10^-4 8.45×10−68.45×10^-6 Band75 4.84×10−44.84×10^-4 8.88×10−68.88×10^-6 | SpTrans0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 BigBird0 1.96×10−41.96×10^-4 5.25×10−65.25×10^-6 | SpTrans25 1.96×10−41.96×10^-4 6.33×10−66.33×10^-6 BigBird25 1.96×10−41.96×10^-4 6.62×10−66.62×10^-6 | SpTrans50 1.96×10−41.96×10^-4 6.84×10−66.84×10^-6 BigBird50 1.96×10−41.96×10^-4 8.07×10−68.07×10^-6 | SpTrans75 4.84×10−44.84×10^-4 8.68×10−68.68×10^-6 Table 18: Direct numerical difference between TileMix and FlashAttention under different precision layouts. A nonzero gap exists even at 0% INT8 due to fused-kernel implementation differences. Config MaxAbsDiff MeanAbsDiff | Config MaxAbsDiff MeanAbsDiff One (fp16acc) 6.59×10−16.59×10^-1 6.99×10−36.99×10^-3 | Global25 (fp16acc) 1.24×10−11.24×10^-1 2.44×10−32.44×10^-3 One (fp32acc) 6.59×10−16.59×10^-1 6.99×10−36.99×10^-3 | Global25 (fp32acc) 1.24×10−11.24×10^-1 2.44×10−32.44×10^-3 FP16 (fp16acc) 2.93×10−42.93×10^-4 1.02×10−51.02×10^-5 | Global50 (fp16acc) 1.80×10−11.80×10^-1 4.29×10−34.29×10^-3 FP16 (fp32acc) 00 00 | Global50 (fp32acc) 1.80×10−11.80×10^-1 4.28×10−34.28×10^-3 AlignSparse25 (fp16acc) 2.38×10−12.38×10^-1 2.71×10−32.71×10^-3 | Global75 (fp16acc) 1.80×10−11.80×10^-1 4.64×10−34.64×10^-3 AlignSparse25 (fp32acc) 2.38×10−12.38×10^-1 2.71×10−32.71×10^-3 | Global75 (fp32acc) 1.80×10−11.80×10^-1 4.64×10−34.64×10^-3 AlignSparse50 (fp16acc) 2.38×10−12.38×10^-1 2.92×10−32.92×10^-3 | RowRand25 (fp16acc) 1.56×10−11.56×10^-1 2.39×10−32.39×10^-3 AlignSparse50 (fp32acc) 2.38×10−12.38×10^-1 2.91×10−32.91×10^-3 | RowRand25 (fp32acc) 1.56×10−11.56×10^-1 2.38×10−32.38×10^-3 AlignSparse75 (fp16acc) 2.38×10−12.38×10^-1 3.82×10−33.82×10^-3 | RowRand50 (fp16acc) 2.27×10−12.27×10^-1 4.34×10−34.34×10^-3 AlignSparse75 (fp32acc) 2.38×10−12.38×10^-1 3.81×10−33.81×10^-3 | RowRand50 (fp32acc) 2.27×10−12.27×10^-1 4.34×10−34.34×10^-3 Band25 (fp16acc) 1.10×10−11.10×10^-1 1.91×10−31.91×10^-3 | RowRand75 (fp16acc) 2.12×10−12.12×10^-1 4.60×10−34.60×10^-3 Band25 (fp32acc) 1.10×10−11.10×10^-1 1.90×10−31.90×10^-3 | RowRand75 (fp32acc) 2.12×10−12.12×10^-1 4.60×10−34.60×10^-3 Band50 (fp16acc) 2.38×10−12.38×10^-1 3.62×10−33.62×10^-3 | SpTrans25 (fp16acc) 1.40×10−11.40×10^-1 1.73×10−31.73×10^-3 Band50 (fp32acc) 2.38×10−12.38×10^-1 3.61×10−33.61×10^-3 | SpTrans25 (fp32acc) 1.41×10−11.41×10^-1 1.72×10−31.72×10^-3 Band75 (fp16acc) 6.59×10−16.59×10^-1 5.21×10−35.21×10^-3 | SpTrans50 (fp16acc) 1.40×10−11.40×10^-1 2.50×10−32.50×10^-3 Band75 (fp32acc) 6.59×10−16.59×10^-1 5.21×10−35.21×10^-3 | SpTrans50 (fp32acc) 1.41×10−11.41×10^-1 2.49×10−32.49×10^-3 BigBird25 (fp16acc) 1.56×10−11.56×10^-1 2.39×10−32.39×10^-3 | SpTrans75 (fp16acc) 6.59×10−16.59×10^-1 5.09×10−35.09×10^-3 BigBird25 (fp32acc) 1.56×10−11.56×10^-1 2.38×10−32.38×10^-3 | SpTrans75 (fp32acc) 6.59×10−16.59×10^-1 5.09×10−35.09×10^-3 BigBird50 (fp16acc) 2.27×10−12.27×10^-1 4.34×10−34.34×10^-3 | BigBird75 (fp16acc) 2.12×10−12.12×10^-1 4.60×10−34.60×10^-3 BigBird50 (fp32acc) 2.27×10−12.27×10^-1 4.34×10−34.34×10^-3 | BigBird75 (fp32acc) 2.12×10−12.12×10^-1 4.60×10−34.60×10^-3 Table 19: Numerical differences compared to full FP16 with FP32 accumulation under different precision layouts and mixing ratios. Each configuration is evaluated with FP16 accumulation and FP32 accumulation. L Max Mean | L Max Mean | L Max Mean | L Max Mean 0 0.000.00 0.000.00 | 8 1.09×10−21.09×10^-2 8.17×10−48.17×10^-4 | 16 1.10×10−21.10×10^-2 1.47×10−31.47×10^-3 | 24 1.11×10−21.11×10^-2 3.52×10−33.52×10^-3 1 1.08×10−51.08×10^-5 1.75×10−61.75×10^-6 | 9 1.09×10−21.09×10^-2 9.81×10−49.81×10^-4 | 17 1.10×10−21.10×10^-2 1.62×10−31.62×10^-3 | 25 1.11×10−21.11×10^-2 3.96×10−33.96×10^-3 2 1.08×10−51.08×10^-5 3.91×10−63.91×10^-6 | 10 1.10×10−21.10×10^-2 1.06×10−31.06×10^-3 | 18 1.10×10−21.10×10^-2 1.81×10−31.81×10^-3 | 26 1.11×10−21.11×10^-2 4.39×10−44.39×10^-4 3 1.09×10−51.09×10^-5 5.58×10−65.58×10^-6 | 11 1.10×10−21.10×10^-2 1.06×10−31.06×10^-3 | 19 1.11×10−21.11×10^-2 2.11×10−32.11×10^-3 | 27 1.11×10−21.11×10^-2 4.98×10−44.98×10^-4 4 1.09×10−51.09×10^-5 6.42×10−66.42×10^-6 | 12 1.10×10−21.10×10^-2 1.10×10−31.10×10^-3 | 20 1.11×10−21.11×10^-2 2.31×10−32.31×10^-3 | 28 1.11×10−21.11×10^-2 5.54×10−45.54×10^-4 5 1.09×10−51.09×10^-5 6.95×10−66.95×10^-6 | 13 1.10×10−21.10×10^-2 1.11×10−31.11×10^-3 | 21 1.11×10−21.11×10^-2 2.60×10−32.60×10^-3 | 29 1.11×10−21.11×10^-2 6.65×10−46.65×10^-4 6 1.09×10−51.09×10^-5 7.20×10−67.20×10^-6 | 14 1.10×10−21.10×10^-2 1.20×10−31.20×10^-3 | 22 1.11×10−21.11×10^-2 2.82×10−32.82×10^-3 | 30 1.16×10−21.16×10^-2 8.42×10−48.42×10^-4 7 1.09×10−51.09×10^-5 7.51×10−67.51×10^-6 | 15 1.11×10−21.11×10^-2 1.35×10−31.35×10^-3 | 23 1.11×10−21.11×10^-2 3.14×10−33.14×10^-3 | 31 1.94×10−21.94×10^-2 1.27×10−51.27×10^-5 Table 21: Layer-wise numerical differences between FP16 and FP32 accumulation on LLaMA 3.1 8B. Max and Mean report the maximum and mean absolute differences at each layer. L Max Mean | L Max Mean | L Max Mean | L Max Mean 0 0.000.00 0.000.00 | 12 5.63×10−25.63×10^-2 1.07×10−41.07×10^-4 | 24 5.82×10−25.82×10^-2 1.18×10−41.18×10^-4 | 36 5.97×10−25.97×10^-2 1.29×10−41.29×10^-4 1 3.23×10−33.23×10^-3 2.30×10−52.30×10^-5 | 13 5.63×10−25.63×10^-2 1.08×10−41.08×10^-4 | 25 5.82×10−25.82×10^-2 1.19×10−41.19×10^-4 | 37 5.95×10−25.95×10^-2 1.31×10−41.31×10^-4 2 6.38×10−36.38×10^-3 3.50×10−53.50×10^-5 | 14 5.62×10−25.62×10^-2 1.08×10−41.08×10^-4 | 26 5.88×10−25.88×10^-2 1.19×10−41.19×10^-4 | 38 5.92×10−25.92×10^-2 1.34×10−41.34×10^-4 3 9.78×10−39.78×10^-3 5.82×10−55.82×10^-5 | 15 5.62×10−25.62×10^-2 1.09×10−41.09×10^-4 | 27 5.90×10−25.90×10^-2 1.20×10−41.20×10^-4 | 39 5.88×10−25.88×10^-2 1.35×10−41.35×10^-4 4 5.10×10−25.10×10^-2 7.92×10−57.92×10^-5 | 16 5.62×10−25.62×10^-2 1.09×10−41.09×10^-4 | 28 5.94×10−25.94×10^-2 1.21×10−41.21×10^-4 | 40 5.82×10−25.82×10^-2 1.37×10−41.37×10^-4 5 5.43×10−25.43×10^-2 8.40×10−58.40×10^-5 | 17 5.63×10−25.63×10^-2 1.10×10−41.10×10^-4 | 29 5.96×10−25.96×10^-2 1.21×10−41.21×10^-4 | 41 5.76×10−25.76×10^-2 1.39×10−41.39×10^-4 6 5.64×10−25.64×10^-2 8.95×10−58.95×10^-5 | 18 5.62×10−25.62×10^-2 1.10×10−41.10×10^-4 | 30 5.96×10−25.96×10^-2 1.22×10−41.22×10^-4 | 42 5.74×10−25.74×10^-2 1.41×10−41.41×10^-4 7 5.65×10−25.65×10^-2 9.94×10−59.94×10^-5 | 19 5.62×10−25.62×10^-2 1.11×10−41.11×10^-4 | 31 5.98×10−25.98×10^-2 1.23×10−41.23×10^-4 | 43 5.71×10−25.71×10^-2 1.44×10−41.44×10^-4 8 5.64×10−25.64×10^-2 1.02×10−41.02×10^-4 | 20 5.62×10−25.62×10^-2 1.12×10−41.12×10^-4 | 32 5.98×10−25.98×10^-2 1.24×10−41.24×10^-4 | 44 3.65×10−23.65×10^-2 1.37×10−41.37×10^-4 9 5.64×10−25.64×10^-2 1.05×10−41.05×10^-4 | 21 5.62×10−25.62×10^-2 1.12×10−41.12×10^-4 | 33 5.98×10−25.98×10^-2 1.25×10−41.25×10^-4 | 45 2.96×10−22.96×10^-2 1.39×10−41.39×10^-4 10 5.64×10−25.64×10^-2 1.06×10−41.06×10^-4 | 22 5.78×10−25.78×10^-2 1.14×10−41.14×10^-4 | 34 5.98×10−25.98×10^-2 1.26×10−41.26×10^-4 | 46 2.98×10−22.98×10^-2 1.76×10−41.76×10^-4 11 5.64×10−25.64×10^-2 1.07×10−41.07×10^-4 | 23 5.79×10−25.79×10^-2 1.17×10−41.17×10^-4 | 35 5.98×10−25.98×10^-2 1.27×10−41.27×10^-4 | 47 8.93×10−48.93×10^-4 1.83×10−51.83×10^-5 Table 22: Layer-wise numerical differences between FP16 and FP32 accumulation on Qwen 2.5 14B. Max and Mean report the maximum and mean absolute differences at each layer. Config Top5 Top10 Top20 Top30 SpTrans25 8.57% 8.63% 8.63% 8.48% SpTrans50 19.19% 19.15% 18.63% 17.76% SpTrans75 21.55% 21.66% 21.60% 21.15% Table 23: Weighted INT8 exposure of heavy-hitter importance under SpTrans precision layouts. Lower values indicate that less high-importance attention mass is routed to INT8. Config MaxAbsDiff MeanAbsDiff | Config MaxAbsDiff MeanAbsDiff One 3.91×10−43.91×10^-4 1.51×10−51.51×10^-5 | Global25 2.93×10−42.93×10^-4 1.21×10−51.21×10^-5 FP16 2.93×10−42.93×10^-4 1.02×10−51.02×10^-5 | Global50 3.91×10−43.91×10^-4 1.36×10−51.36×10^-5 AlignSparse25 3.91×10−43.91×10^-4 1.24×10−51.24×10^-5 | Global75 3.91×10−43.91×10^-4 1.44×10−51.44×10^-5 AlignSparse50 3.91×10−43.91×10^-4 1.27×10−51.27×10^-5 | RowRand25 3.91×10−43.91×10^-4 1.17×10−51.17×10^-5 AlignSparse75 2.93×10−42.93×10^-4 1.37×10−51.37×10^-5 | RowRand50 3.91×10−43.91×10^-4 1.37×10−51.37×10^-5 Band25 3.91×10−43.91×10^-4 1.19×10−51.19×10^-5 | RowRand75 3.91×10−43.91×10^-4 1.46×10−51.46×10^-5 Band50 3.91×10−43.91×10^-4 1.32×10−51.32×10^-5 | SpTrans25 3.91×10−43.91×10^-4 1.15×10−51.15×10^-5 Band75 3.42×10−43.42×10^-4 1.41×10−51.41×10^-5 | SpTrans50 4.39×10−44.39×10^-4 1.21×10−51.21×10^-5 BigBird25 3.91×10−43.91×10^-4 1.17×10−51.17×10^-5 | SpTrans75 4.39×10−44.39×10^-4 1.37×10−51.37×10^-5 BigBird50 3.91×10−43.91×10^-4 1.37×10−51.37×10^-5 | BigBird75 3.91×10−43.91×10^-4 1.46×10−51.46×10^-5 Table 20: Direct comparison between FP16 and FP32 accumulation under different precision layouts and mixing ratios. Appendix G Numerical Analysis The fixed Torch FP16 implementation provides a common reference for comparing kernel schedules and routed arithmetic paths. Reported deviations measure implementation-level output agreement with this reference across differences in quantization, accumulation order, rounding behavior, and reduction schedule. This appendix complements the main numerical analysis in Section 5.4. We examine numerical behavior from six perspectives: (i) kernel-level output deviation on random attention inputs, (i) model-depth and sequence-length effects, (i) INT8 coverage ratio, (iv) direct comparison with FlashAttention, (v) accumulation precision and larger-model accumulation checks, and (vi) static-routing exposure of high-mass attention interactions. G.1 Depth and Sequence-Length Effects Tables 15, 16, and 17 report results for a single-layer model, a 12-layer model, and a 32-layer model, respectively. All experiments use identical random inputs and shared weights across attention implementations. For each sequence length, the Torch Value column reports the maximum absolute logit magnitude produced by the selected Torch FP16 reference, serving as a scale anchor. All other entries report maximum absolute deviation of model logits relative to this reference. Across model depths, deviations generally increase as depth increases, reflecting accumulation of implementation-level differences across layers. Uniform INT8 attention produces larger deviations than mixed-routing configurations. Reducing the fraction of INT8-routed tile groups generally reduces deviation from the fixed Torch FP16 reference. Within this controlled setup, model depth and INT8 coverage produce larger changes in logit deviation than sequence length over the evaluated range. G.2 Direct Comparison with FlashAttention Table 18 reports direct output differences between TileMix and FlashAttention under different precision layouts and INT8 ratios. A nonzero difference exists at 0% INT8 because TileMix and FlashAttention use different fused-kernel schedules. Within most structured layouts, MeanDiff increases with INT8 coverage; repeated MaxDiff or MeanDiff entries indicate equality at the reported numerical precision, not bitwise-identical outputs. G.3 Pattern and Accumulation Effects Table 19 separates the effects of precision layout, INT8 coverage, and accumulation precision. The results show that numerical behavior depends mainly on the precision layout and INT8 coverage ratio, while FP16 and FP32 accumulation produce similar deviations in this setting. Table 20 directly compares FP16 and FP32 accumulation under matched precision layouts. The small differences across configurations indicate that routing layout and INT8 coverage produce much larger numerical effects than accumulation precision in the evaluated setting. G.4 Larger-Model Accumulation Stability We further compare TileMix with FP16 accumulation against TileMix with FP32 accumulation on LLaMA3.1-8B and Qwen2.5-14B. Tables 21 and 22 show that FP16 accumulation remains close to FP32 accumulation across larger models. On LLaMA 3.1 8B, the maximum layer-wise difference remains on the order of 10−210^-2, while most mean differences stay around 10−310^-3 or lower. On Qwen 2.5 14B, maximum differences are larger but remain stable across layers, and mean differences stay around 10−410^-4. Together, these larger-model checks show that FP16-to-FP32 accumulation differences remain smaller than the routed-precision effects characterized above. G.5 Positional Routing and Heavy-Hitter Exposure We quantify how static SpTrans layouts distribute INT8 execution over high-importance interactions while retaining constant-time routing lookup inside the kernel. For each layer-head attention map, we define first-order query-key importance as Iqk=|Pqk∂ℒ∂Pqk|I_qk= |P_qk ∂ P_qk | For each query q, let ℋqτH_q^τ be the smallest key set whose cumulative importance reaches τ%τ\%, where τ∈5,10,20,30τ∈\5,10,20,30\. Let Dqk=1D_qk=1 indicate that position (q,k)(q,k) is routed to INT8. We define the weighted INT8 exposure of heavy hitters as Eτ=∑q∑k∈ℋqτIqkDqk∑q∑k∈ℋqτIqk.E^τ= _q _k _q^τI_qkD_qk _q _k _q^τI_qk. A lower EτE^τ indicates stronger protection of high-importance attention mass from INT8 routing. Table 23 shows that SpTrans25 routes only about 8.5% of the selected high-importance mass to INT8, substantially below its nominal 25% tile-group coverage. Across all three coverage levels, structured spatial routing retains a larger share of high-importance interactions in FP16 without online detection.

Instant research discovery

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

Intelligence

Summary

Entities (14)

Relation Signals (10)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations