Paper deep dive
MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators
Dong Liu, Yanxuan Yu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/18/2026, 3:49:28 AM
Summary
The paper introduces MeshKV, a Network-on-Chip (NoC) based KV cache fabric designed to optimize autoregressive transformer decoding on tiled accelerators. It addresses the bottleneck of irregular KV cache movement and centralized memory traffic by employing three co-designed mechanisms: TaKV (affine striping for load balancing), Mare (multicast routing with duplicate suppression), and Pad (pipelined attention dataflow). Evaluated on an 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B, MeshKV reduces interconnect traffic by up to 58%, improves KV bandwidth utilization by 2.1x, and increases multi-stream throughput by up to 1.9x compared to baseline centralized or shared mesh architectures.
Entities (10)
Relation Signals (12)
MeshKV → evaluatedwith → Mistral-7B
confidence 95% · On our 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B... MeshKV reduces interconnect traffic by up to 58%
MeshKV → evaluatedwith → Llama-2-7b
confidence 95% · On our 8x8 FPGA implementation with LLaMA-2-7B... MeshKV reduces interconnect traffic by up to 58%
MeshKV → improves → kv_bandwidth_utilization
confidence 95% · improves KV bandwidth utilization by 2.1x
MeshKV → improves → multi_stream_throughput
confidence 95% · delivers up to 1.9x multi-stream throughput
MeshKV → reduces → interconnect_traffic
confidence 95% · MeshKV reduces interconnect traffic by up to 58%
MeshKV → uses → TaKV
confidence 95% · MeshKV co-designs (i) TaKV affine striping to spread homes and cut hotspot load
MeshKV → uses → MARE
confidence 95% · MeshKV co-designs (ii) Mare multicast with verified duplicate suppression
MeshKV → uses → Pad
confidence 95% · MeshKV co-designs (iii) Pad, which overlaps prefetch, tile multiply, and streaming softmax
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autoregressive transformer decoding is constrained by irregular key-value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate traffic on centralized memory paths that bottleneck long-context serving. We present MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC. It co-designs (i) TaKV affine striping to spread homes and cut hotspot load, (ii) Mare multicast with verified duplicate suppression, and (iii) Pad, which overlaps prefetch, tile multiply, and streaming softmax behind credit-aligned FIFOs. Together they convert bisection back-pressure into useful KV transfer. On our 8x8 FPGA implementation with LLaMA-2-7B and Mistral-7B at 8K-32K, MeshKV reduces interconnect traffic by up to 58%, improves KV bandwidth utilization by 2.1x, and delivers up to 1.9x multi-stream throughput.
Tags
Links
- Source: https://arxiv.org/abs/2609.19207v1
- Canonical: https://arxiv.org/abs/2609.19207v1
Trouble viewing inline? Open PDF directly →
Full Text
26,808 characters extracted from source content.
Expand or collapse full text
MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators Dong Liu* Affiliation: University of California, Los Angeles Los Angeles, CA, USA pikeliu@ucla.edu Yanxuan Yu* Affiliation: Columbia University New York, NY, USA y3523@columbia.edu Abstract Autoregressive transformer decoding is constrained by irregular key–value (KV) cache movement on tiled accelerators. Prior compression and DRAM-placement systems still concentrate traffic on centralized memory paths that bottleneck long-context serving. We present MeshKV, a KV cache fabric that moves blocks as packetized flows over a lightweight NoC. It co-designs (i) TaKV affine striping to spread homes and cut hotspot load, (i) Mare multicast with verified duplicate suppression, and (i) Pad, which overlaps prefetch, tile multiply, and streaming softmax behind credit-aligned FIFOs. Together they convert bisection back-pressure into useful KV transfer. On our 8×88×8 FPGA implementation with LLaMA-2-7B and Mistral-7B at 8K–32K, MeshKV reduces interconnect traffic by up to %58\%, improves KV bandwidth utilization by 2.1×2.1×, and delivers up to 1.9×1.9× multi-stream throughput. Index Terms: Transformer decoding, KV cache, network-on-chip, accelerator architecture, long-context LLM, multicast routing, spatial locality, FPGA 11footnotetext: Equal contribution. I Introduction LLM decoding is “memory-bound,” but tiled accelerators also stall on on-chip data movement. Autoregressive attention scatters KV across a mesh built for dense GEMM, not all-to-few gather over a growing prefix [1, 2, 3]. ASICs, FPGAs, and chiplet meshes map compute to a 2-D PE array with private SRAM. Prefill amortizes tile hops; decode does not: one query row (dh≈128d_h≈128) meets a KV slab that grows every token. A centralized design routes these fetches through a shared injection point, concentrating long-context KV traffic on the same bisection. Our approach. We treat the KV cache as a communication substrate: each step issues structured, layer-major reads that admit co-designed placement, routing, and compute. MeshKV is a NoC-centric fabric. Unlike DRAM paging or compression [3, 4], it optimizes how resident SRAM blocks traverse the mesh—multicast from each home tile, with prefetch overlapping matmul across credit boundaries. Contributions. 1. TaKV (§V). Affine placement π(ℓ,s)π( ,s) plus a one-cycle occupancy swap. 2. Mare (§VI). Multicast trees, verified duplicate suppression, and a deadlock-free VN0/VN1 split. 3. Pad (§VII). Prefetch / multiply / streaming softmax sized to the NoC RTT. 4. Evaluation (§IX). Hardware-measured traffic, utilization, throughput, and latency at 8K–32K. I Related Work KV cache management and serving. PagedAttention [3], FlexGen [4], and TinyServe [5] virtualize or select KV behind a serving-layer controller; MeshKV distributes it inside a tiled accelerator with private SRAM. Quantization and eviction [6, 7, 8, 9, 10, 11] shrink the bytes per step; MeshKV complements these by improving residual transport. Attention hardware. FlashAttention [2, 12] and FlashDecoding++ [13] tile GPU attention and softmax; ELSA [14] and SpAtten [15] exploit ASIC sparsity. MeshKV targets on-chip communication; Pad uses online softmax in a credit-aligned pipeline. Networks-on-chip and FPGA accelerators. Classic NoCs and the Turn Model [16, 17, 18] supply deadlock theory; Simba [19] uses a 2-D mesh without KV-class flits. DFX [20], FlightLLM [21], and CXL-SpecKV [22] show HBM/CXL KV movement dominates FPGA decode; MeshKV replaces point-to-point KV buses with an attention-aware fabric. I Background and Motivation I-A Autoregressive decoding and the bandwidth wall A decoder with L layers, H heads, head dimension dhd_h, and length T computes, for layer ℓ and head h, aℓ,h=softmax(qℓ,hKℓ,h,≤t⊤dh),oℓ,h=aℓ,hVℓ,h,≤t,a_ ,h=softmax\! ( q_ ,h\,K_ ,h,≤ t d_h ), o_ ,h=a_ ,h\,V_ ,h,≤ t, (1) with qℓ,h∈ℝ1×dhq_ ,h ^1× d_h and cached slabs Kℓ,h,≤t,Vℓ,h,≤t∈ℝt×dhK_ ,h,≤ t,V_ ,h,≤ t ^t× d_h. The cache occupies Γ(T)=2LHdhT⋅bpe bytes, (T)=2LHd_hT·bpe\; bytes, (2) with bpe=2bpe=2 for FP16: ≈ 16 GiB for LLaMA-2-7B (L=32L=32, H=32H=32, dh=128d_h=128) at T=32 768T=32\,768, and every decode step reads all of it. FlashAttention [2, 12] reduces the constant for batched prefill via tiled SRAM reuse, but B=1B=1 decode exposes every byte with no reuse. I-B Tiled accelerators and the spoke–hub pathology Tile-based accelerators (e.g., Simba [19], custom chiplets) map heads and layers across a 2-D PE mesh. Prefill carries dense GEMM traffic; decode issues width-dhd_h gathers from wherever a KV block lives to wherever the compute tile sits. When KV sits at one logical memory controller, all such reads converge at the same injection point. The mesh degenerates to a spoke–hub: H requesters fan in to one hub, consuming 40–60% of link cycles on back-pressure (Figure 1). CentSharedMeshKV002020404060608080100100Link utilization (%)Useful KV transferBack-pressure stallOther Fig. 1: Breakdown of link-cycle utilization at the mesh bisection (T=32KT=32K, LLaMA-2-7B, B=1B=1). Cent expends 41%41\% of bandwidth on back-pressure; MeshKV converts that capacity into useful KV transfer. I-C Observed locality in KV accesses An instrumented FP16 decoder on LLaMA-2-7B [23] at T∈8192,32768T∈\8192,32768\ shows two regularities that motivate MeshKV: Observation 1 (Layer-major temporal locality). Within a decode step, layers are visited in order ℓ=0,…,L−1 =0,…,L-1; a KV line is reused only at step t+1t+1, so the prefetch window is exactly L layer traversals. Observation 2 (Head-group spatial locality). Heads that share a prefix span request identical segments (Mistral-7B GQA uses G=4G=4; LLaMA-2-7B is multi-head). At T=32KT=32K, 78%78\% of segment requests arrive within 12 cycles at the same router, so multicast coalescing does not require wide merge buffers. These observations underlie TaKV placement (§V) and Mare coalescing (§VI): KV_DATA fans out, while PART converges. IV System Architecture Overview Figure 2 deploys MeshKV on a 4×44×4 mesh of tiles (x,y)∈0,…,W−1×0,…,H−1(x,y)∈\0,…,W-1\×\0,…,H-1\ (scalable to 8×88×8). Each tile has a MAC array, four-bank KV SRAM, a partial-sum scratchpad, and a two-VN router. A per-column controller broadcasts descriptors (ℓ,h)( ,h); tiles resolve π(ℓ,s)π( ,s) and inject KV_FETCH on VN1. The controller is off the critical path (4-cycle combinational TaKV). Four-bank KV SRAM lets a tile stream the next window while MACs consume the current flits. A decode step broadcasts descriptors, resolves π(ℓ,s)π( ,s), injects KV_FETCH on VN1, receives KV_DATA at the compute tile, and returns PART on VN0 to a column reduce root. MACarrKVSRAMScrpadR(0,0)MACarrKVSRAMScrpadR(0,1)MACarrKVSRAMScrpadR(0,2)MACarrKVSRAMScrpadR(0,3)MACarrKVSRAMScrpadR(1,0)MACarrKVSRAMScrpadR(1,1)MACarrKVSRAMScrpadR(1,2)MACarrKVSRAMScrpadR(1,3)MACarrKVSRAMScrpadR(2,0)MACarrKVSRAMScrpadR(2,1)MACarrKVSRAMScrpadR(2,2)MACarrKVSRAMScrpadR(2,3)MACarrKVSRAMScrpadR(3,0)MACarrKVSRAMScrpadR(3,1)MACarrKVSRAMScrpadR(3,2)MACarrKVSRAMScrpadR(3,3)bisectionDC0DC1DC2DC3Q2Q0Q3Q1kv_datamcastbranchparthome: π(ℓ,s)=(1,1)π( ,s)=(1,1)BFPf creditHBM2 · 8 GB · 460 GB/stile-local prefetch descriptorsmemoryplane(TaKV)routerplane(Mare)computeplane(Pad) VN1: kv_data multicast VN0: part unicast. HBM prefetch MAC KV SRAM Scrpad Router Fig. 2: 4×44×4 MeshKV tile array (scalable to 8×88×8). Home tile (1,1)(1,1) roots a Mare multicast for KV_DATA; VN0 carries PART reduction; Pad prefetches from HBM. Three planes: TaKV places ℬ(ℓ,s)B( ,s) at load time (with lazy swaps); Mare coalesces, multicasts, and arbitrates VCs; Pad runs Pf/Tm/Rd and credits the router. Multicast trees are therefore rooted at the home tile π(ℓ,s)π( ,s), not at a shared memory port. V Topology-Aware KV Striping (TaKV) Macro-block ℬ(ℓ,s)B( ,s) covers P consecutive tokens of layer ℓ across all heads and both K and V: |ℬ(ℓ,s)|=2HdhP⋅bpebytes,s=⌊j/P⌋.|B( ,s)|=2Hd_hP·bpe , s= j/P . (3) With P=64P=64, H=32H=32, dh=128d_h=128, bpe=2bpe=2, |ℬ|=1MB|B|=1\,MB is the placement and multicast unit. On a W×HW×H mesh, TaKV maps ℬ(ℓ,s)B( ,s) to π(ℓ,s)=(ℓmodW,(a⋅ℓ+s)modH),π( ,s)= ( W,\;(a· +s) H ), (4) with a odd and gcd(a,H)=1 (a,H)=1 (e.g. a=W+1a=W+1). Algorithm 1 is four combinational instructions. Algorithm 1 TaKV static placement. 0: Layer ℓ , segment s, mesh dimensions W,HW,H, constant a (odd, coprime to H) 0: Home-tile coordinates (x,y)(x,y) 1: x←ℓmodWx\;←\; W 2: y←(a⋅ℓ+s)modHy\;←\;(a· +s) H 3: return (x,y)(x,y) Affine π(ℓ,s)π( ,s) does not shorten expected distance to a fixed home: on a symmetric mesh a uniform requester has the same mean hops to any single destination. The gain is spreading homes so injection and bisection load maxeλe _e _e drop, converting the back-pressure stall in Figure 1 into useful KV transfer. The coprime stride a=W+1a=W+1 permutes the y coordinate as ℓ increments, so consecutive layers occupy different columns and consecutive segments of one layer walk distinct rows. Injection therefore spreads even though mean hops to any single home do not shrink. Occupancy swaps further flatten ∑eq¯e2 _e q_e^2. A W×HW×H mesh assigns at most Nmax=⌈L⌈T/P⌉/(WH)⌉N_ = L T/P /(WH) blocks per tile (256256 MB of logical KV at L=32L=32, T=32768T=32768, P=64P=64, W=H=8W=H=8). U280 provides HBM2 (88 GB, 460460 GB/s) and DDR4 (3232 GB); the 1616 GiB slab does not fit in HBM alone. Table I’s 512512 URAMs give eight URAMs (288288 KB) per tile—a four-bank streaming window, not a 11 MB block—so homes promote HBM/DDR data into URAM under tile-local prefetch descriptors. Every K=128K=128 steps (one SRAM eviction epoch), routers sample occupancy q¯e q_e from existing credit counters and swap two homes iff the swap strictly reduces ∑eq¯e2 _e q_e^2 and preserves per-tile capacity NmaxN_ (4 comparisons, 2 register writes, one control cycle). VI Multicast Routing Engine (Mare) MeshKV classifies on-chip traffic into three flit types (Table I): KV_FETCH headers (seg,ℓ,mask,coord)(seg, ,mask,coord) and KV_DATA bodies (512-bit payload, dedup tag) on VN1 decouple fetch coalescence from data fan-out; PART partial logits use VN0. At the home tile π(ℓ,s)π( ,s), requests arriving in window Wcoal=12W_coal=12 cycles merge head masks M(ℓ,s)=⋁r∈Wcoalmaskr,M( ,s)= _r∈ W_coalmask_r, (5) then an XYM spanning tree over destinations (M)D(M) multicasts each payload once per branch ((HlogH)O(H H) build, H=32H=32, +4+4 control cycles before KV_DATA departs). Replication occurs only at tree branches, so a payload crosses any link at most once; the exact-tag plus bloom path is a safety net for overlapping trees and late arrivals. Each router keeps a 16-entry exact tag table of in-flight (seg,ℓ)(seg, ) and a 256256-bit bloom as a prefilter over Wb=96W_b=96 cycles. A duplicate is suppressed only on an exact-table hit; a miss or a full table forwards, so bloom false positives never drop a first-arrival payload (3232 B bloom plus 16 tags). TABLE I: Flit classes in MeshKV. Class Dir. VC Contents KV_FETCH requester→ VN1 seg id, layer, head mask (32 bits), requester coord KV_DATA home→ VN1 512-bit KV payload, in-flight segment tag PART tile→ root VN0 512-bit partial logit accumulator, softmax running stats VI-A Deadlock avoidance MeshKV splits traffic across two virtual networks: VN0 carries unicast PART reductions on dimension-ordered XY routing, while VN1 carries KV_FETCH and KV_DATA multicast on XYM-restricted turns [18]. No flit holds a VN0 buffer while waiting for VN1 resources or vice versa, so inter-network cycles are structurally eliminated; restricted-turn routing removes intra-network cycles within each VN. VII Pipelined Attention Dataflow (Pad) Pad overlaps three stages across segments s=0,…,⌈T/P⌉−1s=0,…, T/P -1 (Figure 3). Pf issues a KV_FETCH for s+1s+1 iff the home injection credit Cπ>0C_π>0 (one outstanding per (seg,ℓ)(seg, )). Tm writes segment logits zs=qℓ,hKℓ,h,s⊤/dhz_s=q_ ,h\,K_ ,h,s / d_h (6) into scratchpad SRAM (≥ 80% MAC when the FIFO is non-empty). FIFO depth F=48F=48 covers RTT 2Lmax+tack≈222L_ +t_ack≈22 cycles plus WcoalW_coal. Rd applies FlashAttention-2 online softmax [2, 12] ms m_s =max(ms−1,rowmax(zs)), = (m_s-1,\,rowmax(z_s)), ℓs _s =ems−1−msℓs−1+rowsum(ezs−ms), =e^m_s-1-m_s _s-1+rowsum(e^z_s-m_s), s _s =ems−1−mss−1+ezs−msVℓ,h,s, =e^m_s-1-m_so_s-1+e^z_s-m_sV_ ,h,s, (7) so V-flits never materialize a full attention matrix. Pf:Tm:Rd:s+1s+1s+2s+2s+3s+3s−1s-1s+1s+1s+2s+2s−2s-2s−1s-1s+1s+1time → 0: Pf prefetchissue kv_fetch(s+1)(s+1) ⋅· credit counterasync FIFO F=48F=48 flitsStage 1: Tm tile multiplyqℓ,h⋅Ksq_ ,h· K_s; logit accumulate in scratchpadasync FIFO F=48F=48 flitsStage 2: Rd stream reduceonline softmax: update (ms,ℓs,s)(m_s, _s,o_s)layero_layer (FP16)kv_fetchVN1 → home tilekv_datafrom home tileRTT2Lmax+tack2L_ +t_ack≈22≈22 cy ← one layeroverlapacrosssegments Fig. 3: Pad three-stage pipeline (Pf/Tm/Rd). Top: segment overlap conceals NoC RTT (≈22≈22 cycles). Bottom: KV prefetch, tile multiply, and online softmax. VIII Implementation We implement MeshKV on a tiled FPGA accelerator: an 8×88×8 interconnect with 512-bit wormhole links, four-bank tile-local KV SRAM, and dh=128d_h=128 systolic PEs, with TaKV, Mare, and Pad in the hardware datapath. We evaluate four configurations on the same platform: Cent, which routes KV traffic through a centralized memory path; Shared, which distributes blocks with round-robin placement and unicast; TaKV-only, which enables affine placement without multicast; and full MeshKV. Workloads are LLaMA-2-7B [23] and Mistral-7B [24] at T∈8192,32768T∈\8192,32768\ and B∈1,4,8B∈\1,4,8\. Latency, throughput, traffic, and utilization are hardware measurements (95% CIs <3%<3\% throughput over five runs). Per-router buffers plus a 256-bit bloom and a 16-entry exact tag table (Table I) require approximately one 18-Kb BRAM equivalent per router. TABLE I: Per-router parameters in the FPGA implementation. Parameter Value Cardinal ports + tile injection 5 Flit width 512 bit Unicast VCs (VN0) 2 Multicast VCs (VN1) 2 Per-VC input FIFO depth 8 flits Segment bloom width 256 bit In-flight exact tags 16 entries Bloom refresh interval WbW_b 96 cycles Coalescing window WcoalW_coal 12 cycles Pad Pf/Tm FIFO depth 48 flits Table I reports implemented router RTL, organized as 8 tiles per quadrant, together with PE and SRAM resources on an 8×88×8 Alveo U280 instance. DSP48E2 is the bottleneck; router, bloom, and tag logic use <5%<5\% of LUTs. TABLE I: Alveo U280 resource utilization for an 8×88×8 MeshKV instance (64 tiles; one attention layer). Percentages vs. DS963 device totals. Resource Used Available (%) LUT 613 K 47% F 820 K 31% BRAM36K (router + control) 768 38% URAM (KV staging SRAM) 512 53% DSP48E2 (MAC arrays) 5 760 64% IX Evaluation IX-A Interconnect, throughput, and latency Table IV reports NoC traffic at T=32KT=32K, B=1B=1, normalized to Cent. Affine TaKV placement reduces traffic to 0.66×0.66×. Mare then suppresses redundant head copies, and the full design reaches 0.42×0.42× (%58\% reduction). Mistral-7B reaches 0.40×0.40× because sliding-window attention localizes some fetches. Table V and Figure 1 decompose bisection link cycles. Cent expends nearly half of those cycles on back-pressure (19%19\% useful KV transfer at T=32KT=32K). MeshKV converts these links into useful transfer: utilization reaches %61\%, a 2.1×2.1× improvement over Shared. Figure 5 reports decode throughput at T=32KT=32K. At B=1B=1 the speedup is 1.35×1.35× from shorter hops and fewer credit stalls. At B=8B=8, streams share segment multicast, and the speedup increases to 1.90×1.90×. Fig. 4 reports per-token latency versus context length (LLaMA-2-7B, B=1B=1). Cent’s injection queue saturates near 8K tokens, after which latency grows steeply. Under MeshKV, latency remains approximately linear in T: TaKV distributes load and Mare reduces duplicate traversals. 4K8K16K32K⋅104· 10^40055101015152020Context TTLatency (μ )CentSharedMeshKV Fig. 4: Latency per token vs. context length (B=1B=1). TABLE IV: Normalized interconnect traffic (Cent= 1.00\,=\,1.00), T=32KT=32K, B=1B=1. Model Cent Shared TaKV-only MeshKV LLaMA-2-7B 1.00 0.81 0.66 0.42 Mistral-7B 1.00 0.79 0.63 0.40 TABLE V: Bisection KV bandwidth utilization (%), LLaMA-2-7B. Configuration T=8KT=8K T=32KT=32K Cent 29 19 Shared 38 29 TaKV-only 52 41 MeshKV (full) 71 61 B=1B=1B=4B=4B=8B=8001122Normalized throughputCentSharedMeshKV Fig. 5: Decode throughput speedup at T=32KT=32K, LLaMA-2-7B. IX-B Energy efficiency At T=32KT=32K, MeshKV reduces link energy by 48%48\% relative to Cent: TaKV shortens paths and Mare eliminates duplicate KV copies, in line with the 58%58\% traffic cut in Table IV. SRAM energy increases 7%7\% because TaKV reads from distributed banks rather than one contiguous array. Net energy per token decreases by %17\% under the activity-based estimate in §VIII. Shared multicast further amortizes the link term at B>1B>1, where concurrent streams reuse the same KV_DATA tree. IX-C Comparison with prior FPGA decoders FPGA LLM accelerators DFX [20] and FlightLLM [21] optimize end-to-end execution and HBM/BRAM management; CXL-SpecKV [22] speculates KV on a disaggregated FPGA. MeshKV implements the on-chip communication substrate for distributed KV movement, not a full FFN-and-norm decoder. Different platforms, mappings, memory systems, and datapaths therefore make absolute latencies incomparable. We report hardware measurements against Cent and Shared on the same 8×88×8 configuration: 58%58\% less interconnect traffic, 2.1×2.1× KV bandwidth utilization over Shared, and up to 1.9×1.9× throughput at B=8B=8 (Tables IV–V, Figure 5). IX-D Ablation and sensitivity Table VI ablates each mechanism at T=32KT=32K, B=8B=8 on LLaMA-2-7B (normalized to full MeshKV). Disabling multicast (Mare unicast) causes the largest regression: throughput falls to 0.62×0.62× and traffic rises to 1.55×1.55×. Removing duplicate suppression and Pad overlap yields 0.82×0.82× and 0.78×0.78× throughput, respectively; adaptive TaKV swaps cost 1111–15%15\% on skewed prefixes. At dh=128d_h=128, P=64P=64 balances TmodPT P stragglers and flit overhead. An 8×88×8 mesh attains 1.9×1.9× over Cent, versus 1.4×1.4× on a 16×416×4 mesh whose longer multicast trees add hop latency. Wcoal=12W_coal=12 cycles merges 78%78\% of heads at the home tile. TABLE VI: Ablation at T=32KT=32K, B=8B=8, LLaMA-2-7B. Mean± 95% CI over five independent hardware runs; normalized to MeshKV. Variant TP Traffic MeshKV 1.00±0.0181.00±0.018 1.00±0.0271.00±0.027 −-bloom 0.82±0.0230.82±0.023 1.38±0.0411.38±0.041 −-TaKV 0.88±0.0160.88±0.016 1.22±0.0291.22±0.029 −-Pad 0.78±0.0250.78±0.025 1.00±0.0211.00±0.021 −-Mare 0.62±0.0140.62±0.014 1.55±0.0441.55±0.044 Cent 0.53±0.0120.53±0.012 2.38±0.0682.38±0.068 X Conclusion MeshKV recasts decode KV movement as packetized mesh flows via TaKV striping, Mare multicast, and credit-aligned Pad. Hardware measurements on FPGA datapaths show up to 58% less interconnect traffic, 2.1×2.1× KV bandwidth utilization, and 1.9×1.9× multi-stream throughput relative to same-platform baselines. As context lengths grow, MeshKV is a promising communication-aware KV fabric for long-context decode on tiled accelerators. References [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §I. [2] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35, p. 16344–16359. Cited by: §I, §I, §I-A, §VII. [3] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP), Cited by: §I, §I, §I. [4] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang (2023) FlexGen: high-throughput generative inference of large language models with a single GPU. In International Conference on Machine Learning, p. 31094–31116. Cited by: §I, §I. [5] D. Liu and Y. Yu (2025) TinyServe: query-aware cache selection for efficient LLM serving. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 12529–12537. Cited by: §I. [6] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu (2024) KIVI: a tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning (ICML), Cited by: §I. [7] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami (2024) KVQuant: towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [8] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen (2023) H2_2O: heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Cited by: §I. [9] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen (2024) SnapKV: LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §I. [10] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2024) Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), Cited by: §I. [11] D. Liu and Y. Yu (2024) LLMEasyQuant: scalable quantization for parallel and distributed LLM inference. arXiv preprint arXiv:2406.19657. Cited by: §I. [12] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), Cited by: §I, §I-A, §VII. [13] K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y. Dong, and Y. Wang (2024) FlashDecoding++: faster large language model inference with asynchronization, flat GEMM optimization, and heuristics. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §I. [14] T. J. Ham, Y. Lee, S. H. Seo, S. Kim, H. Choi, S. J. Jung, and J. W. Lee (2021) ELSA: hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In International Symposium on Computer Architecture (ISCA), Cited by: §I. [15] H. Wang, Z. Zhang, and S. Han (2021) SpAtten: efficient sparse attention architecture with cascade token and head pruning. In IEEE International Symposium on High Performance Computer Architecture (HPCA), Cited by: §I. [16] L. Benini and G. De Micheli (2002) Networks on chips: a new SoC paradigm. Computer 35 (1), p. 70–78. Cited by: §I. [17] W. J. Dally and B. Towles (2004) Principles and practices of interconnection networks. Morgan Kaufmann. Cited by: §I. [18] C. J. Glass and L. M. Ni (1992) The turn model for adaptive routing. In International Symposium on Computer Architecture (ISCA), p. 278–287. Cited by: §I, §VI-A. [19] Y. S. Shao, J. Clemons, R. Venkatesan, B. Zimmer, M. Fojtik, N. Jiang, B. Keller, A. Klinefelter, N. Pinckney, P. Raina, S. G. Tell, Y. Zhang, W. J. Dally, J. S. Emer, C. T. Gray, B. Khailany, and S. W. Keckler (2019) Simba: scaling deep-learning inference with multi-chip-module-based architecture. In 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Cited by: §I, §I-B. [20] S. Hong, S. Moon, J. Kim, S. Lee, M. Kim, D. Lee, and J. Kim (2022) DFX: a low-latency multi-FPGA appliance for accelerating transformer-based text generation. In 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), Cited by: §I, §IX-C. [21] S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang, Y. Dai, J. Li, Z. Wang, R. Zhang, K. Wen, X. Ning, and Y. Wang (2024) FlightLLM: efficient large language model inference with a complete mapping flow on FPGAs. In ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA), Cited by: §I, §IX-C. [22] D. Liu and Y. Yu (2026) CXL-SpecKV: a disaggregated FPGA speculative KV-cache for datacenter LLM serving. In Proceedings of the 2026 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, p. 56–66. Cited by: §I, §IX-C. [23] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §I-C, §VIII. [24] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023) Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: §VIII.