Paper deep dive
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
Athos Georgiou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 7:01:11 AM
Summary
This study presents a comprehensive benchmark of production LLM inference on AMD Instinct MI325X GPUs using vLLM. It evaluates four models (Llama-3.1-405B, DeepSeek V3.2, Qwen3-VL-235B, Kimi-K2.5) across three architectural families (Dense+GQA, MoE+GQA, MoE+MLA). Key findings include the necessity of architecture-aware optimization, specifically that MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both. The AMD AITER runtime is critical for competitive MLA throughput but introduces variability. The study confirms that active parameter count drives throughput and identifies a memory-bandwidth bottleneck causing throughput saturation at specific concurrency levels.
Entities (10)
Relation Signals (8)
AMD Instinct MI325X → runs → vLLM
confidence 95% · We present a systematic cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs ... using the vLLM serving framework.
Llama 3.1-405B → usesarchitecture → Grouped Query Attention
confidence 95% · Llama-3.1-405B (Dense+GQA, 405B active)
DeepSeek V3.2 → usesarchitecture → Multi-Head Latent Attention
confidence 95% · DeepSeek V3.2 (MoE+MLA, 37B active)
Kimi k2.5 → usesarchitecture → Multi-Head Latent Attention
confidence 95% · Kimi-K2.5 (MoE+MLA, 32B active)
Grouped Query Attention → benefitsfrom → KV cache offloading
confidence 90% · GQA models benefit from both [KV cache offloading and standard block sizes]
Multi-Head Latent Attention → incompatiblewith → KV cache offloading
confidence 90% · MLA models ... cannot use KV cache offloading
Multi-Head Latent Attention → requires → AITER
confidence 90% · The AMD AITER runtime is required for competitive MLA inference throughput
AITER → increases → measurement variability
confidence 85% · AITER ablation ... reveals ... 2–16× higher measurement variability
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present a cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs, benchmarking four models spanning 235B to 1 trillion parameters across three architectural families (MoE+MLA, Dense+GQA, MoE+GQA) on an 8-GPU cluster with 2TB aggregate HBM3e using vLLM v0.14.1. Our results demonstrate that architecture-aware optimization is essential: MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both. The AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations. A controlled AITER ablation on Llama-3.1-405B (n=5 per condition) reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically. Under text-only workloads, Llama-405B and DeepSeek V3.2 achieve comparable peak throughput (15,944 and 15,343 tok/s) despite an order-of-magnitude difference in active parameters. Under vision workloads, Qwen3-VL-235B reaches 47,873 tok/s, 6.5x higher than Kimi-K2.5 (7,327 tok/s). Active parameter count per token is associated with inference throughput, though confounded by differences in quantization, AITER acceleration, and tensor parallelism. All four models exhibit a common throughput saturation point consistent with a memory-bandwidth bottleneck (~500 concurrent for short sequences, ~100-200 for longer sequences). All models maintain 100% HTTP-level success rates through 1,000 concurrent users, processing 18.9 million tokens across 17,406 requests without failures.
Tags
Links
- Source: https://arxiv.org/abs/2603.10031v1
- Canonical: https://arxiv.org/abs/2603.10031v1
Trouble viewing inline? Open PDF directly →
Full Text
121,065 characters extracted from source content.
Expand or collapse full text
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study Athos Georgiou athos.georgiou@nca-it.com (Technical Report – February 23, 2026) Abstract Large language model (LLM) inference at frontier scale demands careful co-optimization of model architecture, hardware capabilities, and serving-system configuration, yet systematic benchmarking studies on AMD accelerators remain scarce. We present a systematic cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs, benchmarking four models spanning 235 billion to 1 trillion parameters across three architectural families (MoE+MLA, Dense+GQA, MoE+GQA) on an 8-GPU cluster with 2 TB aggregate HBM3e using vLLM v0.14.1. Our results demonstrate that architecture-aware optimization is essential. On the current ROCm stack, MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both. The AMD AITER runtime is required for competitive MLA inference throughput, with a Triton fallback available at substantially reduced performance, and must be selectively disabled for architectures with incompatible attention head configurations. A controlled AITER ablation on Llama-3.1-405B (n=5n=5 per condition) reveals a modest 3–5% throughput benefit at high concurrency but 2–16× higher measurement variability, confirming that AITER’s large speedups target MoE/MLA kernels specifically. Under text-only workloads, Llama-3.1-405B (Dense+GQA, 405B active) and DeepSeek V3.2 (MoE+MLA, 37B active) achieve comparable peak throughput at 15,944 and 15,343 tok/s respectively, despite an order-of-magnitude difference in active parameters. Under vision workloads, Qwen3-VL-235B (MoE+GQA, 22B active) reaches 47,873 tok/s, 6.5× higher than Kimi-K2.5 (MoE+MLA, 32B active, 7,327 tok/s); these totals include image tokens from the vision encoder and are not directly comparable to the text-only results. Active parameter count per token is associated with inference throughput across the models tested, though confounded by differences in quantization, AITER acceleration, and tensor parallelism (Section 6). All four models exhibit a common throughput saturation point within a given workload, consistent with a memory-bandwidth bottleneck (∼ 500 concurrent for short sequences, ∼ 100–200 for longer sequences). All models maintain 100% HTTP-level request success rates (HTTP 200 with valid response structure) through 1,000 concurrent users, processing 18.9 million tokens across 17,406 requests without failures. 1 Introduction Large language models (LLMs) have rapidly evolved from research prototypes into production infrastructure, powering applications ranging from conversational agents and code generation to scientific reasoning and multimodal understanding [29, 13]. As model scale has grown from billions to trillions of parameters, the challenge of efficient inference serving has become a critical bottleneck for real-world deployment. While much attention has focused on training-side scaling laws, the operational reality of serving these models at scale, where throughput, latency, and hardware utilization must be simultaneously optimized, remains comparatively underexplored. The inference landscape is further complicated by the rapid diversification of model architectures. Dense transformer models with Grouped-Query Attention (GQA) [1], Mixture-of-Experts (MoE) architectures with hundreds of billions of total parameters [26, 15], and novel attention mechanisms such as Multi-head Latent Attention (MLA) [9] each impose fundamentally different demands on memory hierarchy, parallelism strategies, and kernel-level optimization. A deployment configuration that yields excellent throughput for a dense GQA model may produce suboptimal or even erroneous results when applied to an MoE model with MLA, yet systematic studies of these architecture-specific serving behaviors remain scarce. Simultaneously, the GPU accelerator landscape is broadening beyond a single vendor. AMD’s Instinct MI325X accelerators, based on the CDNA 3 architecture, offer 256 GB of HBM3e memory per device with 6.0 TB/s bandwidth, presenting a compelling alternative for large-scale inference workloads. The ROCm software ecosystem has matured to support production inference frameworks such as vLLM [16], yet comprehensive benchmarking studies on AMD hardware for state-of-the-art LLMs, particularly trillion-parameter models, remain limited in scope. In this work, we present a cross-architecture benchmark study of LLM inference on AMD Instinct MI325X GPUs, systematically evaluating four architecturally diverse models spanning from 235 billion to 1 trillion parameters using the vLLM serving framework. Our testbed comprises an 8-GPU cluster with 2 TB of aggregate HBM3e, and our evaluation covers concurrency scaling from single-request latency through 1,000 concurrent users, stress testing under diverse workload profiles, and architecture-specific optimization. The models under study comprise Kimi-K2.5 (1T total, 32B active; MoE+MLA) [20], DeepSeek V3.2 (685B total, 37B active; MoE+MLA) [10, 11], Llama-3.1-405B (405B dense; GQA) [13], and Qwen3-VL-235B (235B total, 22B active; MoE+GQA) [22], collectively representing the major architectural paradigms in contemporary LLM design. Our study makes the following key contributions: 1. Cross-architecture MI325X benchmark study for LLM inference at scale. We provide the first academic cross-architecture evaluation of production LLM serving on AMD Instinct MI325X hardware with architecture-specific configuration characterization, covering four frontier models across three architectural families (dense GQA, MoE+GQA, MoE+MLA), with workloads ranging from single requests to 1,000 concurrent users. Industry benchmarks such as SemiAnalysis InferenceMAX [25] provide multi-model MI325X throughput comparisons, but do not characterize the architecture-specific configuration constraints that govern deployment. All configurations achieve 100% success rates under stress testing, processing 18.9 million tokens across 17,406 requests without failures. 2. Architecture-aware optimization requires fundamentally different strategies. We demonstrate that MoE, MLA, and GQA architectures demand distinct serving configurations. On the current ROCm stack, MLA models require block size 1 and are incompatible with KV cache offloading, while GQA models benefit from KV offloading and standard block sizes. The AMD AI Tensor Engine for ROCm (AITER) is required for competitive production MLA inference throughput on ROCm; a Triton MLA fallback exists but delivers substantially lower performance, making AITER a practical necessity for production deployments. A controlled A/B ablation on the GQA-based Llama-3.1-405B (n=5n=5 independent server restarts per condition) reveals a modest 3–5% throughput benefit at high concurrency but a 2–16× increase in measurement variability (coefficient of variation), confirming that AITER’s documented 2–3× speedups are specific to MoE and MLA kernels rather than general attention acceleration. AITER must be disabled for MLA configurations with incompatible head-count constraints. These findings challenge the assumption that a single serving configuration can be applied across architectures. 3. Trillion-parameter model deployment on production GPU clusters. We demonstrate successful deployment and benchmarking of Kimi-K2.5, a 1 trillion parameter MoE model, on a cluster of 4 MI325X GPUs using INT4 quantization-aware training (QAT) weights. This represents, to our knowledge, the first published inference benchmark of a trillion-parameter model on MI325X (CDNA 3), achieving 7,327 tok/s throughput at 500 concurrent requests with 100% reliability. 4. Empirical validation that active parameters drive throughput at frontier scale. It is well established in MoE literature that active parameter count per token, not total parameter count, governs inference compute cost [15, 3]. Our results provide quantitative validation of this principle at frontier scale on MI325X, though the comparison is confounded by other experimental variables (Section 6). Qwen3-VL-235B (22B active) achieves 47,873 tok/s despite having more total parameters than models with lower throughput, while the 1T-parameter Kimi-K2.5 (32B active) achieves throughput comparable to models one-third its total size. This empirical confirmation at frontier scale has practical implications for model selection in throughput-sensitive deployments. 5. Workload-dependent throughput saturation. All four models, despite spanning a 4× range in total parameters and employing three different architectural paradigms, exhibit a common throughput saturation point within a given workload on our 8-GPU MI325X cluster. The saturation threshold is workload-dependent: ∼ 500 concurrent for the stress test workload (500-token input, 200-token output), confirmed by fine-grained sweeps from 500 to 1,000 in steps of 50, and ∼ 100–200 concurrent for longer-sequence workloads (2,048-token input, 512-token output). This common saturation across architecturally diverse models within a given workload suggests a memory-bandwidth bottleneck, consistent with the DRAM bandwidth saturation dynamics described by Recasens et al. [23]. The remainder of this paper is organized as follows. Section 2 surveys related work in LLM serving systems, model architectures, and quantization techniques. Section 3 describes the system architecture and hardware platform. Section 4 details the optimization techniques employed. Section 5 presents our experimental methodology, including workload design and metrics. Section 6 reports our experimental results across all models and workload configurations. Section 7 discusses the implications of our findings, and Section 8 concludes with directions for future work. 2 Related Work Our work sits at the intersection of LLM serving systems, model architecture innovation, hardware-aware optimization, and quantization for inference. We survey each area and position our contributions within the existing literature. 2.1 LLM Serving Systems The efficient serving of large language models has emerged as a critical systems challenge, driven by the autoregressive nature of text generation and the enormous memory footprint of modern models. Continuous batching and iteration-level scheduling. Orca [31] introduced iteration-level scheduling for transformer-based generative models, enabling the serving system to add and remove requests from a batch at each decoding iteration rather than waiting for an entire batch to complete. This continuous batching approach, combined with selective batching that applies batching only to compatible operations, achieved up to 36.9× throughput improvement over NVIDIA FasterTransformer on GPT-3 175B. Orca’s design established the foundational scheduling paradigm adopted by subsequent serving systems. Memory-efficient KV cache management. Building on Orca’s scheduling innovations, vLLM [16] addressed the critical memory management challenge for KV caches. The PagedAttention algorithm, inspired by virtual memory and paging in operating systems, partitions the KV cache of each sequence into fixed-size blocks that can be stored in non-contiguous memory. This approach achieves near-zero memory waste (under 4%), enabling 2–4× throughput improvements over prior systems including Orca and FasterTransformer. vLLM has become the de facto standard for LLM serving, supporting a wide range of models and hardware backends including AMD ROCm. Our work uses vLLM as the serving framework and provides a systematic cross-architecture evaluation of its performance on MI325X hardware across architecturally diverse frontier models, complementing industry benchmarks from SemiAnalysis InferenceMAX [25] and MLPerf [24] with architecture-specific configuration characterization. Alternative serving frameworks. SGLang [33] introduced RadixAttention for KV cache reuse across structured language model programs, achieving up to 5× throughput over vLLM for workloads with prefix sharing. HuggingFace Text Generation Inference (TGI) provides a production-ready serving solution with continuous batching and tensor parallelism support. NVIDIA’s TensorRT-LLM offers vendor-specific optimizations including custom attention kernels, inflight batching, and FP8/FP4 quantization for NVIDIA GPUs. While these systems have been extensively benchmarked on NVIDIA hardware, systematic evaluations on AMD accelerators remain limited. Our study fills this gap by providing detailed performance characterization of vLLM on MI325X across multiple model architectures. 2.2 Model Architectures The models evaluated in this study span three major architectural paradigms: dense transformers with grouped-query attention, mixture-of-experts models, and models employing multi-head latent attention. Each architecture presents distinct serving challenges. The transformer foundation. The transformer architecture [29] established self-attention as the dominant paradigm for sequence modeling. As models scaled to hundreds of billions of parameters, the KV cache required for autoregressive generation became a primary memory bottleneck, motivating architectural innovations to reduce its footprint. Attention mechanism variants. Multi-Query Attention (MQA) [27] proposed sharing key and value heads across all query heads, dramatically reducing KV cache size and memory bandwidth requirements during decoding. Grouped-Query Attention (GQA) [1] generalized this approach by using an intermediate number of key-value heads, achieving quality close to standard multi-head attention with speed approaching MQA. GQA has been widely adopted in production models including the Llama family [13]. In our evaluation, Llama-3.1-405B employs GQA in a dense architecture, while Qwen3-VL-235B combines GQA with MoE. We find that GQA models benefit from KV cache offloading and standard PagedAttention block sizes, achieving the highest absolute throughput in our study. Multi-head Latent Attention (MLA). DeepSeek-V2 [9] introduced Multi-head Latent Attention, which compresses the KV cache into a low-rank latent representation, reducing KV cache memory by 93.3% compared to standard multi-head attention while maintaining or improving model quality. MLA has been adopted by DeepSeek-V3 [10] and Kimi-K2.5 [20]. However, MLA introduces significant serving constraints: on the current ROCm stack it requires block size 1 for PagedAttention and is incompatible with KV cache offloading, and imposes specific attention head distribution requirements for tensor parallelism and hardware-accelerated kernels. While individual constraints have been documented in scattered GitHub issues (vLLM, AITER), our work consolidates these into a systematic characterization across multiple MLA models on MI325X and demonstrates their cumulative impact on deployment configuration. Mixture-of-Experts (MoE). The Sparsely-Gated Mixture-of-Experts architecture [26] demonstrated that model capacity can scale independently of computational cost by routing each token to a sparse subset of expert networks. Mixtral 8x7B [15] popularized MoE for LLMs, using 8 experts with 2 selected per token to achieve 47B total parameters with only 13B active. DeepSeek-V3 [10] (subsequently updated as V3.2 [11]) scaled this to 685B parameters with 37B active, while Kimi-K2.5 [20] reaches 1 trillion parameters with 384 experts and 32B active per token. It is well established that active parameter count, rather than total count, governs per-token inference compute [15, 3]. Our benchmarks corroborate this at frontier scale on MI325X across the models tested, though the comparison is confounded by other experimental variables (see Section 6 for details), with practical implications for model selection in production deployments. 2.3 Hardware Acceleration for LLM Inference Attention kernel optimization. FlashAttention [5] introduced an IO-aware exact attention algorithm that tiles attention computation to minimize HBM reads and writes, achieving 2–4× wall-clock speedups for transformer training and inference. FlashAttention-2 [6] further improved parallelism and work partitioning, reaching 50–73% of theoretical peak FLOPs on NVIDIA A100 GPUs. These kernel-level optimizations are critical for inference throughput, and their availability (or absence) on specific hardware platforms directly impacts serving performance. AMD Instinct and ROCm ecosystem. AMD’s Instinct MI300X and MI325X accelerators, based on the CDNA 3 architecture, provide high-bandwidth memory (up to 256 GB HBM3e at 6.0 TB/s per device on MI325X) designed for large-scale AI workloads. The ROCm (Radeon Open Compute) software stack provides an open-source platform for GPU computing, including support for PyTorch, vLLM, and custom kernel libraries. The AMD AI Tensor Engine for ROCm (AITER) provides optimized kernels for MoE and attention operations on Instinct hardware, with AMD reporting 2–3× inference speedups for compatible architectures [2]. However, AITER compatibility is architecture-dependent. On the current ROCm stack, AITER is required for competitive production MLA inference throughput, consistent with AMD’s documentation and community experience; a Triton MLA fallback exists but delivers substantially lower performance, making controlled ablation impractical. We find AITER must be entirely disabled for Kimi-K2.5 due to MXFP4 hardware requirements (CDNA 4 only) and attention head count constraints (the AITER MLA backend supports exactly 16 or 128 heads per rank; supported values may vary by AITER version). Prior benchmarking studies of LLM inference on AMD hardware have been limited in scope, typically evaluating single models or narrow workload ranges. Our work provides the first academic multi-model evaluation spanning dense, MoE, and MLA architectures on MI325X with architecture-specific configuration characterization. 2.4 Quantization for LLM Inference Model quantization has become essential for deploying large models within GPU memory constraints, with techniques spanning post-training quantization (PTQ) and quantization-aware training (QAT). Weight quantization. GPTQ [12] demonstrated accurate post-training quantization to 3–4 bits per weight using approximate second-order information, enabling 175B-parameter models to fit in a single GPU for the first time. AWQ [18] introduced activation-aware weight quantization, identifying that protecting only 1% of salient weight channels (determined by activation distributions) significantly reduces quantization error. AWQ received the MLSys 2024 Best Paper Award and has become widely adopted for INT4 weight-only quantization. Weight-activation quantization. SmoothQuant [30] enabled W8A8 (8-bit weights and activations) quantization by smoothing activation outliers through an offline mathematical transformation that migrates quantization difficulty from activations to weights. FP8 quantization has emerged as a practical format for LLM inference, with recent studies demonstrating that FP8 weight and activation quantization (W8A8-FP) is effectively lossless across model scales for the Llama family. FP8 quantization reduces memory consumption by approximately 50% compared to FP16/BF16, enabling larger models and batch sizes on fixed hardware. Quantization in our study. Our evaluation spans multiple quantization strategies dictated by architectural constraints: FP8 for Llama-3.1-405B and DeepSeek V3.2, BF16 for Qwen3-VL-235B (whose vision encoder dimensions are incompatible with FP8 block quantization kernels on both ROCm and CUDA platforms), and INT4 QAT compressed tensors for Kimi-K2.5. We find that quantization format selection is not merely a precision-performance trade-off but is constrained by architecture-specific compatibility with hardware kernels: an underappreciated dimension of deployment planning. 2.5 KV Cache Optimization Efficient KV cache management is central to high-throughput LLM serving. Beyond PagedAttention’s memory-efficient allocation [16], recent work has explored KV cache compression, eviction, and offloading strategies [32]. KV cache offloading to CPU memory extends effective context capacity for memory-bound workloads, and vLLM supports this through its native offloading backend. However, we demonstrate that KV cache offloading compatibility is architecture-dependent: GQA models (Llama-3.1-405B, Qwen3-VL-235B) successfully utilize offloading, while MLA models (DeepSeek V3.2, Kimi-K2.5) are incompatible with the offloading connector on the current ROCm stack. vLLM’s planned offloading redesign (RFC #22605) would also benefit MLA models by supporting their compressed latent KV cache format. This architectural dependency has significant implications for context length scaling and workload planning. 2.6 Positioning of This Work The MLPerf Inference benchmark [24] provides a standardized framework for cross-platform ML inference evaluation. AMD submitted MI325X results for Llama 2 70B in MLPerf Inference v5.0, with v5.1 (September 2025) expanding coverage to include Mixtral 8x7B and Llama 2 70B Interactive scenarios. However, the benchmark suite does not currently cover the full architectural diversity examined in our study: specifically, MoE routing, multi-head latent attention, and trillion-parameter model deployment. LLM-Inference-Bench [4] benchmarks eight models across seven hardware platforms, including AMD MI300X, using vLLM and TensorRT-LLM, but evaluates only models up to 72B parameters, does not include MI325X, and does not reach 1,000-user concurrency. SemiAnalysis InferenceMAX [25] provides nightly multi-model, multi-hardware throughput-latency Pareto frontiers including MI325X, but does not characterize architecture-specific configuration constraints or provide fixed-concurrency scaling analysis. MoE-Inference-Bench [3] systematically evaluates MoE inference throughput across varying active/total parameter ratios, but covers only models up to 70B on NVIDIA H100. Existing academic LLM serving benchmarks have otherwise predominantly focused on NVIDIA hardware with single-architecture evaluations. Our work contributes the first academic cross-architecture comparative benchmark on MI325X at 1,000-concurrent-user scale, encompassing (1) four architecturally diverse frontier models (dense GQA, MoE+GQA, MoE+MLA), (2) characterization of architecture-specific constraints governing serving configuration (block size, AITER compatibility, KV offloading, tensor parallelism), (3) the first published inference benchmark of a trillion-parameter model on MI325X (CDNA 3), acknowledging AMD’s prior MI355X benchmarks and ORNL’s MI250X training work, and (4) empirical characterization of workload-dependent throughput saturation behavior (consistent with the memory-bandwidth bottlenecks modeled analytically by LIMINAL [8] and observed empirically by Recasens et al. [23]) across all four models on our 8-GPU MI325X cluster. Speculative decoding [17], an orthogonal optimization for autoregressive generation, is not evaluated in this study but represents a promising direction for future work. 3 System Architecture and Hardware Platform This section describes the hardware platform, inference engine, and model architectures evaluated in our study. Our experimental setup targets a production-representative configuration comprising an 8-GPU AMD Instinct MI325X cluster running vLLM v0.14.1, serving four frontier-scale large language models spanning diverse architectural families. 3.1 AMD Instinct MI325X Platform All experiments were conducted on a single-node server equipped with eight AMD Instinct MI325X accelerators. The MI325X is built on AMD’s CDNA 3 architecture (gfx942, the instruction set architecture identifier for CDNA 3) and represents the current generation of AMD data center GPUs optimized for large-scale AI inference workloads. Table 1 summarizes the key specifications. Table 1: AMD Instinct MI325X specifications and 8-GPU cluster aggregate resources. Specification Per GPU 8-GPU Cluster HBM3e Capacity 256 GB 2 TB Memory Bandwidth 6.0 TB/s 48 TB/s FP16 Compute 1,307 TFLOPS 10.5 PFLOPS Architecture CDNA 3 (gfx942) The MI325X’s 256 GB HBM3e capacity per accelerator is a critical enabler for frontier-scale model serving. With tensor parallelism across eight GPUs, even the largest models leave substantial per-GPU headroom: DeepSeek V3.2 in FP8 requires only ∼ 83 GiB of weight memory per GPU (as reported by vLLM’s model loading log), consuming approximately 35% of each GPU’s 256 GB capacity (noting that 83 GiB ≈ 89 GB) and leaving the remainder available for KV cache and batch state. This eliminates the need for KV cache offloading to CPU memory in most deployment scenarios, reducing architectural complexity and latency. LLM inference is fundamentally memory-bandwidth-bound rather than compute-bound [21]. The MI325X’s 6.0 TB/s memory bandwidth per accelerator (48 TB/s aggregate) directly translates to higher token throughput, particularly at large batch sizes where memory access patterns dominate runtime. The CDNA 3 microarchitecture provides hardware-level support for FP8 matrix operations, enabling quantized inference without dedicated quantization accelerators. The system runs ROCm 6.4.2 with RCCL 2.26.6 for multi-GPU communication. NUMA balancing is disabled (kernel.numa_balancing=0) to prevent the operating system from migrating memory pages between NUMA nodes, which would degrade GPU communication latency. The software stack is containerized using Docker 29.1.5 with ROCm support, ensuring reproducible deployments. 3.2 vLLM Inference Engine We use vLLM v0.14.1 [16] as the inference serving engine. vLLM implements several key techniques that enable efficient high-throughput LLM serving: PagedAttention. vLLM’s core memory management innovation is PagedAttention, which manages KV cache memory using a paging mechanism inspired by virtual memory systems in operating systems. Rather than pre-allocating contiguous memory blocks for each sequence’s KV cache, PagedAttention divides the cache into fixed-size blocks that can be allocated and freed independently. This virtually eliminates memory fragmentation and enables near-optimal memory utilization, allowing more concurrent sequences to be served from the same GPU memory budget. Continuous Batching. Unlike static batching approaches that wait for an entire batch to complete before processing new requests, vLLM implements continuous (or iteration-level) batching. New requests are inserted into the running batch at each decoding step as slots become available, maximizing GPU utilization and minimizing queuing delays. V1 Engine and Chunked Prefill. vLLM v0.14.1 uses the V1 engine architecture, in which chunked prefill is always enabled and cannot be disabled. Chunked prefill splits long prompt processing into smaller chunks that are interleaved with decode steps from other sequences. This prevents long prompts from monopolizing the GPU and allows decode-phase sequences to maintain low inter-token latency even when new long-context requests arrive. The chunk size is controlled by --max-num-batched-tokens, which determines the maximum number of tokens processed in a single scheduler iteration. Multi-Step Scheduling. vLLM supports multi-step scheduling via the --num-scheduler-steps parameter, which batches multiple decode steps before returning to the scheduler. This reduces CPU-side scheduling overhead and improves GPU utilization, with values of 10–15 providing measurable throughput gains before diminishing returns. All models are served through vLLM’s OpenAI-compatible API endpoint, containerized in Docker images (vllm/vllm-openai-rocm:latest for stable models; rocm/vllm-dev:nightly for Kimi-K2.5 which requires a nightly build, specifically v0.16.0rc1.dev88). 3.3 Model Architectures We evaluate four frontier-scale models spanning three architectural families: Mixture-of-Experts with Multi-head Latent Attention (MoE+MLA), dense transformer with Grouped-Query Attention (Dense+GQA), and MoE with GQA (MoE+GQA). Table 2 provides a comparative overview. Table 2: Evaluated model architectures. Active parameters are per-token for MoE models. Per-GPU memory is from vLLM startup logs on the 8-GPU MI325X cluster at the deployed precision and tensor parallelism. Model Total Active Architecture Attention Context Precision Per-GPU Params Params Length Memory DeepSeek V3.2 685B ∼ 37B MoE + MLA MLA 160K FP8 ∼ 83 GiB Llama-3.1-405B 405B 405B Dense + GQA GQA 128K FP8 ∼ 112 GiB Qwen3-VL-235B 235B ∼ 22B MoE + GQA GQA 256K BF16 ∼ 58 GiB Kimi-K2.5 1T ∼ 32B MoE + MLA MLA 256K INT4 QAT ∼ 145 GiB DeepSeek V3.2 (685B). DeepSeek V3.2 [10, 11] is a Mixture-of-Experts model that uses Multi-head Latent Attention (MLA) to compress key-value pairs into a low-rank latent space, reducing the per-token KV cache footprint relative to standard multi-head attention. The model activates approximately 37B of its 685B total parameters per token via its expert routing mechanism. On our MI325X cluster, DeepSeek V3.2 requires mandatory configuration of --block-size 1 for the MLA KV cache format and the AITER_ENABLE_VSKIP=0 environment variable to prevent HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION errors in fused MoE kernels on CDNA 3. The model is served in FP8 precision with AITER acceleration enabled. Llama-3.1-405B. Meta’s Llama-3.1-405B-Instruct [13] is the largest dense (non-MoE) transformer in our evaluation. It employs Grouped-Query Attention (GQA), which shares key-value heads across multiple query heads to reduce KV cache memory requirements while maintaining attention quality. As a dense model, all 405B parameters are activated for every token, making it the most compute-intensive model per token in our study. FP8 quantization is essential to fit this model within the 8-GPU cluster’s memory budget while leaving room for KV cache at practical context lengths. Qwen3-VL-235B. Qwen3-VL-235B-A22B-Instruct [22] is a vision-language model combining a Mixture-of-Experts language backbone with a vision encoder (ViT). The MoE architecture activates only ∼ 22B of its 235B total parameters per token, yielding the lowest active parameter count in our evaluation. The model uses GQA for its attention mechanism. A notable constraint is that its vision encoder’s intermediate MLP dimension of 4304 is not divisible by 128 (the block size used by FP8 block quantization kernels), making it incompatible with FP8 quantization. This is a model-architecture constraint, not a platform-specific limitation; the same incompatibility has been reported on NVIDIA hardware (vLLM Issues #30934, #26589). Consequently, it must be served in BF16 precision. Despite this, its low active parameter count enables the highest throughput among all evaluated models. Kimi-K2.5 (1T). Moonshot AI’s Kimi-K2.5 [20] is the largest model in our study at 1 trillion total parameters. It employs an MoE architecture with 384 experts (8 selected per token), activating approximately 32B parameters per forward pass. Like DeepSeek V3.2, it uses MLA for attention. However, AITER cannot be enabled for Kimi-K2.5 on MI325X: enabling AITER encounters a fatal error during CUDA graph capture (“MXFP4 is not available on your device”), because the AITER MLA backend’s MXFP4 (microscaling FP4) quantization pathway requires hardware support available only on MI350X and later (CDNA 4). Additionally, Kimi-K2.5’s 64-head MLA configuration creates a tensor parallelism constraint: with TP==8, each GPU receives only 8 attention heads, which falls outside the AITER MLA backend’s supported head counts (exactly 16 or 128 per rank; supported values may vary by AITER version). The official Moonshot AI deployment guide specifies TP==8 with the Triton MLA fallback at reduced performance. In our evaluation, Kimi-K2.5 is deployed with TP==4 and AITER disabled, utilizing only half of the available GPUs; the TP==4 choice was made to match the AITER head count constraint (64 / 4 = 16 heads per GPU) during initial deployment attempts before the MXFP4 limitation was identified, and was retained for consistency. The model ships with native INT4 Quantization-Aware Training (QAT) via the compressed-tensors format, including a 400M-parameter MoonViT vision encoder. It requires the vLLM nightly build (rocm/vllm-dev:nightly) as stable releases do not yet support its architecture. Table 3 summarizes the deployment configuration for each model, including the AITER component flags, tensor parallelism degree, and block size required by the architecture. Table 3: Per-model deployment configuration on the MI325X cluster. AITER columns indicate which kernel components are enabled (1) or disabled (0). Model AITER MHA MLA MoE Block TP Size DeepSeek V3.2 1 0 1 1 1 8 Llama-3.1-405B 1a 1 0 0 16 8 Qwen3-VL-235B 1a 1 0 1 16 8 Kimi-K2.5 0b – – – 1 4 a For Llama and Qwen3-VL, AITER is enabled explicitly (VLLM_ROCM_USE_AITER=1). Note that VLLM_ROCM_USE_AITER defaults to 0 (disabled) when unset. DeepSeek also sets it explicitly. b Kimi-K2.5 requires AITER disabled (VLLM_ROCM_USE_AITER=0) due to attention head count incompatibility; AITER=1 encounters a fatal error during CUDA graph capture. Detailed command-line flags for each model’s deployment are provided in Table 11 (Section 5). The architectural diversity across these four models, spanning dense and sparse parameter activation, MLA and GQA attention mechanisms, text-only and vision-language modalities, and FP8, BF16, and INT4 precision formats, provides a comprehensive testbed for evaluating the AMD MI325X platform’s versatility as an inference accelerator for frontier-scale models. 4 Optimization Techniques Deploying frontier-scale models on the MI325X cluster requires a combination of quantization, memory management, kernel-level acceleration, parallelism configuration, and scheduling optimizations. This section details the techniques applied and their model-specific considerations. 4.1 Quantization Strategies Quantization reduces the numerical precision of model weights and activations, decreasing memory consumption and improving computational throughput at the cost of potential accuracy degradation. We employ three distinct quantization strategies depending on model architecture and compatibility constraints, summarized in Table 4. Table 4: Quantization strategy and compatibility for each model. FP8 KV refers to whether --kv-cache-dtype fp8 can be used. Model Method Precision FP8 KV Memory Savings DeepSeek V3.2 FP8 W8A8 Noa ∼ 50% Llama-3.1-405B FP8 W8A8 Yesb ∼ 50% Qwen3-VL-235B None (BF16) W16A16 Noc – Kimi-K2.5 INT4 QAT W4 Nod ∼ 75% aMLA backend uses fp8_ds_mla format automatically; --kv-cache-dtype fp8 is incompatible. bFP8 KV cache is architecturally supported but was not enabled in our benchmarks; savings reflect FP8 weights only. cVision encoder intermediate dimension (4304) not divisible by 128 (FP8 block quantization block size); incompatible with FP8 kernels on both ROCm and CUDA platforms. dMLA architecture incompatible with standard FP8 KV cache format. FP8 Quantization (W8A8). Standard FP8 quantization (--quantization fp8) converts both weights and activations from 16-bit to 8-bit floating point, yielding approximately 50% memory reduction with minimal accuracy loss for most workloads [19]. On the AMD ROCm platform, we additionally evaluated Per-Token Per-Channel FP8 (PTPC-FP8, --quantization ptpc_fp8), which applies per-token scaling to activations and per-channel scaling to weights. PTPC-FP8 provides improved accuracy over standard FP8 by adapting the dynamic range to each token’s activation distribution and each output channel’s weight distribution independently, and is the recommended FP8 mode for ROCm since vLLM v0.7.3. The MI325X’s CDNA 3 architecture implements the OCP FP8 standard, using the E4M3 format (4 exponent bits, 3 mantissa bits, dynamic range ± 448) for both weights and activations during inference [19]. Two FP8 quantization modes are available in vLLM for ROCm: per-tensor scaling (fp8) and per-token-per-channel scaling (ptpc_fp8). Both achieve identical memory reduction; PTPC-FP8 provides better numerical accuracy by adapting dynamic range independently per token and output channel. Our benchmarks use per-tensor fp8 quantization. FP8 is critical for fitting Llama-3.1-405B on the cluster: at BF16 the per-GPU weight memory would be approximately 224 GiB (extrapolating from the measured FP8 footprint), leaving minimal headroom within each GPU’s 256 GB HBM for KV cache and runtime buffers. With FP8, the per-GPU weight footprint drops to ∼ 112 GiB, enabling deployment with substantial KV cache capacity. For DeepSeek V3.2, FP8 reduces the per-GPU weight footprint from an estimated ∼ 180 GiB to ∼ 83 GiB. INT4 Quantization-Aware Training. Kimi-K2.5 ships with native INT4 QAT quantization using the compressed-tensors format [20]. Unlike post-training quantization, QAT incorporates quantization effects during training, enabling more aggressive compression (4-bit weights) while preserving model quality. This reduces the 1T-parameter model to ∼ 145 GiB per GPU (TP==4), fitting comfortably within four MI325X GPUs (1 TB aggregate HBM at TP==4). No additional quantization flags are needed; vLLM automatically detects the compressed-tensors format. Vision Encoder Constraints. Qwen3-VL-235B’s vision encoder (ViT) contains MLP layers with dimensions not divisible by 128 (e.g., the intermediate dimension of 4304, where 4304/128=33.6254304/128=33.625), which is the block size requirement for FP8 block quantization kernels. This is a model-architecture constraint rather than a platform-specific limitation: the same dimension incompatibility has been reported on NVIDIA hardware (vLLM Issues #30934, #26589). Attempting FP8 quantization produces a runtime error due to this alignment constraint. The model must therefore be served entirely in BF16. Despite this precision penalty, the MoE architecture’s low active parameter count (∼ 22B) enables competitive throughput. 4.2 KV Cache Management The KV cache stores key and value tensors from prior tokens during autoregressive generation and often dominates GPU memory consumption at long context lengths and high concurrency. We evaluate two complementary strategies: KV cache quantization and KV cache offloading to CPU memory. FP8 KV Cache. For models with GQA-based attention, the KV cache can be independently quantized to FP8 (--kv-cache-dtype fp8), reducing its memory footprint by approximately 50% relative to BF16. This is orthogonal to weight quantization and can be combined with FP8 weights for cumulative savings. Our Llama-3.1-405B benchmarks used FP8 weight quantization only; FP8 KV cache was not enabled, yielding approximately 50% weight memory savings. However, FP8 KV cache is incompatible with both MLA-based models in our evaluation. DeepSeek V3.2’s ROCMAiterMLASparseBackend uses a specialized fp8_ds_mla format that is automatically selected by vLLM; specifying --kv-cache-dtype fp8 produces a ValueError. Similarly, Kimi-K2.5’s MLA implementation does not support the standard FP8 KV cache interface. CPU Offloading. vLLM supports offloading KV cache blocks to CPU memory via --kv-offloading-backend native, with a configurable buffer size (--kv-offloading-size). This extends effective KV cache capacity beyond GPU HBM at the cost of increased latency from CPU–GPU data transfers. In our experiments, we configure offloading buffers of 64 GiB for models that support it. On the current ROCm stack (vLLM v0.14.1), KV cache offloading is not supported for MLA models, producing a KeyError at runtime. GQA models successfully use offloading on the same ROCm stack, indicating this limitation is MLA-specific rather than a blanket ROCm constraint. vLLM’s planned offloading redesign (RFC #22605), which proposes a separated-process architecture with CUDA IPC handles, would also benefit MLA models by supporting their compressed latent KV cache format. Table 5 summarizes compatibility. Table 5: KV cache optimization compatibility by model and attention architecture. Model Attention FP8 KV CPU Offload Block Size DeepSeek V3.2 MLA No No 1 Llama-3.1-405B GQA Yes Yes 16 Qwen3-VL-235B GQA Noa Yes 16 Kimi-K2.5 MLA No No 1 aFP8 KV is architecturally compatible with GQA but blocked by the vision encoder’s FP8 incompatibility. In practice, the MI325X’s 256 GB HBM capacity per GPU (2 TB aggregate) substantially reduces the need for KV cache offloading. For the models and context lengths evaluated (up to 32K tokens with 1,000 concurrent requests), all workloads fit within HBM without offloading. We note, however, that offloading flags also serve as a workaround for a GEMM kernel compatibility error (RuntimeError: wrong! device_gemm … does not support this GEMM problem) that affects Llama-3.1-405B and Qwen3-VL-235B under certain AITER configurations. 4.3 AITER Kernel Optimization AMD’s AI Tensor Engine for ROCm (AITER) provides optimized compute kernels specifically designed for the Instinct GPU family. AITER replaces generic GPU kernels with hand-tuned implementations that exploit CDNA 3 microarchitectural features, yielding substantial performance improvements for key inference primitives. Performance Impact. According to AMD’s documentation, AITER provides the following speedups for DeepSeek V3/R1 workloads [2]: 2.1× overall inference acceleration, 2× faster block-scale GEMM operations, and 3× faster fused Mixture-of-Experts execution. These gains are most pronounced for MoE models where the expert routing and sparse matrix operations are dominant computational bottlenecks. We note that on the current ROCm stack, AITER is required for competitive production MLA inference throughput; a Triton MLA fallback exists (VLLM_ROCM_USE_AITER=0) but delivers substantially lower performance, making AITER a practical necessity for production deployments. See Section 6.5 for details. Component-Level Control. AITER is activated via the master environment variable VLLM_ROCM_USE_AITER=1 (which defaults to 0/disabled when unset). When enabled, it activates the following component flags: • VLLM_ROCM_USE_AITER_LINEAR – Quantization and GEMM operations • VLLM_ROCM_USE_AITER_MOE – Fused Mixture-of-Experts kernels • VLLM_ROCM_USE_AITER_RMSNORM – Accelerated RMS normalization • VLLM_ROCM_USE_AITER_MHA – Multi-Head Attention kernels • VLLM_ROCM_USE_AITER_MLA – Multi-head Latent Attention kernels • VLLM_ROCM_USE_AITER_FP8BMM – FP8 batched matrix multiplication Individual components can be selectively disabled when encountering model-specific incompatibilities (e.g., VLLM_ROCM_USE_AITER_MLA=0 for MLA accuracy issues under certain TP/DP configurations). VSKIP Configuration Requirement. A known issue (AITER Issue #1143, October 2025) affects AITER on MI300X/MI325X hardware: the AITER_ENABLE_VSKIP flag defaults to true when unset, causing HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION errors during inference with DeepSeek models. This must be explicitly disabled (AITER_ENABLE_VSKIP=0) for stable operation. The root cause is an incompatibility in the VSKIP-enabled fused MoE kernel variant on CDNA 3 hardware. We confirm this workaround is required on MI325X with vLLM v0.14.1. Model-Specific AITER Behavior. AITER’s applicability varies significantly across models (see Table 3). DeepSeek V3.2 benefits from full AITER acceleration (MLA + MoE kernels) with the VSKIP workaround. Llama-3.1-405B uses AITER’s MHA kernels but not MLA or MoE, as it is a dense GQA model. Qwen3-VL-235B uses AITER’s MHA and MoE kernels. Kimi-K2.5, however, must run with AITER entirely disabled (VLLM_ROCM_USE_AITER=0). Enabling AITER encounters a fatal error during CUDA graph capture (“MXFP4 is not available on your device”): the AITER MLA backend’s MXFP4 (microscaling FP4) quantization pathway requires hardware support available only on MI350X and later (CDNA 4), not on MI325X (CDNA 3). Independently, its 64-head MLA configuration also falls outside the AITER MLA backend’s supported head counts (exactly 16 or 128 per rank; supported values may vary by AITER version). FP8 BMM Warmup. AITER’s FP8 batched matrix multiplication kernels require pre-compilation on first invocation, adding approximately 3 minutes of warmup time to the initial model load for FP8-quantized models. Subsequent starts use cached compiled kernels and incur no additional overhead. 4.4 Tensor Parallelism Configuration Tensor parallelism (TP) [28] distributes model layers across multiple GPUs by partitioning weight matrices along specific dimensions, enabling models that exceed single-GPU memory to be served with low latency. All models in our evaluation require multi-GPU serving, but the optimal TP configuration differs based on architectural constraints. Default TP==8 Configuration. Three of the four models (DeepSeek V3.2, Llama-3.1-405B, and Qwen3-VL-235B) are deployed with TP==8, utilizing all eight MI325X GPUs. This maximizes the available memory bandwidth (48 TB/s aggregate) and memory capacity, and is the natural configuration for the 8-GPU cluster. Inter-GPU communication uses RCCL 2.26.6 with NCCL minimum channels set to 112 (NCCL_MIN_NCHANNELS=112) for high-throughput all-reduce operations. For large TP configurations, quantized all-reduce (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=FP) and BF16-to-FP16 casting (VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=1) provide additional communication bandwidth savings. Kimi-K2.5 TP==4 Constraint. Kimi-K2.5 represents a notable exception to the TP==8 default. The model’s MLA architecture has 64 attention heads. Under TP==8, each GPU would receive 64/8=864/8=8 heads, which falls outside the AITER MLA backend’s supported head counts (exactly 16 or 128 per rank; supported values may vary by AITER version). With AITER disabled, TP==8 is possible using the Triton MLA fallback (as specified in the official Moonshot AI deployment guide), but at reduced performance. In our evaluation, we use TP==4 with AITER disabled. The TP==4 choice was made during initial deployment to match the AITER head count constraint (64/4=1664/4=16 heads per GPU) before the MXFP4 hardware limitation was identified (Section 4.3), and was retained for consistency across benchmark runs. We note that TP==8 with the Triton MLA fallback would be a valid alternative that utilizes the full cluster’s bandwidth. Using only four GPUs, Kimi-K2.5 has access to only 1 TB of HBM and 24 TB/s of aggregate bandwidth, compared to 2 TB and 48 TB/s for TP==8 models. Block Size. MLA-based models (DeepSeek V3.2 and Kimi-K2.5) require --block-size 1 for their KV cache allocation on the ROCm/AITER stack. This is a platform-specific constraint: on NVIDIA hardware, FlashMLA uses block size 64, and Intel Gaudi uses block size 128 for MLA models. GQA-based models (Llama-3.1-405B and Qwen3-VL-235B) use the default block size of 16, which amortizes allocation overhead and enables more efficient memory management. 4.5 Concurrency and Scheduling Maximizing throughput under concurrent load requires tuning vLLM’s batching and scheduling parameters. The key parameters are --max-num-seqs (maximum concurrent sequences per batch), --max-num-batched-tokens (maximum tokens per scheduler iteration), and --num-scheduler-steps (decode steps per scheduling round). Batch Size Configuration. The --max-num-seqs parameter controls the upper bound on concurrent sequences. We evaluate configurations ranging from low-latency (128–256 sequences) to maximum-throughput (2048–4096 sequences). For our high-throughput configuration, we set --max-num-seqs 2048 with --max-num-batched-tokens 65536, which provides sufficient batch capacity to saturate the MI325X cluster’s memory bandwidth. Table 6 summarizes the configuration profiles. Table 6: Scheduling configuration profiles and their trade-offs. Profile max-num-seqs max-batched-tokens scheduler-steps Low latency 256 8,192 1 Balanced 512 16,384 10 High throughput 1,024 32,768 15 Maximum 2,048 65,536 15 Multi-Step Scheduling. The --num-scheduler-steps parameter batches multiple decode iterations before the scheduler re-evaluates the batch composition. Setting this to 15 reduces CPU-side scheduling overhead, as the scheduler runs once per 15 decode steps rather than once per step. Values above 20 yield diminishing returns, as the overhead savings plateau while responsiveness to new requests decreases. GPU Memory Utilization. The --gpu-memory-utilization parameter controls the fraction of GPU HBM allocated to the KV cache pool after model weights are loaded. We use 0.95 for maximum-throughput configurations, reserving only 5% of HBM for runtime overhead and temporary allocations. Lower values (0.85–0.90) are appropriate for workloads requiring memory headroom for variable-length sequences or vision encoder activations. Chunked Prefill Interaction. Since vLLM V1 always enables chunked prefill, the --max-num-batched-tokens parameter also controls the prefill chunk size. Lower values (2,048) favor inter-token latency by processing smaller chunks and yielding back to decode more frequently. Higher values (32,768+) favor time-to-first-token (TTFT) and overall throughput by processing more prompt tokens per scheduler iteration, at the cost of slightly increased ITL for concurrent decode sequences. 5 Experimental Methodology This section describes the benchmark framework, test environment, and workload design used to evaluate large-scale LLM inference on AMD Instinct MI325X GPUs with vLLM. 5.1 Benchmark Framework We developed a unified progressive benchmark that systematically evaluates inference serving performance across five phases of increasing intensity: 1. WARMUP – Initialize the model, warm GPU caches, and establish a single-request baseline. 2. BASELINE – Measure clean metrics at concurrency==1 to establish reference p99 latency. 3. SCALING – Sweep concurrency from 5 to 200 to characterize the throughput–latency trade-off. 4. STRESS – Test edge cases including long output generation (500+ tokens), long context prefill (4K–8K tokens), and multi-image vision workloads. 5. SATURATION – Find the breaking point by pushing concurrency from 150 to 1,000 concurrent requests. This progressive design ensures that each phase builds on prior results: the BASELINE p99 latency serves as the reference threshold for DEGRADED status detection (p99 >2×>2× baseline), and the SCALING phase throughput informs SATURATED status detection (throughput <1.05×<1.05× previous level). Prompt Construction. Benchmarks use a fixed deterministic text prompt (a 4-sentence passage on machine learning concepts) repeated to reach the target input token count using a ∼ 4 characters-per-token heuristic. Prompts are not randomized, which may interact with vLLM’s prefix caching optimizations. Vision benchmarks append image URLs to the same text prompt. Warmup Protocol. Each benchmark run begins with a single warmup request (1 concurrent request, 100 input tokens, 50 output tokens) to trigger initial model compilation. No post-warmup cooldown is applied between the warmup and measurement phases. Warmup results are recorded but labeled separately as Phase 1 (WARMUP) data. Inter-phase cooldowns of 3–5 seconds are applied between subsequent measurement phases. Concurrency Model. All concurrent requests are dispatched using Python’s ThreadPoolExecutor with max_workers set to the target concurrency level. Each worker issues a synchronous HTTP POST to the vLLM OpenAI-compatible /v1/chat/completions endpoint with a 300-second timeout. This approach accurately models real-world API gateway patterns where multiple client connections are served simultaneously. Client-Side Measurement Validity. To rule out client-side bottlenecks, we benchmarked the client against a local echo server returning instant responses at concurrency levels from 100 to 2,000. The client sustained over 885 requests/second with 100% success at all concurrency levels including 2,000 concurrent threads, with p99 latency under 25 ms at concurrency levels ≥500 (matching the benchmark’s saturation range). In contrast, vLLM benchmarks at saturation (∼ 500 concurrent) produce at most ∼ 19 requests/second for the fastest model (Qwen3-VL). The client’s measured ceiling exceeds actual benchmark request rates by ∼ 50×, confirming that the observed throughput saturation is server-side. Metrics Collection. For each test, we collect the following metrics from successful responses: • Latency percentiles: p50, p95, and p99, computed from sorted response times. • Total throughput: (Tprompt+Tcompletion)/twall(T_prompt+T_completion)/t_wall in tokens per second, where TpromptT_prompt and TcompletionT_completion are cumulative token counts from the API usage response and twallt_wall is the wall-clock duration. • Output throughput: Tcompletion/twallT_completion/t_wall in tokens per second, reflecting pure generation speed. • Success rate: fraction of requests returning HTTP 200 without client-side exceptions. This metric does not validate response content, JSON structure, or whether the model generated the requested number of output tokens. A response returning zero completion tokens with HTTP 200 would be counted as successful. Throughput metrics (tok/s) reflect actual tokens generated, providing an implicit output quality signal. Status Classification. Each test result is classified into one of four status categories, summarized in Table 7. Table 7: Benchmark status indicators and their thresholds. Status Threshold Interpretation OK Success ≥95%≥ 95\%, latency normal Normal operation DEGRADED p99 >2×>2× baseline p99 Expected latency increase under concurrent load SATURATED Throughput <1.05×<1.05× previous Throughput plateau reached FAILING Success <95%<95\% Requests are failing; reduce load Status labels use phase-relative criteria: the SCALING phase classifies requests as DEGRADED when p99 latency exceeds twice the single-request baseline, while the SATURATION phase uses throughput-relative classification only (SATURATED when throughput plateaus within 5% of the previous level). The baseline_p99 threshold is only passed to the SCALING phase; consequently, DEGRADED status cannot trigger in the SATURATION or STRESS phases, and status labels are not directly comparable across phases. Test Multipliers. The framework supports three intensity modes: quick (0.5×0.5× request count), default (1×1×), and thorough (3×3×). All stress test results reported in this paper use the thorough (3×3×) multiplier to ensure statistical robustness, while validation results use the quick (0.5×0.5×) multiplier for rapid correctness checks. Multi-Run Reproducibility Protocol. To characterize measurement variance, we performed multiple independent benchmark runs per model using a standardized workload (100 requests, 2,048 input tokens, 512 output tokens, quick mode). Five runs were completed for all four models. Each run used a fresh vLLM server restart, including full model reloading and CUDA graph recapture, to ensure statistical independence between runs. Confidence intervals are computed using the t-distribution at the 95% level with df=n−1df=n-1, appropriate for small sample sizes where the population variance is unknown. 5.2 Test Environment All experiments were conducted on a single server equipped with eight AMD Instinct MI325X GPUs. The complete hardware and software specifications are listed in Table 8. Table 8: Test environment specifications. Component Specification GPU 8× AMD Instinct MI325X Architecture CDNA 3 (gfx942) VRAM per GPU 256 GB HBM3e Total VRAM 2 TB Memory Bandwidth (per GPU) 6.0 TB/s Aggregate Bandwidth 48 TB/s FP16 Compute (per GPU) 1,307 TFLOPS ROCm 6.4.2-120 vLLM 0.14.1 Docker 29.1.5 RCCL 2.26.6 Python 3.10+ (inside container) Container (stable models) vllm/vllm-openai-rocm:latest† Container (Kimi-K2.5) rocm/vllm-dev:nightly† System RAM 256 GB+ CPU Cores 64+ NUMA Balancing Disabled † digests: vllm-openai-rocm sha256:236900d57300..., vllm-dev sha256:e8ce7f6d74a0.... Full digests and model checkpoint revisions are listed in Table 9. Table 9: Pinned model checkpoint revisions and container image digests for reproducibility. Artifact Revision / Digest DeepSeek-V3-0324 e9b33add7688 Llama-3.1-405B-Instruct be673f326cab Qwen3-VL-235B-A22B 710c13861be6 Kimi-K2-Instruct 1cbe779b5c9d vllm-openai-rocm sha256:236900d573001f7713e1526db1000dbaec660d025645528ecc614d811d25c5a vllm-dev:nightly sha256:e8ce7f6d74a05d11678920df707a5665d5998b438288da31406c9d65521a855 All models were served through Docker containers with ROCm device passthrough (/dev/kfd, /dev/dri), shared memory via --ipc=host, and model weights cached on the host filesystem. NUMA balancing was disabled (kernel.numa_balancing=0) to prevent the OS from migrating memory pages across NUMA domains, which degrades multi-GPU communication latency. 5.3 Workload Design We evaluated four large language models spanning three distinct architectural families. Model specifications and architectural details are presented in Table 2 (Section 3). 5.3.1 Validation Tests Validation tests use the quick (0.5×0.5×) multiplier and serve two purposes: (1) verifying functional correctness of the deployment (all API endpoints return valid responses) and (2) establishing preliminary scaling curves for rapid comparison. Validation concurrency sweeps cover levels from 5 to 100, and saturation tests extend to 500 concurrent requests. 5.3.2 Stress Tests Stress tests use the thorough (3×3×) multiplier for comprehensive evaluation. The workload categories are: • Concurrency scaling: Sweep from 5 to 200 concurrent requests with 500-token input prompts and 200-token output generation, measuring throughput and latency at each level. • Long output generation: 50 concurrent requests generating 500 output tokens each, measuring sustained generation throughput. • Long context prefill: 25 concurrent requests with 4,000-token and 8,000-token input contexts, measuring prefill throughput. • Vision workloads (Qwen3-VL, Kimi-K2.5 only): Multi-image requests with 3–5 images per request, high-concurrency vision at 100 concurrent, and sustained vision load at 50 concurrent over 450 requests. • Saturation: Extreme concurrency at 150, 200, 300, 500, 750, and 1,000 concurrent requests to identify peak throughput and the saturation point. Table 10: Primary benchmark workloads by model. Vision-capable models were benchmarked with their native multimodal workload; text-only models used a text-only workload. Image tokens (processed by the vision encoder during prefill, not generated by the decoder) inflate total throughput for vision workloads, making total tok/s not directly comparable across workload types. Model Type Input Output Images Llama-3.1-405B Text 500 100 0 DeepSeek V3.2 Text 500 100 0 Qwen3-VL-235B Vision 100 200 1 Kimi-K2.5 Vision 100 200 1 Workload comparability. Each image adds approximately 1,000–1,400 tokens to the total token count (varying by model vision encoder), making throughput_total not directly comparable across workload types. All cross-architecture comparisons in Section 6 are therefore restricted to within-workload groups: text models (Llama-3.1-405B, DeepSeek V3.2) and vision models (Qwen3-VL-235B, Kimi-K2.5). 5.3.3 Architecture-Specific Configuration Each model required architecture-specific vLLM flags, summarized in Table 11. Table 11: Architecture-specific vLLM configuration flags. Model Key Flags DeepSeek V3.2 VLLM_ROCM_USE_AITER=1, AITER_ENABLE_VSKIP=0, --block-size 1, --quantization fp8 Llama-3.1-405B --quantization fp8, --max-model-len 32768 Qwen3-VL-235B VLLM_USE_TRITON_FLASH_ATTN=0, --kv-offloading-backend native, --kv-offloading-size 64 Kimi-K2.5 VLLM_ROCM_USE_AITER=0, VLLM_USE_TRITON_FLASH_ATTN=0, --block-size 1, --tensor-parallel-size 4 Notable configuration constraints include: (1) MLA-based models (DeepSeek V3.2, Kimi-K2.5) require --block-size 1 and do not support KV cache offloading or FP8 KV cache; (2) Kimi-K2.5 is deployed with TP==4 (see Section 4.4 for the rationale and limitations of this choice); and (3) Qwen3-VL does not support FP8 quantization due to a model-architecture dimension constraint (intermediate MLP dimension of 4304 not divisible by 128, the FP8 block quantization block size). Memory Measurement. All memory figures reported in this paper are per-GPU weight memory as measured from vLLM’s startup log line “Model loading took X GiB,” which reports the weight memory allocated on a single GPU after model loading but before KV cache pre-allocation. These measurements were confirmed via dedicated memory verification benchmarks (Phase 3) that captured vLLM logs and cross-referenced with rocm-smi readings for all four models. Total GPU VRAM usage (visible via rocm-smi) is substantially higher because vLLM pre-allocates nearly all remaining VRAM for KV cache and CUDA graph capture buffers. For MoE models, reported parameter counts (e.g., 685B, 235B, 1T) represent architectural totals that sum all expert parameters independently. The actual stored parameters (and thus the GPU memory footprint) are smaller because MoE architectures share routing, embedding, and attention layers across experts. 6 Results and Analysis We present comprehensive benchmark results for four large language models deployed on 8× AMD Instinct MI325X GPUs using vLLM. All stress test results use the thorough (3×3×) multiplier unless otherwise noted. Across all models, we processed over 18.9 million tokens across 17,406 requests with a 100% HTTP-level success rate. 6.1 Per-Model Performance Tables 12 and 13 summarize peak performance metrics for each model, separated by workload type. Table 12: Peak performance summary: text-only workload (500-token input, 100-token output) on 8× MI325X. Peak throughput measured at the concurrency level that maximizes total tok/s during saturation testing. Saturation onset confirmed by fine-grained sweeps (500–1,000 in steps of 50, 3 runs per level). Model Peak tok/s Output tok/s Peak Conc. Sat. Point Success Llama-3.1-405B 15,944 3,673 500 500 100% DeepSeek V3.2 15,343 1,239 500 500 100% Table 13: Peak performance summary: vision workload (100-token text + 1 image, 200-token output) on 8× MI325X. Total tok/s includes ∼ 1,000–1,400 image tokens per request and is not directly comparable to text-only results. Peak throughput measured at the concurrency level that maximizes total tok/s during saturation testing. Saturation onset confirmed by fine-grained sweeps (500–1,000 in steps of 50, 3 runs per level). Model Peak tok/s Output tok/s Peak Conc. Sat. Point Success Qwen3-VL-235B 47,873 7,140 500 500 100% Kimi-K2.5 7,327 867 500 500 100% 6.1.1 Qwen3-VL-235B-A22B (MoE + GQA) Qwen3-VL achieved the highest total throughput among the vision models tested, reaching 47,873 tok/s at 500 concurrent requests. Approximately 77% of these tokens are image tokens processed by the vision encoder rather than output tokens generated by the LLM decoder (output throughput: 7,140 tok/s). This result is striking given that Qwen3-VL has the smallest total parameter count (235B) among the models tested. A key contributing factor is its MoE architecture with only 22B active parameters per token, combined with Grouped-Query Attention (GQA), which enables exceptionally efficient batching. Table 14 shows the concurrency scaling profile. Table 14: Qwen3-VL-235B concurrency scaling results (stress test, 3×3× multiplier). Conc. Total tok/s Output tok/s p99 Latency Status 10 3,586 535 3.78s DEGRADED 25 7,728 1,153 4.36s DEGRADED 50 13,489 2,012 5.09s DEGRADED 100 21,567 3,217 6.33s DEGRADED 200 26,674 3,978 8.92s DEGRADED The model exhibited strong sublinear scaling up to 200 concurrent requests, with throughput increasing 13.6× from 5 to 200 concurrent (a 40× concurrency increase). Stress testing revealed strong prefill performance: 14,193 tok/s for 4K-token contexts and efficient multi-image processing (3–5 images per request handled without failures). Long output generation reached 1,959 output tok/s. 6.1.2 Llama-3.1-405B-Instruct (Dense + GQA) As the only dense model in our evaluation, Llama-3.1-405B provides a reference point for understanding MoE benefits. It achieved a peak throughput of 15,944 tok/s at 500 concurrent requests, comparable to DeepSeek V3.2 (15,343 tok/s) despite having an order of magnitude more active parameters (405B vs. 37B), consistent with MoE sparsity’s potential to match dense model throughput with far fewer active parameters. Table 15 shows the scaling profile. Table 15: Llama-3.1-405B concurrency scaling results (stress test, 3×3× multiplier). Conc. Total tok/s Output tok/s p99 Latency Status 5 423 153 6.15s DEGRADED 10 851 322 6.23s DEGRADED 25 1,953 738 6.80s DEGRADED 50 3,428 1,296 7.75s DEGRADED 100 5,254 1,986 10.08s DEGRADED 150 6,927 2,619 11.45s DEGRADED Llama-3.1-405B demonstrated the most predictable scaling behavior, with p99 latency increasing only 2.4× (from 6.15s to 14.49s under the scaling workload of 500-token input, 200-token output) across the full concurrency range. Stress tests showed efficient long-context prefill at 8,240 tok/s for 4K contexts and 6,794 tok/s for 8K contexts. Long output generation achieved 1,224 output tok/s. 6.1.3 DeepSeek V3.2 (MoE + MLA) DeepSeek V3.2 reached a peak throughput of 15,343 tok/s at 500 concurrent requests, comparable to Llama-3.1-405B despite having 685B total parameters (37B active). The MLA attention mechanism enables competitive throughput at scale but exhibits earlier throughput saturation during the scaling phase relative to GQA-based models. Table 16 presents the scaling results. Table 16: DeepSeek V3.2 concurrency scaling results (stress test, 3×3× multiplier). Conc. Total tok/s Output tok/s p99 Latency Status 5 461 106 8.19s OK 10 1,235 162 8.12s OK 25 3,013 477 8.72s DEGRADED 50 4,607 802 11.34s DEGRADED 100 7,160 981 13.75s DEGRADED 200 7,266 1,045 13.95s DEGRADED Scaling throughput peaked at 7,266 tok/s at 200 concurrent during the scaling phase, with the saturation-phase tests pushing this to 15,343 tok/s at 500 concurrent. Stress tests showed 4,274 tok/s for 4K-context workloads and 3,372 tok/s for 8K-context workloads, with long output generation at 230 output tok/s. The p99 latency increased moderately from 8s to 14s across the scaling range. 6.1.4 Kimi-K2.5 (MoE + MLA, 1T Parameters) Kimi-K2.5 is the largest model in our evaluation at 1 trillion total parameters with 32B active per token. It achieved a peak throughput of 7,327 tok/s at 500 concurrent requests. As the only model requiring TP==4 (due to MLA attention head constraints) and running without AITER acceleration, its throughput is lower than models with full kernel optimization support. Table 2 presents the concurrency scaling profile. Table 17: Kimi-K2.5 concurrency scaling results (stress test, 3×3× multiplier).222The concurrency 1 entry (587 tok/s) uses the vision-enabled baseline workload. The text-only baseline throughput is 151 tok/s. The status reversal from DEGRADED (concurrency 10–200) to OK (concurrency 500) reflects different classification criteria: the scaling phase labels requests DEGRADED when p99 latency exceeds twice the single-request baseline, while the saturation phase uses throughput-relative classification only (see Tables 4 and 23). Conc. Total tok/s Output tok/s p99 Latency Status 1 587 37 2.73s OK 10 670 79 25.40s DEGRADED 25 896 106 47.56s DEGRADED 50 1,632 193 52.02s DEGRADED 100 2,656 314 63.86s DEGRADED 200 3,754 444 74.89s DEGRADED 500 7,327 867 103.34s OK The model exhibited excellent linear scaling from 5 to 200 concurrent requests, with consistent throughput from 500 to 1,000 concurrent (∼ 7,300 tok/s). Vision workloads performed well: multi-image processing (3 images) reached 1,475 tok/s with 77.2s p99 latency, while sustained vision load (50 concurrent, 450 requests) maintained 836 tok/s over 17 minutes. Critically, the system maintained a 100% success rate at all concurrency levels including 1,000 concurrent requests. 6.2 Architecture Comparison Our evaluation spans three architectural families, enabling direct comparison of their inference characteristics. Tables 18 and 19 summarize the key differences, separated by workload type. Table 18: Architecture comparison: text-only workload. Throughput ratios are relative to Llama-3.1-405B (Dense+GQA) as the baseline. Architecture Model Active Params Peak tok/s Ratio Dense + GQA Llama-3.1-405B 405B 15,944 1.00× MoE + MLA DeepSeek V3.2 37B 15,343 0.96× Table 19: Architecture comparison: vision workload. Throughput ratios are relative to Kimi-K2.5 (MoE+MLA) as the baseline. Architecture Model Active Params Peak tok/s Ratio MoE + GQA Qwen3-VL-235B 22B 47,873 6.53× MoE + MLA Kimi-K2.5 32B 7,327 1.00× Text workload: Dense+GQA vs. MoE+MLA. Llama-3.1-405B (Dense+GQA, 405B active) and DeepSeek V3.2 (MoE+MLA, 37B active) achieve nearly identical total throughput on the text workload (15,944 vs. 15,343 tok/s), despite DeepSeek having only 9% of Llama’s active parameters. This parity suggests that DeepSeek’s MoE sparsity advantage is offset by MLA’s constraints on the current ROCm stack, including the block-size-1 requirement and inability to use KV cache offloading. However, Llama achieves substantially higher output throughput (3,673 vs. 1,239 tok/s), indicating differences in decoder generation speed. Vision workload: MoE+GQA vs. MoE+MLA. Among the vision models, Qwen3-VL-235B (MoE+GQA, 22B active) achieves 6.5× the total throughput of Kimi-K2.5 (MoE+MLA, 32B active). Even in output throughput (7,140 vs. 867 tok/s, an 8.2× ratio), Qwen3-VL substantially outperforms Kimi-K2.5. This gap reflects several compounding factors: Qwen3-VL’s fewer active parameters (22B vs. 32B), its GQA attention enabling KV cache offloading and standard block sizes, AITER-enabled acceleration (vs. disabled for Kimi), and higher tensor parallelism degree (TP==8 vs. TP==4). Active Parameters and Throughput. A key finding is that active parameters per token are more consistently associated with throughput than total parameters across the models tested. Tables 20 and 21 quantify this relationship. Table 20: Throughput normalized by active parameter count: text-only workload. Higher values indicate more efficient utilization of active compute. Model Active (B) Peak tok/s tok/s per B Llama-3.1-405B 405 15,944 39 DeepSeek V3.2 37 15,343 415 Table 21: Throughput normalized by active parameter count: vision workload. Higher values indicate more efficient utilization of active compute. Vision values include image tokens in the numerator and are not directly comparable to text-workload values. Model Active (B) Peak tok/s tok/s per B Qwen3-VL-235B 22 47,873 2,176 Kimi-K2.5 32 7,327 229 Within the text workload, DeepSeek V3.2 achieves 415 tok/s per billion active parameters, 10.6× higher than dense Llama-3.1-405B (39 tok/s/B), quantifying the throughput efficiency of MoE sparsity per active parameter. Within the vision workload, Qwen3-VL achieves 2,176 tok/s/B, 9.5× higher than Kimi-K2.5 (229 tok/s/B); the Kimi-K2.5 gap is partially attributable to its disabled AITER acceleration and TP==4 constraint. The substantially higher absolute tok/s/B values for vision models reflect the inclusion of image tokens in the numerator and are not directly comparable to the text-workload values. Llama-3.1-405BDeepSeek V3.2Qwen3-VL-235BKimi-K2.5101001,000Text WorkloadVision Workload394152,176229Throughput per Billion Active Params (tok/s/B)Dense + GQAMoE + MLAMoE + GQAMoE + MLA (Vision) Figure 1: Throughput normalized by active parameter count (tok/s per billion active parameters), grouped by workload type. Left pair: text-only workload; right pair: vision workload. Values are not directly comparable across workload types because vision total tok/s includes image tokens. Y-axis uses logarithmic scale. Data from primary stress-test benchmark (3×3× multiplier). 6.3 Scaling and Saturation Analysis 02002004004006006008008001,0001,00005,0005,00010,00010,00015,00015,00020,00020,000Saturation regionConcurrent RequestsThroughput (tok/s)Llama-3.1-405BDeepSeek V3.2 (a) Text-only workload 02002004004006006008008001,0001,000020,00020,00040,00040,000Saturation regionConcurrent RequestsThroughput (tok/s)Qwen3-VL-235BKimi-K2.5 (b) Vision workload Figure 2: Throughput scaling as a function of concurrent requests, separated by workload type. Both panels show saturation at approximately 500 concurrent requests and flat throughput through 1,000. All models maintain 100% success rates. Data from primary stress-test benchmark (3×3× multiplier). Linear Scaling Region. All four models exhibit sublinear throughput scaling up to 200 concurrent requests within their respective workloads. Among text models, Llama-3.1-405B scales to 6,927 tok/s at 150 concurrent before slight degradation at 200, while DeepSeek V3.2 shows strong scaling from 461 tok/s at 5 concurrent to 7,266 tok/s at 200 concurrent. Among vision models, Qwen3-VL scales most efficiently, achieving a 13.6× throughput increase from 5 to 200 concurrent, while Kimi-K2.5 achieves a 6.4× increase from baseline to 200 concurrent (587 to 3,754 tok/s). Saturation Behavior. A notable finding is that within each workload type, all models reach peak throughput at approximately 500 concurrent requests and plateau through 1,000. A fine-grained concurrency sweep from 500 to 1,000 in steps of 50 (three independent runs per level, 200 requests each) confirms flat throughput across this entire range: Qwen3-VL varies by 0.65%, Kimi-K2.5 by 1.2%, DeepSeek V3.2 by 1.9%, and Llama-3.1-405B by 2.0% (relative range (max−min)/mean( - )/mean across per-level mean throughputs, three runs per concurrency level), with no performance cliff or non-monotonic behavior. Tables 4 and 23 detail the saturation-phase behavior. Table 22: Saturation-phase throughput (tok/s) and p99 latency at extreme concurrency levels: text-only workload.444Status labels use phase-relative criteria: the SCALING phase classifies requests as DEGRADED when p99 latency exceeds twice the single-request baseline, while the SATURATION phase uses throughput-relative classification only (SATURATED when throughput plateaus within 5% of the previous level). Consequently, status labels are not directly comparable across phases. Note that DeepSeek V3.2 at concurrency 1000 shows OK status despite lower throughput than at 500; the non-monotonic recovery (13,218 → 14,148 tok/s) exceeds the 5% growth threshold, preventing SATURATED classification. Conc. Llama-405B DeepSeek tok/s p99 tok/s p99 150 10,320 6.15s 8,355 5.77s 200 11,519 7.35s 10,864 5.73s 300 13,937 9.09s 12,719 7.35s 500 15,944 11.78s 15,343 9.08s 750 15,693 12.01s 13,218 10.84s 1000 15,319 12.28s 14,148 9.99s Table 23: Saturation-phase throughput (tok/s) and p99 latency at extreme concurrency levels: vision workload. Total tok/s includes image tokens processed by the vision encoder. Conc. Qwen3-VL Kimi-K2.5 tok/s p99 tok/s p99 150 23,557 8.50s 3,628 69.74s 200 29,557 8.96s 4,528 74.44s 300 37,850 10.50s 5,820 86.81s 500 47,873 12.37s 7,327 103.34s 750 47,092 12.53s 7,304 103.66s 1000 47,252 12.50s 7,309 103.64s Reliability Under Extreme Load. The most significant operational finding is that all four models maintained a 100% HTTP-level success rate (every request returned HTTP 200 with a valid response structure; this metric does not validate output quality or token count correctness) across all concurrency levels, including 1,000 simultaneous requests. No request failures were observed even at saturation, indicating that vLLM’s continuous batching and scheduling mechanisms gracefully handle overload by queuing excess requests rather than rejecting them. Latency Trade-offs. The latency cost of high-concurrency throughput is manageable. Llama-3.1-405B showed the most predictable latency growth: p99 increased from 6.15s at 5 concurrent to 14.49s at 200 concurrent (2.4×) under the scaling workload (500-token input, 200-token output; note that the saturation-phase workload in Table 4 uses a shorter 100-token output). Qwen3-VL p99 latency increased only 2.6× from 5 to 200 concurrent despite a 13.6× throughput gain, representing the best throughput-per-latency-cost ratio among all models. DeepSeek V3.2 p99 latency increased moderately from 8s to 14s across the scaling range. 01001002002003003004004005005006006007007008008009009001,0001,00002020404060608080100100Concurrent Requestsp99 Latency (seconds)Qwen3-VL-235BLlama-3.1-405BDeepSeek V3.2Kimi-K2.505005001,0001,00005510101515Concurrent Requestsp99 Latency (s)Detail: 0–15s range Figure 3: p99 latency as a function of concurrent requests. Text models (Llama-3.1-405B, DeepSeek V3.2) use a 500-token input / 100-token output workload; vision models (Qwen3-VL, Kimi-K2.5) use a 100-token input + 1 image / 200-token output workload. Latency values are not directly comparable across workload types. All models show sublinear latency growth: throughput increases faster than latency, yielding positive scaling efficiency at all tested concurrency levels. Inset shows the 0–15s range for Qwen3-VL, Llama-3.1-405B, and DeepSeek V3.2 (Kimi-K2.5 latencies of 25–103s compress these curves in the main plot). Data from primary stress-test benchmark (3×3× multiplier). 6.4 Measurement Reproducibility To characterize measurement variance, we conducted multiple independent benchmark runs per model using a standardized workload (100 requests, 2,048 input tokens, 512 output tokens). Each run used a fresh vLLM server restart to ensure independence. Five runs were completed for all four models. Confidence intervals use the t-distribution at the 95% level. Table 24: Multi-run reproducibility statistics (peak throughput at the concurrency level maximizing mean tok/s). CoV is the coefficient of variation (stdev/mean). This workload differs from the primary benchmarks (Tables 12 and 13), which use 17,406 requests with a 3×3× multiplier; values are not directly comparable. Confidence intervals use the t-distribution at 95%. Model n Peak Conc. Mean tok/s CI95 CoV Qwen3-VL-235B 5 1000 11,218 ± 32 0.2% Llama-3.1-405B 5 750 6,808 ± 336 4.0% DeepSeek V3.2 5 1000 5,786 ± 842 11.7% Kimi-K2.5 5 1000 952 ± 4 0.4% 01001002002003003004004005005006006007007008008009009001,0001,00002,0002,0004,0004,0006,0006,0008,0008,00010,00010,00012,00012,000Concurrent RequestsThroughput (tok/s)Qwen3-VL-235BLlama-3.1-405BDeepSeek V3.2Kimi-K2.5 Figure 4: Throughput vs. concurrency with 95% confidence interval error bars from n=5n=5 independent runs per model. DeepSeek V3.2 exhibits the widest error bars (CoV up to 11.7% at peak concurrency, reaching 50.8% at concurrency 10), while Qwen3-VL and Kimi-K2.5 show near-deterministic behavior. Data from multi-run reproducibility workload (100 requests, 2,048 input / 512 output tokens); not directly comparable to the primary stress-test benchmark. Table 24 reveals a substantial spread in measurement stability. Qwen3-VL exhibits near-deterministic behavior (CoV << 0.3% across all concurrency levels), indicating that throughput measurements for GQA-based MoE models are highly reproducible with minimal repetition. Llama-3.1-405B shows moderate variance (CoV ≈ 4%), consistent with the larger memory footprint and denser compute patterns of a 405B-parameter model. DeepSeek V3.2 exhibits the highest variance (CoV up to 11.7% at peak concurrency, and reaching 50.8% at concurrency 10 where individual run throughputs ranged from 1,485 to 5,193 tok/s), possibly attributable to MLA’s latent-space projections and MoE routing stochasticity interacting with AITER kernel scheduling, though the specific mechanism has not been profiled. Kimi-K2.5 shows very low variance (CoV ≈ 0.4%), comparable to Qwen3-VL despite also being an MLA-based MoE model; its stability may reflect the absence of AITER kernel scheduling non-determinism (AITER is disabled for Kimi on MI325X). These results suggest that single-run benchmarks are adequate for GQA models but that MLA-based models with AITER enabled benefit from multi-run averaging to obtain stable throughput estimates. 6.5 Optimization Impact Several vLLM and hardware-specific optimizations significantly influenced performance. We analyze their contributions below. 6.5.1 AITER Kernel Acceleration The AMD AI Tensor Engine for ROCm (AITER) library provides optimized kernels for MoE and attention operations on CDNA 3 architecture. Table 25 summarizes AITER status across models. Table 25: AITER acceleration status across models. DeepSeek V3.2 requires AITER for MLA attention on ROCm. Kimi-K2.5 cannot use AITER due to MLA head count incompatibility with MI325X. Model Workload AITER Peak tok/s Notes Llama-3.1-405B Text Enabled 15,944 MHA kernels (dense model) DeepSeek V3.2 Text Enabled 15,343 MLA + AITER kernels Qwen3-VL-235B Vision Enabled 47,873 MHA + MoE kernels Kimi-K2.5 Vision Disabled 7,327 MLA head incompatibility DeepSeek V3.2 with AITER enabled achieves higher throughput than Kimi-K2.5 without AITER; however, these models used different workloads (text vs. vision), making direct throughput comparison invalid. The throughput difference also reflects confounding factors including total parameter count (685B vs. 1T), quantization format (FP8 vs. INT4), tensor parallelism degree (TP==8 vs. TP==4), and GPU count. MLA Ablation Infeasibility. We attempted a controlled ablation study comparing AITER-enabled and AITER-disabled inference for MLA models on MI325X. While a Triton MLA fallback exists (VLLM_ROCM_USE_AITER=0), it delivers substantially lower performance than the AITER MLA backend, making it unsuitable as a controlled comparison (the performance gap conflates AITER’s contribution with the fallback path’s inherent limitations). Consequently, AITER’s isolated contribution to MLA inference throughput cannot be cleanly measured on the current ROCm software stack. GQA Model Ablation. Because MLA ablation is infeasible, we performed a controlled A/B comparison on Llama-3.1-405B (GQA attention, where AITER can be toggled). Both conditions use n=5n=5 independent server restarts with an identical workload (100 requests, 2,048 input / 512 output tokens) at six concurrency levels. Table 26 presents the comparison. Table 26: AITER ablation on Llama-3.1-405B (GQA). Both conditions use n=5n=5 independent server restarts with identical workload. AITER provides a modest throughput benefit at single-request and high concurrency (++3–10%), no benefit at mid-concurrency, and consistently higher variance. Conc. Enabled tok/s Disabled tok/s Diff. En. CoV Dis. CoV 1 150 ± 9 137 ± 1 ++10.0% 4.69% 0.38% 10 1,084 ± 23 1,092 ± 3 −-0.8% 1.73% 0.23% 50 4,340 ± 50 4,380 ± 12 −-0.9% 0.93% 0.22% 100 6,955 ± 136 6,682 ± 8 ++4.1% 1.57% 0.10% 200 6,871 ± 230 6,663 ± 17 ++3.1% 2.69% 0.20% 500 6,972 ± 137 6,676 ± 15 ++4.4% 1.58% 0.18% 02,0002,0004,0004,0006,0006,0008,0008,000Throughput (tok/s)AITER Ablation: Llama-3.1-405B (n=5n=5 runs per condition)AITER EnabledAITER Disabled1510501002005000551010+10.0%−-1.1%−-0.8%−-0.9%+4.1%+3.1%+4.4%Concurrency LevelAITER Speedup (%) Figure 5: AITER ablation on Llama-3.1-405B. Top: grouped bar chart comparing throughput with AITER enabled vs. disabled (n=5n=5 runs, error bars show 95% CI). Bottom: percentage speedup from AITER. AITER provides 10% benefit at concurrency 1, negligible effect at mid-concurrency, and 3–5% at high concurrency. Data from multi-run reproducibility workload (100 requests, 2,048 input / 512 output tokens). The ablation reveals a concurrency-dependent pattern. At single-request latency (concurrency 1), AITER provides a 10% throughput improvement, likely from optimized single-stream kernel dispatch. At mid-concurrency (5–50), there is no meaningful difference (<<1%). At high concurrency (100–500), AITER provides a consistent 3–5% throughput benefit. Across all concurrency levels, disabling AITER dramatically reduces measurement variability: CoV drops from 0.6–4.7% (enabled) to 0.1–0.4% (disabled), a 2–16× reduction. This confirms that AITER kernel scheduling introduces non-determinism on CDNA 3 even for GQA models, consistent with the elevated variance observed for AITER-enabled DeepSeek V3.2 (CoV 11.7%, Table 24). Implications. These results indicate that AITER’s documented 2–3× speedups [2] are specific to MoE expert routing and MLA attention kernels; for standard GQA attention, the benefit is modest (3–10% depending on concurrency). For GQA models, AITER’s most pronounced effect is increased measurement variability. AITER’s critical role for MLA models is as a performance enabler, without which throughput drops to impractical levels via the Triton MLA fallback, making it a practical necessity for production deployments rather than merely an optional optimization. The AITER VSKIP optimization must be disabled on MI300X/MI325X (AITER_ENABLE_VSKIP=0) to prevent HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION errors in fused MoE kernels. 6.5.2 Quantization and Memory Savings FP8 quantization provides approximately 50% memory savings compared to FP16/BF16, enabling larger batch sizes and KV cache capacity. Table 27 compares memory usage across precision formats. Table 27: Per-GPU weight memory by precision format and load times on MI325X. Values from vLLM startup log. FP16/BF16 rows are theoretical estimates. Model Precision Per-GPU Memory Load Time Notes DeepSeek V3.2 FP8 ∼ 83 GiB ∼ 71s FP8 warmup +3 min DeepSeek V3.2 FP16 ∼ 180 GiB – 2.2× memory (est.) Llama-3.1-405B FP8 ∼ 112 GiB ∼ 344s Dense model Llama-3.1-405B BF16 ∼ 224 GiB – 2× FP8 measured (est.) Qwen3-VL-235B BF16 ∼ 58 GiB ∼ 112s FP8 incompatible (ViT) Kimi-K2.5 INT4 QAT ∼ 145 GiB ∼ 146s Compressed-tensors FP8 quantization is particularly impactful for dense models: Llama-3.1-405B at BF16 would require an estimated ∼ 224 GiB per GPU (extrapolating from the measured FP8 footprint of ∼ 112 GiB), leaving minimal headroom within each GPU’s 256 GB capacity. FP8 halves the weight footprint, making single-node deployment feasible with substantial KV cache headroom. For MoE models, the savings are proportionally smaller because expert parameters are sparse, but FP8 still reduces DeepSeek V3.2 from an estimated ∼ 180 GiB to ∼ 83 GiB per GPU. The MI325X’s 256 GB HBM3e per GPU provides substantial headroom. DeepSeek V3.2 in FP8 requires only ∼ 83 GiB of per-GPU weight memory, consuming approximately 35% of each GPU’s capacity (noting that 83 GiB ≈ 89 GB) and leaving the remainder available for KV cache and batch buffers. Memory Footprint Derivation. Table 28 presents a derivation of expected per-GPU weight memory from model parameters, quantization format, and tensor parallelism degree, compared against measured values from vLLM’s startup log (“Model loading took X GiB”).555All “Measured” values are per-GPU weight memory as reported by vLLM’s model loading log, confirmed via Phase 3 memory verification benchmarks. The “Expected” column computes (Stored Params×Bytes per Param)/TP degree(Stored Params×Bytes per Param)/TP degree, which represents the naive sharding estimate. Discrepancies arise from embedding layer duplication across TP ranks, FP8 scale metadata, and communication buffer overhead. Table 28: Per-GPU weight memory derivation. “Stored Params” is the total count in GPU memory; for MoE models this equals all expert weights. Expected and Measured are per-GPU values (total divided by TP). Model Total Active Stored Precision TP Expected Measured Params Params Params (GiB) (GiB) DeepSeek V3.2 685B 37B 685B FP8 8 ∼ 87 ∼ 83 Llama-3.1-405B 405B 405B 405B FP8 8 ∼ 52 ∼ 112a Qwen3-VL-235B 235B 22B 235B BF16 8 ∼ 59 ∼ 58 Kimi-K2.5 1T 32B 1T INT4 QAT 4 ∼ 140 ∼ 145 aThe 2.2× gap between expected (∼ 52 GiB) and measured (∼ 112 GiB) per GPU for Llama-3.1-405B is due to embedding layer duplication across TP ranks, FP8 scale factor metadata, and RCCL communication buffers. The expected per-GPU memory footprint equals (Stored Params×Bytes/Param)/TP(Stored~Params×Bytes/Param)/TP, where MoE models store all expert parameters even though only a subset are active per token. Effective bytes per parameter vary by format: FP8 requires ∼1.02 1.02 bytes (1 byte per weight plus scale factor overhead), BF16 requires 2.0 bytes, and INT4 QAT requires ∼0.56 0.56 bytes (0.5 bytes per weight plus FP16 scales per quantization group). Embedding layers and layer norms remain in BF16 regardless of quantization format but constitute <<1% of total parameters. In practice, measured per-GPU memory exceeds the naive estimate for dense models (Llama-3.1-405B) due to embedding table duplication across TP ranks and FP8 metadata; MoE models show closer agreement because expert weights dominate and shard evenly. 6.5.3 KV Cache Management KV cache management strategies differ by attention mechanism: • GQA models (Qwen3-VL, Llama-3.1-405B): Support KV cache offloading to system RAM, enabling larger effective batch sizes. Qwen3-VL uses --kv-offloading-backend native with 64 GB offloading, which contributes to its exceptional throughput. Llama-3.1-405B supports FP8 KV cache (--kv-cache-dtype fp8), though this was not enabled in our benchmarks. • MLA models (DeepSeek V3.2, Kimi-K2.5): Do not support KV cache offloading on the current ROCm stack (an MLA-specific limitation in vLLM v0.14.1; vLLM’s planned offloading redesign, RFC #22605, would also benefit MLA models). However, MLA inherently uses less KV cache memory per token, and the MI325X’s large HBM capacity compensates. Llama-3.1-405B [-1pt]405B active DeepSeek V3.2 [-1pt]37B active 05,0005,00010,00010,00015,00015,00020,00020,00015,94415,343Peak Throughput (tok/s)Dense + GQAMoE + MLA (a) Text-only workload Qwen3-VL-235B [-1pt]22B active Kimi-K2.5 [-1pt]32B active 020,00020,00040,00040,00047,8737,327Peak Throughput (tok/s)MoE + GQAMoE + MLA (b) Vision workload Figure 6: Peak throughput comparison separated by workload type. (a) Text-only workload (500-token input, 100-token output). (b) Vision workload (100-token text + 1 image, 200-token output). Total tok/s for vision models includes image tokens processed by the vision encoder and is not directly comparable to text models. Data from primary stress-test benchmark (3×3× multiplier). Key Insight. Within each workload type, active parameter count per token is consistently associated with inference throughput. In the text workload, DeepSeek V3.2 (37B active) matches Llama-3.1-405B (405B active) in total throughput despite using only 9% of the active parameters. In the vision workload, Qwen3-VL (22B active) achieves 6.5× the throughput of Kimi-K2.5 (32B active). These patterns are consistent with LLM inference on MI325X being fundamentally memory-bandwidth-bound: fewer active parameters means less data movement per token. However, these comparisons are confounded by differences in quantization format, AITER acceleration status, and tensor parallelism degree, so the active-parameter relationship cannot be fully isolated. 7 Discussion Our benchmark results reveal several findings with implications for production LLM deployment on AMD hardware and for the broader understanding of inference serving across diverse model architectures. 7.1 Architecture-Aware Optimization: One Size Does Not Fit All A central finding is that serving configuration must be tailored to model architecture; a single default configuration cannot achieve optimal (or even correct) results across the architectures we evaluate. The divergence between MLA and GQA models is particularly stark: • MLA constraints. Both MLA models (DeepSeek V3.2 and Kimi-K2.5) require --block-size 1 on the ROCm/AITER stack for their compressed latent KV cache (other platforms use larger block sizes), are incompatible with KV cache offloading on the current ROCm stack (vLLM’s offloading redesign, RFC #22605, would also benefit MLA models), and impose specific attention head distribution requirements for tensor parallelism. Kimi-K2.5 is further constrained to TP==4 in our evaluation (see Section 4.4 for the rationale), forgoing half the cluster’s bandwidth and memory. • GQA flexibility. GQA models (Llama-3.1-405B and Qwen3-VL-235B) benefit from KV cache offloading, support FP8 KV cache quantization (where vision encoder constraints permit), and operate with standard block sizes. This flexibility contributes to Qwen3-VL’s 47,873 tok/s peak under its vision workload. Among vision models, Qwen3-VL achieves 6.5× the throughput of Kimi-K2.5, while among text models, Llama-3.1-405B (also GQA) matches DeepSeek V3.2 (MLA) at comparable throughput despite 10× more active parameters. • AITER selectivity. AITER is required for competitive production MLA inference throughput on ROCm; a Triton MLA fallback exists but delivers substantially lower performance, making controlled ablation impractical (the performance gap conflates AITER’s contribution with the fallback path’s limitations). Our controlled ablation on Llama-3.1-405B (GQA, n=5n=5 per condition) demonstrates that AITER provides a modest throughput benefit for standard attention models: 3–5% at high concurrency and ∼ 10% in total throughput at single-request latency (though output throughput, i.e. pure generation speed, shows no meaningful difference, indicating the gain is confined to prefill acceleration), but no benefit at mid-concurrency (Table 26), while increasing measurement variability by 2–16× (coefficient of variation) (CoV 0.6–4.7% enabled vs. 0.1–0.4% disabled). This confirms that AMD’s documented 2–3× speedups [2] are specific to MoE and MLA kernels; GQA models see only single-digit percentage gains. AITER must be entirely disabled for Kimi-K2.5 due to MXFP4 hardware requirements (CDNA 4 only) and head count incompatibilities. Even for AITER-compatible deployments, VSKIP must be disabled (a known issue documented in AITER Issue #1143) to prevent HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION errors in fused MoE kernels on CDNA 3. These findings challenge the implicit assumption in many deployment guides that a single set of vLLM flags suffices for all models. Production deployments should implement architecture-detection logic that selects block size, AITER configuration, KV cache strategy, and tensor parallelism degree based on the model’s attention mechanism and expert structure. 7.2 AMD MI325X Platform Viability The MI325X cluster demonstrates strong viability as an inference platform for frontier-scale models. Several hardware characteristics prove particularly advantageous: Memory capacity eliminates offloading necessity. The 256 GB HBM3e per GPU (2 TB aggregate) is sufficient to host all models tested, including the 1T-parameter Kimi-K2.5, with substantial headroom for KV cache. DeepSeek V3.2 in FP8 requires only ∼ 83 GiB of per-GPU weight memory (as reported by vLLM’s model loading log), consuming approximately 35% of each GPU’s 256 GB capacity (noting that 83 GiB ≈ 89 GB) and leaving the remainder for KV cache and batch state. This eliminates the need for CPU memory offloading in most practical scenarios, simplifying deployment architecture and removing offloading-induced latency. Aggregate bandwidth supports high throughput. The 48 TB/s aggregate memory bandwidth (6.0 TB/s per GPU) enables the high token throughput observed across all models. Qwen3-VL’s 47,873 tok/s peak (vision workload, including ∼ 1,000 image tokens per request) demonstrates that the platform can sustain high throughput for frontier-scale multimodal models. For text-only workloads, the platform achieves up to 15,944 tok/s (Llama-3.1-405B). Trillion-parameter model support. To our knowledge, this study represents the first published inference benchmark of a trillion-parameter model on MI325X (CDNA 3). AMD has published Kimi-K2 benchmarks on MI355X (CDNA 4), and Oak Ridge National Lab trained a trillion-parameter model on MI250X GPUs [7]. Kimi-K2.5’s [20] 1T parameters fit within four MI325X GPUs using INT4 QAT quantization, achieving 7,327 tok/s at 500 concurrent requests with 100% reliability through 1,000 concurrent. This demonstrates that the MI325X platform can serve the largest publicly available models without multi-node communication overhead. Power and thermal behavior under load. GPU-level monitoring during the Kimi-K2.5 benchmark runs (3 runs, 1-second sampling via rocm-smi) provides insight into power and thermal characteristics. Table 29 summarizes the per-GPU power and thermal measurements. Table 29: Power and thermal characteristics during Kimi-K2.5 inference (TP==4, 3 runs, 1-second sampling via rocm-smi). Active GPUs are the 4 GPUs participating in tensor parallelism; idle GPUs are the remaining 4 GPUs on the node. Metric Active GPUs (4) Idle GPUs (4) Mean power (W/GPU) 662 118 Min power (W/GPU) 129 — Max power (W/GPU) 798 — % of TDP (1,000 W) ∼ 66% ∼ 12% Mean junction temp (°C) 64 — Peak junction temp (°C) 76 — Mean HBM3e temp (°C) 56 — Peak HBM3e temp (°C) 66 — Hardware utilization ∼ 99.5% — FLOPs utilization 1.1% — The four idle GPUs consumed a combined 472 W, representing ∼ 15% of total system GPU power (3.1 kW) for zero productive work. This quantifies the practical cost of MLA’s tensor parallelism constraint: the TP==4 requirement leaves half the node’s GPUs idle and their associated power budget unutilized. Both junction and HBM3e temperatures remain well within the MI325X’s thermal envelope, indicating substantial thermal headroom under production workloads. Active GPUs reported ∼ 99.5% hardware utilization despite achieving only 1.1% FLOPs utilization (Table 30), reinforcing that the GPUs are fully occupied servicing memory transfers rather than compute operations, a direct consequence of the memory-bandwidth-bound nature of autoregressive inference discussed in Section 6. 7.3 Memory Bandwidth as the Primary Bottleneck Under the primary stress test workloads (500-token input, 100-token output for text models; 100-token input + 1 image, 200-token output for vision models), all four models exhibit throughput saturation at approximately 500 concurrent requests on our 8-GPU MI325X cluster, despite spanning a 4× range in total parameters and employing three different architectural paradigms. However, this saturation point is workload-dependent: under the multi-run workload (2,048-token input, 512-token output), saturation occurs earlier, at approximately 100–200 concurrent requests, as longer sequences consume more memory bandwidth per request. The consistent saturation across architecturally diverse models within a given workload, combined with the sub-15% FLOPs utilization (Table 30), strongly suggests that the bottleneck at high concurrency is memory bandwidth rather than compute capacity. The MI325X cluster provides 10.5 PFLOPS of FP16 compute but only 48 TB/s of aggregate memory bandwidth. Recasens et al. [23] have shown that DRAM bandwidth saturation dynamics drive model-dependent saturation on other hardware platforms, and the LIMINAL analytical model [8] identifies memory bandwidth as the primary barrier for LLM decode performance across GPU architectures. Table 30 quantifies the compute utilization for each model at peak output throughput. Table 30: FLOPs utilization at peak output throughput. Each output token requires approximately 2×Pactive2× P_active FLOPs. MI325X peak compute: 2,615 TFLOPS FP8 and 1,307 TFLOPS BF16 per GPU. Model Active Precision GPUs Output tok/s Utilization Qwen3-VL-235B 22B BF16 8 7,140 3.0% Llama-3.1-405B 405B FP8 8 3,673 14.2% DeepSeek V3.2 37B FP8 8 1,239 0.4% Kimi-K2.5 32B INT4 QAT† 4 867 1.1% † 4 QAT weights are dequantized to BF16 for compute; utilization computed against BF16 peak (1,307 TFLOPS/GPU). Even the most compute-intensive model (Llama-3.1-405B) utilizes only 14% of peak FP8 compute, while MoE models operate at 0.4–3% utilization, confirming that autoregressive decoding is overwhelmingly memory-bandwidth-bound [21]. This has several implications. First, for a given workload, the common saturation point means that adding more concurrent requests beyond the saturation threshold yields no throughput benefit. For our stress test workload, this threshold is ∼ 500; for longer-sequence workloads, saturation occurs earlier. Second, optimizations that reduce memory traffic, such as MoE sparsity (fewer active parameters per token), quantization (fewer bytes per parameter), and MLA’s compressed KV cache (smaller memory reads per attention operation), are more impactful than optimizations that increase raw compute throughput. Third, the 100% success rate at saturation (no request failures at 1,000 concurrent) indicates that vLLM’s scheduling and queuing mechanisms [16] gracefully handle overload conditions, converting excess concurrency into increased latency rather than failures. 7.4 Transferability to AMD Instinct MI300X Our findings transfer directly to the AMD Instinct MI300X because both accelerators share the CDNA 3 microarchitecture (gfx942 ISA) and identical compute unit count (304 CUs). All software-level configurations (AITER kernel flags, block size constraints, tensor parallelism requirements) apply without modification. The primary hardware differences are memory capacity (192 GB HBM3 on MI300X vs. 256 GB HBM3e on MI325X) and memory bandwidth (5.3 TB/s vs. 6.0 TB/s). All four models fit on an 8-GPU MI300X node. However, the MI300X’s ∼ 12% lower bandwidth should translate to a roughly proportional reduction in peak throughput, and reduced per-GPU capacity leaves less headroom for KV cache at high concurrency. 7.5 Limitations This study has several limitations that should be considered when interpreting the results: Single cluster configuration. All experiments were conducted on a single 8-GPU MI325X node. Multi-node scaling behavior, which introduces inter-node communication overhead and pipeline parallelism considerations, remains unexplored. The TP==4 constraint for Kimi-K2.5 further limits our evaluation to half the cluster for that model. Model selection. We evaluate four models across three architectural families, but many important architectures remain untested, including standard transformer models smaller than 100B parameters, state-space models (Mamba [14]), and hybrid architectures. Our findings about architecture-specific optimization may not generalize to all model families. Single-platform evaluation. This study focuses exclusively on the AMD MI325X platform. Cross-platform comparisons require identical workloads, software versions, and measurement methodology, and are beyond the scope of this work. Single vLLM version. All experiments use vLLM v0.14.1 (with nightly builds for Kimi-K2.5). The vLLM project evolves rapidly, and performance characteristics may change significantly across versions as ROCm support matures, AITER kernels are updated, and scheduling algorithms improve. Our architecture-specific configuration findings (block size, AITER flags) are version-dependent and should be re-validated on future releases. Evaluation scope. This study evaluates serving throughput and tail latency under concurrent load. Time to First Token (TTFT) and output quality metrics are orthogonal concerns studied elsewhere [10, 20] and are outside the scope of this work. Software maturity considerations. The ROCm and AITER ecosystem is production-capable and continues to mature, with active development in areas that will further streamline deployment. Remaining considerations include the VSKIP workaround requiring explicit disabling, vision encoder FP8 incompatibilities due to dimension constraints, and the need for nightly builds for Kimi-K2.5 support. These are active areas of development expected to be addressed in subsequent ROCm and AITER releases. 7.6 Practical Deployment Considerations Our experience deploying and benchmarking these models yields several practical recommendations for production MI325X deployments: 1. Architecture detection before configuration. Determine the model’s attention mechanism (MLA vs. GQA), sparsity (dense vs. MoE), and expert count before selecting vLLM flags. On the current ROCm stack, MLA models require --block-size 1 and cannot use --kv-offloading-backend native. 2. AITER validation per model. Enable AITER globally (VLLM_ROCM_USE_AITER=1) but validate correctness for each model. Set AITER_ENABLE_VSKIP=0 for all MLA deployments. Disable AITER entirely for models with incompatible attention head counts. 3. Quantization format compatibility. Verify FP8 kernel compatibility before assuming FP8 quantization is available. Vision encoder dimension constraints and MLA-specific KV cache formats may force alternative precision formats. 4. Concurrency planning. For short-sequence workloads (∼ 500-token input), target approximately 500 concurrent requests for maximum throughput on the 8-GPU MI325X cluster; saturation occurs earlier (∼ 100–200 concurrent) for longer sequences (2,048+ tokens). Additional concurrency beyond the saturation point adds latency without throughput benefit. 5. Container and build management. Maintain separate container images for stable and nightly vLLM builds, as some frontier models require nightly support before stable release integration. 8 Conclusion and Future Work We have presented a systematic cross-architecture benchmark study of large language model inference on AMD Instinct MI325X GPUs, evaluating four frontier-scale models spanning 235 billion to 1 trillion parameters on an 8-GPU cluster with 2 TB aggregate HBM3e using the vLLM serving framework. 8.1 Summary of Contributions Our study makes five key contributions: 1. Cross-architecture MI325X inference benchmark at scale. We provide systematic performance characterization for four models across three architectural families (dense GQA, MoE+GQA, MoE+MLA) under workloads from single-request latency to 1,000 concurrent users, processing 18.9 million tokens across 17,406 requests with 100% HTTP-level success. 2. Architecture-aware optimization is essential. We demonstrate that MLA and GQA architectures require fundamentally different serving configurations. On the current ROCm stack, MLA models require block size 1 and cannot use KV cache offloading; GQA models benefit from offloading and standard block sizes. The AITER runtime is required for competitive production MLA inference throughput on ROCm; a Triton MLA fallback exists but delivers substantially lower performance, making AITER a practical necessity for production deployments. A controlled ablation on Llama-3.1-405B (GQA, n=5n=5 per condition) shows a modest 3–5% throughput benefit at high concurrency but 2–16× higher measurement variability (coefficient of variation) with AITER enabled (Table 26), confirming that AITER’s large speedups target MoE and MLA kernels specifically. AITER must be entirely disabled for Kimi-K2.5 due to MXFP4 hardware requirements (CDNA 4 only) and attention head count constraints. 3. Trillion-parameter model on MI325X (CDNA 3). We deploy and benchmark Kimi-K2.5 [20], a 1T-parameter MoE model, on four MI325X GPUs using INT4 QAT quantization, achieving 7,327 tok/s at 500 concurrent requests with 100% reliability through 1,000 concurrent. 4. Active parameter count is associated with throughput. Active parameter count per token is more consistently associated with inference throughput than total parameter count within each workload type, though confounded by other experimental variables (Section 6). Within the text workload, DeepSeek V3.2 (MoE+MLA, 37B active) matches Llama-3.1-405B (Dense+GQA, 405B active) at 15,343 vs. 15,944 tok/s despite having only 9% of the active parameters. Within the vision workload, Qwen3-VL-235B (MoE+GQA, 22B active) achieves 6.5× the throughput of Kimi-K2.5 (MoE+MLA, 32B active). 5. Workload-dependent throughput saturation. All four models, despite a 4× range in total parameters and three distinct architectural paradigms, exhibit a common throughput saturation point within a given workload on our 8-GPU MI325X cluster, consistent with a memory-bandwidth bottleneck. The saturation threshold is workload-dependent: ∼ 500 concurrent for the stress test workload (500-token input, 100-token output for text; 100-token input + 1 image, 200-token output for vision) and ∼ 100–200 concurrent for longer-sequence workloads (2,048-token input, 512-token output). Prior work has shown that DRAM bandwidth saturation dynamics drive model-dependent saturation on other hardware platforms [23]. 8.2 Future Work Several directions would extend and strengthen the findings presented here: Multi-node scaling. Our evaluation is limited to a single 8-GPU node. Extending to multi-node configurations would introduce inter-node communication overhead (InfiniBand or RoCE) and enable evaluation of pipeline parallelism and expert parallelism strategies for MoE models. Understanding whether the ∼ 500-concurrent saturation point shifts proportionally with additional nodes is of direct practical interest. Pipeline and expert parallelism. The current study uses tensor parallelism exclusively. Pipeline parallelism could enable higher throughput for dense models like Llama-3.1-405B by overlapping computation across pipeline stages, while expert parallelism could improve MoE model serving by distributing experts across GPUs independently of the attention parallelism degree. Broader model coverage. Extending the evaluation to additional architectures, including state-space models (e.g., Mamba [14]), hybrid attention-SSM architectures, and smaller models in the 7B–70B range, would test the generality of our architecture-aware optimization findings and active-parameter throughput relationship. Speculative decoding. Speculative decoding [17] is orthogonal to our evaluated optimizations and could further improve single-request latency and throughput, particularly for MLA models where per-token generation speed is lower. Evaluating draft-model selection and acceptance rates across our four architectures would complement the throughput-focused results presented here. Disaggregated prefill and decode. Separating prefill and decode phases onto different GPU pools (an emerging approach in production serving systems) could improve overall throughput by dedicating compute-optimized GPUs to prefill and bandwidth-optimized GPUs to decode. The MI325X’s high memory bandwidth would make it a strong candidate for the decode phase, suggesting a heterogeneous deployment where prefill runs on compute-dense accelerators. Streaming inference and per-token latency characterization. Our benchmarks use non-streaming requests, measuring end-to-end latency and aggregate throughput but not per-token metrics such as Time to First Token (TTFT) and Inter-Token Latency (ITL). These metrics are critical for interactive applications where perceived responsiveness depends on how quickly the first token arrives and how smoothly subsequent tokens stream. Re-running the benchmark suite with streaming enabled would characterize prefill latency (TTFT) and decode cadence (ITL) across all four architectures, potentially revealing architecture-specific differences; for example, whether MLA’s compressed KV cache format affects prefill latency differently than GQA’s standard attention. Comprehensive power and thermal profiling. GPU-level power and thermal monitoring in this study was collected only for Kimi-K2.5 (Table 29). Extending rocm-smi monitoring to all four models across the full concurrency range would enable cross-architecture power efficiency comparisons (tokens per watt), characterize how power draw scales with concurrency, and quantify the energy cost differences between dense, MoE+GQA, and MoE+MLA inference. This data would also inform capacity planning for datacenter power budgets. Automated architecture-aware configuration. Our findings on architecture-specific optimization requirements motivate the development of automated configuration selection systems that detect model architecture (attention mechanism, expert structure, head count) and automatically select optimal vLLM parameters, AITER flags, tensor parallelism degree, and quantization format. Such a system would reduce the deployment expertise barrier and prevent the silent correctness and performance issues we encountered during manual configuration. Acknowledgments The authors thank Vultr for providing the 8-GPU AMD Instinct MI325X cluster used in this study. We also thank the open-source vLLM and ROCm communities for their contributions to the inference software stack. References [1] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai (2023) GQA: training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §2.2. [2] AMD (2024) AITER: AI tensor engine for ROCm. Note: https://github.com/ROCm/aiterOptimized inference kernels for AMD Instinct GPUs Cited by: §2.3, §4.3, §6.5.1, 3rd item. [3] K. T. Chitty-Venkata, S. Howland, G. Azar, D. Soboleva, N. Vassilieva, S. Raskar, M. Emani, and V. Vishwanath (2025) MoE-Inference-Bench: performance evaluation of mixture of expert large language and vision models. arXiv preprint arXiv:2508.17467. Cited by: item 4, §2.2, §2.6. [4] K. T. Chitty-Venkata, S. Raskar, M. Emani, V. Vishwanath, and A. K. Somani (2024) LLM-Inference-Bench: inference benchmarking of large language models on AI accelerators. In SC’24 Workshops: International Conference for High Performance Computing, Networking, Storage, and Analysis, Note: arXiv preprint arXiv:2411.00136 Cited by: §2.6. [5] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Cited by: §2.3. [6] T. Dao (2023) FlashAttention-2: faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. Cited by: §2.3. [7] S. Dash, I. Lyngaas, J. Yin, X. Wang, R. Egele, G. Cong, F. Wang, and P. Balaprakash (2023) Optimizing distributed training on Frontier for large language models. arXiv preprint arXiv:2312.12705. Note: Demonstrates trillion-parameter model training on AMD MI250X GPUs at Oak Ridge National Lab (Frontier) Cited by: §7.2. [8] M. Davies, N. Crago, K. Sankaralingam, and C. Kozyrakis (2025) LIMINAL: exploring the frontiers of LLM decode performance. arXiv preprint arXiv:2507.14397. Cited by: §2.6, §7.3. [9] DeepSeek-AI (2024) DeepSeek-V2: a strong, economical, and efficient Mixture-of-Experts language model. arXiv preprint arXiv:2405.04434. Cited by: §1, §2.2. [10] DeepSeek-AI (2024) DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1, §2.2, §2.2, §3.3, §7.5. [11] DeepSeek-AI (2025) DeepSeek-V3.2: pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Cited by: §1, §2.2, §3.3. [12] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2023) GPTQ: accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), Cited by: §2.4. [13] A. Grattafiori, A. Dubey, A. Jauhri, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §1, §1, §2.2, §3.3. [14] A. Gu and T. Dao (2023) Mamba: linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Cited by: §7.5, §8.2. [15] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, et al. (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: item 4, §1, §2.2. [16] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), Cited by: §1, §2.1, §2.5, §3.2, §7.3. [17] Y. Leviathan, M. Kalman, and Y. Matias (2023) Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.17192. Cited by: §2.6, §8.2. [18] J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han (2024) AWQ: activation-aware weight quantization for LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), Note: Best Paper Award Cited by: §2.4. [19] P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, et al. (2022) FP8 formats for deep learning. arXiv preprint arXiv:2209.05433. Cited by: §4.1, §4.1. [20] Moonshot AI (2026) Kimi-K2.5: scaling reinforcement learning with LLMs. Technical Report. Note: https://github.com/MoonshotAI/Kimi-K2.5 Cited by: §1, §2.2, §2.2, §3.3, §4.1, §7.2, §7.5, item 3. [21] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean (2023) Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §3.1, §7.3. [22] Qwen Team (2025) Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. Cited by: §1, §3.3. [23] P. G. Recasens, F. Agullo, Y. Zhu, C. Wang, E. K. Lee, O. Tardieu, J. Torres, and J. Ll. Berral (2025) Mind the memory gap: unveiling GPU bottlenecks in large-batch LLM inference. arXiv preprint arXiv:2503.08311. Note: IEEE CLOUD 2025. Shows model-dependent DRAM-bandwidth-driven saturation on different hardware platforms Cited by: item 5, §2.6, §7.3, item 5. [24] V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, et al. (2020) MLPerf inference benchmark. In ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), p. 446–459. Cited by: §2.1, §2.6. [25] SemiAnalysis (2025) InferenceMAX: multi-model, multi-hardware nightly LLM inference benchmarks. Note: https://inferencemax.semianalysis.com/Launched October 2025. Covers MI325X, H200, B200, MI355X with throughput-latency Pareto frontiers Cited by: item 1, §2.1, §2.6. [26] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Cited by: §1, §2.2. [27] N. Shazeer (2019) Fast transformer decoding: one write-head is all you need. arXiv preprint arXiv:1911.02150. Cited by: §2.2. [28] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro (2019) Megatron-LM: training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. Cited by: §4.4. [29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §1, §2.2. [30] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han (2023) SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), Cited by: §2.4. [31] G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022) Orca: a distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Cited by: §2.1. [32] J. Yuan, H. Liu, S. Zhong, Y. Chuang, S. Li, G. Wang, D. Le, H. Jin, V. Chaudhary, Z. Xu, Z. Liu, and X. Hu (2024) KV cache compression, but what must we give in return? A comprehensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.01527. Note: EMNLP 2024 Findings Cited by: §2.5. [33] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kober, C. Shi, Y. Sheng, et al. (2024) SGLang: efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104. Cited by: §2.1.