Paper deep dive

CCCL: In-GPU Compression-Coupled Collective Communication

Chon Lam Lao, Zhiying Xu, Zhuang Wang, Ziming Mao, Delong Meng, Jia Zhen, Jun Wu, Ion Stoica, Yida Wang, Yang Zhou

Year: 2026Venue: arXiv preprintArea: cs.DCType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%

Last extracted: 4/27/2026, 4:36:13 AM

Summary

CCCL (Compression-augmented Collective Communication Library) is a built-in compression-based library designed to reduce communication overhead in LLM workloads. It integrates GPU-native compression (CCCL-CORE) directly into the NCCL data path, enabling seamless, transparent adoption without user-side code changes. By leveraging the high bandwidth of HBM to perform lightweight exponent-based compression, CCCL reduces communication volume and improves the communication-computation overlap. The system achieves up to 3x effective NVLink bandwidth utilization and improves end-to-end throughput in vLLM PD disaggregation workloads by up to 10.1%.

Entities (7)

allreduce · operation · 100%CCCL · library · 100%CCCL-CORE · component · 100%HBM · memory · 100%NCCL · library · 100%NVLink · interconnect · 100%vLLM · framework · 100%

Relation Signals (5)

CCCL → contains → CCCL-CORE

confidence 100% · The design of CCCL centers around two tightly coupled components... CCCL-CORE: GPU compression engine.

CCCL → improvesthroughputof → vLLM

confidence 100% · improves end-to-end throughput in vLLM PD disaggregation workloads by up to 10.1%

CCCL → integrateswith → NCCL

confidence 100% · integrates with NCCL to eliminate the data coalescing stage

CCCL → supports → allreduce

confidence 100% · supports operations such as allreduce, alltoall, and send/recv

CCCL-CORE → leverages → HBM

confidence 90% · leveraging HBM’s bandwidth for lightweight, in-place data transformations

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Collective communication incurs significant overhead in LLM workloads. Although overlapping communication with computation in application-level is a common strategy, it often requires substantial code modifications and is impractical for many workloads (e.g., tensor and expert parallelism). We present CCCL, a built-in compression-based collective communication library that supports operations such as allreduce, alltoall, and send/recv without requiring any user-side changes, thereby enabling seamless adoption in existing applications. CCCL tightly fuses compression kernels to minimize memory accesses and integrates with NCCL to eliminate the data coalescing stage, making it fast enough (up to 3x NVLink bandwidth) to sustain communication. Our evaluation shows that CCCL improves end-to-end throughput in vLLM PD disaggregation workloads by up to 10.1% and microbenchmark throughput by up to 30%.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

41,856 characters extracted from source content.

Expand or collapse full text

CCCL: IN-GPU COMPRESSION-COUPLED COLLECTIVE COMMUNICATION ChonLam Lao 1 † Zhiying Xu 1 † Zhuang Wang 1 † Ziming Mao 2 Delong Meng 1 † Jia Zhen 1 † Jun Wu 1 † Ion Stoica 2 Yida Wang 1 † Yang Zhou 3 ABSTRACT Collective communication incurs significant overhead in LLM workloads. Although overlapping communication with computation in application-level is a common strategy, it often requires substantial code modifications and is impractical for many workloads (e.g., tensor and expert parallelism). We present CCCL, a built-in compression-based collective communication library that supports operations such as allreduce, alltoall, and send/recv without requiring any user-side changes, thereby enabling seamless adoption in existing applications. CCCL tightly fuses compression kernels to minimize memory accesses and integrates with NCCL to eliminate the data coalescing stage, making it fast enough (up to 3× NVLink bandwidth) to sustain communication. Our evaluation shows that CCCL improves end-to-end throughput in vLLM PD disaggregation workloads by up to 10.1% and microbenchmark throughput by up to 30%. 1 INTRODUCTION Large Language Models (LLMs) have nurtured a flour- ishing ecosystem of applications, including code gener- ation (Qwen et al., 2025), chatbots (DeepSeek-AI et al., 2025), agents (Guo et al., 2024), and search engines (Xiong et al., 2024). These large LLMs exhibit remarkable capabil- ities but, at the same time, demand enormous computational resources. This rapid scaling introduces a new challenge: communica- tion. As training and inference are distributed across multi- ple accelerators, inter-GPU communication has become a dominant bottleneck (Rajasekaran et al., 2024; Cao et al., 2024). Although modern GPUs provide immense computa- tional throughput, communication and computation rarely overlap effectively in practice (Chang et al., 2024; Zhang et al., 2025a). Profiling large-scale LLM training shows that GPUs spend most of the communication phase idle, achieving only about a 20% communication–computation overlap ratio (see Section §2.1). This poor overlap arises from several factors, including: (1) intrinsic dependency constraints in model execution—such as tensor parallelism, where each layer’s computation de- pends on the results of the previous layer’s communica- tion—and (2) coarse-grained GPU kernel scheduling that prevents fine-grained interleaving between communication † This work does not relate to the position at Amazon. 1 Amazon Web Services, USA 2 University of California, Berkeley, CA, USA 3 University of California, Davis, CA, USA. Correspondence to: Yang Zhou<yangzhou@ucdavis.edu>. and computation. Together, these factors make communi- cation–computation overlap inherently inefficient, leaving significant GPU capacity underutilized. Moreover, most deep learning frameworks rely on NVIDIA’s NCCL library (Hu et al., 2025) for collective communication. NCCL provides convenient APIs—such asallreduce(),send(), andrecv()—that abstract away transport details and hardware differences. However, this convenience comes at a cost: NCCL implements these collectives as CUDA kernels that occupy GPU streaming multiprocessors (SMs), limiting the system’s ability to over- lap communication and computation and ultimately leading to underutilized GPU resources (Hwang et al., 2023; Shah et al., 2025). Recent systems have explored bypassing NCCL and imple- menting communication using low-level GPU primitives to fuse it more tightly with computation (Zhao et al., 2025). For example, DeepEP co-designs computation and com- munication to achieve effective overlapping, significantly improving efficiency. However, such approaches require invasive code modifications and deep understanding of GPU kernel scheduling and dependencies—effort that is difficult to maintain and generalize. These two extremes expose a fundamental tradeoff: NCCL’s simplicity sacrifices efficiency, while manual co-design de- livers efficiency at the cost of programmability and portabil- ity. To bridge this gap, we build CCCL, a Compression- augmented Collective Communication Library that inte- grates GPU-native compression directly into NCCL’s data 1 arXiv:2604.17172v1 [cs.DC] 19 Apr 2026 path while preserving an NCCL-compatible interface. By compressing data on the sender and decompressing it on the receiver, CCCL reduces communication volume while leveraging otherwise idle GPU resources for compression work. Compression and decompression are pipelined with communication and computation to further minimize latency and improve overlap. This design is particularly well suited to modern GPUs: high-bandwidth memory (HBM) provides several terabytes per second of throughput—3–10× higher than NVLink or NIC bandwidth—offering abundant head- room for in-memory compression. By preserving NCCL’s familiar API and execution model, CCCL enables appli- cations to benefit from compression transparently while achieving higher bandwidth utilization and improved com- munication–computation overlap. The design of CCCL cen- ters around two tightly coupled components that together enable efficient, modular integration: CCCL-CORE: GPU compression engine. A standalone, high-throughput compression module that operates entirely within the GPU. It performs localized exponent-based com- pression using per-block frequency estimation, avoiding costly global synchronization and reducing global memory access from three rounds to one. This lightweight design achieves near-HBM throughput and high compression ratios (up to 70%), forming the building block for system-level integration. Compression–communication co-design. A system layer that embeds CCCL-CORE directly into NCCL’s collective pipeline. Compression, transmission, and decompression are fused into a single GPU execution context, executing at the warp level to overlap computation and communication seamlessly. The co-design applies compression selectively to inter-GPU data paths while bypassing local transfers, en- suring both performance and correctness across collectives such as allreduce, alltoall, and send/recv. Together, these two components bridge the tradeoff between efficiency and programmability. CCCL-CORE provides an efficient, reusable GPU compression primitive, while the co- design layer integrates it transparently into existing NCCL collectives. This combination enables up to 3×higher ef- fective NVLink bandwidth utilization and substantially im- proves communication–computation balance. Our evalua- tion shows that CCCL improves end-to-end throughput by up to 10.1% in vLLM parameter-disaggregation workloads and achieves up to 30% higher microbenchmark throughput compared to standard NCCL. 051015202530 Rank 0 4 8 12 16 20 24 Overlap (%) Computation-Communication Overlap Figure 1. The Comm-Comp Overlap Ratio is low. 2 MOTIVATION AND BACKGROUND 2.1 The Challenge of Overlapping Communication and Computation Communication and computation are tightly coupled in large-scale ML workloads, yet achieving effective overlap between them remains challenging (DeepSeek-AI et al., 2025). To illustrate, we profile the training of a 39.1B GPT model (TP=8, DP=4) using Megatron-LM (Narayanan et al., 2021) on 32 A100 GPUs interconnected by 800 Gbps NICs (Figure 1). Using Meta’s HolisticTraceAnalysis (Bhatna- gar et al., 2023), we observe that during all communication phases, only about 20% of the time overlaps with computa- tion. In other words, GPUs spend most of the communica- tion duration without performing any useful compute, and vice versa—leaving significant potential overlap unrealized. This limited overlap stems from intrinsic dependency con- straints in ML execution. For example, under tensor par- allelism (TP), the computation of each layer cannot begin until the communication of the previous layer completes. 2.2 NCCL or Custom Kernels: A Fundamental Tradeoff Collective communication libraries (CCLs) have become the de facto standard for distributed machine learn- ing in PyTorch-based frameworks such as Megatron- LM (Narayanan et al., 2021), ColossalAI (Li et al., 2023), and DeepSpeed (Rasley et al., 2020).Their simple and portable APIs (e.g.,allreduce()and allgather()) abstract away low-level communication details. However, this abstraction comes at a cost: while li- braries like NCCL provide ease of use and portability, their coarse-grained design often limits GPU efficiency (Shah et al., 2025; Hwang et al., 2023). For example, when an NCCL collective operation is in- voked, it launches one or more CUDA kernels on the GPU. 2 These kernels occupy the streaming multiprocessors (SMs) to perform communication tasks such as data copying and reduction over NVLink or RDMA. However, ML workloads are already saturated with compute kernels that also use the same SMs, causing interference and preventing effective overlap between communication and computation. More- over, NCCL’s general-purpose design hides communication semantics from the framework, preventing tight coordina- tion between the two phases. As a result, communication and computation are often executed sequentially rather than concurrently. Although NCCL provides asynchronous prim- itives, scheduling remains managed internally by the GPU, which lacks global visibility into computation dependencies and resource contention—leading to suboptimal overlap and memory bandwidth utilization. In addition, when collective operations handle small mes- sages, the per-call kernel launch and scheduling overhead can dominate the total communication time, making it non- negligible compared to the actual data transfer. Together, these limitations result in poor GPU utilization and minimal communication–computation overlap. To address the overlap problem, developers may bypass NCCL and manually fuse communication and compu- tation using low-level CUDA communication primitives. This approach enables fine-grained coordination between computation and data transfer, allowing systems such as DeepEP (Zhao et al., 2025) to achieve near-perfect over- lap and high GPU efficiency in specialized workloads like MoE training. However, this performance comes at a steep cost: implementing and maintaining such designs requires intrusive code modifications and deep expertise across GPU programming, LLM model internals, and the entire training codebase. This tension motivates the need for a communication library that achieves fine-grained overlap without sacrificing pro- grammability. We present CCCL, a compression-augmented collective communication library that integrates lightweight compression directly into NCCL’s data path. CCCL retains the same familiar APIs (e.g.,allreduce()) while reduc- ing communication volume and dynamically balancing com- putation and communication on the GPU. By co-designing compression with collective execution, CCCL achieves effi- cient overlap without requiring any changes to model code or training frameworks. 2.3 Opportunities Arising from the HBM–Communication Bandwidth Gap Modern GPUs are equipped with high-bandwidth mem- ory (HBM) that provides extremely fast access for on-device computation, reaching terabytes per second (TB/s). In contrast, communication links such as NVLink, PCIe, and RDMA NICs deliver substantially lower band- ++ 16-byte data chunk L/ L128/ Simple L/ L128/ Simple Comp. Table C D Receive Send RecvReduceSend Recv Decomp. Reduce Comp. Send 16-byte data chunk 16-byte data chunk Figure 2. CCCL Overview. width—typically in gigabytes per second (GB/s)—resulting in a significant gap that often limits communication effi- ciency. For instance, on NVIDIA A100 GPUs, NVLink provides up to 300 GB/s per direction (600 GB/s bidirectional) of intra- node communication bandwidth per GPU, whereas HBM delivers up to 1.6 TB/s of memory bandwidth. The gap widens on newer architectures—GH200 GPUs offer roughly 450 GB/s per direction (900 GB/s bidirectional) of NVLink bandwidth versus about 3.3 TB/s of HBM bandwidth. The difference is even more pronounced for PCIe-connected GPUs such as the L40S, which achieve around 30 GB/s. For inter-node communication, RDMA data transfers over a are limited to roughly 50GB/s (400 Gbps NIC). Overall, HBM bandwidth is typically 3–10×higher than NVLink bandwidth and an order of magnitude higher than PCIe or NIC bandwidth. This imbalance highlights a key characteristic of modern GPU systems: while memory band- width continues to scale rapidly, communication bandwidth remains comparatively constrained. This disparity creates an opportunity to shift part of the communication workload into the memory domain by leveraging HBM’s bandwidth for lightweight, in-place data transformations such as com- pression and decompression. Motivated by this observation, we design CCCL, which exploits excess HBM bandwidth to reduce communication volume through in-GPU compres- sion while preserving the NCCL programming model. 3 CCCL OVERVIEW CCCL(Compression-augmentedCollective Communication Library) is a unified framework that integrates compression directly into GPU collective com- munication. Its goal is to accelerate large-scale distributed 3 workloads by reducing communication volume, while remaining fully compatible with existing NCCL-based systems. CCCL can be adopted transparently without any application-side changes—a simpleLDPRELOAD substitution replaces NCCL and preserves its original API (e.g., allreduce, alltoall, send/recv). CCCL is built upon two complementary components that serve distinct purposes. The first, CCCL-CORE, is a reusable GPU compression library that performs lightweight, localized compression and decompression opti- mized for collective workloads. It can operate as an indepen- dent library, providing a high-level API for efficient in-GPU data compression across various applications. The second component, referred to simply as CCCL, extends this core by integrating compression seamlessly into the collective communication pipeline. This integration enables compres- sion, transmission, and decompression to be executed in a tightly pipelined and overlapped manner, achieving high throughput without requiring user intervention. Figure 2 illustrates the overall architecture. Data flowing between GPUs is compressed on the sender, transmitted in reduced form over NVLink or RDMA, and decompressed on the receiver before entering computation. Because com- pression is performed within the GPU using otherwise idle SM resources, CCCL leverages the bandwidth gap between HBM and interconnects to balance computation and com- munication. This tight fusion improves effective NVLink bandwidth by up to 3×and achieves end-to-end through- put gains of up to 10% in vLLM parameter-disaggregation workloads. 4 CCCL-CORE: THE GPU COMPRESSION ENGINE At the heart of CCCL lies CCCL-CORE, a GPU-native com- pression engine designed to reduce communication volume with minimal overhead. It exploits the statistical structure of floating-point tensors to achieve high compression ra- tios while maintaining throughput sufficient to overlap with communication. This section describes the principle behind its design, the inefficiencies in na ̈ ıve GPU implementations, and the architectural optimizations that make CCCL-CORE efficient enough to serve as the foundation of end-to-end compression–communication co-design. 4.1 Compression Principle A floating-point value is composed of three components: fraction, exponent, and sign. Among them, the fraction and sign are nearly random and thus difficult to compress, whereas the exponent tends to follow a narrow, highly skewed distribution. Inbf16, the exponent has 8 bits, corresponding to 256 possible symbols, but in practice only a small subset of them occur frequently during training or inference. This structure allows the exponent to be com- pressed by up to 70% (Zhang et al., 2025b), providing a substantial opportunity to reduce communication volume in collective workloads that frequently exchange gradient or activation tensors. CCCL-CORE implements component-wise compression based on this principle, leveraging (Meta AI Research, 2026) as a building block while extending it with GPU-optimized integration for collective communication. It separates ex- ponent bits from the fraction and sign, entropy-encodes the exponent using a frequency table, and transmits the remain- ing components in raw form. The challenge lies in perform- ing this process efficiently on GPUs: building frequency tables, compressing large tensors, and assembling variable- length outputs all incur costly global memory accesses and synchronization. 4.2 Baseline GPU Compression Pipeline A na ̈ ıve GPU implementation of exponent-based compres- sion typically involves three full rounds of global memory access: 1.Frequency table construction. The input tensor must be scanned once to collect the frequency of each expo- nent symbol and construct a global table. Forbf16, 256 exponent values are possible. After table construc- tion, the compressible exponents are separated from the fraction and sign bits and rearranged into contigu- ous memory regions, which are then written back to global memory. 2. Block-wise compression. Once the table is available, multiple thread blocks perform compression indepen- dently using the global table. Each block loads its input data from global memory and writes compressed re- sults to temporary global buffers. Because each block produces variable-length output, these intermediate results cannot be directly written to the final buffer, leading to a second round of global memory access. 3.Indexing and coalescing. After compression com- pletes, offsets and headers must be computed to coa- lesce variable-length outputs into a contiguous output buffer, incurring a third global memory write. This baseline design functions correctly but performs poorly on GPUs. Each additional round of global memory traversal substantially increases latency, and synchronization between blocks prevents fine-grained parallelism. Consequently, throughput remains below interconnect bandwidth, mak- ing it difficult to overlap compression with communication. 4 4.3 Localized Frequency Tables CCCL-CORE addresses these inefficiencies by replacing global coordination with localized frequency estimation and reducing global memory accesses to a single round. Instead of sharing a single global frequency table, each thread block constructs its own local table by sampling a small portion of its assigned data range (e.g., the first 256 KB). This localized estimation accurately captures the local exponent distribution while avoiding a full dataset traversal. Because exponents in neural network tensors exhibit sta- ble distributions across nearby regions, local estimation achieves compression ratios comparable to the global-table baseline (within 7%), while eliminating one entire memory pass. Each block then uses its local table for both com- pression and decompression, removing the need for global lookups or synchronization. To further simplify encoding, CCCL-CORE assigns fixed-length bit codes to all exponent symbols, avoiding conditional table lookups and minimizing control divergence within GPU warps. 4.4 Reducing Global Memory Access By integrating local frequency estimation into the compres- sion kernel, CCCL-CORE fuses the computation stages of the baseline pipeline. Each thread block directly compresses its input into a preallocated region of global memory, elim- inating the need for intermediate buffers and the coalesc- ing phase. The total number of global memory accesses is thereby reduced from three to one, yielding up to 3× higher throughput compared to DietGPU while maintain- ing similar compression effectiveness. This optimization allows CCCL-CORE to sustain compression throughput that matches or exceeds NVLink bandwidth, a necessary condi- tion for overlapping compression and communication. 4.5 Integration with Communication While CCCL-CORE can be used as a standalone compres- sion library, its full benefit is realized when integrated with collective communication. In this integrated mode, com- pressed outputs are streamed directly into the communica- tion pipeline rather than coalesced in global memory, allow- ing compression and transmission to proceed in a pipelined manner. This system-level co-design, including warp-level overlap and selective compression across collectives, is de- scribed in detail in Section §5. 5 COMPRESSION–COMMUNICATION CO-DESIGN CCCL extends CCCL-CORE beyond standalone compres- sion by integrating it directly into the GPU collective com- munication path. This integration enables compression, transmission, and decompression to be executed in a unified pipeline, allowing communication to proceed at reduced bandwidth while maintaining full compatibility with exist- ing NCCL APIs. Integration into the NCCL data path. The core modi- fication in CCCL occurs within NCCL’s data processing layer, where collective operations such asallreduce, alltoall, andsend/recvare implemented. Specifi- cally, CCCL integrates compression and decompression kernels into the data flow previously handled by the CopyReducePacksfunction, which processes data in fixed-size chunks (128×8×256 elements). This function loads data from source buffers, performs the required collec- tive operation (e.g., sum, min, max), and stores results back into global memory. By fusing compression kernels into this stage, CCCL removes redundant memory transfers between communication and computation, allowing compressed data to be transmitted directly over NVLink or RDMA. Warp-level execution and compatibility. Both compres- sion and communication execute at the warp level, with each warp handling a single compression or decompression block. This design aligns naturally with GPU scheduling, allowing each warp to independently process data segments and issue communication requests without global synchro- nization. The warp-level structure also preserves portability: since NCCL’s collectives are themselves implemented as warp-cooperative kernels, no architecture-specific warp tun- ing is required to support new GPU generations. In practice, this design achieves effective overlap between compression and data transfer within the same kernel launch, leveraging otherwise idle SM cycles during communication. Selective compression across collective stages. Not all data in collective operations benefit equally from compres- sion. For point-to-point primitives such assendandrecv, CCCL always applies compression before transmission and decompression upon reception. However, multi-step col- lectives such as allreduce require finer control because data passes through several intermediate aggregation stages, each potentially involving multiple sources and destinations. CCCL therefore applies compression selectively: on the sender side, compression is performed only when a GPU transmits data to another node or writes to a remote FIFO buffer. During intermediate reduction steps that combine locally available data with received partial results, only the remote data are decompressed and merged, while local GPU data remain uncompressed. This policy minimizes unneces- sary computation while ensuring that all cross-node traffic is transmitted in compressed form. Similarly, on the receiver side, decompression is triggered only for data that originate from remote peers, while intra-GPU or final output paths bypass decompression entirely. 5 Handling data alignment. Because NCCL processes data in chunked segments, CCCL applies compression only to regions that align with these boundaries (e.g., 32 KB). Any remainder data that fall outside of these boundaries are trans- ferred in uncompressed form to avoid partial-block handling overhead. This selective policy simplifies synchronization and preserves correctness across heterogeneous message sizes. Intra- vs. inter-node communication. Within a single node, compressed data can be exchanged directly between GPUs via NVLink. For inter-node communication over RDMA, CCCL writes compressed outputs into a GPU FIFO buffer that the host CPU subsequently reads and transmits across the network interface. This design maintains compat- ibility with existing NCCL transport mechanisms, allowing CCCL to support both intra- and inter-node collectives with- out modification to the user-level API. Overall, CCCL transforms NCCL’s sequential communica- tion–computation pipeline into a tightly coupled, warp-level co-design that fuses compression, transfer, and decompres- sion. By compressing only where it reduces data movement and overlapping all stages within a single GPU execution context, CCCL achieves high utilization of both memory and compute resources, enabling up to 3×effective NVLink bandwidth in practice. 6 EVALUATION In this section, we evaluate CCCL in three aspects: vLLM parameter disaggregation, microbenchmarks, and a break- down of each design component. 6.1 Experimental Setup Our experiments are conducted on multiple AWS EC2 in- stance types. The default setup usesg6einstances equipped with NVIDIA L40S GPUs. We also evaluate CCCL onp4d instances with eight A100 GPUs and onp5eninstances with H200 GPUs to examine performance across different GPU architectures. For the communication library CCCL, we fix the configura- tion to use 4 channels (4 SMs). We use NCCL v2.23.4 as the baseline in all experiments with its default configurations (e.g., number of channels, number of SMs). For fair com- parison, we include an additional configuration, denoted as NCCL (matched), which aligns NCCL’s configuration with CCCL by also using 4 SMs. For the high-level compression library (CCCL-core), we evaluate it independently to measure its compression and decompression performance. We use DietGPU (Meta AI Research, 2026) as the baseline and report throughput as the primary performance metric across different message sizes. 7680153602560051200 Input Length (tokens) 0 2 4 6 8 10 CCCL Performance Improvement (%) 6.4% 8.2% 10.1% 6.8% Figure 3. PD Disaggregation. For the application-level evaluation, we integrate CCCL with parameter disaggregation invLLM(Kwon et al., 2023), demonstrating CCCL ’s performance benefits in realistic distributed inference workloads. 6.2 CCCL on PD Disaggregation We evaluate the performance of PD aggregation in CCCL and compare it against the original NCCL used in vLLM. The experiments are conducted under the 1 prefill + 3 decode configuration, which aligns with the default setup in vLLM PXDY. We vary the input sequence length from 7,680 to 51,200 tokens. Our results in Figure 3 show that CCCL achieves up to 9% improvement in performance over NCCL. This gain primarily comes from the reduced latency in transferring the KV cache from the prefill server to the decode servers. Since CCCL improves the communication efficiency by roughly 30%, the end-to-end application speedup can be estimated as the product of the network bottleneck portion and the microbenchmark improvement. Specifically, for an input length of 7,680, we observe that KV cache transmission accounts for 23% of the total execution time—consistent with the expected overall speedup. Importantly, these improvements require no changes to user code. CCCL only replaces the underlying NCCL implemen- tation via LDPRELOAD, enabling seamless integration with existing vLLM deployments. 6.3 CCCL Performance We evaluate CCCL’s performance on three repre- sentative collectives—send/recv,alltoall, and allreduce—that are widely used in NCCL, as shown in Figure 4, Figure 5, and Figure 6. We compare against two baselines: (1) the default NCCL configuration, and (2) NCCL (matched), which uses 4 SMs to align with CCCL’s 6 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 2 GB Tensor Size 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Throughput CCCL NCCL (matched) NCCL (default) Figure 4. SendRecv 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 2 GB Tensor Size 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Throughput CCCL NCCL (matched) NCCL (default) Figure 5. AlltoAll configuration. As shown in Figure 4, CCCL consistently outperforms both baselines in the send/recv operation across all message sizes, achieving over 20% performance improvement for messages larger than 32 MB. This gain stems from CCCL’s ability to significantly reduce the communication volume for large messages. When compression efficiency reaches around 20%, and assuming perfect overlap, the resulting performance gain aligns closely with the observed improve- ment. For smaller messages (<32 MB), compression intro- duces additional overhead that dominates the relatively short communication time, making compression less beneficial. A similar trend is observed for thealltoallcol- lective (Figure 5), which essentially combines multiple send/recvoperations. Due to increased communication and scheduling overhead on the GPU side, the improve- ment is slightly lower than that ofsend/recv, showing approximately 18% gain for large messages (>32 MB). In contrast, CCCL underperforms both NCCL baselines in theallreduceoperation (Figure 6). This is because each GPU must perform compression and decompression at every communication stage. For instance, in a ring all- 2 MB8 MB 32 MB 128 MB512 MB 2 GB Tensor Size 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Throughput CCCL NCCL (matched) NCCL (default) Figure 6. AllReduce 32 KB 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 2 GB Tensor Size 0 250 500 750 1000 1250 Bandwidth (GBps) H200 NVLink H200 CCCL Optimal A100 NVLink A100 CCCL Optimal 400Gbps RDMA Figure 7. NVLink Bandwidth reduce operation withkGPUs, each GPU executes roughly 2kcompression/decompression operations. Since perfect overlap between computation and communication is not yet achieved, CCCL incurs additional overhead in this scenario. Optimizing compression overlap for collective operations remains an area of future work. 6.4 Can CCCL Be Applied to A100/H200? So far, we have presented results on L40S GPUs. However, CCCL can also be applied to A100 and H200 GPUs with communication-computation overlapping, as illustrated in Figure 7. We report the optimal CCCL bandwidth measured without communication overlap—that is, when compres- sion is performed and the resulting data remains in GPU memory. Theoretically, with perfect overlap between com- pression and communication, the achievable performance can approach this optimal bandwidth. Taking a 2 GB message as an example, CCCL on H200 achieves an optimal throughput of up to 1300 GB/s, while NVLink provides approximately 400 GB/s per direc- tion—over a 3×potential improvement. This demonstrates the substantial benefits of integrating CCCL with overlap- 7 32 KB 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 1 GB Tensor Size 0 200 400 600 800 1000 Throughput (GBps) CCCL-core DietGPU Improvement 0 20 40 60 80 100 Improvement (%) Figure 8. H200 Throughput 32 KB 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 1 GB Tensor Size 0% 20% 40% 60% 80% 100% Compressed Size (% of Original) CCCL-core DietGPU Figure 9. H200 Compression Ratio 32 KB 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 2 GB Tensor Size 0 100 200 300 400 Throughput (GBps) CCCL-core DietGPU Improvement 0 25 50 75 100 125 150 Improvement (%) Figure 10. A100 Throughput 32 KB 128 KB512 KB 2 MB8 MB 32 MB 128 MB512 MB 1 GB Tensor Size 0% 20% 40% 60% 80% 100% Compressed Size (% of Original) CCCL-core DietGPU Figure 11. A100 Compression Ratio ping execution. To fully exploit this potential, however, the GPU must efficiently support such overlap, for instance through techniques like warp specialization. A similar trend is observed on A100 GPUs, where CCCL’s optimal bandwidth reaches around 700 GB/s—over 2× higher than NVLink bandwidth.For commonly used RDMA connections, such as 400 Gbps NICs (NVIDIA Corporation, 2021), achieving this overlap is even more fea- sible, further underscoring CCCL’s potential for deployment across diverse GPU platforms. 6.5 CCCL-core Performance We further evaluate the performance ofCCCL-corein- dependently and compare it against DietGPU, as shown in Figures 10 and 11. The experimental results on H200 GPUs (Figures 8 and 9) show improvements in both com- pression throughput and overall performance. Although CCCL-coreproduces slightly larger compressed data (i.e., a lower compression ratio)—by less than 7% compared to DietGPU—it achieves more than 20% higher through- put. This gain comes fromCCCL-core’s design, which reduces global memory accesses and thereby improves per- formance significantly. While the frequency table becomes localized (and thus less globally representative of the input data, leading to a marginally lower compression ratio), the throughput improvement is substantial, reaching at least 18% for 128 MB messages. A similar trend is observed on A100 GPUs. The compressed data size remains consistent (Figure 11) because the com- pression algorithm itself is independent of the GPU architec- ture. However, the throughput on A100 (Figure 10) shows even greater gains over DietGPU, achieving up to 150% performance improvement for smaller message sizes such as 512 KB. 7 CONCLUSION CCCL revisits collective communication through the lens of in-GPU compression. By introducing CCCL-CORE, a lightweight GPU compression engine, and integrating it di- rectly into NCCL’s collective pipeline, CCCL reduces com- munication volume and overlaps compression with transmis- sion to better utilize GPU resources. This design achieves up to 3×higher effective bandwidth and tangible end-to-end speedups, demonstrating that GPU-aware compression can 8 fundamentally reshape the balance between computation and communication in large-scale distributed learning. REFERENCES Bhatnagar, A., Coutinho, B., Feng, X., Liu, Y., Lin, S.-H., and Feng, L.Holistic trace analysis (hta). https://github.com/facebookresearch/ HolisticTraceAnalysis, 2023.Accessed: 2025-10-20. Cao, J., Guan, Y., Qian, K., Gao, J., Xiao, W., Dong, J., Fu, B., Cai, D., and Zhai, E. Crux: Gpu-efficient communica- tion scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, ACM SIG- COMM ’24, p. 1–15, New York, NY, USA, 2024. Asso- ciation for Computing Machinery. ISBN 9798400706141. doi: 10.1145/3651890.3672239. URLhttps://doi. org/10.1145/3651890.3672239. Chang, L.-W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Jiang, Z., Lin, H., Jin, X., and Liu, X. Flux: Fast software-based communication overlap on gpus through kernel fusion, 2024. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Wang, Q., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y., Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z., Zhang, Z., Hao, Z., Gou, Z., Ma, Z., Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437. Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24, 2024. ISBN 978-1-956792-04-1. doi: 10.24963/ijcai.2024/ 890. URLhttps://doi.org/10.24963/ijcai. 2024/890. Hu, Z., Shen, S., Bonato, T., Jeaugey, S., Alexander, C., Spada, E., Dinan, J., Hammond, J., and Hoefler, T. De- mystifying nccl: An in-depth analysis of gpu communica- tion protocols and algorithms. In 2025 IEEE Symposium on High-Performance Interconnects (HOTI), p. 48–59, 2025. doi: 10.1109/HOTI66940.2025.00024. Hwang, C., Park, K., Shu, R., Qu, X., Cheng, P., and Xiong, Y.ARK: GPU-driven code execution for distributed deep learning.In 20th USENIX Sympo- sium on Networked Systems Design and Implemen- tation (NSDI 23), p. 87–101, Boston, MA, April 2023. USENIX Association. ISBN 978-1-939133-33-5. URLhttps://w.usenix.org/conference/ nsdi23/presentation/hwang. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), p. 1207–1216, Stan- ford, CA, 2000. Morgan Kaufmann. Li, S., Liu, H., Bian, Z., Fang, J., Huang, H., Liu, Y., Wang, B., and You, Y. Colossal-ai: A unified deep learning sys- tem for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing, ICPP ’23, p. 766–775, New York, NY, USA, 2023. Asso- ciation for Computing Machinery. ISBN 9798400708435. doi: 10.1145/3605573.3605613. URLhttps://doi. org/10.1145/3605573.3605613. Meta AI Research. DietGPU.https://github.com/ facebookresearch/dietgpu , 2026.GitHub repository, accessed 2026-03-07. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, 9 P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model train- ing on gpu clusters using megatron-lm. In Proceed- ings of the International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, SC ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10. 1145/3458817.3476209. URLhttps://doi.org/ 10.1145/3458817.3476209. NVIDIA Corporation.NVIDIA ConnectX-7 Eth- ernetDatasheet.https://w.nvidia. com/content/dam/en-z/Solutions/ networking/ethernet-adapters/ connectx-7-datasheet-Final.pdf ,2021. Accessed: 2026-03-08; ConnectX-7 SmartNIC up to 400 Gb/s bandwidth. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URLhttps: //arxiv.org/abs/2412.15115. Rajasekaran, S., Ghobadi, M., and Akella, A. Cassini: network-aware job scheduling in machine learning clus- ters.In Proceedings of the 21st USENIX Sympo- sium on Networked Systems Design and Implementation, NSDI’24, USA, 2024. USENIX Association. ISBN 978- 1-939133-39-7. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, p. 3505–3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. URL https://doi-org.ezp-prod1.hul.harvard. edu/10.1145/3394486.3406703. Shah, A., Jangda, A., Li, B., Rocha, C., Hwang, C., Jose, J., Musuvathi, M., Saarikivi, O., Cheng, P., Zhou, Q., Dathathri, R., Maleki, S., and Yang, Z. Msccl++: Re- thinking gpu communication abstractions for cutting- edge ai applications, 2025. URLhttps://arxiv. org/abs/2504.09014. Xiong, H., Bian, J., Li, Y., Li, X., Du, M., Wang, S., Yin, D., and Helal, S. When search engine services meet large language models: Visions and challenges. IEEE Transactions on Services Computing, 17(6):4558–4577, 2024. doi: 10.1109/TSC.2024.3451185. Zhang, S., Zheng, N., Lin, H., Jiang, Z., Bao, W., Jiang, C., Hou, Q., Cui, W., Zheng, S., Chang, L.- W., Chen, Q., and Liu, X.Comet: Fine-grained computation-communication overlapping for mixture-of- experts, 2025a. Zhang, T., Hariri, M., Zhong, S., Chaudhary, V., Sui, Y., Hu, X., and Shrivastava, A. 70% size, 100% accuracy: Lossless LLM compression for efficient GPU inference via dynamic-length float (DFloat11). In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URLhttps://openreview.net/ forum?id=xdNAVP7TGy. Zhao, C., Zhou, S., Zhang, L., Deng, C., Xu, Z., Liu, Y., Yu, K., Li, J., and Zhao, L. Deepep: an efficient expert- parallel communication library.https://github. com/deepseek-ai/DeepEP, 2025. 10

Instant research discovery