Paper deep dive
EMA: Elastic and Performance Transparent Memory Across GPUs
Yi Xu, Tian Xia, Ion Stoica
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/24/2026, 4:37:55 AM
Summary
The paper introduces EMA, a memory sharing system for multi-GPU servers that enables elastic and performance-transparent memory sharing between GPUs. EMA addresses the mismatch between static GPU provisioning and dynamic memory demands of workloads like LLM inference by allowing GPUs to borrow and reclaim memory from directly connected peers via 'memory slices'. It ensures Local-Remote Symmetry (indistinguishable performance for borrowers) and Static-Dynamic Symmetry (guaranteed baseline performance for lenders), achieving up to 52% throughput improvement and 96% of the performance of a system with double capacity.
Entities (8)
Relation Signals (6)
EMA ā ensures ā Local-Remote Symmetry
confidence 95% Ā· EMA ensures performance transparency... we term this property LocalāRemote Symmetry
EMA ā ensures ā Static-Dynamic Symmetry
confidence 95% Ā· For lenders, borrowed resources remain reclaimable on demand... a property we call StaticāDynamic Symmetry
EMA ā implements ā Memory Slice
confidence 95% Ā· We introduce the memory slice abstraction... EMA, a prototype memory system... realizes elastic, performance-transparent memory sharing
EMA ā optimizes ā LLM Inference
confidence 90% Ā· We present EMA, a prototype memory system for LLM inference
EMA ā uses ā NVLink
confidence 90% Ā· Slices are established only across two GPUs with direct links... high-bandwidth interconnects like NVLink
Pie ā issimilarto ā EMA
confidence 80% Ā· Pie... facilitates memory capacity expansion... EMA... realizes elastic, performance-transparent memory sharing
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2X capacity, and maintains latency similar to the static local baseline.
Tags
Links
- Source: https://arxiv.org/abs/2609.27040v1
- Canonical: https://arxiv.org/abs/2609.27040v1
Trouble viewing inline? Open PDF directly ā
Full Text
79,006 characters extracted from source content.
Expand or collapse full text
EMA: Elastic and Performance Transparent Memory Across GPUs Yi Xu Affiliation: UC Berkeley Tian Xia Affiliation: UC Berkeley Ion Stoica Affiliation: UC Berkeley Abstract Multi-GPU servers have become the standard building block of modern data centers, providing aggregated capacity through high-bandwidth interconnects. At the same time, workloads such as LLM inference exhibit highly dynamic memory demands, which can cause one GPU to exhaust its local memory while others remain underutilized. This mismatch motivates a model of elastic resource sharing across GPUs. We present EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity. EMA ensures performance transparency for both borrowers and lenders. For borrowers, prefetching hides remote access costs so that applications experience remote and local memory as indistinguishable in performance. For lenders, borrowed resources remain reclaimable on demand, guaranteeing that performance never falls below that of static partitioning. While our design focuses on memory, the same principle naturally extends to other GPU resources. Our evaluation shows that EMA improves individual user throughput by up to 52%, achieves 96% of the throughput of a system provisioned with 2Ć2Ć capacity, and maintains latency similar to the static local baseline. 1 Introduction In recent years, AI and data analytics increasingly rely on GPU acceleration. Large language models (LLMs), video processing, and graph analytics now enable a wide range of applications, from natural language understanding and recommendation to scientific and real-time media analysis. These workloads are growing beyond what a single accelerator can handle, both in memory footprint and in computational demand. To meet this demand, modern deployments place multiple GPUs on the same server. Multi-GPU servers provide the resources needed to host massive models and sustain high throughput, while also enabling efficient intra-server communication through high-bandwidth interconnects like NVLink. Cloud providers typically expose these resources as fixed instance types with predefined bundles [30, 37, 7, 8, 9]. Despite the breadth of available configurations, many workloads exhibit strong time variation in resource demand. For example, in LLM inference, the KV cache grows with the number and length of concurrent queries, which creates bursts in memory pressure even when average demand is moderate. Under todayās model of static provisioning, a model running on one device can exhaust its local HBM while other GPUs in the same node remain partially idle. Because instance assignments are fixed after allocation [25, 18, 45], capacity that could alleviate pressure is stranded, which leads to underutilization for some users and shortages for others, as well as unnecessary operational cost. Figure 1: GPU Memory as an Elastic Resource This work pursues a simple idea: when a task runs out of memory on the local GPU, it should be able to borrow the available memory of other GPUs on the same machine. When the lender later needs that memory, it can reclaim it (Figure 1). The key is that such borrowing must be performance transparent for both the borrower and the lender. From the perspective of the borrowerās application, the borrowed memory is exposed as part of local memory, preserving the applicationās latency and throughput; we term this property LocalāRemote Symmetry. At the same time, the lender must be able to reclaim its full static allocation at any point, so its own application proceeds unaffected, a property we call StaticāDynamic Symmetry. Together, these symmetries enable GPUs to flexibly share capacity without introducing overhead. We introduce EMA, a prototype memory system for LLM inference that realizes elastic, performance-transparent memory sharing across GPUs in a node. EMA organizes memory available to share into memory slices. Our key insight is that a memory slice provides a uniform capacity abstraction across GPUs: for the application, allocations placed in a slice deliver the same performance regardless of whether they reside in local or remote memory. EMA forms a memory slice as a dedicated segment of memory shared exclusively between two peers, with each contributing a portion. Slices are established only across two GPUs with direct links. With a private link, the available bandwidth is dedicated to the pair. EMA sequences transfers across the two peers. It decides when data moves and how much is sent in each direction. This sequencing effectively divides the fixed bandwidth between the peers. As a result, the duration of each transfer becomes predictable. Leveraging knowledge of the application behavior, the data in the slices is prefetched to the accessing GPU before use, ensuring that remote and local memory accesses are indistinguishable in performance from the applicationās perspective. The size of a slice is determined by hardware and application configuration, while the share allocated to each peer is dynamically negotiated at runtime, allowing expansion by borrowing or contraction by reclaiming as needed. EMA runs a compute process on each GPU and a memory slice process on each slice. Memory slice processes handle address translation and prefetching. By aggregating multiple slices, they form an elastic, performance-transparent memory pool that enables each GPU to extend its usable capacity beyond local HBM by borrowing from directly connected peers. On real workloads, EMA improves throughput by up to 52% over static partitioning. It achieves 96% of the throughput of a system provisioned with 2Ć KV-cache capacity, while maintaining similar per-token latency. Although our focus is memory for LLM inference, the approach generalizes to other resources and workloads. Treating a multi-GPU node as a pool that can lend resources on demand points toward cross-GPU virtualization, where the aggregate capacity of a node is flexibly and transparently shared rather than statically partitioned. In summary, this paper makes the following contributions: ā¢ We identify the limitations of existing GPU resource management approaches and introduce new principles for memory sharing in multi-GPU environments. ā¢ We present EMA, the first system to enable performance-transparent memory sharing across GPUs. ā¢ We introduce the memory slice abstraction, which ensures that from the borrowerās perspective remote memory behaves as local, and from the lenderās perspective it is as if the borrowing had never occurred. ā¢ We perform a set of evaluations of EMA on real-world LLM workloads, demonstrating its effectiveness over existing solutions. 2 Background Next, we describe the inherent conflict between the fluctuating nature of GPU workloads and the static partitioning methods adopted by cloud providers. This mismatch leads to missed opportunities to optimize resource utilization and accommodate more workloads within the same hardware configuration. 2.1 Fluctuation of GPU Workloads GPU resource usage fluctuates across tasks due to workload dynamics. In neural network training, variations in data complexityāsuch as sequence lengths in NLP or input resolutions in vision, change memory and compute demands even at fixed batch size [41, 33]. Real-time simulations fluctuate with environment detail, physics, and user interactions while meeting strict latency constraints [34, 39]. Video rendering varies with scene complexity (objects, lighting, textures, effects), with adaptive resolution and tile-based rendering adjusting resource allocation [15, 43]. In particular, as a widely adopted solution across various applications [44, 22, 54], LLM inference also exhibits fluctuations in memory usage. Most of the GPU memory is allocated to store large model weights and the KV cache. While the size of the model weights remains constant, the memory footprint of the KV cache is dynamically allocated, growing and shrinking over time. We show the memory usage fluctuations of two inference engines in Figure 2. The memory usage increases significantly with a higher number of concurrent requests or longer output sequences, as each tokenās KV Cache consumes memory proportional to the model dimension. This can quickly exhaust the allocated GPU memory. In contrast, when a request completes processing, the associated KV cache memory is fully released, resulting in a noticeable drop in memory utilization. The magnitude of this drop varies depending on the total length of the processed request. These fluctuations in memory allocation and release make the memory utilization of LLM inference highly volatile and unpredictable. Figure 2: Memory Utilization Variation. We simulate memory usage for LLM inference with a maximum batch size of 256 on the ShareGPT workload. The blue and orange lines show per-user usage, while the green line shows aggregate usage across users. Aggregate usage exceeds total capacity far less often than static partitioning exceeds individual capacities. For simplicity, we assume that each user exclusively occupies a single GPU. In scenarios where multiple users statically share a GPU, the same analysis applies by treating their aggregated resource demand as a single composite user. Throughout the rest of this paper, we use the terms āuser,ā āmodel,ā and āGPUā interchangeably. 2.2 Inefficiency of Static Partition Modern data centers increasingly deploy multi-GPU nodes, where multiple high-performance GPUs (e.g., A100, H100) share a motherboard and are interconnected via high-bandwidth NVLink [37, 38, 36]. These nodes, common in cloud platforms such as AWS [48], Azure [5], and GCP [16], as well as in on-premises systems [26, 60], typically house 2 to 8 GPUs per node. Emerging rack-scale platforms push this idea much further: NVIDIAās GraceāBlackwell NVLink systems (GB200/GB300) wire entire racks into a single NVLink domain. A GB200 NVL72 integrates 36 Grace CPUs with 72 Blackwell GPUs in a rack, while the GB300 NVL72 similarly unifies 36 Grace CPUs with 72 Blackwell Ultra GPUs [7, 9, 8]. Such nodes enable in compute-intensive tasks like deep learning training, HPC simulations, and large-scale inference. For lighter workloads, users often rent GPU subsets in integral units on the same node [48], typically with a minimum of one GPU on most cloud platforms. These subsets are statically partitioned for the duration of a userās session. Although partitions may change when users leave and new ones arrive, they remain fixed during extended sharing periods, leading to several drawbacks. External Fragmentation. Although Azure offers fractional GPUs as an option [27], these instance types are restricted to a specific GPU model (A10) and require specialized drivers for proper functionality. Otherwise, users are required to allocate an integer multiple of GPUs, which reduces flexibility, particularly when the estimated resource requirements do not align with the capacity of a single GPU. For example, hosting models that require only 30 GB of memory (including weights and KV cache for maximum concurrency) on an A100 GPU with 40 GB of device memory result in 10 GB of external fragmentationāunused memory that remains inaccessible to the user. Internal Fragmentation. Even within the memory allocated to the user, the fluctuations discussed in Section 2.1 can still lead to internal fragmentation, leaving portions of memory unused and idle at times. In Figure 2, these internal fluctuations, occurring at varying rates, can lead to periods where statically allocated memory experiences shortages for 49.5%49.5\% of the time, while resources allocated to other users remain underutilized. However, we observed that the aggregated memory usage of the two users is more likely to stay within the total memory capacity for 27.5%27.5\% of the time, highlighting a potential improvement in resource sharing efficiency. Wasted Bandwidth. The NVLink interconnect within a node typically utilizes mesh topologies, enabling GPU-to-GPU bandwidths of hundreds of GB per second. For instance, NVLink 2.0 on V100 GPUs achieves over 300 GB/s, while A100 and H100 GPUs can exceed 600 GB/s and 900GB/s in certain configurations [37, 38]. More importantly, these connections exist between every arbitrary pair of GPUs, resulting in (n2) n2 dedicated links for a system with n GPUs. In GB200 NVL72 systems, NVLink Switch exposes all 72 GPUs as one logical accelerator. Each GPU can inject and receive up to about 900 GB/s into the fabric in each direction (ā 1.8 TB/s bidirectional), shared across communication with its peers. However, under static partitioning, the bandwidth between GPUs assigned to different users remains entirely idle, as the isolation abstraction provided by cloud environments ensures no communication across user boundaries. These unused interconnect resources present an opportunity to enable memory sharing across users. 2.3 Performance Transparent Swapping Accessing memory beyond the physical local HBM capacity poses significant challenges. This is due to the high overhead commonly associated with traditional swapping mechanisms. Pie [58], as a prototype implementation aimed at addressing this challenge, facilitates memory capacity expansion for LLM inference through performance-transparent swapping. Pie dynamically partitions KV caches into two parts, storing some layers on the GPUās HBM and others on the CPUās DRAM. Due to the sequential and cyclic access pattern across layers during LLM inference, Pie observes that a given layer will not be accessed again until all other layers have been accessed. Pie exploits this latency window between two consecutive accesses to the same layer to perform swapping: as long as the data is swapped back before the next access, the application does not perceive that a swap-out occurred. This effectively increases the available memory capacity without incurring performance overhead. By pipelining data transfer with computation, Pie overlaps compute latency with swapping latency, creating the illusion of a larger external memory that behaves with the same performance characteristics as the GPUās native HBM. The size of this transparent expanded memory is determined by the product of compute latency and interconnect bandwidth, which defines the maximum amount of data that can be transferred between two consecutive accesses to the same layer. The expansion ratio is defined as the ratio of total usable memory, consisting of performance-transparent remote memory and the local HBM portion, to the capacity of that local HBM portion. 3 A Path to Cross-User Memory Sharing To improve GPU usage efficiency, we advocate for memory sharing across users. We first define two objectives, staticādynamic symmetry and localāremote symmetry, which capture the guarantees required for performance-transparent sharing. We then articulate two principles that make these objectives achievable: applications must expose preemptable points and tolerate some level of information exchange. Static partitioning simplifies system design and guarantees predictable performance, but it cannot adapt to the dynamic nature of GPU workloads, leaving memory underutilized at times and insufficient at others. Sharing unused capacity across users can smooth out usage spikes and improve efficiency, as illustrated in Figure 2. Our goal is to achieve elastic and transparent sharing, which we formalize as a double-symmetry objective. ā¢ Static-Dynamic Symmetry: Each user must maintain performance that is at least equivalent to what they would achieve under a static allocation. ā¢ Local-Remote Symmetry: From a performance perspective, access to remote and local resources should be indistinguishable to users. To achieve these goals, cross-user memory sharing must follow certain principles. Preemptible Semantics. Any resources shared from one user must remain āpreemptible,ā meaning the user can reclaim those resources on demand. This ensures that each user retains their original static allocation as a guaranteed lower bound, even in shared environments. To enable this, applications must define clear preemption points - specific stages in the workload process where interruptions will not cause inconsistency and can safely allow for exit or recovery. The more frequent these preemption points, the smoother the resource sharing process becomes. Most GPU workloads inherently support this: LLM inference can be preempted at any time, training workloads can be interrupted between batches or epochs, video processing can pause between frames, and graph analytics can halt between algorithm iterations. Information Exchange. Applications must tolerate a certain level of information exchange regarding their resource requirements and operational characteristics. The access patterns of the resource recipient can influence the extent to which resources can be shared transparently, and the lending user might observe variability in resource availability as a result. Similarly, applications experiencing different levels of resource shortages may benefit unequallyāfor instance, a heavily memory-bound process gains significantly from additional memory, whereas a less memory-bound process does not. To enable efficient and fair resource allocation under this principle, applications must share key resource demands and operational characteristics. Figure 3: An Illustration of EMA. EMA leverages the fact that these principles naturally hold in LLM inference to realize the double-symmetry objective. Preemption is always feasible because queries are stateless and idempotent, allowing a user to reclaim previously borrowed memory by preempting requests from others that are using it. For information exchange, the system only needs to expose the latency window and changes in dynamic memory requirements over time. This exchange reveals limited information about workload behavior, and it is used solely for coordination within EMA rather than being broadly exposed across GPUs. Our prototype demonstrates the effectiveness of these principles in practice. As shown in Figure 3, the user memory space is divided into two parts: private memory and memory slices. We define private memory as the memory that is dedicated to one user, and memory slice segments as chunk of memory each user contributes for sharing. 4 Memory Slice: Sharing Between Two GPUs The memory slice abstraction serves as the foundation for cross-user memory sharing. Its structure ensures staticādynamic symmetry and localāremote symmetry: each GPU always retains at least the static portion of memory it contributed to the slice, and the slice delivers uniform performance to applications regardless of whether a given location resides in local or remote memory. Each memory slice is exclusively shared between two users that are directly connected via a high-speed interconnect such as NVLink, with the associated bandwidth dedicated solely to their use. Each slice is jointly reserved, granting both users the ability to use it as available and needed. Individual memory segments within the slice are assigned to a single user on demand and may be reassigned over time. When a segment is assigned, a virtual-to-physical mapping table ensures that only the assigned user can access it. Formally, we denote a memory slice using the indices of the GPUs that contribute to it. Suppose GPU i and GPU j are directly interconnected with bandwidth B. We define the shared memory slice as ā_ā(i,j) mem\_slice(i,j), which consists of two parts: ā_āiāj mem\_slice_iā j: the portion contributed by GPU iās HBM, and ā_ājāi mem\_slice_jā i: the portion contributed by GPU jās HBM. For example, ā_ā0ā3 mem\_slice_0ā 3 and ā_ā3ā0 mem\_slice_3ā 0 together form ā_ā(0,3) mem\_slice(0,3), as shown in Figure 3. Regardless of where the data physically reside, both GPU i and GPU j can store the data in ā_ā(i,j) mem\_slice(i,j) and access them with performance comparable to accessing their respective local HBM. Figure 4 shows an example of memory slice sharing between GPU i and GPU j. Figure 4: Memory and Bandwidth Sharing in a Memory Slice. The upper half shows physical GPU i and the lower half shows physical GPU j, which are directly interconnected with a certain bandwidth. The yellow region belongs logically to GPU i, and the blue region belongs logically to GPU j. Assume that ā_āiāj=miāi+miāj mem\_slice_iā j=m_i+m_ij, where miāim_i is the portion of memory actually utilized by GPU i, and miājm_ij is the portion utilized by GPU j. Similarly, let ā_ājāi=mjāi+mjāj mem\_slice_jā i=m_ji+m_j, where mjāim_ji is the portion utilized by GPU i, and mjājm_j is the portion utilized by GPU j. The unidirectional bandwidth from GPU i to GPU j is BiājB_iā j, which is composed of two components: Biāj=Biāji+BiājjB_iā j=B_iā j^i+B_iā j^j where BiājiB_iā j^i and BiājjB_iā j^j represent the bandwidth portion that GPU i and GPU j can utilize, respectively. Similarly, the unidirectional bandwidth from GPU j to GPU i is: Bjāi=Bjāi+BjāijB_jā i=B_jā i^i+B_jā i^j with BjāiiB_jā i^i and BjāijB_jā i^j defined in the same way. Assuming GPU i have a latency window LiL_i (see Section 2.3), and GPU j have a latency window LjL_j. To ensure that swapping remains transparent, the data transfer must complete within the available latency window. This requires the following conditions to hold: BjāiĆLiā„mjāi B_jā i^iĆ L_iā„ m_ji BiājĆLjā„miāj B_iā j^jĆ L_jā„ m_ij In order to swap in a certain amount of data, the local GPU must have sufficient memory space to accommodate incoming data. This implies that, in order to fully utilize available bandwidth, during the same time interval, the amount of data swapped in must equal the amount of data swapped out. As a result, the following constraints must also hold: BiājiĆLiā„mjāi B_iā j^iĆ L_iā„ m_ji BjāijĆLjā„miāj B_jā i^jĆ L_jā„ m_ij Given the bandwidth decomposition: Bjāi+Bjāij=Bjāi,Biāji+Biāj=BiājB_jā i^i+B_jā i^j=B_jā i, B_iā j^i+B_iā j^j=B_iā j We arrive at the following conclusion (full derivation provided in Appendix A): mjāiLi+miājLj=Bjāi=Biāj=B m_jiL_i+ m_ijL_j=B_jā i=B_iā j=B (1) The terms mjāim_ji and miājm_ij represent the physical memory capacity that one GPU has borrowed from the other. If mjāi=miājm_ji=m_ij, then the borrowing is effectively neutralized, as each GPU lends and borrows the same amount, making the sharing redundant. Meaningful sharing occurs only when there is a significant difference between mjāim_ji and miājm_ij. This raises the question of how much memory each GPU should borrow from the other. The decision must be guided by the double symmetry objectives: static-dynamic symmetry and local-remote symmetry. 4.1 Static-Dynamic Symmetry At any given time, each GPU is guaranteed having access to at least its original physical memory capacity, regardless of whether the allocated memory resides locally or remotely. For LLM inference, workloads can be preempted anytime, allowing borrowing and reclamation to happen immediately as needed. Reclamation operates on logical ownership: a GPU regains capacity by preempting regions that are logically owned by its peer, irrespective of whether those regions reside on its own device or on the peer. If GPU j preempts GPU i to reclaim k GB, then GPU j regains k GB from mjāim_ji or miāim_i. In theory, this allows mjājm_j or miājm_ij to increase by k. In practice, data at different locations may be dependent with each other. Suppose GPU i has an expansion ratio r, which means that its effective memory capacity is r times its local HBM. In the context of LLM inference, making memory expansion performance-transparent corresponds to each request placing m layers remotely and n layers locally. The ratio (m+n)/n(m+n)/n must match the expansion factor r (see Section 2.3). When GPU i releases k GB from the remote side, there remain (rā1)ā k(r-1)Ā· k GB of local data from the same request but different layers. These local entries are only useful if the corresponding remote KV cache is present; otherwise, retaining them wastes capacity. We therefore treat the (rā1)ā k(r-1)Ā· k GB as being in the dependency chain of the reclaimed k GB, and require GPU i to release it as well. If (rā1)ā k>k(r-1)Ā· k>k, the system releases more local memory than can be reborrowed remotely, resulting in wasted capacity that could have otherwise been used by GPU jās data. This excess memory must either be garbage collected into GPU iās private memory or left underutilized. On the other hand, if (rā1)ā k<k(r-1)Ā· k<k, then GPU i lacks sufficient local memory to support the full k GB increase in miājm_ij, leading to underutilized bandwidth because there is not enough memory capacity for data to fully saturate the transfer. Therefore, to avoid frequent garbage collection and to keep both memory and bandwidth fully utilized, we enforce the condition (rā1)ā k=k(r-1)Ā· k=k, which yields a fixed expansion ratio of r=2r=2. Because the expansion ratio is fixed at 2, we know that miāj=mjājm_ij=m_j and miāi=mjāim_i=m_ji, which leads to miāj+miāi=mjāj+mjāim_ij+m_i=m_j+m_ji, meaning that both GPUs contribute an equal amount of memory to the memory slice. Memory slice sizes are negotiated collaboratively, and the allocation ratio can adapt dynamically throughout the sharing period. Given an expansion ratio of 2 for both users, when GPU j reclaims k GB from GPU i, GPU i releases k/2k/2 GB on GPU j, and simultaneously releases k/2k/2 GB on its own local memory. This is because releasing k/2k/2 GB on GPU j corresponds to freeing half of the associated layers, making the corresponding k/2k/2 GB on GPU i no longer useful and therefore also released. This implies that, for a memory slice of total capacity ā_ā(i,j) mem\_slice(i,j), each GPU adjusts its memory usage by modifying its share within this unified address space. Building on Eq. (1), we generalize it to capture changing shares, where mvm_v can take any value in the range 0ā¤mvā¤mjāi+miāj0ā¤ m_vā¤ m_ji+m_ij: mjāiāmvLi+miāj+mvLjā¤B. m_ji-m_vL_i+ m_ij+m_vL_jā¤ B. mjāi+mvLi+miājāmvLjā¤B. m_ji+m_vL_i+ m_ij-m_vL_jā¤ B. We conclude that setting L=minā”(Li,Lj)L= (L_i,L_j) yields the maximum capacity of a memory slice, given by (see detailed proof in Appendix B): ā_ā(i,j)=2Ć(mjāi+miāj)=2āBā L mem\_slice(i,j)=2Ć(m_ji+m_ij)=2BĀ· L With this memory layout, satisfying the staticādynamic symmetry is straightforward. When a GPU requires additional memory, it first attempts to allocate from free memory. If the allocation succeeds, it proceeds with using the memory. If the allocation fails, the GPU checks whether it is currently using at least as much memory as its statically allocated capacity. If not, indicating that it previously lent underutilized memory to the other GPU (i.e., for GPU i, mjāi<miājm_ji<m_ij), it can reclaim the borrowed portion by preempting a number of requests on GPU j. This is done by asking the memory allocator to identify and evict appropriate requests of GPU j until GPU i has reclaimed enough memory to satisfy the allocation request, or the reclamation has made its usage achieve its static capacity (i.e., mjāi=miājm_ji=m_ij). 4.2 Local-Remote Symmetry Memory slices are designed for dynamic sharing between two users, based on their memory demands over time. As workloads progress, memory is allocated and deallocated as needed. This dynamic behavior leads to memory layouts where segments belonging to different users become interleaved within each slice. However, this interleaving poses a challenge. Previously, we derived the condition mjāi+miāj=Bā Lm_ji+m_ij=BĀ· L By static-dynamic symmetry, we have mjāi=miāim_ji=m_i, and substituting gives miāi+miāj=Bā Lm_i+m_ij=BĀ· L This shows that the capacity each GPU contributes to the slice (e.g., for GPU i, miāi+miājm_i+m_ij) is determined jointly by the effective bandwidth B and the latency windows LiL_i and LjL_j of both users. When data from different GPUs are interleaved within a memory slice, each swap can retrieve only a small fragment at a time. These small transfers often lead to lower effective bandwidth B, as memory copy overhead dominates. To preserve transparency in such cases, the system must either reduce the usage of remote memory for each GPU or improve interconnect bandwidth efficiency. To maintain high interconnect bandwidth efficiency, data copy operations must be performed in large chunks. EMA applies a slab allocator to keep each userās data physically contiguous. It then exploits applicationās predictable access patterns to group temporally adjacent data (e.g., consecutive layers) together. This way, even when data from different users are interleaved, they can still be transferred in relative large chunks with high bandwidth utilization. 4.2.1 Layer-major Allocation EMA uses a slab allocator when managing memory slice. Slabs are relatively large, contiguous memory chunks composed of tens to hundreds of KV Cache blocks. Similar to blocks, slab allocation is applied uniformly across all layers. Each slab is allocated exclusively to a user. When a task exhausts its allocated memory, it requests additional slabs from the slice. In contrast, when a task releases memory upon completing its operations, the freed slabs are returned to the slice for reallocation to either user. When performing reclamation to enforce staticādynamic symmetry, the memory allocator simply selects a slab belonging to the borrowing GPU that contains the fewest number of active requests. As shown in Figure 5(a), the memory slice (0,1) spans across the physical HBM of both GPU 0 and GPU 1. When computation begins, during allocation, each GPU gets allocated a number of slabs, with half of the layers residing on GPU 0 and others on GPU 1. During computation, each GPU ensures that the slabs of the upcoming layers are prefetched in time. This approach enables memory sharing between two GPUs while keeping each GPUās data relatively contiguous, but it must balance two opposing constraints: when workloads are highly dynamic, slab contiguity decreases, and in the worst case neighboring slabs always map to the other GPU, forcing swaps at single-slab granularity; because swapping now occurs one slab at a time, slabs must be large enough to fully utilize interconnect bandwidth. On the other hand, slabs cannot be too large or they suffer from internal fragmentation, transfer unused data during swapping, and reduce sharing efficiency. Since slabs are private to each GPU, any unused space cannot be allocated to the other GPU until the entire slab becomes free and reallocated as a whole, so if slabs are too large, sharing is rarely triggered. (a) Layer-major Allocation (b) Slab-major Allocation (c) Layer-group-major Allocation Figure 5: Memory Layout in Memory Slice. 4.2.2 Slab-major Allocation To address this challenge, EMA introduces the concept of a layer group, which refers to a set of layers that are logically consecutive. EMA ensures that these layers are also physically contiguous in memory. As a result, from both the logical and physical perspectives, a layer group behaves like a single large layer for the purpose of swap management. Current LLM inference engines do not ensure physical contiguity across layers. As shown in Figure 5(a), memory is contiguous only within a layer, where slab i and slab i+1i+1 are consecutive, while logically consecutive layers are often not. Since intra-layer contiguity is a redundant property of layer-major allocation, EMA adopts a slab-major strategy to make logically consecutive layers physically contiguous by placing all layers of a slab together. We maintain the same logical-to-physical layer mapping, meaning that at any point in time, half of the layers are physically located on the local GPU and the other half on the remote GPU. To achieve this, we allocate the first half of the layers in a slab contiguously on the local GPU, and the second half contiguously on the remote GPU. As a result, when a group of layers is swapped, the layers involved are guaranteed to be physically contiguous within each slab, enabling efficient bulk transfers. Under this strategy, the number of swaps issued for each model is determined deterministically by the number of slabs allocated to it. With the simplified abstraction of treating each layer group as a ābig layer,ā EMA applies the same swapping logic as before. The key difference is that, instead of using actual physical and logical layer indices, EMA operates on layer group indices. For example, as shown in Figure 5(b), under a layer group size of 2, physical layer group 0ā2 corresponds to physical layers 0ā5, while logical layer group 2 corresponds to logical layers 4 and 5. At each layer group boundary, EMA checks whether swapping should be triggered. Since each swap is performed at the granularity of a layer group, and each scheduling step occurs after the computation of an entire layer group, the latency ratio between swapping and computation remains unchanged. As a result, the performance transparency of the original design is preserved. This design has two sources of overhead. First, each swap requires a buffer equal to the transferred chunk, which was one layer per model but becomes one layer group with grouping. Second, the effective latency window of LLM inference is proportional to (kā1)/k(k-1)/k, where k is the number of layers [58]; with larger groups, k can drop to 10 or fewer, making the window slightly smaller. Because the expansion size is the product of the latency window and the bandwidth, this reduction directly limits performance-transparent expansion. Thus, while larger groups improve transfer efficiency, they also increase overhead, so group size must be carefully bounded. 4.2.3 Layer-group-major Allocation Slab-major allocation provides the guarantee that swapping occurs at the granularity of a single layerās single slab size multiplied by the layer group size. However, it enforces contiguity in an unnecessary dimension: it places each layer group contiguously next to the next one. This is inefficient, as two layer groups are never swapped together. EMA transforms the memory layout to take advantage of contiguity in a best-effort manner. Specifically, for slab i and layer group j, instead of placing slab i, layer group j+1j+1 adjacent to it, we place slab i+1i+1, layer group j next to it. We refer to this allocation strategy as layer-group-major. For example, as shown in Figure 5(c), for GPU 0, slab 0 of logical layers 2 and 3 is placed contiguously with slab 1 of those same layers. This layout enables the swap of slab 0 and slab 1 to be combined into a single transfer. Under this hierarchical layout, memory contiguity is exploited in two dimensions. Within each layer group, swapping occurs at least at the granularity of a full group. Across the slice, EMA allocates slabs for each model as contiguously as possible. If one slab holds a full group of size kāSkS (k: group size, S: per-layer slab size) and each layer spans m slabs, the effective swap size ranges from kāSkS to mākāSmkS. This provides a significant improvement over layer-major allocation, where the swapping size ranges from S to māSmS, and slab-major allocation, where the swapping size is fixed at kāSkS. Because transfers occur in large contiguous chunks, this memory layout is able to drive the interconnect close to full utilization in practice. 5 Multi-GPU: Aggregating Memory Slices The memory slice serves as the basic unit of sharing. Multiple memory slices can coexist independently on a single GPU, since each GPU typically has multiple independent interconnects with different peers. As a result, in multi-GPU sharing scenarios, each memory slice operates independently and does not interfere with others. 5.1 Process Address Space EMA assumes one compute process per GPU, responsible for reading from local HBM and executing computations on model weights, KV cache, and ephemeral data. If a GPU is directly interconnected with n peers willing to participate in memory sharing, it spawns n memory-slice processes, one for each peer. Each compute process is provided with an Elastic Address Space (EAS) abstraction that extends beyond its guaranteed Local Address Space (LAS). The LAS corresponds to the fixed capacity of GPU-local memory, while the EAS includes reclaimable slices contributed by peers. The availability of EAS memory depends on system load and sharing state, so its capacity is not guaranteed. This abstraction allows processes to allocate working sets larger than their LAS while maintaining the ability to fall back on guaranteed local memory. Each memory-slice process manages its assigned slice, consisting of equal portions from the two participating GPUs, and exposes it as a shared region accessible to both compute processes. Ownership within the slice is logically partitioned: for example, in Figure 6, data in light green are owned by process 0 but physically distributed across GPU 0 and GPU 1, while data in dark green are owned by process 1 with a similarly distributed layout. A memory-slice process also handles address translation on behalf of its compute process. When an allocation falls within its range, the slice process issues a virtual address, predicts upcoming accesses, allocates a physical location, prepares a directly accessible address, and prefetches the data ahead of time. Thus, when the compute process accesses the memory, it uses the prepared address without additional latency. For objects spanning multiple slices, access proceeds once all relevant slice processes complete preparation; due to the performance-transparent design, this coordination occurs in advance and does not add to the access latency. Figure 6: Multiple Memory Slice in Multi-GPU Sharing 5.2 Scheduling A request can arrive at any time and may be scheduled to use any memory slice. Scheduling is not independent: assigning one request to a slice changes how much memory remains available for other processes. For example, if a request on process 0 is scheduled to slice (0,1), then process 1 will have less memory available, while processes 2 and 3 will have more. Intuitively, scheduling to slice (0,1) is preferable if process 1 is unlikely to use that memory and processes 2 and 3 are expected to need it. In EMA, the key is to avoid driving any slice to full utilization. When a slice reaches 100%, preemptions occur for two reasons: (1) if the scheduling process already occupies more than 50%, its own ongoing requests are preempted because capacity is exhausted; (2) if it occupies less than 50%, staticādynamic symmetry forces reclamation, preempting the peerās requests. Thus, good scheduling should minimize the likelihood that utilk,i+utilk,j=100%util_k,i+util_k,j=100\%. Formally, suppose process i has n slices. For each slice k, let utilk,iutil_k,i and utilk,jutil_k,j denote the utilizations of process i and its peer j. By construction, utilk,i+utilk,jā¤100%util_k,i+util_k,jā¤ 100\%, and preemption begins once this sum reaches 100%. We formulate scheduling as an edge-constrained, unknown-length load-balancing problem. To avoid preemption, we adopt a local greedy algorithm that assigns each request to the slice k with the smallest utilk,i+utilk,jutil_k,i+util_k,j. As shown in Appendix C, this algorithm achieves provably good performance bounds. 5.3 Bursty Workload Preemption Control The staticādynamic symmetry can cause excessive preemptions. For example, if user A temporarily exceeds its static capacity, user B may reclaim memory, only for A to use it again when B releases, and then be preempted once more as Bās demand rises. This back-and-forth can repeat endlessly. To prevent such cycles, EMA caps allocations after a preemption: if a user is preempted when its usage exceeds 50%, the slice limits that userās future allocations to the preempted level for a sliding window of length T. If another preemption occurs within that window at a lower level k1<k0k_1<k_0, the cap resets to k1%k_1\%; if no further preemptions occur, the cap is lifted once T expires. This mechanism reduces disruption under dynamic workloads and high memory pressure, while still allowing aggressive sharing when preemptions are unlikely. 6 Evaluation In this section, we answer the following questions: (a) LLaMA13B - Alpaca on 4ĆA100-40GB (b) LLaMA13B - ShareGPT on 4ĆA100-40GB (c) OPT30B - Alpaca on 8ĆA100-80GB (d) OPT30B - ShareGPT on 8ĆA100-80GB Figure 7: Throughput vs. Request Rate on Heavily-Loaded GPUs 1. Throughput Gains: How much throughput improvement can EMA provide compared to a state-of-the-art inference system with statically partitioned resources under different load? (Ā§6.2) 2. Memory Utilization: How much does EMA improve memory utilization, and how do these improvements affect overall system throughput? (Ā§6.3) 3. Capacity Expansion and Latency Comparison: How much performance transparent memory is EMA able to provide? How does the latency of EMA compare to the static baseline? (Ā§6.4) 4. LocalāRemote Symmetry: How important is local-remote symmetry for memory sharing, and how much performance improvement does this property provide? (Ā§6.5) 5. StaticāDynamic Symmetry: How important is static-dynamic symmetry for memory sharing? How much improvement does it offer globally and for each GPU? (Ā§6.6) 6. Memory Slice Study: How to achieve high performance with the same slice size? (Ā§6.7) 6.1 Experiment Setup Hardware Configuration. Each user is assigned a single GPU operating independently. The experiments are carried out on NVIDIA servers with 4ĆA100-40GB and 8ĆA100-80GB GPUs, where the suffix denotes the local HBM capacity. Within each node, all GPU pairs are interconnected via NVLink, offering up to 600 GB/s bidirectional bandwidth (300 GB/s per direction under concurrent traffic). Experiment Setup. Each GPU runs one model independently. We evaluate ShareGPT [49] and Alpaca [52] on LLaMA [53] and OPT [63], both consisting of real LLM service texts. ShareGPT includes longer input and output sequences than Alpaca. Request arrivals follow a Poisson distribution with varying rates, and all datasets are tokenized as in prior work [29, 58]. We compare the performance of the following systems: ā¢ EMA: We build EMA on top of vLLM [29]. Unless otherwise specified, we allow each GPU to contribute its full available KV cache capacity to its memory slices. For example, on an A100-40GB running Llama-13B, the model weights occupy approximately 26GB of HBM, and we reserve 1GB for ephemeral data usage, leaving up to 13GB for KV cache. In a 4-GPU setup, each GPU has 3 peers and can divide its 13 GB contribution equally across them, i.e., 13āGBĆ·3ā4.3āGB13\,GB 3ā 4.3\,GB per slice, resulting in a potential expanded capacity of 13 GB for each GPU. For all experiments except those in Section 6.4 on latencyāexpansion, we maintain HBM utilization at 90%. ā¢ vLLM: Each GPU uses its full memory and independently hosts a vLLM instance. ā¢ Pie: The system was originally designed to offload the KV cache to the CPU. We modified it so that one GPU runs the compute process, while the other GPUs on the same node remain idle and serve purely as passive memory devices. This configuration represents an upper bound of EMA, as the compute GPU is guaranteed access to 2Ć its local capacity for the KV cache. 6.2 Throughput Gains In this section, we discuss how much throughput improvement can EMA provide compared to the static version backed by vLLM on each GPU. We fix the request rates for models 0ā6 at 10 for Alpaca and 0.5 for ShareGPT, while gradually increasing the request rate to model 7. We report individual performance of the heavily loaded model in Figure 7. At lower request rates, EMA shows virtually no improvement over vLLM, since the workload is less memory-bound and rarely encounters memory insufficiency. As the request rate increases, however, the improvement becomes significant, exceeding 50% for both LLaMA and OPT models on Alpaca workloads at request rates of 30 and 20, respectively. Beyond these points, the gains flatten as performance becomes compute-bound. For ShareGPT, which is more memory-intensive due to longer output lengths, the improvements continue to grow steadily with higher request rates. Across all settings, EMA achieves 96% of the performance of Pie. Pie has equal access to remote and local memory, making it equivalent to vLLM with a 2Ć larger KV cache. This provides an optimal baseline: memory borrowing is maximized in Pie, where only the heavily loaded GPU consumes memory while all other devices remain free. By contrast, EMA delivers comparable performance despite having no guarantee of access to remote memory. This observation raises a key question: why does the presence of lightly loaded models have no measurable effect on the performance gains of their more heavily loaded counterparts? Figure 8: Throughput vs. Request Rate on Lightly-Loaded GPUs We analyze how the improvement changes when the load on lightly loaded GPUs (vLLM and EMA-light) is varied, as shown in Figure 8. In this setup, the heavily loaded GPU receives a fixed request rate of 30 for Alpaca and 2.5 for ShareGPT (we report only LLaMA results due to space limitations; OPT follows the same trend). Because the request rate of the heavily loaded model does not change, its memory and compute intensity remain constant throughout the workload. Consequently, vLLM-heavy performance remains unchanged, and EMA-heavy improvement depends only on the effectiveness of memory sharing. For Alpaca, the load on lightly loaded models has no impact on the heavily loaded GPU, even when the lightly loaded model itself becomes memory-bound (e.g., at ratio 1:2 with request rate 15). The improvement remains above 50% across ratios from 1:6 to 1:2. A reduction is observed only when the ratio reaches 1:1, where all GPUs operate under the same load, causing the improvement to drop to 20%. For ShareGPT, the heavily loaded GPU is more sensitive to the load on lightly loaded models. Its improvement begins to decrease at a ratio of 1:3. In addition, the lightly loaded models achieve smaller or no improvement. These results indicate that memory sharing improves performance on both lightly and heavily loaded GPUs, and the gains are not achieved at the expense of other devices. Sharing is more effective when workloads have moderate memory requirements, as opportunities diminish when all devices operate under high memory pressure. Furthermore, because sharing is performed at request granularity, longer contexts require larger memory chunks, which are difficult to allocate under high memory usage. As a result, when memory utilization is high, sharing is more effective for shorter requests. 6.3 Memory Utilization and Performance Breakdown In this section, we analyze where EMAās performance gains come from. We measure per-GPU memory utilization and throughput on a node with four A100-40GB GPUs, where each GPU hosts a LLaMA-13B model. Alpaca requests are issued to the four GPUs at request rates of 40, 30, 20, and 10 requests/s, respectively. The results are shown in Figure 9. From this experiment, we have three key findings. First, borrowing memory has little to no impact on the lenderās performance, which shows the effectiveness of staticādynamic symmetry. Second, we show that the throughput improvement results directly from an increase in effective memory capacity. Among the peers within a node, more heavily loaded GPUs are able to borrow additional memory from underutilized ones. In some cases, a GPUās memory utilization can reach up to 200%, where 100% corresponds to its physical HBM capacity and the additional 100% comes from memory borrowed from its peers. This redistribution leads to more efficient memory allocation across the system: heavily loaded GPUs gain access to more memory, while lightly loaded GPUs are limited to only what they need. Third, given the dynamic nature of the system, even GPUs that are heavily loaded may experience periods of lower memory utilization. During these moments, other peers are allowed to temporarily use the available memory. As a result, even lightly loaded GPUs can benefit and achieve better performance compared to a baseline where each GPU statically owns and uses only its local memory. On GPU 0, GPU 1 and GPU 2, EMA achieves improvements of 1.3Ć, 1.3Ć and 1.2Ć. GPU 3 remains 0.95Ć of the static version performance. The 5% overhead mostly comes from memory allocation. (a) Per GPU Memory Utilization (b) Per GPU Throughput Figure 9: Higher Memory Utilization, Higher Throughput 6.4 Latency Comparison Remote memory sharing is traditionally known to trade off latency for higher throughput. However, EMA achieves higher throughput without compromising latency by ensuring localāremote symmetry. In this section, we compare vLLM and EMA under the same amount of on-device HBM. As shown in Figure 10, EMA provides an Elastic Address Space (EAS) that doubles the available on-device KV-cache capacity. With 4ĆA100-40GB GPUs running the 13B model, each GPU reaches up to 26 GB, and with 8ĆA100-80GB GPUs running the 30B model, each GPU reaches 40 GB. (The corresponding model sizes are 26.6 GB and 58.8 GB.) We find that EMA maintains localāremote symmetry by sustaining nearly the same compute latency while varying the expanded memory size. With large expansions (e.g., 20 GB on A100-80GB), EMA shows a 10% increase in latency. This is primarily due to larger batch sizes enabled by the expanded capacity, which increase per-iteration compute time, and DMA synchronization and coordination overhead between the memory slice processes. Even so, the added capacity to EAS still improves overall throughput. (a) LLaMA13B - ShareGPT on 4ĆA100-40GB (b) OPT30B - ShareGPT on 8ĆA100-80GB Figure 10: Performance Comparison with Larger Cache Size 6.5 LocalāRemote Symmetry In this section, we evaluate the importance of localāremote symmetry in memory sharing and quantify the performance improvement this property provides. We implement a variant of EMA that forgoes localāremote symmetry. Rather than prefetching remote data ahead of use, it performs on-demand swapping of an entire KV-cache layer only when accessed. Compared to HUVM [32], this variant offers a stronger design under a GPU-only memory setting. While HUVM services page faults incrementally at the granularity of individual pages, our variant transfers the full layer in a single operation, allowing one large memcpy to eliminate the substantial per-page overhead of on-demand paging. From Figure 11, we find that local-remote symmetry is important in both throughput and latency, as computation must stall whenever accessed data resides on remote memory and must be swapped in on the critical path. (a) LLaMA-13B - Alpaca Throughput (b) LLaMA-13B - Alpaca Per Token Compute Latency Figure 11: LocalāRemote Symmetry 6.6 StaticāDynamic Symmetry EMA ensures that each GPU, when underutilized, contributes its free memory to memory slices for sharing. However, when it requires any of the contributed memory back, it must be able to logically reclaim it immediately. This subsection evaluates the performance impact of enforcing the staticādynamic symmetry guarantee. We compare the performance of EMA with and without this guarantee. Specifically, we report the individual GPU throughput, the number of preemptions caused by out-of-memory conditions, and the number of preemptions triggered by memory reclamation. We find that with an effective reclamation strategy, EMA with staticādynamic symmetry achieves aggregate throughput within ā5% to +10% relative to the version without it, while additionally providing the symmetry guarantee. (a) LLaMA-13B - ShareGPT Throughput (b) LLaMA-13B - ShareGPT Number of Preemptions Figure 12: Static-Dynamic Symmetry 6.7 Maximizing Performance per Slice EMA relies on the abstraction of memory slices. Before the system becomes compute-bound, increasing the combined capacity of memory slices always leads to higher performance. In this section, we study how to configure a memory slice to achieve higher performance with the same capacity. Figure 13 shows the throughput of OPT models evaluated on ShareGPT. (a) OPT-13B (b) OPT-30B Figure 13: Throughput under Different Group and Slab Size We fix the layer group size, and vary the slab size from 1 to 100. We find that when the slab size is small, EMA frequently stalls at each layer boundary, resulting in higher per-token latency and reduced throughput. On the other hand, with a large slab size, slab utilization and the opportunity for memory sharing are significantly compromised. This reduces the effective memory available to each model and ultimately lowers throughput. We also study the impact of layer group size by varying it from 1 to 4. A smaller group size leads to performance degradation for the same reason as smaller slab size: the smaller granularity of data transfer results in lower effective bandwidth and increased waiting time. However, a group size that is too large leads to memory inefficiency, as it requires larger swap-in/out buffers. The buffer size must accommodate one entire group of layers in flight, which increases memory overhead and reduces available space for actual usage. Therefore, we find that a group size of 4 and a slab size of 10 usually yield consistently good performance across workloads and models, and we set these as the default values. 7 Related Work General Model Serving Systems. Prior work aims to improve throughput, latency, and elastic scaling for inference pipelines [31, 11, 40, 50, 10, 62, 61]. Transformer stacks such as FasterTransformer, LightSeq, TurboTransformers, DeepSpeed-Inference, and Accelerate optimize kernels, scheduling, and batching to increase single-node efficiency [35, 57, 14, 2, 21]. These systems treat GPU memory as a fixed resource, focusing on how it is used rather than how much is available. In practice, capacity pressure, especially from LLM KV caches, still forces queuing, preemption, or model downsizing, motivating mechanisms that expand usable memory rather than only improve scheduling. Recomputation and Offloading/Swapping to CPU. To stretch capacity without new hardware, many approaches trade extra compute or I/O for memory headroom. Recomputation and checkpointing reduce peak activation footprint during training [6, 19, 28, 23, 12, 13], while swapping pipelines move tensors between GPU and host or NVMe [1, 51, 29, 46, 20, 47, 56, 42, 59]. All of these are fundamentally reactive: data is brought back only when an access occurs. This makes expanded capacity helpful but not transparentāapplications observe a clear performance delta between local HBM and slower tiers. Swapping/Sharing Between GPUs. A complementary direction borrows memory from peer GPUs. HUVM demonstrates that idle HBM on neighboring devices can be harvested over high-bandwidth interconnects to expand effective capacity [32], and object-aware managers further explore multi-GPU placement and migration policies [24]. AQUA [55] enables preemptive prompt scheduling by offloading KV cache to peer GPUs over NVLink instead of host DRAM, using a profiler to classify GPUs as memory producers or consumers. Harvest [17] treats unused peer HBM as an opportunistic cache tier for MoE expert weights and KV cache entries, retrieving them over GPU-to-GPU interconnects on demand. However, these designs still move data reactively, whether triggered by scheduling-driven page-ins or cache misses, so remote HBM operates as a faster-than-CPU but still observably slower tier rather than a truly uniform extension of local memory. By contrast, EMA treats peer memory as a near tier that can be made performance-transparent during compute by proactively prefetching data ahead of use, and additionally guarantees that lenders can reclaim shared memory on demand without performance loss. 8 Conclusion In conclusion, we present the objectives and principles of cross-GPU memory sharing and introduce EMA. By exploiting predictable memory access patterns, the high bandwidth of modern interconnects, and real-time information, EMA optimizes memory utilization in unbalanced and highly dynamic environments. It allows each user to transparently access peersā memory with performance indistinguishable from local memory, thereby achieving higher throughput without sacrificing latency. References [1] R. Abhyankar, Z. He, V. Srivatsa, H. Zhang, and Y. Zhang InferCept: efficient intercept support for augmented large language model inference. In Forty-first International Conference on Machine Learning, Cited by: Ā§7. [2] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, et al. (2022) Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1ā15. Cited by: Ā§7. [3] Y. Azar and A. Epstein (2005) Convex programming for scheduling unrelated parallel machines. In Proceedings of the thirty-seventh annual ACM symposium on Theory of computing, p. 331ā337. Cited by: Appendix C. [4] Y. Azar and L. Epstein (1997) On-line load balancing of temporary tasks on identical machines. In Proceedings of the Fifth Israeli Symposium on Theory of Computing and Systems, p. 119ā125. Cited by: Appendix C. [5] M. Azure (2025) Microsoft azure cloud services. Note: https://azure.microsoft.com Cited by: Ā§2.2. [6] T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016) Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Cited by: Ā§7. [7] N. Corporation (2025) GB200 nvl72. Note: https://w.nvidia.com/en-us/data-center/gb200-nvl72/Accessed: 2025-09-01 Cited by: Ā§1, Ā§2.2. [8] N. Corporation (2025) NVIDIA gb300 nvl72. Note: https://w.nvidia.com/en-us/data-center/gb300-nvl72/Accessed: 2025-09-01 Cited by: Ā§1, Ā§2.2. [9] N. Corporation (2025) NVLink & nvswitch. Note: https://w.nvidia.com/en-us/data-center/nvlink/Accessed: 2025-09-01 Cited by: Ā§1, Ā§2.2. [10] D. Crankshaw, G. Sela, X. Mo, C. Zumar, I. Stoica, J. Gonzalez, and A. Tumanov (2020) InferLine: latency-aware provisioning and scaling for prediction serving pipelines. In Proceedings of the 11th ACM Symposium on Cloud Computing, p. 477ā491. Cited by: Ā§7. [11] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica (2017) Clipper: a \low-latency\ online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), p. 613ā627. Cited by: Ā§7. [12] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. RĆ© (2022) Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35, p. 16344ā16359. Cited by: Ā§7. [13] T. Dao (2023) Flashattention-2: faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. Cited by: Ā§7. [14] J. Fang, Y. Yu, C. Zhao, and J. Zhou (2021) Turbotransformers: an efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, p. 389ā402. Cited by: Ā§7. [15] E. Games (2025) Unreal engine. Note: https://w.unrealengine.com Cited by: Ā§2.1. [16] Google (2025) Google cloud platform. Note: https://cloud.google.com/Accessed: 2025-01-16 Cited by: Ā§2.2. [17] N. Gopal and K. Kaffes (2026) Harvest: opportunistic peer-to-peer gpu caching for llm inference. arXiv preprint arXiv:2602.00328. Cited by: Ā§7. [18] T. Griggs, X. Liu, J. Yu, D. Kim, W. Chiang, A. Cheung, and I. Stoica (2024) M\ āelange: cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527. Cited by: Ā§1. [19] J. Herrmann, O. Beaumont, L. Eyraud-Dubois, J. Hermann, A. Joly, and A. Shilova (2019) Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory. arXiv preprint arXiv:1911.13214. Cited by: Ā§7. [20] C. Huang, G. Jin, and J. Li (2020) Swapadvisor: pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, p. 1341ā1355. Cited by: Ā§7. [21] HuggingFace (2024) Hugging face accelerate. Note: https://huggingface.co/docs/accelerate/indexAccessed: 2024-06-13 Cited by: Ā§7. [22] InData Labs (2023) Best applications of large language models. Note: https://indatalabs.com/blog/large-language-model-appsAccessed: 2025-01-16 Cited by: Ā§2.1. [23] P. Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, J. Gonzalez, K. Keutzer, and I. Stoica (2020) Checkmate: breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems 2, p. 497ā511. Cited by: Ā§7. [24] Y. Jiang, H. Wu, J. Zhao, and Y. Chen (2025) OASIS: object-aware multi-gpu memory management. In Proceedings of the IEEE International Symposium on High-Performance Computer Architecture (HPCA), Cited by: Ā§7. [25] Y. Jiang, F. Fu, X. Yao, G. He, X. Miao, A. Klimovic, B. Cui, B. Yuan, and E. Yoneki (2025) Demystifying cost-efficiency in llm serving over heterogeneous gpus. arXiv preprint arXiv:2502.00722. Cited by: Ā§1. [26] A. Kalra and Y. Moukhtar (2024) Comparative analysis of on-premises and cloud hosting solutions. Cited by: Ā§2.2. [27] V. Kanchanahalli (2020) Power your azure gpu workstations with flexible gpu partitioning. Note: Microsoft Azure BlogAccessed: 2025-01-16 Cited by: Ā§2.2. [28] V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro (2023) Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5, p. 341ā353. Cited by: Ā§7. [29] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p. 611ā626. Cited by: 1st item, Ā§6.1, Ā§7. [30] A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. Tallent, and K. Barker (2019) Evaluating modern gpu interconnect: pcie, nvlink, nv-sli, nvswitch and gpudirect. arXiv preprint arXiv:1903.04611. Cited by: Ā§1. [31] Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, et al. (2023) \alpaserve\: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), p. 663ā679. Cited by: Ā§7. [32] Y. Miao, Z. Qiu, J. Zhang, S. Jiang, et al. (2022) HUVM: harvesting unused gpu memory. In USENIX Annual Technical Conference (ATC), Cited by: Ā§6.5, Ā§7. [33] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. (2018) Mixed precision training. arXiv preprint arXiv:1710.03740. Cited by: Ā§2.1. [34] B. Mirtich (1996) Impulse-based dynamic simulation of rigid body systems. PhD thesis, University of California, Berkeley. Cited by: Ā§2.1. [35] NVIDIA (2024) FasterTransformer. Note: https://github.com/NVIDIA/FasterTransformerAccessed: 2024-06-13 Cited by: Ā§7. [36] NVIDIA (2025) NVIDIA a100 tensor core gpu. Note: https://w.nvidia.com/en-us/data-center/a100/ Cited by: Ā§2.2. [37] NVIDIA (2025) NVIDIA dgx a100. Note: https://w.nvidia.com/en-us/data-center/dgx-a100/ Cited by: Ā§1, Ā§2.2, Ā§2.2. [38] NVIDIA (2025) NVIDIA h100 tensor core gpu architecture. Note: https://w.nvidia.com/en-us/data-center/h100/ Cited by: Ā§2.2, Ā§2.2. [39] NVIDIA (2025) NVIDIA physx. Note: https://developer.nvidia.com/physx-sdk Cited by: Ā§2.1. [40] C. Olston, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V. Rajashekhar, S. Ramesh, and J. Soyke (2017) Tensorflow-serving: flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139. Cited by: Ā§7. [41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: Ā§2.1. [42] X. Peng, X. Shi, H. Dai, H. Jin, W. Ma, Q. Xiong, F. Yang, and X. Qian (2020) Capuchin: tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, p. 891ā905. Cited by: Ā§7. [43] M. Pharr, W. Jakob, and G. Humphreys (2016) Physically based rendering: from theory to implementation. Morgan Kaufmann. Cited by: Ā§2.1. [44] PixelPlex (2023) Top 10 real-life applications of large language models. Note: https://pixelplex.io/blog/llm-applications/Accessed: 2025-01-16 Cited by: Ā§2.1. [45] A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing (2021) Pollux: co-adaptive cluster scheduling for goodput-optimized deep learning. In 15th \USENIX\ Symposium on Operating Systems Design and Implementation (\OSDI\ 21), Cited by: Ā§1. [46] J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He (2021) \zero-Offload\: democratizing \billion-scale\ model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), p. 551ā564. Cited by: Ā§7. [47] M. Rhu, N. Gimelshein, J. Clemons, A. Zulfiqar, and S. W. Keckler (2016) VDNN: virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), p. 1ā13. Cited by: Ā§7. [48] A. W. Services (2025) AWS cloud services. Note: https://aws.amazon.com Cited by: Ā§2.2, Ā§2.2. [49] ShareGPT Team (2023) ShareGPT. External Links: Link Cited by: Ā§6.1. [50] H. Shen, L. Chen, Y. Jin, L. Zhao, B. Kong, M. Philipose, A. Krishnamurthy, and R. Sundaram (2019) Nexus: a gpu cluster engine for accelerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, p. 322ā337. Cited by: Ā§7. [51] Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. RĆ©, I. Stoica, and C. Zhang (2023) Flexgen: high-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, p. 31094ā31116. Cited by: Ā§7. [52] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023) Stanford alpaca: an instruction-following llama model. Cited by: Ā§6.1. [53] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. RoziĆØre, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: Ā§6.1. [54] UbiOps (2023) Top 6 current llm applications and use cases. Note: https://ubiops.com/llm-use-cases/Accessed: 2025-01-16 Cited by: Ā§2.1. [55] A. Vijaya Kumar, G. Antichi, and R. Singh (2025) AQUA: network-accelerated memory offloading for LLMs in scale-up GPU domains. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), p. 48ā62. Cited by: Ā§7. [56] L. Wang, J. Ye, Y. Zhao, W. Wu, A. Li, S. L. Song, Z. Xu, and T. Kraska (2018) Superneurons: dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming, p. 41ā53. Cited by: Ā§7. [57] X. Wang, Y. Xiong, Y. Wei, M. Wang, and L. Li (2020) LightSeq: a high performance inference library for transformers. arXiv preprint arXiv:2010.13887. Cited by: Ā§7. [58] Y. Xu, Z. Mao, X. Mo, S. Liu, and I. Stoica (2024) Pie: pooling cpu memory for llm inference. arXiv preprint. External Links: 2411.09317, Link Cited by: Ā§2.3, Ā§4.2.2, Ā§6.1. [59] H. Yang, J. Zhou, Y. Fu, X. Wang, R. Roane, H. Guan, and T. Liu (2024) ProTrain: efficient llm training via memory-aware techniques. arXiv preprint arXiv:2406.08334. Cited by: Ā§7. [60] S. Younus, K. Kumar, I. Ali, and A. A. Laghari (2024) Systematic analysis of on-premise and cloud services. International Journal of Cloud Computing. Note: Accessed: 2025-01-16 External Links: Link Cited by: Ā§2.2. [61] G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022) Orca: a distributed serving system for \transformer-based\ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), p. 521ā538. Cited by: Ā§7. [62] H. Zhang, Y. Tang, A. Khandelwal, and I. Stoica (2023) \shepherd\: Serving \dnns\ in the wild. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), p. 787ā808. Cited by: Ā§7. [63] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. (2022) Opt: open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. Cited by: Ā§6.1. Appendix Appendix A Proof of Capacity, Latency and Bandwidth Relationships To prove that the capacity requirement in a memory slice must be satisfied to enable localāremote symmetry in memory sharing, we begin with the following four inequalities: Bjāiā Li B_jā i^iĀ· L_i >mjāi >m_ji (1) Biājā Lj B_iā j^jĀ· L_j >miāj >m_ij (2) Biājiā Li B_iā j^iĀ· L_i >mjāi >m_ji (3) Bjāijā Lj B_jā i^jĀ· L_j >miāj >m_ij (4) These indicate that each GPU must be able to transmit and receive the same amount of data in both directions. We also have bandwidth decomposition equations: Bjāi+Bjāij B_jā i^i+B_jā i^j =Bjāi =B_jā i (5) Biāji+Biāj B_iā j^i+B_iā j^j =Biāj =B_iā j (6) Let us define: Ī±=BjāiiBjāi,Ī²=BiājjBiājĪ±= B_jā i^iB_jā i, Ī²= B_iā j^jB_iā j Then: Bjāi=Ī±āBjāi,Bjāij=(1āĪ±)āBjāiB_jā i^i=Ī± B_jā i, B_jā i^j=(1-Ī±)B_jā i Biāj=Ī²āBiāj,Biāji=(1āĪ²)āBiājB_iā j^j=Ī² B_iā j, B_iā j^i=(1-Ī²)B_iā j Substituting into inequalities (1)ā(4), we get: Ī±āBjāiā Li Ī± B_jā iĀ· L_i >mjāi >m_ji (1āĪ±)āBjāiā Lj (1-Ī±)B_jā iĀ· L_j >miāj >m_ij (1āĪ²)āBiājā Li (1-Ī²)B_iā jĀ· L_i >mjāi >m_ji Ī²āBiājā Lj Ī² B_iā jĀ· L_j >miāj >m_ij From these, we obtain constraints: mjāiBjāiāLi<Ī±<1āmiājBjāiāLjāmjāiLi+miājLj<Bjāi m_jiB_jā iL_i<Ī±<1- m_ijB_jā iL_j m_jiL_i+ m_ijL_j<B_jā i miājBiājāLj<Ī²<1āmjāiBiājāLiāmjāiLi+miājLj<Biāj m_ijB_iā jL_j<Ī²<1- m_jiB_iā jL_i m_jiL_i+ m_ijL_j<B_iā j Therefore, a necessary and sufficient condition for all inequalities (1)ā(4) to hold is: mjāiLi+miājLj<minā”(Bjāi,Biāj) m_jiL_i+ m_ijL_j< (B_jā i,B_iā j) Tightest Case. When the above inequality becomes equality, we achieve the tightest case: mjāiLi+miājLj=Bjāi=Biāj=B m_jiL_i+ m_ijL_j=B_jā i=B_iā j=B Appendix B Proof of Per GPU Contribution to Memory Slice Proposition. Let mjāi,miājā„0m_ji,m_ijā„ 0 and denote S=mjāi+miājS=m_ji+m_ij. Suppose that for any mvā[0,S]m_vā[0,S], mjāiāmvLiār+miāj+mvLjārā¤B,mjāi+mvLiār+miājāmvLjārā¤B, m_ji-m_vL_ir+ m_ij+m_vL_jrā¤ B, m_ji+m_vL_ir+ m_ij-m_vL_jrā¤ B, where Liārā¤LiL_irā¤ L_i and Ljārā¤LjL_jrā¤ L_j are individual application latency windows. Then Smaxā(Liār,Ljār)=Bā minā”(Liār,Ljār).S_ (L_ir,L_jr)=BĀ· (L_ir,L_jr). Moreover, maxLiārā¤LiLjārā¤Ljā”Smaxā(Liār,Ljār)=Bā minā”(Li,Lj), _ subarraycL_irā¤ L_i\\ L_jrā¤ L_j subarrayS_ (L_ir,L_jr)=BĀ· (L_i,L_j), achieved when Liār=Ljār=minā”(Li,Lj).L_ir=L_jr= (L_i,L_j). Proof. For any fixed (mjāi,miāj)(m_ji,m_ij) with total S=mjāi+miājS=m_ji+m_ij, each inequality can be written as mjāiLiār+miājLjārĀ±mvā(1Liārā1Ljār). m_jiL_ir+ m_ijL_jr\;Ā±\;m_v\! ( 1L_ir- 1L_jr ). Requiring both inequalities for all mvā[0,S]m_vā[0,S] is equivalent to mjāiLiār+miājLjār+Sā|1Liārā1Ljār|ā¤B. m_jiL_ir+ m_ijL_jr+S | 1L_ir- 1L_jr |\;ā¤\;B. (2) For a given total S, the term mjāiLiār+miājLjār m_jiL_ir+ m_ijL_jr is minimized by assigning all of S to the side with the larger reported rate. Without loss of generality, assume Liārā„LjārL_irā„ L_jr. Then the minimum is SLiār SL_ir, and since |1Liārā1Ljār|=1Ljārā1Liār | 1L_ir- 1L_jr |= 1L_jr- 1L_ir, the left-hand side of (2) simplifies to SLiār+Sā”(1Ljārā1Liār)=SLjār. SL_ir+S ( 1L_jr- 1L_ir )= SL_jr. Thus feasibility requires S/Ljārā¤BS/L_jrā¤ B, i.e., Sā¤BāLjār=Bāminā”(Liār,Ljār).Sā¤ B\,L_jr=B\, (L_ir,L_jr). This bound is tight: with Liārā„LjārL_irā„ L_jr, the allocation (mjāi,miāj)=(S,0)(m_ji,m_ij)=(S,0) achieves equality when S=BāLjārS=BL_jr. Therefore, Smaxā(Liār,Ljār)=Bāminā”(Liār,Ljār).S_ (L_ir,L_jr)=B\, (L_ir,L_jr). Finally, since Liārā¤LiL_irā¤ L_i and Ljārā¤LjL_jrā¤ L_j, maximizing over all admissible reported rates gives maxLiārā¤LiLjārā¤Ljā”Smaxā(Liār,Ljār)=Bāminā”(Li,Lj), _ subarraycL_irā¤ L_i\\ L_jrā¤ L_j subarrayS_ (L_ir,L_jr)=B\, (L_i,L_j), which is attained by setting L=Liār=Ljār=minā”(Li,Lj)L=L_ir=L_jr= (L_i,L_j), then: mjāi+miāj=Bā L m_ji+m_ij=BĀ· L Appendix C Proof of Scheduling Algorithm Informal description. We formulate allocation on memory slices as an online scheduling problem. Each slice has normalized capacity 1. Setting. Let m be the number of GPUs, V=1,ā¦,mV=\1,ā¦,m\. For every unordered pair i,jāV\i,j\ā V with iā jiā j, there is a shared memory slice of normalized capacity 11 that both GPUs can allocate from. The utilization of slice i,j\i,j\ at time t equals its current allocation (since capacity is 11): ui,jā(t)=alloci,jā(t)ā[0,1].u_\i,j\(t)\;=\;alloc_\i,j\(t)\;ā[0,1]. Jobs and demand. Requests (ājobsā) arrive online; each job r is owned by a GPU irāVi_rā V, has an arrival time arāā¤ā„0a_r _ā„ 0, and a time-varying (possibly non-monotone) demand function dr:ā¤ā„0āā¤ā„0,drā(t)=units held by ārā at time āt.d_r:Z_ā„ 0 _ā„ 0, d_r(t)=units held by r at time t. At each tick t, the allocator observes Īādrā(t)=drā(t)ādrā(tā1) d_r(t)=d_r(t)-d_r(t-1). If Īādrā(t)>0 d_r(t)>0, that many new units must be placed on slices ir,j\i_r,j\ with jā irjā i_r; if Īādrā(t)<0 d_r(t)<0, that many units are released. A job departs once drā(t)=0d_r(t)=0 for all subsequent t. Online allocation objective. At each time t, the allocator must irrevocably assign every positive increment to a slice incident to the owner GPU, without knowledge of future demands. We track: Congā(A)=suptā„0maxi,jā”ui,jAā(t)(peak utilization under allocator A), Cong(A)= _tā„ 0\, _\i,j\u^A_\i,j\(t) (peak utilization under allocator $A$), and the number of slice-saturation events Smax(A)=#(t,i,j)|ui,jA(tā)<1,ui,jA(t)=1.S (A)=\# \(t,\i,j\)\ |\ u_\i,j\^A(t^-)<1,\ u_\i,j\^A(t)=1 \. The competitive ratio compares an online allocator A to an offline optimal OPT that knows all arrivals, releases, and demands in advance: Ļā”(A)=supdemand sequencesCongā(A)Congā(OPT).Ļ(A)= _demand sequences Cong(A) Cong( OPT). GreedyāLeast-Loaded (GLL). When GPU i receives a unit, it scans its mā1m\!-\!1 incident slices i,j\i,j\ and chooses the one with minimum utilization ui,jā(t)u_\i,j\(t). This requires only per-slice counters adjacent to i and Oā”(m)O(m) work per decision. Azar & Epstein [4, 3] prove maxt,i,jā”ui,jGLLā(t) _t,\i,j\u^GLL_\i,j\(t) ā¤m+32āmaxt,i,jāui,jOPTā(t) \;ā¤\; m+32\, _t,\i,j\u OPT_\i,j\(t) ā¹Smaxā(GLL) S \! (GLL ) ā¤m+32āSmaxā(OPT). \;ā¤\; m+32\,S ( OPT). Thus GLL may cause up to Īā”(m) (m) more slices to saturate than an omniscient allocator, yet it needs only local state and Oā”(m)O(m) work per decision. Interpretation. The worst case arises from adversarial demand sequences that force myopic early placements to later coalesce on the same slices, causing avoidable saturations that an omniscient allocator could have steered around. Despite this Īā”(m) (m) worst-case gap, GLL is appealing in practice because it is decentralized, needs only local state, and incurs low decision overhead.