Paper deep dive
ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform
Xiaoxiao Jiang, Suyi Li, Sheng Yao, Tianyu Feng, Lingyun Yang, Dapeng Nie, Haoran Yang, Wei Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/4/2026, 10:45:11 AM
Summary
ServerlessT2I is a serverless-native system designed to efficiently serve Text-to-Image (T2I) workflows by decomposing monolithic GPU functions into loosely coupled model functions represented as Directed Acyclic Graphs (DAGs). It addresses limitations in existing platforms such as high scaling overhead, exposed communication complexity, and lack of multi-tenant fairness. Key innovations include a GPU-resident data plane that harvests slack memory to reduce loading and communication overheads, and a fair scheduler that balances tenant fairness with throughput using virtual time (vTime). Evaluation shows up to 2x higher request rates and 3x GPU resource savings compared to state-of-the-art systems.
Entities (8)
Relation Signals (6)
ServerlessT2I → decomposes → T2I Workflow
confidence 95% · ServerlessT2I decomposes a T2I workflow into loosely coupled model functions
ServerlessT2I → implements → Fair Scheduler
confidence 92% · ServerlessT2I further introduces a fair scheduler for multi-tenant serving
ServerlessT2I → uses → GPU-Resident Data Plane
confidence 92% · ServerlessTI introduces a unified GPU-resident data plane to absorb these costs
Fair Scheduler → optimizes → Multi-tenant Fairness
confidence 90% · achieves fairness at model function granularity while allowing workload-aware decisions
GPU-Resident Data Plane → reduces → Model Loading Overhead
confidence 90% · builds a data plane that reduces model loading and data communication overheads
ServerlessT2I → outperforms → Existing T2I Systems
confidence 88% · sustains up to 2x higher request rates than existing T2I workflow serving systems
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. \sys{} further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2$\times$ higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3$\times$ GPU resources while satisfying service level objectives (SLOs).
Tags
Links
- Source: https://arxiv.org/abs/2607.26566v1
- Canonical: https://arxiv.org/abs/2607.26566v1
Trouble viewing inline? Open PDF directly →
Full Text
94,245 characters extracted from source content.
Expand or collapse full text
ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform Xiaoxiao Jiang†∗, Suyi Li†∗#, Sheng Yao†, Tianyu Feng†, Lingyun Yang†, Dapeng Nie, Haoran Yang, Wei Wang† †Hong Kong University of Science and Technology Alibaba Group Abstract. Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. ServerlessT2I further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2× higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3× GPU resources while satisfying service level objectives (SLOs). 1. Introduction Text-to-image (T2I) workflows built on diffusion models are a cornerstone of modern image generation (Xu et al., 2025; Ju et al., 2024; Zhang et al., 2023; OpenAI, 2025a, b; Modal, 2025), underpinning commercial services (OpenAI, 2025b; Midjourney, 2025; Adobe, 2025; Li et al., 2025b) that serve millions of users at more than 10K requests per second (RPS) in production (Lin et al., 2025). In a mainstream public cloud platform, we observe growing demand for serverless T2I deployments driven by two workload characteristics. First, unlike conventional large language model (LLM) services exposed through standard APIs (OpenAI, 2020), T2I applications are highly customized. Professional creators compose unique workflows from diverse diffusion models and adapters (e.g., LoRAs (Hu et al., 2022)) based on their application needs; recent Alibaba production traces (Li et al., 2025b; Lin et al., 2025) reveal 31,133 distinct workflows in a single 20-day period. Second, T2I demand is often ad hoc and bursty. Serverless deployment addresses these needs by letting users upload custom workflows while delegating provisioning, elastic scaling, and accounting to the platform. Consequently, a serverless T2I platform must efficiently multiplex customized workflows under dynamic traffic, all while hiding infrastructure management details like GPU assignment and data movement from the user. However, current serverless cloud platforms fall short in T2I serving. As a common practice, users are required to compose workflows using third-party tools such as ComfyUI (ComfyUI, 2025) or Diffusers (von Platen et al., 2022), and then deploy the entire T2I workflow as a single, monolithic GPU function (Cloud, 2026b, c; AI, 2026). This practice obscures internal model executions and data exchanges, leading to three problems that burden users and limit platform efficiency. First, while existing works enable multi-GPU parallelism for T2I inference (Li et al., 2025b, 2024; Fang et al., 2024), they target limited workflows and lack abstractions for flexible model placement and tensor movement. This forces users building customized workflows to manually manage GPU assignment and inter-GPU communication, violating the serverless principle of hiding infrastructure details (Schleier-Smith et al., 2021). Second, T2I workflows contain large models of sizes up to tens of GiB, making scale-out bottlenecked by loading overhead (cold starts (Chai et al., 2025)). Standard techniques that overlap loading with inference fail when the loading overhead dominates; in a Flux (Labs, 2024) workflow, per-model loading is 3.2×–13.5× slower than the model’s single inference pass. Third, GPU scarcity makes multi-tenant T2I serving backlog-prone during peak hours (Sheng et al., 2024; Documentation, 2026): on our platform, tens of thousands of requests can queue, and major users can experience up to 10% of requests backlogged. This exposes a fair scheduling problem: because user-composed workflows have heterogeneous resource demands, treating them as uniform requests allows costlier workflows to consume a disproportionate share of GPU time. Despite these challenges, serverless T2I inference for customized workflows remains largely unexplored. Prior systems (vllm-project, 2026; sgl-project, 2026; Fang et al., 2024; Li et al., 2025b; Agarwal et al., 2024; Li et al., 2024; Lu et al., 2026) optimize individual T2I workflows through kernel optimization, parallelization, or in-workflow caching, but do not address the multi-tenant challenge of serving thousands of distinct workflows under dynamic traffic. We propose ServerlessT2I, a serverless-native system for T2I workflow inference. ServerlessT2I represents workflows as model DAGs (directed acyclic graphs), making individual model invocations and their data dependencies the fundamental units for execution, scaling, and scheduling. ServerlessT2I introduces three system components, each addressing one challenge. Workflow DAG Representation. ServerlessT2I exposes a new serverless programming interface that defines a user-platform contract for customized T2I workflows. Through this interface, a T2I workflow is converted from a monolithic GPU function into an explicit DAG of model function invocations: users implement each model as a model function that encapsulates model loading and execution and declares its inputs and outputs; ServerlessT2I uses these declarations to infer the workflow DAG. This DAG gives ServerlessT2I the data dependency information needed to parallelize independent model functions and materialize intermediate tensors across GPUs, without requiring users to manage GPU placement or inter-GPU communication. The same abstraction also lets ServerlessT2I control model scaling and fine-grained resource accounting while preserving a familiar serverless programming style, allowing users to focus on the high-level application logic—a key benefit provided by serverless T2I deployment. GPU-Resident Data Plane. Decomposing a T2I workflow into model functions, while enabling fine-grained scaling and resource accounting, puts model loading and inter-model data transfers on the critical path. ServerlessT2I introduces a unified GPU-resident data plane to absorb these costs. Our key insight is that T2I inference is typically compute-bound and leaves substantial GPU memory on the table: in our production cluster, the P95 GPU memory usage of a diffusion model is 35 GiB, only 36% of a modern NVIDIA H20 GPU. The system runtime can harvest this unused GPU memory for cached weights and inter-model tensor communication. ServerlessT2I introduces three mechanisms in the data plane to implement this insight (§5). First, because diffusion-model loading is often much slower than one inference pass, ServerlessT2I caches the first few layers of each model and overlaps loading of the remaining layers with ongoing inference computation, using a lightweight profiling step to determine the number of layers needed to hide loading latency. Second, ServerlessT2I materializes inter-model dependencies through GPU-resident transfers and callback-based data fetching, enabling efficient tensor exchange and flexible communication patterns while remaining transparent to users. Third, ServerlessT2I manages GPU memory as a unified logical address space shared by active inference, tensor communication, and cached weights. At its core is model weight virtualization, which decouples a model’s logical weights from their physical placement in GPU memory. This enables on-demand allocation of weights across disjoint memory regions. Also, ServerlessT2I treats weights as evictable, spilling them to host memory under pressure. Together, these mechanisms cut scale-out and communication overhead without exposing the runtime data path to users. Fair Scheduling. In view of the request backlogs during peak hours and the need for fair scheduling, ServerlessT2I defines fairness for multi-tenant T2I serving by the amount of GPU service each tenant receives, rather than by request counts. For each model function execution, ServerlessT2I measures the GPU time spent on model loading, computation, and tensor transfer, and charges this cost to the owning tenant as vTime. The scheduler prioritizes tenants with lower cumulative vTime, but strictly enforcing this order can rule out dispatch choices that would improve serving efficiency, e.g., reduce queuing delay. ServerlessT2I therefore uses a two-stage scheduler that bounds unfairness while preserving scheduling flexibility. In each scheduling round, the first stage admits only tenants whose cumulative vTime is within an operator-configured slack of the least-served tenant. The second stage ranks ready model functions from this eligible set using queuing delay, execution cost, and remaining workflow work. This design achieves fairness at model function granularity while allowing workload-aware decisions that improve efficiency. We prototyped ServerlessT2I and evaluated its decomposition, data plane, and scheduler under realistic workload variation. Our evaluation spans 20 representative T2I workflows, including SD3.5 (stabilityai, 2025), Z-Image (Team et al., 2025), Flux (Lab, 2025; Labs, 2024), and various adapters. Using production traces on a multi-GPU testbed, we compare ServerlessT2I against state-of-the-art serving systems (vllm-project, 2026; von Platen et al., 2022). ServerlessT2I sustains up to 2× higher request rates at the same GPU budget, reduces GPU demand by up to 3× at a fixed rate, and meets up to 7× tighter SLOs while tolerating 2× higher burstiness. Microbenchmarks confirm that the data plane effectively hides loading and communication overheads, and that the scheduler successfully balances tenant fairness with throughput. 2. Background and Motivation Figure 1. Basic Workflow and Workflows augmented with ControlNet (Zhang et al., 2023) and LoRA (Hu et al., 2022). 2.1. A Primer on Text-to-Image Workflow Basic Text-to-Image Workflows. As shown in Fig. 1-top, a basic text-to-image generation workflow consists of three types of models: text encoder, base diffusion model, and decoder-only variational autoencoder (VAE). The process begins with the text encoder, which encodes a text prompt into a sequence of text embeddings. The system then initializes a latent tensor with random Gaussian noise. Conditioned on the text embeddings, the base diffusion model iteratively refines this tensor through a series of denoising steps. Finally, the denoised latent representation is passed to the VAE decoder, which reconstructs the output image in pixel space. Adapter-augmented Workflows. Production T2I workflows often augment the basic pipeline with adapter models to provide fine-grained control over visual attributes such as spatial structure, artistic style, and illumination (Li et al., 2025b; Zhang et al., 2023; Hu et al., 2022; Zhang et al., 2025b; Zhang, 2025; Ye et al., 2023). From a systems perspective, two classes of adapters are most relevant, as shown in Fig. 1-bottom. First, tandem adapters, such as ControlNet (Zhang et al., 2023), execute alongside the base diffusion model at each denoising step and inject spatial conditioning signals such as edges or depth maps. These adapters complicate serving because their parameter sizes are often comparable to the base model, which increases both model-loading latency and inference latency; moreover, maximizing throughput often requires parallelizing the adapter and base model across GPUs, introducing nontrivial synchronization and data communication (Li et al., 2025b). Second, weight-update adapters, such as LoRA (Hu et al., 2022), modify the base model’s weights before inference. While they do not invoke an additional model, they introduce loading overhead: as users may rely on many such adapters, they are typically fetched and applied before inference on demand (Li et al., 2025b; Lin et al., 2025). 2.2. Serverless T2I Workflows Serving in Production T2I workflows are becoming a major serverless workload: in a one-month trace from our platform in March 2026, they account for 28% of total GPU usage, the largest share among GPU workloads. We identify two key reasons. Prevalence of Customized T2I Workflows. Unlike DNN and LLM services, which are typically exposed through standardized APIs (OpenAI, 2020), T2I applications are often built as customized workflows, especially by professional creators with specialized visual requirements. These workflows span diverse applications, such as virtual try-on (Choi et al., 2021) and image editing (Jiang et al., 2026). Recent Alibaba production traces (Li et al., 2025b; Lin et al., 2025) show this diversity at scale, with 31,133 distinct workflows recorded over 20 days; we observe similar trends on our platform. In addition to aesthetic customization, users may incorporate T2I-specific parallelization techniques to accelerate workflow execution, such as ControlNet parallelization (Li et al., 2025b) and sequence parallelism (Li et al., 2025b, 2024; Fang et al., 2024). These techniques exploit the data dependencies within T2I workflows to parallelize adapter execution and base model inference. For example, in Fig. 1-bottom, ControlNet produces intermediate results that must be transferred to specific layers of the base model at each denoising step; otherwise, the base model stalls waiting for unavailable inputs. Exploiting this parallelism therefore requires specialized system support (Li et al., 2025b; Suyi32, 2025). Dynamic Workload Traffic. Fig. 2-left plots the request volume per hour in our production cluster, normalized by the peak hourly load observed from March 1 to March 14, 2026. We include the Azure Functions trace (Shahrad et al., 2020) as a comparison baseline. Our T2I workload is visibly more bursty: whereas the Azure trace maintains a relatively stable baseline at roughly 60% of its peak, the T2I trace fluctuates over a much wider range. We further quantify workload variability using the coefficient of variation (CoV) of request inter-arrival times (IATs). A Poisson arrival process has an IAT CoV of 1, while values above 1 indicate burstier arrivals (Shahrad et al., 2020). Fig. 2-right shows the distribution of IAT CoV values in our T2I trace and the Azure trace. Both workloads exhibit substantial request variability: 75% of functions in the T2I trace have an IAT CoV above 1, compared with 60% in Azure Functions. Figure 2. Left: Invocations per hour, normalized to the peak. Right: CDF of CoV values across functions. 2.3. Limitations of Current Practices Develop and Deploy a T2I Workflow. Serverless T2I services are currently provided in the cloud (Cloud, 2026a; HuggingFace, 2026), where users first build a workflow with existing tools, such as ComfyUI (ComfyUI, 2025) or Diffusers (von Platen et al., 2022). They then provision GPU resources, upload the entire workflow as a GPU function (Cloud, 2026b; HuggingFace, 2026), and deploy it in the cloud. After that, the platform follows the standard serverless computing paradigm: it manages a shared GPU cluster and automatically scales the GPU functions in response to request traffic. Users are billed on a pay-as-you-go basis by accounting GPU usage based on the GPU specification and execution duration (Li et al., 2023a; Amazon, 2025; Cloud, 2025; HuggingFace, 2026). However, we identify following limitations, each of which undermines an essential property of serverless computing (Schleier-Smith et al., 2021). L1: High Scaling Overhead. Current serverless T2I platforms typically deploy each workflow as a single GPU function (Cloud, 2026a; HuggingFace, 2026). This practice makes the workflow, rather than an individual model component, the unit of scaling. When load increases, the platform must therefore replicate the full workflow even if only one component is the actual bottleneck. This coarse granularity increases auto-scaling overhead***Like (Yu et al., 2025b), we exclude the delay of fetching a remote container image for cold starts, which can take extra seconds to minutes to complete. in both startup latency and GPU memory footprint. For example, scaling a basic Flux1-Schnell workflow on H800 with Diffusers (von Platen et al., 2022) takes 1.1 seconds even when loading from pinned host memory, adding 110% overhead relative to model inference under default settings. It also consumes 42% more GPU memory than scaling only the base diffusion model, which is typically the bottleneck. We observe similar behavior in vLLM-Omni (vllm-project, 2026). The root cause is common across existing systems (vllm-project, 2026; sgl-project, 2026; von Platen et al., 2022; Fang et al., 2024; Li et al., 2025b, 2024): they reuse Diffusers’ design (von Platen et al., 2022), which follows a “single-file” abstraction (Diffusers, 2025b) and packages the entire generation workflow, including the base model, adapters, and control logic, as a monolithic unit. L2: Exposed Communication Complexity. High performance T2I execution often relies on multi-GPU parallelization, but existing systems force serverless users to handle the resulting communication complexity when composing workflows. In existing systems (Li et al., 2024, 2025b; Fang et al., 2024), the communication logic is tightly coupled to framework internals. Adapting these techniques to a user’s customized workflow therefore requires substantial systems expertise and engineering (Fang et al., 2024; vllm-project, 2026; Li et al., 2025b, 2024). In practice, users must reason directly about GPU placement, synchronization, and data movement at runtime (Grokipedia, 2026). This requirement exposes low-level resource management during workflow development, conflicting with the serverless principle of hiding infrastructure from users (Schleier-Smith et al., 2021). For example, combining ControlNet parallelization with sequence parallelism requires users to understand framework internals, tensor sharding, and low-level distributed communication (Suyi32, 2025), although these details should be hidden behind a serverless abstraction (Jonas et al., 2017). L3: Limited Support for Multi-tenant Serving. In a serverless platform, multiple tenants share a GPU pool to execute their workflows. However, GPUs are scarce resources, so bursty demand can quickly create request backlogs during peak periods; in our production cluster, tens of thousands of requests can queue, and major users can see up to 10% of requests backlogged. Backlog makes fairness a scheduling requirement: one tenant should not consume a disproportionate share of GPU service and delay others. Existing per-tenant quotas, such as request-per-minute (RPM) limits (Documentation, 2026; OpenAI, 2025c), provide isolation but are not work-conserving: they can throttle a tenant even when GPUs are idle. Worse, quotas account for requests rather than GPU consumption, which mismatches T2I workflows whose costs vary widely: as shown in Fig. 3-left, the inference latency of eight popular workflows***Equal contribution; # Corresponding authorspans up to 16×. While T2I serving systems (Li et al., 2025b; von Platen et al., 2022; Fang et al., 2024; vllm-project, 2026; sgl-project, 2026) can be efficient in single-instance deployments, they largely lack scheduling mechanisms that ensure both fairness and efficiency in a multi-tenant cloud. Figure 3. CDF of latency and GPU memory usage of eight workflows†footnotemark: on H800 at model level and workflow level. **footnotetext: Qwen-Image, Z-Image, Z-Image-Turbo, Flux1-Schnell, Flux1-Dev, SD3, SD3.5, and SDXL with default settings in (von Platen et al., 2022), e.g., resolutions (1024×1024). 3. Motivation and System Overview Figure 4. Resource utilization. Recent Alibaba trace (Lin et al., 2025) profiles GPU memory but omits SM util. and host memory. 3.1. Key Insight Compute-bound T2I leaves harvestable GPU memory. T2I inference is typically compute-bound: even an inference batch of one can saturate a high-end GPU (Li et al., 2024, 2025b; Lin et al., 2025). Unlike LLM serving, which reserves substantial GPU memory for runtime state such as the KV cache needed to serve a large batch of requests (Kwon et al., 2023), T2I inference maintains much less runtime state and therefore leaves a substantial fraction of GPU memory unused. For example, in Flux workflows, runtime state accounts for only 7.6% of the total model memory footprint. Fig. 3-right further shows that both end-to-end workflows and individual diffusion models use only a small fraction of the memory available on a modern GPU. Meanwhile, we observe the same pattern in our production workloads. As shown in Fig. 4, GPU SM utilization during T2I function execution is high, with the P50 reaching 100%. In contrast, GPU memory usage is much lower: the P50 is only 16 GiB and the P95 is 35 GiB. Since modern GPUs provide 80–141 GiB of memory, a large fraction of GPU memory remains idle in production. Another production trace from Alibaba reports a similar pattern, with P50 and P95 memory usage of 30 GiB and 36 GiB, respectively (Lin et al., 2025). We further observe that host memory is significantly underutilized on these servers, motivating the design described in §5.1. How does it work in ServerlessT2I? ServerlessT2I turns this slack memory into a backend data plane for serverless T2I serving (§5). At a high level, the data plane serves two roles. First, it caches model weights in otherwise idle GPU memory, reducing model loading overhead for fast scaling. Second, because ServerlessT2I decomposes a T2I workflow into a DAG of model functions (§4), the data plane provides GPU-resident buffers for fast data movement along DAG edges. This keeps critical-path inter-model communication efficient. As a benefit of DAG execution, the exposed function boundaries also enable fine-grained resource accounting for fair scheduling (§6). 3.2. System Overview ServerlessT2I treats a T2I workflow as a model DAG: individual models are exposed as functions, and workflow execution is a sequence of function invocations with data dependencies. System Architecture. Fig. 5 shows the architecture of ServerlessT2I. At the frontend, users implement model functions, compose them into workflows, and register the workflows with the system ( -.9pt 1⃝). ServerlessT2I provides a serverless-style programming interface with T2I-specific abstractions for workflow composition, hiding the complex data movement required by DAG execution from users and addressing L2. After registration, users invoke a workflow with inputs such as text prompts through OpenAI-compatible APIs (OpenAI, 2020) ( -.9pt 2⃝). At the backend, ServerlessT2I maintains workflow state and uses a fair scheduler to dispatch ready function executions across distributed GPU executors (L3). Each executor manages its GPU memory as a unified address space for active model inference runtime, weight caching for fast scaling (L1), and efficient communication between dependent model functions ( -.9pt 5⃝). Host memory acts as a secondary storage tier (§5.1). Figure 5. System Overview. Life of a Request. When a request arrives, the control plane instantiates the corresponding workflow DAG ( -.9pt 3⃝) and tracks the readiness of each model function. A vTime tracker accounts for each tenant’s resource share to support fair scheduling, as described in §6. The scheduler selects ready functions, i.e., functions whose inputs are available, and dispatches them to GPU executors ( -.9pt 4⃝). After executing a function, the executor reports completion to the control plane and exposes the output through the data plane. The control plane then releases downstream functions once their inputs become ready; this process repeats until the workflow output is returned to the user. 4. Programming Interface Design Objectives. First, ServerlessT2I’s programming interfaces resemble those provided by commercial serverless platforms (AWS, 2025; GoogleCloudPlatform, 2026; Azure-Samples, 2026), which reduces the additional learning curve for users. Second, ServerlessT2I abstracts complex data communication behind intuitive APIs, making data communication transparent to users and hiding the underlying infrastructure as well as the complexity of operating it (Schleier-Smith et al., 2021). With ServerlessT2I, individual models are implemented as functions and users can use an orchestrator function to assemble them into a workflow in a declarative manner. ServerlessT2I’s runtime handles execution and communication in the shared resource pool while respecting the workflow’s data dependencies. Model Function Development. Fig. 6 illustrates how to implement a Flux1-Dev model in ServerlessT2I. Developers implement three methods: setup_io(), load_model(), and execute(). For setup_io(), ServerlessT2I provides two primitives, add_input and add_output, to specify the model’s input and output interfaces. The other two methods encapsulate model loading and inference, respectively. As in prior systems (sgl-project, 2026; vllm-project, 2026; Fang et al., 2024; Li et al., 2025b), users can reuse model implementation code from Diffusers (von Platen et al., 2022) when implementing these methods, which also preserves compatibility with inference optimizations such as torch.compile(). Under the hood, ServerlessT2I organizes the three methods under a ModelFn class, the scaling and management unit at runtime. Assemble Models as a Workflow. ServerlessT2I uses an orchestrator function to assemble models as a workflow, similar to the design of Azure Durable function (Azure, 2026). ServerlessT2I adopts a declarative workflow programming model. The I/O interfaces declared by setup_io in each ModelFn are sufficient for ServerlessT2I to parse the data dependencies and apply topological sorting to infer the workflow DAG. For example, in Fig. 7, prompt_embeds is the output of text_enc, which serves as the input for controlnet and flux, meaning both of them depend on the output of text_enc for inference. At runtime, ServerlessT2I materializes the data communication while hiding the implementation details from users, as elaborated in §5.2. Function-as-a-Service Paradigm. In ServerlessT2I, ModelFn corresponds to the notion of a function in the Function-as-a-Service paradigm. It abstracts model components in a T2I workflow as small, single-purpose functions that are exposed to the serverless platform, which scales them on demand. ModelFn is also stateless and with random seeds fixed, the same method inputs always produce the same outputs, allowing safe retries for fault tolerance. ⬇ 1## Developers start here ## 2def setup_io(self) -> None: 3 # define inputs 4 add_input("latents") 5 add_input("prompt_embeds") 6 add_input("control_outputs", callback=True) 7 # define outputs 8 add_output("noise_pred") 9 10def load_model(model_path): 11 transformer = FluxTransformer2DModel.from_pretrained( 12 model_path, torch_dtype=torch.bfloat16, 13 ) 14 return "transformer": transformer 15 16@torch.no_grad() 17def execute(model_components, **kwargs): 18 transformer = model_components["transformer"] 19 noise_pred = transformer.forward(**kwargs) 20 return "noise_pred": noise_pred 21 22## Invisible from developers ## 23class ModelFn: 24 setup_io = setup_io 25 initialize = load_model 26 execute = execute Figure 6. A simplified implementation of Flux function. ⬇ 1def compose_workflow(): 2 # create model function instances 3 text_enc = ModelFn(model_path=model_path) 4 flux = ModelFn(model_path=model_path) 5 controlnet = ModelFn(model_path=controlnet_path) 6 vae = ModelFn(model_path=model_path) 7 ... 8 # connect function instances 9 latents = random_latents_generator(seed) 10 prompt_embeds = text_enc(prompt) 11 for i in range(num_steps): # iterative denoising 12 control_outputs = controlnet(latents, prompt_embeds) 13 noise_pred = flux(latents, prompt_embeds, control_outputs) 14 latents = denoise(noise_pred, latents) 15 output_img = vae(latents, mode="decode") 16 ... Figure 7. A simplified Flux workflow composition. 5. Data Plane ServerlessT2I’s data plane builds on the insight in §3.1: harvested GPU memory can reduce two costs on the critical path, model loading and data movement across functions. We describe how to reduce model loading overhead (§5.1) and provide efficient data communication (§5.2), then explain how a unified logical address space manages GPU memory (§5.3). 5.1. Model Loading Challenge: Model loading remains on the critical path. Decomposing a T2I workflow into model functions gives the platform control over each model, but it does not remove model loading during scale out. As workflows have large models along the critical path, loading remains a bottleneck. Prior systems reduce loading overhead by overlapping model loading with execution in two ways. One approach transfers later layers of a model while its earlier layers are executing (Bai et al., 2020; Li et al., 2025a; Yu et al., 2025b; Jiang et al., 2026). Another approach loads later stage models in a workflow while earlier stage models are executing (Mahgoub et al., 2022). Both approaches require enough execution time to hide loading, which often does not hold for T2I workflows. For example, in a basic Flux1-Dev workflow, loading text_encoder, text_encoder_2, transformer, and the VAE decoder from pinned host memory to an NVIDIA H800 GPU takes 5×, 13.5×, 3.2×, and 6.3× their respective inference latencies. Even with ideal overlap, layer by layer loading reduces Flux1-Dev end-to-end latency by only 13%. Prewarming later stage models also leaves a large bottleneck: even with weights cached in host memory, model loading still accounts for 44% of end-to-end latency. Prewarming further interferes with multi-tenant scheduling by injecting bursts of loading traffic that may delay other tenants. Caching the first layers. ServerlessT2I reduces model loading overhead with a multi tier cache spanning GPU memory, host memory, and external storage. GPU memory is the fastest tier, caching model weights for immediate execution. Host memory is the second tier, holding weights that can be loaded into GPU memory on demand. External storage is the last tier with nearly unlimited capacity but much lower bandwidth. Consistent with prior observations (Lin et al., 2025; Li et al., 2025b), workflow popularity is skewed. In our trace replay, caching the 15 most popular workflows in underutilized host memory serves 99% of model loads from host memory, as detailed in Appendix A. Accordingly, we focus on host to GPU loading. The key idea is to split each model across the two tiers: cache its first layers in GPU memory and keep the remaining layers in host memory, as shown in Fig. 8-left. Consider a model with L uniform layers. ServerlessT2I preloads the first LeL_e layers into GPU memory and keeps the remaining L−LeL-L_e layers in host memory. When a request arrives, inference starts immediately on the cached first layers, while loading the remaining layers from host memory asynchronously. Let TloadT_load denote the loading latency of one layer and TcompT_comp denote its inference latency. Because Tload≫TcompT_load T_comp, the end-to-end latency Te2eT_e2e as a function of LeL_e is (1) Te2e(Le)=max(LTcomp,(L−Le)Tload+Tcomp). T_e2e(L_e)= (LT_comp,(L-L_e)T_load+T_comp). Eq. 1 gives the minimum prefix length Le′L_e required to fully hide model loading: (2) Le′=⌈L(Tload−Tcomp)+TcompTload⌉. L_e = L\! (T_load-T_comp )+T_compT_load . Computing Le′L_e requires only two profiled quantities: the per-layer loading latency TloadT_load and the per-layer inference latency TcompT_comp. For diffusion models and their ControlNets, profiling is lightweight because they are typically composed of uniform transformer blocks with similar costs. Eq. 1 and Eq. 2 can be extended to models with non-uniform layers. Figure 8. Left: A simplified illustration of ServerlessT2I’s loading mechanism. Right: An example of data fetching. 5.2. Communication Challenge: Model DAG execution requires efficient and flexible tensor communication. Once a workflow is decomposed into functions, intermediate tensors must be communicated across functions. This poses two challenges. First, each workflow execution transfers a large volume of data—on the order of GiBs (Li et al., 2025b)—and 99% of the transferred objects are tensors. Second, specialized parallelization techniques introduce complex communication patterns in which both correctness and performance depend on consuming data at the right time. For example, if computation blocks while waiting for data to be produced, the system may lose much of the performance benefit from parallel execution (§2). Existing serverless data planes are a poor fit for T2I workflow execution. Data planes that rely on host memory (Yu et al., 2023) perform poorly for CUDA tensors: the communication latency for an SD3 ModelFn is 28× its execution latency due to PCIe transfers, serialization, and socket overhead. Recent work builds data planes atop high-speed interconnects such as NVLink and RDMA (Wu et al., 2026; Yu et al., 2026), but exposing these links through collective-style APIs introduces multiple synchronization points to preserve correctness. In our measurements, these synchronization points add up to 20% inference latency when running Flux1-Schnell (Labs, 2024), a computation-intensive model, on two NVIDIA H800 GPUs, in both the basic workflow and the ControlNet-augmented workflow. Efficient and flexible data fetching. ServerlessT2I combines NVSHMEM (NVIDIA, 2025b) with callback-based fetching to provide efficient and flexible GPU communication. NVSHMEM is a natural substrate for ServerlessT2I because it provides one-sided GPU communication over NVLink and RDMA, supports GPU-initiated transfers, and exposes a symmetric heap abstraction that avoids explicit remote-address management. In ServerlessT2I, each executor reserves a fixed-size NVSHMEM arena on its GPU using nvshmem_malloc(APIs, 2026) and manages the resulting symmetric heap with a buddy allocator(Knowlton, 1965, 1966), returning device pointers usable by both local CUDA kernels and remote NVSHMEM operations. On top of this substrate, ServerlessT2I allows users to mark a ModelFn input as a callback when the input is needed only partway through execution. For example, line 6 in Fig. 6 declares control_outputs with callback=True to support ControlNet parallelization (§2.1). The runtime wraps this input and fires the callback when downstream models consume it, overlapping communication with earlier computation. Compared with manually managed peer-memory access (Zone, 2026), this design better matches T2I workflows, where intermediate tensors are produced and consumed at fine-grained, model-dependent points during inference. At runtime, a producer writes its output tensor into local NVSHMEM and publishes the tensor metadata, such as its pointer and shape, to downstream consumers. When a consumer first needs the tensor, ServerlessT2I allocates a destination block from the consumer’s local NVSHMEM arena and pulls the tensor bytes directly from the producer’s remote address using a one-sided NVSHMEM operation. The data path stays in GPU memory, avoiding PCIe round trips and socket communication. The callback abstraction controls when a transfer occurs: the fetch is issued only when execution reaches the program point where the tensor is needed. Fig. 8-right presents an example of data fetching in ControlNet parallelization, where ControlNet execution is interleaved with the base Flux model, as shown in Fig. 1-bottom. At runtime, the output of ControlNet layer 0 is consumed partway through Flux layer 0. Rather than blocking until this output is available, Executor 1 begins executing Flux layer 0 and registers a fetch callback. When Flux reaches this point, the callback fires. By then, Executor 0 has produced the ControlNet layer 0 output and placed it in its NVSHMEM communication buffer ( -.9pt 1⃝); the corresponding tensor metadata is forwarded to Executor 1 ( -.9pt 2⃝), which uses it to issue a one-sided NVSHMEM fetch into its local tensor store ( -.9pt 3⃝) before Flux consumes the tensor ( -.9pt 4⃝). This callback-based fetch path allows Flux to overlap its computation with ControlNet execution. Without it, Flux would have to wait until the relevant ControlNet output was materialized, eliminating the parallelism between the two models. Note that tensor metadata is tiny, on the order of KiB, and executors piggyback it on completion notifications, allowing the control plane to track tensors with little overhead. Figure 9. Left: An example of model weight virtualization. Right: Difference of service received for two backlogged tenants. 5.3. Unified GPU Memory Management The shared use of GPU memory requires each executor to coordinate three memory consumers within its local GPU: active inference state, communication buffers (§5.2), and model weights (§5.1). Active inference state, such as inputs and intermediate activations, is framework-specific and allocated through the deep learning framework runtime, i.e., PyTorch in ServerlessT2I. In contrast, communication buffers are allocated and managed by NVSHMEM. As a result, local GPU memory is divided into two disjoint allocation domains, which we refer to as the PyTorch region and the NVSHMEM region. ServerlessT2I introduces model weight virtualization, a software indirection layer that decouples a model’s logical weights from their physical placement in GPU memory. Similar to how virtual memory in operating systems decouples a process’s logical address space from physical memory frames, ServerlessT2I decouples a model’s logical layers from the physical GPU memory that stores their weights. Model weights can transparently reside in either the PyTorch region or the NVSHMEM region, while model execution accesses them through the same logical interface. This indirection bridges the two otherwise isolated regions into a unified pool for cached weights, exploiting the underutilized GPU memory (§3.1). How does it work? We walk through an example in Fig. 9-left, to show how ServerlessT2I executes model inference and manages GPU memory. From the perspective of model execution, the model still consists of a conventional sequence of logical layers, as seen by the PyTorch forward() pass. Before executing each layer, ServerlessT2I consults a Layer Lookup Table to translate the logical layer into the corresponding tensor objects that represent its weights. Each tensor object contains the necessary metadata, such as shape and data type, together with a data pointer to the underlying physical GPU memory. This physical memory may be allocated from either the PyTorch region or the NVSHMEM region. The CUDA kernels invoked by the framework then execute normally, using the data pointers of both active inference state and model weights to access the appropriate physical memory. Weights allocated by PyTorch are ordinary tensor objects and can be used directly, whereas weights stored in the NVSHMEM region require pointer rebinding. For each NVSHMEM allocation that holds a layer’s weights, ServerlessT2I wraps the device pointer as a CUDA tensor using PyTorch C++’s torch::from_blob, creating a tensor view over the NVSHMEM-backed memory. ServerlessT2I then rebinds the model’s parameter entries to these tensors and records them in the Tensor Object entries of the Layer Lookup Table. Consequently, forward() calls access NVSHMEM-resident weights as ordinary PyTorch tensors, and CUDA kernels treat them identically to tensors allocated by PyTorch’s CUDA allocator. ServerlessT2I tracks model weights at layer granularity and treats cached weights as elastic state. When GPU memory pressure arises, for example because a new model must be loaded from host memory, ServerlessT2I evicts cached weights layer by layer until the demand is satisfied. An executor orders its GPU-resident models by recency and reclaims weights from the least recently used (LRU) models first, evicting layers in reverse layer order to preserve first layers that enable overlapping loading with inference. This eviction policy trades a modest latency increase for reclaimed memory while striving to preserve cached weights for fast scale-out. Can NVSHMEM region be elastic? The NVSHMEM region size should be specified at initialization (NVIDIA, 2025a), and resizing it at runtime is impractical because NVSHMEM maintains a symmetric heap with identical size and layout across all GPUs, making any adjustment require cluster-wide coordination (NVIDIA, 2025c). This creates a fundamental tension: an undersized NVSHMEM region risks deadlock when in-flight operations cannot allocate space for data communication, stalling the entire system; an oversized region, however, squeezes the space available for cached weights without model weight virtualization, which we quantitatively analyze in §8.3. Backend-as-a-Service Paradigm. ServerlessT2I’s data plane follows the Backend-as-a-Service paradigm in serverless computing (Schleier-Smith et al., 2021). It exposes model caching and data communication as managed backend services, allowing users to benefit from them without building or operating these components themselves. In T2I workflow execution, all intermediate data is immutable: intermediate tensors produced are consumed once and never updated (Li et al., 2025b; von Platen et al., 2022), which obviates consistency protocols and simplifies fault tolerance. The data plane reclaims tensors when no downstream ModelFn requires them. We use expandable_segment (Notes, 2026) in the PyTorch region and our buddy allocator in the NVSHMEM region to mitigate memory fragmentation. If an executor fails, ServerlessT2I reconstructs lost data by re-executing the affected ModelFns and loading model weights, following a similar approach to prior cluster computing frameworks (Moritz et al., 2018; Zaharia et al., 2012; Yu et al., 2023). 6. Ensure Fairness in Multi-tenant Serving Resource fairness is critical in multi-tenant serving systems because GPUs are scarce cloud resources, and tenant requests can remain backlogged during peak demand (Sheng et al., 2024). Yet existing T2I serving systems provide limited support for resource-aware scheduling (vllm-project, 2026; sgl-project, 2026; Fang et al., 2024; Li et al., 2025b; Agarwal et al., 2024; Li et al., 2024; Lu et al., 2026). This gap is especially problematic for T2I workloads: because workflows differ substantially in GPU demand (Fig. 3-left), two backlogged tenants that submit requests at the same rate may still receive very different amounts of service. We illustrate this effect by augmenting Diffusers (von Platen et al., 2022), a representative T2I serving system, with three scheduling policies: FIFO, SJF, and an adapted version of VTC (Sheng et al., 2024), a fairness-oriented scheduler originally designed for LLM serving. We run the experiment on a four-H800 testbed with two tenants, each issuing requests at 2 RPS. One tenant invokes a basic SD3-medium (Esser et al., 2024) workflow, while the other invokes a basic Flux1-Dev (Labs, 2024) workflow. We define the service received by a tenant as its cumulative GPU time. Fig. 9-right reports the service difference between the two tenants over the interval in which both remain backlogged. None of the policies provides satisfactory fairness. FIFO accounts for requests rather than service: despite identical request rates, the Flux1-Dev tenant receives substantially more GPU time because each Flux1-Dev request is much more expensive, with 10× the inference latency of SD3-medium. SJF exhibits the opposite bias, giving more service to the SD3-medium tenant because it consistently favors shorter requests. Adapting VTC reduces the imbalance by counting T2I workflow operations, analogous to its use of decoding steps in LLM serving (Sheng et al., 2024), but operation counts remain an inaccurate proxy for GPU service because T2I models differ widely in per-operation cost (Fig. 3-left). Fine-grained Fairness Accounting with vTime. ServerlessT2I accounts for service at the granularity of each ModelFn execution. Whenever a ModelFn runs on a GPU, ServerlessT2I charges its owner a vTime equal to the wall-clock GPU time consumed, including computation, model loading, and tensor transfer. At dispatch time, the scheduler charges an estimated vTime derived from historical measurements so that scheduling decisions can proceed immediately; once execution completes, the estimate is replaced with the measured GPU time, keeping accounting faithful to realized usage. If a denoising step launches parallel ModelFns (e.g., a base model and a ControlNet), the tenant is charged for both. Fairness Scheduling with vTime. ServerlessT2I tracks the cumulative vTime of each tenant and prioritizes those that have received less service. For a tenant that newly enters the system or returns after inactivity, ServerlessT2I performs a vTime lift, initializing its vTime to the minimum among active tenants. Without this lift, a returning tenant’s stale, artificially low vTime would grant it repeated priority until it catches up, converting past absence into a scheduling advantage. To balance fairness and serving efficiency, ServerlessT2I allows a configurable slack so that the scheduler retains scheduling flexibility. This is realized as a two-layer scheduler. Layer 1: Fairness filter. The scheduler maintains cumulative vTime vuv_u for each tenant u. At dispatch time, it forms a candidate set C of tenants: a tenant u is included if vu−minu′vu′≤Δv_u- _u v_u ≤ , where Δ is an operator-configured slack. Only ModelFns belonging to tenants in C proceed to Layer 2. A smaller Δ enforces stricter fairness, while a larger Δ gives Layer 2 a wider pool of candidates to optimize throughput. Layer 2: Priority scoring. Strict fairness alone can hurt serving efficiency. For example, the scheduler may interleave the execution of two workflows to ensure fair resource allocation, increasing the latency of both. To improve serving efficiency, among the eligible ModelFns, Layer 2 ranks each ModelFn n by a normalized priority score score(n)=w^(n)−r^(n)−e^(n)score(n)= w(n)- r(n)- e(n), where w^(n) w(n), r^(n) r(n), and e^(n) e(n) denote its waiting time, the remaining critical-path time of its workflow, and its execution latency, respectively. The scheduler dispatches the highest-scoring ModelFn. The terms −r^(n)- r(n) and −e^(n)- e(n) favor ModelFns with shorter remaining work or lower execution latency. The term w^(n) w(n) prevents starvation. The selected ModelFn is dispatched to an executor that can run it more efficiently, e.g., one with its model already cached to reduce loading overhead or its inputs available in the local GPU memory to reduce communication overhead. Algorithm 1 summarizes the scheduling process, including vTime lift, candidate-set construction, priority scoring, and vTime correction after execution. Input: Per-tenant cumulative vTime vuv_u; fairness slack Δ Function Schedule() // invoked when an executor is idle: // vTime lift for new/returning tenants foreach tenant u with ready ModelFns do if u is new or returning then vu←minu′∈Activevu′v_u← _u v_u end foreach // Layer 1: fairness filter R←R← tenants with ready ModelFns vmin←minu∈Rvuv_ ← _u∈ R\,v_u C←u∈R∣vu−vmin≤ΔC←\u∈ R v_u-v_ ≤ \ // Layer 2: priority scoring N←N← ready ModelFns owned by tenants in C n⋆←argmaxn∈Nw^(n)−r^(n)−e^(n)n ← _n∈ N\; w(n)- r(n)- e(n) // Dispatch and charge estimated vTime T^n⋆← T_n ← estimated execution time of n⋆n vowner(n⋆)+=T^n⋆v_owner(n ) += T_n Dispatch n⋆n to best-fit executor Function OnComplete(n,Tn,\,T) // T: measured GPU time of n: vowner(n)+=T−T^nv_owner(n) +=T- T_n // correct vTime Algorithm 1 Scheduling Algorithm Fairness bound. Since ServerlessT2I serves DAG-structured workflows, a tenant may have outstanding requests but no schedulable work when all pending ModelFns are blocked on predecessors. We thus call a tenant eligible-backlogged over [t1,t2)[t_1,t_2) if it has at least one ready ModelFn at every point in the interval. This refines the backlogged condition in VTC for LLM serving (Sheng et al., 2024), which only requires a queued request. This distinction is necessary for DAGs, where dependencies can leave outstanding work with no ready computation. Let LmaxL_max denote the largest vTime charge of any single dispatch, and define U=Δ+LmaxU= +L_max. Consider any two tenants f and g that remain eligible-backlogged during [t1,t2)[t_1,t_2). For tenant u, let Wu(t1,t2)=vu(t2)−vu(t1)W_u(t_1,t_2)=v_u(t_2)-v_u(t_1) denote the service it receives during this interval. Then ServerlessT2I guarantees |Wf(t1,t2)−Wg(t1,t2)| |W_f(t_1,t_2)-W_g(t_1,t_2)| ≤|vf(t2)−vg(t2)|+|vf(t1)−vg(t1)| ≤|v_f(t_2)-v_g(t_2)|+|v_f(t_1)-v_g(t_1)| ≤2(Δ+Lmax)=2U. ≤ 2( +L_max)=2U. 7. Implementation We have implemented ServerlessT2I with a FastAPI (FastAPI, 2025) frontend, which exposes a programming interface for users to compose and register T2I workflows (§4). Users invoke their workflows with image generation parameters, such as prompts and reference images, similar to the OpenAI API (OpenAI, 2020). ServerlessT2I’s backend runtime consists of a control plane and distributed executors (Fig. 5), totaling 4,000 lines of Python code. The data plane is implemented in 3,000 lines of Python and C++/CUDA code, built on PyTorch and NVSHMEM (NVIDIA, 2025b). Aside from CUDA tensors, communication between the scheduler and distributed executors is facilitated via ZeroMQ (zeromq, 2025). 8. Evaluation We evaluate ServerlessT2I with the following highlights: • ServerlessT2I outperforms state-of-the-art baselines in controlled end-to-end evaluation, sustaining up to 2× higher request rates, satisfying 7× more stringent SLOs, reducing GPU requirements by up to 3×, or tolerating 2× higher burst traffic, all while maintaining 90% SLO attainment (§8.2). • Our microbenchmarks isolate the benefits of ServerlessT2I’s designs: it reduces model loading overhead, enables efficient parallelization with minimal effort, and explores a tradeoff between fairness and efficiency. (§8.3 & §8.4). • ServerlessT2I’s DAG execution adds negligible overhead (§8.5). 8.1. Setup Testbed and Workloads. By default, we use a testbed of 32 NVIDIA H800 GPUs and a scaled real-world T2I production trace collected from our production cluster (Fig. 2-left). To evaluate under diverse conditions, we vary request rates, SLO targets, traffic burstiness, and testbed sizes, covering a broad range of traffic patterns and performance requirements. Metrics. Our primary metric is SLO attainment: the fraction of requests completed within their specified latency deadline. We set the default deadline to 3× the solo inference latency of each workflow (SLO Scale ==3). Unlike prior works (Li et al., 2025b, 2024; Agarwal et al., 2024; Fang et al., 2024), ServerlessT2I does not alter the computation of T2I inference and we have validated the identity of generated images. Baselines. We primarily compare ServerlessT2I with vLLM-Omni and Diffusers , which are representative state-of-the-art T2I serving systems (von Platen et al., 2022; Diffusers, 2025a; vllm-project, 2026). Following current practices, we deploy each workflow as a monolithic GPU function (§2.2). Since these systems were originally designed as standalone inference frameworks, we adapt them to the serverless setting and evaluate three deployment variants: • NoCache (No caching) executes each request to a workflow in a GPU function without any caching on GPU. • WCache (Workflow cache) utilizes slack GPU memory to cache entire workflows. A GPU function is terminated when the cached workflow is evicted. • MCache (Model cache) utilizes slack GPU memory to cache individual models within workflows. Because caching is performed at a finer model granularity, this variant can accommodate more workflows in GPU memory. Note that WCache and MCache are augmented versions of NoCache that utilize slack GPU memory and use the same LRU caching policy as ServerlessT2I for a fair comparison. Workflows and Settings. We compose T2I workflows using six popular base models: SD3.5-Large (stabilityai, 2025), Z-Image (Team et al., 2025), Z-Image-Turbo (Team et al., 2025), Flux1-Dev (Labs, 2024), Flux1-Schnell (Labs, 2024), and Flux2-Klein (Lab, 2025). They exhibit diverse computational characteristics, with parameter counts spanning 6B to 12B and denoising steps ranging from 4 to 50. We set up two settings, as detailed in Table 1, randomly assigning workflows to the top-tier tenant traffic from our production trace. In S1, baselines are based on vLLM-Omni, as it provides little support for adapters (vllm-project, 2026) and does not support Flux1-Schnell. In S2, baselines use Diffusers. Table 1. Evaluation settings: S1 includes basic workflows, where each workflow consists of text encoders, a diffusion model, and a decoder. S2 further includes adapter-augmented workflows, which extend the basic workflows with different ControlNet and LoRA adapters. Setting Diffusion Models No. Workflows S1 SD3.5-Large, Z-Image, Z-Image-Turbo Flux1-Dev, Flux2-Klein 5 S2 SD3.5-Large, Z-Image, Z-Image-Turbo Flux1-Dev, Flux1-Schnell, Flux2-Klein 20 8.2. End-to-end Evaluation Figure 10. End-to-end evaluation of Setting 1 (S1). All baselines are implemented on top of vLLM-Omni (vllm-project, 2026). Figure 11. End-to-end evaluation of Setting 2 (S2). All baselines are implemented on top of Diffusers (von Platen et al., 2022). As Fig. 10 and Fig. 11 show, ServerlessT2I consistently achieves higher SLO attainment than the baselines in S1 and S2. We use a controlled evaluation methodology: each experiment varies one workload or deployment factor while holding the others fixed. Overall, ServerlessT2I sustains up to 2× higher request rate, satisfying up to 7× stringent SLOs, saving up to 3×3× GPU resources, and tolerating 2× higher burst traffic, all while maintaining SLO attainment for 90% of the requests. These gains are partly driven by ServerlessT2I’s model-loading design: data-plane weight caching accelerates 99%/86% of model loads in S1/S2, leaving only 1%/14% to require full model loading. Among the baselines, MCache and WCache consistently outperform NoCache, confirming ServerlessT2I’s insight that fine-grained use of idle GPU memory improves serving efficiency. SLO Attainment vs. Rate Scale. We first vary the request rate scale while fixing the SLO scale, testbed size, and traffic burstiness as the default values. Fig. 10(a) and Fig. 11(a) show that ServerlessT2I preserves high SLO attainment over a wider load range than all baselines. In S1, ServerlessT2I maintains 100% attainment up to a rate scale of 1.5. By contrast, the strongest baseline, MCache, falls below 90% once the rate scale reaches 1.0, and NoCache drops to only 10% at rate scale 2.0. In S2, ServerlessT2I again remains above 90% through rate scale 1.5, whereas MCache reaches 85% at the same load and the other baselines are below 50%. These results show that whole-workflow caching alone is insufficient under increasing load: even when some requests hit in cache, monolithic deployments still pay high loading and scaling costs when the active working set changes. ServerlessT2I avoids this cliff by decomposing workflows and reusing GPU-resident model state across requests. SLO attainment vs. SLO Scale. We next vary the SLO scales while fixing the rate scale and testbed size, using the scaled original production trace. Fig. 10(b) shows that ServerlessT2I meets substantially tighter SLOs in S1: at an SLO scale of 3.0, ServerlessT2I completes all requests within deadline, whereas WCache and MCache require scale 24.0 to exceed 90% attainment, and NoCache never reaches 90% even at the loosest SLO. This gap reveals that the baselines are not merely short of compute capacity; their tail latency is dominated by model and workflow loading overheads, which only very loose deadlines can hide. The same trend holds in S2 (Fig. 11(b)): ServerlessT2I achieves 97% attainment at SLO scale 8.0, while MCache and WCache require scales of 16.0 and 18.0, respectively. Even as adapters increase workflow diversity, ServerlessT2I converts deadline slack into SLO attainment more efficiently than baselines. SLO Attainment vs. Testbed Size. We next vary the number of GPUs while keeping the workload fixed. In Fig. 10(c), ServerlessT2I meets the 90% SLO attainment target with only 8 GPUs; the strongest vLLM-Omni baseline requires 24 GPUs to match this, MCache requires 32, and NoCache remains below 90% even at 32 GPUs. In S2, ServerlessT2I reaches 98% attainment with 16 GPUs, whereas the strongest Diffusers baseline requires 24 GPUs to exceed 90% and other baselines remain far below target at 32 GPUs. These results show that ServerlessT2I’s gains extend beyond latency: scaling individual models rather than entire workflows reduces over-provisioning and allows the cluster to operate as a shared GPU pool. SLO Attainment vs. CoV. Finally, we evaluate robustness to bursty traffic by varying the coefficient of variation (CoV) of request arrivals while fixing the average rate scale, SLO scale, and testbed size. Following prior works (Li et al., 2023b; Gujarati et al., 2020), we partition the original trace into time windows, fit arrivals to a Gamma process, and resample at scaled CoV values to control burstiness. Higher CoV increases short-term queue buildup, stressing autoscaling and cache replacement. As shown in Fig. 10(d), ServerlessT2I maintains at least 96% attainment up to CoV scale 4.0, while MCache falls below 90% between scales 2.4–4.0 and WCache is below 90% at the lowest CoV. In S2, ServerlessT2I stays above 90% through CoV scale 5.0, whereas MCache drops to 89% at scale 2.5. The comparison between S1 and S2 indicates that adapter-heavy workloads make burst handling more sensitive to cache granularity: workflow-level caching cannot react quickly when bursts shift demand across variants, while ServerlessT2I absorbs these shifts by reusing shared model components. GPU Memory Utilization. Figures 10(e) and 11(e) show average GPU memory utilization at runtime. ServerlessT2I consistently achieves the highest utilization, followed by MCache, WCache, and NoCache. This ordering reflects their caching granularities: ServerlessT2I caches at the granularity of model layers, MCache at models, and WCache at entire workflows. Finer granularity reduces internal fragmentation by better utilizing residual memory. In S1 at rate scale 1.5, NoCache, WCache, MCache, and ServerlessT2I achieve 13%, 29%, 71%, and 88% utilization, respectively. In S2 at rate scale 1.75, they achieve 30%, 41%, 62%, and 84%, respectively. 8.3. Data Plane Figure 12. Left: Latency breakdown of workflow execution. ppl.: pipeline. Right: Normalized inference latency w/ and w/o parallelization. Opt.: theoretically optimal latency. Model Loading. We elaborate on the model loading design in §5.1. Fig. 12-left reports the latency breakdown of a basic Flux1-Schnell workflow request, including model loading and inference. We define two performance bounds: a lower bound where loading and inference execute serially without pipelining (w/o ppl.), and an upper bound where model loading latency is fully hidden (w/o load). We show a spectrum of caching configurations: ppl.(0) indicates no layers are pre-cached in the GPU, equivalent to existing layer-wise pipelined loading (Bai et al., 2020; Yu et al., 2025b). By caching Le′L _e layers in the GPU (ppl.(Le′L _e)), ServerlessT2I reduces loading latency by 90% and end-to-end latency by 21% relative to existing methods, rivaling the upper bound. Even when half of the Le′L _e layers are evicted (ppl.(Le′/2L _e/2)), ServerlessT2I still reduces loading latency by 62%. Communication. ServerlessT2I’s data plane enables efficient and flexible communication for T2I-specific parallelization (§2.2). We validate these properties by measuring the speedups achieved by ServerlessT2I’s parallelization, which would be substantially reduced if either property were absent. As shown in Fig. 12-right, ServerlessT2I exploits parallelism to accelerate Z-Image workflow executions on NVIDIA H800 GPUs. Its speedup approaches the theoretically optimized performance and is consistent with prior results (Li et al., 2025b, 2024; Fang et al., 2024), indicating that ServerlessT2I’s data plane supports efficient, timely data exchange. Model weight virtualization. As described in §5.3, model weight virtualization allows cached weights to reside in either GPU memory region, addressing the limitations of a statically sized NVSHMEM region. We validate this on an 8-GPU testbed using the ControlNet-augmented workflows in S2 at a rate scale of 0.5. As shown in Fig. 13-left, an undersized NVSHMEM region can stall the system when in-flight operations cannot allocate space for data communication, even after all weights have been evicted from the NVSHMEM region. An oversized NVSHMEM region, conversely, crowds out the space for cached model weights without model weight virtualization, increasing the volume of loaded weight by up to 3× and degrading SLO attainment by up to 50%. We also verified that repeated offloading and reloading of parameters does not fragment the PyTorch or NVSHMEM regions: even at 97% peak memory utilization, we observed zero allocation stalls, retries, or out-of-memory events. Figure 13. Left: Effectiveness of model weight virtualization. Right: Fair scheduler with varying slacks. 8.4. Tradeoff between Fairness and Efficiency ServerlessT2I’s scheduler exposes a configurable fairness slack to balance fairness and efficiency. We set a microbenchmark on an 8-GPU testbed with two tenants, issuing basic Z-Image-Turbo and Flux1-Schnell workflows, respectively. In Fig. 13-right, ServerlessT2I bounds the service difference according to the configured slack when requests are backlogged. Larger slack values permit greater transient imbalance, but improve scheduling flexibility and hence serving efficiency: SLO attainment increases from 62% with strict fairness (Slack=0) to 75%, 81%, and 84% with slack values of 5, 10, and 20, respectively. 8.5. System Overhead Execution Overhead. Decomposing a monolithic workflow into a model DAG introduces overhead from data communication and control-plane coordination. We quantify this overhead by comparing ServerlessT2I with monolithic baselines on S1’s workflows. For each workflow, we measure the execution latency of requests that generate the identical image. Compared with Diffusers, on which ServerlessT2I builds its model components, ServerlessT2I adds only 3% average latency overhead. While vLLM-Omni reduces execution latency by 8% relative to ServerlessT2I, it falls short in the end-to-end evaluation (§8.2). ServerlessT2I at scale. To show ServerlessT2I remains efficient at large scale, we conduct simulation-based experiments on a 256-GPU setup under high concurrency, with 600 inflight requests. The simulator models ServerlessT2I’s procedures with request latencies matching measured values. Across the workflows in S1, ServerlessT2I incurs only 3.3% overhead of total execution time, indicating that neither the control plane nor the data plane becomes a bottleneck at this scale. 9. Discussion and Related Work Can ServerlessT2I serve video generation models? While video generation also uses diffusion models, its serving characteristics differ substantially from T2I, placing it outside ServerlessT2I’s scope. First, video models exhibit different loading and computation profiles. For the transformer of Wan2.2-14B (Wan et al., 2025), the one-step denoising inference latency for a 480P video on an H800 is 2× higher than its loading latency, so loading can be pipelined with computation. Second, video generation service is typically exposed through APIs rather than user-composed workflows, a poor fit for serverless execution. As evidence, Wan2.2-14B (Mar. 2025) has only 2 community adapters on HuggingFace, whereas Z-Image (Nov. 2025) has 135, both released by the same Alibaba Tongyi team. Model Serving in Serverless Clouds. To our knowledge, prior serverless model-serving systems target DNN and LLM inference rather than T2I workflows (Hu et al., 2025a). To reduce model loading latency, PipeSwitch and Torpor (Bai et al., 2020; Yu et al., 2025b) overlap host-to-GPU model loading with inference, but this is less effective for T2I workflows, where loading can still dominate the execution time of individual stages. BlitzScale and FaaScale (Zhang et al., 2025a; Yu et al., 2026) speed up loading by transferring parameters over high-speed GPU interconnects instead of PCIe. However, this design assumes a small set of repeatedly loaded model types and spare interconnect bandwidth, both of which are less suitable for serverless T2I serving, where workflows use diverse models and interconnects are also needed for intermediate tensor communication. ServerlessLLM (Fu et al., 2024) reduces LLM loading latency with a multi-tier path across disk, host memory, and GPU memory. As discussed in §5, ServerlessT2I instead focuses on memory-to-GPU loading and GPU-resident communication, where pinned memory alone is insufficient. Its disk-to-memory optimization is complementary to ServerlessT2I. ServerlessLLM (Fu et al., 2024), DeepServe (Hu et al., 2025a), and Medusa (Zeng et al., 2025) also optimize LLM-specific state such as KV-cache management and CUDA-graph materialization. These techniques are valuable, but they do not address the workflow heterogeneity and data-movement bottlenecks of serverless T2I serving. T2I Workflow Serving Systems. Existing T2I workflow serving systems (Diffusers, 2025a; BentoML, 2025; sgl-project, 2026; vllm-project, 2026) accelerate individual workflow execution, but they do not target serverless deployment. As a result, they inherit the limitations of deploying T2I workflows as monolithic GPU functions on serverless platforms (§2.3). Nirvana (Agarwal et al., 2024) reduces denoising steps through cached images; DistriFusion (Li et al., 2024) and xDiT (Fang et al., 2024) exploit multi-GPU parallelism; Katz (Li et al., 2025b) parallelizes ControlNets and asynchronously loads LoRAs; TetriServe (Lu et al., 2026) and TridentServe (Xia et al., 2025) adapt sequence parallelism for latency SLOs. However, several of these systems (Xia et al., 2025; Lu et al., 2026; Agarwal et al., 2024; Li et al., 2024) do not support the adapters commonly used in production workloads (Li et al., 2025b; Lin et al., 2025). ServerlessT2I is complementary to these systems: it targets efficient T2I workflow serving on serverless platforms and focuses on serverless-specific challenges in scaling, data movement, and multi-tenant scheduling. Other Model Serving Systems. Prior work on model serving has improved latency (Crankshaw et al., 2017; Wang et al., 2023a; She et al., 2026; Schroeder et al., 2026; Zhu et al., 2022; Jiang et al., 2025; Zhang et al., 2026b; Wang et al., 2026), throughput (Ahmad et al., 2024; Yang et al., 2022; Ruan et al., 2026; Yu et al., 2025a), and resource efficiency (Zhang et al., 2019; Wang et al., 2021; Gunasekaran et al., 2022; Yang et al., 2025; Wang et al., 2023b) across DNNs and LLMs (Yu et al., 2022; Agrawal et al., 2024; Duan et al., 2024; Wu et al., 2024; Mei et al., 2025; Hu et al., 2025b; Yao et al., 2025a; Oliaro et al., 2025; Zhang et al., 2026a; Yu et al., 2025c; He et al., 2025; Yao et al., 2025b; Chen et al., 2025a; Srivatsa et al., 2025; Chen et al., 2025b; Gao et al., 2025; Gong et al., 2025). ServerlessT2I complements this line of work by focusing on T2I workflow serving, which has different computation characteristics. KunServe (Cheng et al., 2026) is the closest to our data plane design, but it is specific to LLM serving: it observes redundant LLM parameters and selectively drops them to free memory for KV cache. 10. Conclusions We presented ServerlessT2I, an efficient serverless inference system for T2I workflows. ServerlessT2I has three key designs: (1) a programming interface that allows users to compose workflows as a model DAG; (2) a unified data plane that harvests slack GPU memory for efficient model loading and data communication; and (3) a fairness-aware scheduler for multi-tenant serverless serving. Overall, ServerlessT2I substantially improves T2I workflow serving efficiency. Under the same GPU budget, it sustains up to 2× higher request rates than existing serving systems; at a fixed request rate, it reduces GPU requirements by up to 3× while meeting SLOs for more than 90% of requests. References Adobe (2025) Create with Adobe Firefly generative AI. Note: https://w.adobe.com/products/firefly.html Cited by: §1. S. Agarwal, S. Mitra, S. Chakraborty, S. Karanam, K. Mukherjee, and S. K. Saini (2024) Approximate caching for efficiently serving text-to-image diffusion models. In Proc. USENIX NSDI, Cited by: §1, §6, §8.1, §9. A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee (2024) Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In Proc. USENIX OSDI, Cited by: §9. S. Ahmad, H. Guan, B. D. Friedman, T. Williams, R. K. Sitaraman, and T. Woo (2024) Proteus: A high-throughput inference-serving system with accuracy scaling. In Proc. ACM ASPLOS, Cited by: §9. A. S. AI (2026) Deploy models with Amazon SageMaker Serverless Inference. Note: https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html Cited by: §1. Amazon (2025) AWS Lambda pricing. Note: https://aws.amazon.com/lambda/pricing/ Cited by: §2.3. N. N. APIs (2026) Memory Management. Note: https://docs.nvidia.com/nvshmem/api/gen/api/memory.html Cited by: §5.2. AWS (2025) Adapt your own inference container for Amazon SageMaker AI. Note: https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-inference-container.html Cited by: §4. M. Azure (2026) Azure documentation. Note: https://learn.microsoft.com/en-us/azure/durable-task/common/durable-task-sequence Cited by: §4. Azure-Samples (2026) Azure Functions PyTorch ML multi-model image classification with Remote Build and Azure File integration. Note: https://github.com/Azure-Samples/azure-functions-pytorch-image-identify/blob/master/classify/__init__.py Cited by: §4. Z. Bai, Z. Zhang, Y. Zhu, and X. Jin (2020) PipeSwitch: fast pipelined context switching for deep learning applications. In Proc. USENIX OSDI, Cited by: §5.1, §8.3, §9. BentoML (2025) comfy-pack: Serving ComfyUI Workflows as APIs. Note: https://w.bentoml.com/blog/comfy-pack-serving-comfyui-workflows-as-apis Cited by: §9. X. Chai, T. Zhou, K. Hu, J. Tan, T. Bie, A. Shen, D. Shen, Q. Xing, S. Song, T. Yang, L. Gao, F. Yu, Z. He, D. Du, Y. Xia, K. Chen, and Y. Chen (2025) Fork in the road: reflections and optimizations for cold start latency in production serverless systems. In Proc. USENIX OSDI, Cited by: §1. H. Chen, W. Xie, B. Zhang, J. Tang, J. Wang, J. Dong, S. Chen, Z. Yuan, C. Lin, C. Qiu, Y. Zhu, Q. Ou, J. Liao, X. Chen, Z. Ai, Y. Wu, and M. Zhang (2025a) KTransformers: unleashing the full potential of cpu/gpu hybrid inference for moe models. In Proc. SOSP, Cited by: §9. L. Chen, D. Feng, E. Feng, Y. Wang, R. Zhao, Y. Xia, P. Xu, and H. Chen (2025b) Characterizing mobile soc for accelerating heterogeneous llm inference. In Proc. SOSP, Cited by: §9. R. Cheng, Y. Lai, X. Wei, R. Chen, and H. Chen (2026) KunServe: parameter-centric memory management for efficient memory overloading handling in llm serving. In Proc. EuroSys, Cited by: §9. S. Choi, S. Park, M. Lee, and J. Choo (2021) VITON-hd: high-resolution virtual try-on via misalignment-aware normalization. In Proc. CVPR, Cited by: §2.2. A. Cloud (2025) Alibaba Cloud Function Compute Billing Overview. Note: https://w.alibabacloud.com/help/en/functioncompute/fc/product-overview/billing-overview-of-fc Cited by: §2.3. A. Cloud (2026a) Build a text-to-image service with comfyui and sd/flux using function compute. Note: https://w.alibabacloud.com/help/en/functioncompute/fc/use-cases/building-a-text-to-image-service-that-uses-comfyui-and-sd-flux-through-function-compute?spm=a2c63.p38356.help-menu-2508973.d_4_0_0.62b54f6bCLyXzC&scm=20140722.H_2872195._.OR_help-T_intl˜en-V_1 Cited by: §2.3, §2.3. A. Cloud (2026b) Create a gpu function. Note: https://w.alibabacloud.com/help/en/functioncompute/fc/user-guide/creating-a-gpu-function/ Cited by: §1, §2.3. G. Cloud (2026c) Google Cloud Serverless Computing. Note: https://cloud.google.com/serverless Cited by: §1. ComfyUI (2025) ComfyUI: the most powerful and modular visual ai engine and application.. GitHub. Note: https://github.com/comfyanonymous/ComfyUI Cited by: §1, §2.3. D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica (2017) Clipper: A low-latency online prediction serving system. In Proc. USENIX NSDI, Cited by: §9. H. Diffusers (2025a) Create a server. Note: https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/create_a_server.md Cited by: §8.1, §9. H. Diffusers (2025b) Philosophy. Note: https://huggingface.co/docs/diffusers/en/conceptual/philosophy Cited by: §2.3. A. C. Documentation (2026) Set function quotas. Note: https://w.alibabacloud.com/help/en/functioncompute/fc/user-guide/overview-of-configuring-the-maximum-number-of-on-demand-instances Cited by: §1, §2.3. J. Duan, R. Lu, H. Duanmu, X. Li, X. Zhang, D. Lin, I. Stoica, and H. Zhang (2024) MuxServe: flexible spatial-temporal multiplexing for multiple LLM serving. In Proc. ICML, Cited by: §9. P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, and R. Rombach (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Proc. ICML, Cited by: §6. J. Fang, J. Pan, X. Sun, A. Li, and J. Wang (2024) XDiT: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01738. Cited by: §1, §1, §2.2, §2.3, §2.3, §2.3, §4, §6, §8.1, §8.3, §9. FastAPI (2025) FastAPI. Note: https://github.com/fastapi/fastapi Cited by: §7. Y. Fu, L. Xue, Y. Huang, A. Brabete, D. Ustiugov, Y. Patel, and L. Mai (2024) ServerlessLLM: Low-Latency serverless inference for large language models. In Proc. USENIX OSDI, Cited by: §9. S. Gao, Q. Wang, S. Zeng, Y. Lu, and J. Shu (2025) WEAVER: efficient multi-llm serving with attention offloading. In Proc. ATC, Cited by: §9. L. Gong, K. Liu, X. Li, S. Han, P. P. C. Lee, Y. Hu, and D. Feng (2025) HyperGen: optimizing generative inference with long prompts for resource-constrained systems. In Proc. APSys, Cited by: §9. GoogleCloudPlatform (2026) Google Cloud Platform Python Samples. Note: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/run/image-processing/main.py Cited by: §4. Grokipedia (2026) Multi-gpu support in comfyui. Note: https://grokipedia.com/page/Multi-GPU_Support_in_ComfyUI Cited by: §2.3. A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace (2020) Serving DNNs like Clockwork: Performance predictability from the bottom up. In Proc. USENIX OSDI, Cited by: §8.2. J. R. Gunasekaran, C. S. Mishra, P. Thinakaran, B. Sharma, M. T. Kandemir, and C. R. Das (2022) Cocktail: A multidimensional optimization for model serving in cloud. In Proc. USENIX NSDI, Cited by: §9. Y. He, H. Yang, Y. Lu, A. Klimović, and G. Alonso (2025) Resource multiplexing in tuning and serving large language models. In Proc. ATC, Cited by: §9. E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In Proc. ICLR, Cited by: §1, Figure 1, §2.1, §2.1. J. Hu, J. Xu, Z. Liu, Y. He, Y. Chen, H. Xu, J. Liu, J. Meng, B. Zhang, S. Wan, G. Dan, Z. Dong, Z. Ren, C. Liu, T. Xie, D. Lin, Q. Zhang, Y. Yu, H. Feng, X. Chen, and Y. Shan (2025a) DEEPSERVE: serverless large language model serving at scale. In Proc. ATC, Cited by: §9. Z. Hu, V. Murthy, Z. Pan, W. Li, X. Fang, Y. Ding, and Y. Wang (2025b) HedraRAG: co-optimizing generation and retrieval for heterogeneous rag workflows. In Proc. SOSP, Cited by: §9. HuggingFace (2026) Using GPU Spaces. Note: https://huggingface.co/docs/hub/en/spaces-gpus Cited by: §2.3, §2.3. W. Jiang, S. Subramanian, C. Graves, G. Alonso, A. Yazdanbakhsh, and V. Dadu (2025) RAGO: systematic performance optimization for retrieval-augmented generation serving. In Proc. ISCA, Cited by: §9. X. Jiang, S. Li, L. Yang, T. Feng, Z. Di, W. Lu, G. Zhu, X. Lin, K. Liu, Y. Yu, et al. (2026) FlashPS: efficient generative image editing with mask-aware caching and scheduling. In Proc. EuroSys, Cited by: §2.2, §5.1. E. Jonas, Q. Pu, S. Venkataraman, I. Stoica, and B. Recht (2017) Occupy the cloud: distributed computing for the 99%. In Proc. SoCC, Cited by: §2.3. X. Ju, X. Liu, X. Wang, Y. Bian, Y. Shan, and Q. Xu (2024) BrushNet: a plug-and-play image inpainting model with decomposed dual-branch diffusion. In Proc. ECCV, Cited by: §1. K. C. Knowlton (1965) A fast storage allocator. Commun. ACM. Cited by: §5.2. K. C. Knowlton (1966) A programmer’s description of l6. Commun. ACM. Cited by: §5.2. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proc. SOSP, Cited by: §3.1. B. F. Lab (2025) FLUX.2: frontier visual intelligence. Note: https://bfl.ai/blog/flux-2 Cited by: §1, §8.1. B. F. Labs (2024) FLUX. Note: https://github.com/black-forest-labs/flux Cited by: §1, §1, §5.2, §6, §8.1. M. Li, T. Cai, J. Cao, Q. Zhang, H. Cai, J. Bai, Y. Jia, M. Liu, K. Li, and S. Han (2024) DistriFusion: Distributed parallel inference for high-resolution diffusion models. In Proc. IEEE/CVF CVPR, Cited by: §1, §1, §2.2, §2.3, §2.3, §3.1, §6, §8.1, §8.3, §9. S. Li, H. Lu, T. Wu, M. Yu, Q. Weng, X. Chen, Y. Shan, B. Yuan, and W. Wang (2025a) Toppings: cpu-assisted, rank-aware adapter serving for LLM inference. In Proc. USENIX ATC, Cited by: §5.1. S. Li, W. Wang, J. Yang, G. Chen, and D. Lu (2023a) Golgi: performance-aware, resource-efficient function scheduling for serverless computing. In Proc. SoCC, Cited by: §2.3. S. Li, L. Yang, X. Jiang, H. Lu, Z. Di, W. Lu, J. Chen, K. Liu, Y. Yu, T. Lan, G. Yang, L. Qu, L. Zhang, and W. Wang (2025b) Katz: efficient workflow serving for diffusion models with many adapters. In Proc. USENIX ATC, Cited by: §1, §1, §1, §2.1, §2.1, §2.1, §2.2, §2.2, §2.3, §2.3, §2.3, §3.1, §4, §5.1, §5.2, §5.3, §6, §8.1, §8.3, §9. Z. Li, L. Zheng, Y. Zhong, V. Liu, Y. Sheng, X. Jin, Y. Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica (2023b) AlpaServe: statistical multiplexing with model parallelism for deep learning serving. In Proc. USENIX OSDI, Cited by: §8.2. Y. Lin, S. Wu, S. Luo, H. Xu, H. Shen, C. Ma, M. Shen, L. Chen, C. Xu, L. Qu, and K. Ye (2025) Understanding diffusion model serving in production: a top-down analysis of workload, scheduling, and resource efficiency. In Proc. ACM SoCC, Cited by: §1, §2.1, §2.2, Figure 4, §3.1, §3.1, §5.1, §9. R. Lu, S. He, W. Tan, S. Li, R. Wu, J. J. Ma, A. Chen, and M. Chowdhury (2026) TetriServe: efficiently serving mixed dit workloads. In Proc. ASPLOS, Cited by: §1, §6, §9. A. Mahgoub, E. B. Yi, K. Shankar, S. Elnikety, S. Chaterji, and S. Bagchi (2022) ORION and the three rights: sizing, bundling, and prewarming for serverless DAGs. In Proc. USENIX OSDI, Cited by: §5.1. Y. Mei, Y. Zhuang, X. Miao, J. Yang, Z. Jia, and R. Vinayak (2025) Helix: serving large language models over heterogeneous GPUs and network via max-flow. In Proc. ASPLOS, Cited by: §9. Midjourney (2025) Midjourney AI. Note: https://w.midjourney.com/explore Cited by: §1. Modal (2025) How OpenArt scaled their Gen AI art platform on hundreds of GPUs. Note: https://modal.com/blog/openart-case-study Cited by: §1. P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica (2018) Ray: a distributed framework for emerging AI applications. In Proc. USENIX OSDI, Cited by: §5.3. P. D. Notes (2026) CUDA semantics. Note: https://docs.pytorch.org/docs/2.12/notes/cuda.html Cited by: §5.3. NVIDIA (2025a) NVIDIA OpenSHMEM Library (NVSHMEM) Documentation, Environment Variables. Note: https://docs.nvidia.com/nvshmem/api/gen/env.html Cited by: §5.3. NVIDIA (2025b) NVIDIA OpenSHMEM Library (NVSHMEM) Documentation. Note: https://docs.nvidia.com/nvshmem/api/index.html Cited by: §5.2, §7. NVIDIA (2025c) NVSHMEM APIs, Memory Management. Note: https://docs.nvidia.com/nvshmem/api/gen/api/memory.html Cited by: §5.3. G. Oliaro, X. Miao, X. Cheng, V. Kada, R. Gao, Y. Huang, R. Delacourt, A. Yang, Y. Wang, M. Wu, C. Unger, and Z. Jia (2025) FlexLLM: a system for co-serving large language model inference and parameter-efficient finetuning. arXiv preprint arXiv:2402.18789. Cited by: §9. OpenAI (2020) OpenAI API. Note: https://openai.com/index/openai-api/ Cited by: §1, §2.2, §3.2, §7. OpenAI (2025a) Introducing 4o Image Generation. Note: https://openai.com/index/introducing-4o-image-generation/ Cited by: §1. OpenAI (2025b) OpenAI DALLůE 2. Note: https://openai.com/index/dall-e-2/ Cited by: §1. OpenAI (2025c) Rate limits. Note: https://developers.openai.com/api/docs/guides/rate-limits Cited by: §2.3. C. Ruan, Y. Chen, D. Tian, Y. Shi, Y. Wu, J. Li, and C. Li (2026) Libra: flexible request partitioning and scheduling for serving unbalanced and dynamic LLM workloads. In Proc. NSDI, Cited by: §9. J. Schleier-Smith, V. Sreekanti, A. Khandelwal, J. Carreira, N. J. Yadwadkar, R. A. Popa, J. E. Gonzalez, I. Stoica, and D. A. Patterson (2021) What serverless computing is and should become: the next phase of cloud computing. Commun. ACM. Cited by: §1, §2.3, §2.3, §4, §5.3. L. G. Schroeder, A. Desai, A. Cuadron, K. Chu, S. Liu, M. Zhao, S. Krusche, A. Kemper, M. Zaharia, and J. E. Gonzalez (2026) VCache: verified semantic prompt caching. In Proc. ICLR, Cited by: §9. sgl-project (2026) SGLang diffusion. Note: https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen Cited by: §1, §2.3, §2.3, §4, §6, §9. M. Shahrad, R. Fonseca, I. Goiri, G. Chaudhry, P. Batum, J. Cooke, E. Laureano, C. Tresness, M. Russinovich, and R. Bianchini (2020) Serverless in the wild: characterizing and optimizing the serverless workload at a large cloud provider. In Proc. ATC, Cited by: §2.2, §2.2. J. She, Z. Li, H. Du, S. Wu, W. Zheng, E. P. Xing, Z. Liu, H. Yao, J. Xue, and Q. Ho (2026) PLA-serve: a prefill-length-aware LLM serving system. In Proc. MLSys, Cited by: §9. Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica (2024) Fairness in serving large language models. In Proc. USENIX OSDI, Cited by: §1, §6, §6, §6, §6. V. Srivatsa, Z. He, R. Abhyankar, D. Li, and Y. Zhang (2025) Preble: efficient distributed prompt scheduling for LLM serving. In Proc. ICLR, Cited by: §9. stabilityai (2025) stable-diffusion-3.5-large. Note: https://huggingface.co/stabilityai/stable-diffusion-3.5-large Cited by: §1, §8.1. Suyi32 (2025) distrifuser-controlnet. Note: https://github.com/Suyi32/distrifuser-controlnet/commit/e2099655ef0052e1f5f030007f52b25faf06b4 Cited by: §2.2, §2.3. I. Team, H. Cai, S. Cao, R. Du, P. Gao, S. Hoi, Z. Hou, S. Huang, D. Jiang, X. Jin, L. Li, Z. Li, Z. Li, D. Liu, D. Liu, J. Shi, Q. Wu, F. Yu, C. Zhang, S. Zhang, and S. Zhou (2025) Z-image: an efficient image generation foundation model with single-stream diffusion transformer. External Links: 2511.22699 Cited by: §1, §8.1. vllm-project (2026) VLLM omni. Note: https://github.com/vllm-project/vllm-omni Cited by: §1, §1, §2.3, §2.3, §2.3, §4, §6, Figure 10, §8.1, §8.1, §9. P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, D. Nair, S. Paul, W. Berman, Y. Xu, S. Liu, and T. Wolf (2022) Diffusers: state-of-the-art diffusion models. GitHub. Note: https://github.com/huggingface/diffusers Cited by: §1, §1, §2.3, §2.3, §2.3, §4, §5.3, §6, Figure 11, §8.1, §2.3. T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X. Huang, X. Xu, Y. Kou, Y. Lv, Y. Li, Y. Liu, Y. Wang, Y. Zhang, Y. Huang, Y. Li, Y. Wu, Y. Liu, Y. Pan, Y. Zheng, Y. Hong, Y. Shi, Y. Feng, Z. Jiang, Z. Han, Z. Wu, and Z. Liu (2025) Wan: open and advanced large-scale video generative models. External Links: 2503.20314, Link Cited by: §9. J. Wang, W. Xie, M. Zhang, B. Zhang, J. Dong, Y. Zhu, C. Lin, J. Tang, Y. Han, Z. Ai, X. Chen, Y. Wu, and C. Jiang (2026) From prefix cache to fusion rag cache: accelerating llm inference in retrieval-augmented generation. Proc. ACM Manag. Data. Cited by: §9. L. Wang, L. Yang, Y. Yu, W. Wang, B. Li, X. Sun, J. He, and L. Zhang (2021) Morphling: Fast, near-optimal auto-configuration for cloud-native model serving. In Proc. ACM SoCC, Cited by: §9. Y. Wang, K. Chen, H. Tan, and K. Guo (2023a) Tabi: An efficient multi-level inference system for large language models. In Proc. ACM EuroSys, Cited by: §9. Y. Wang, B. Feng, Z. Wang, T. Geng, K. Barker, A. Li, and Y. Ding (2023b) MGG: Accelerating graph neural networks with fine-grained intra-kernel communication-computation pipelining on multi-GPU platforms. In Proc. USENIX OSDI, Cited by: §9. B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin (2024) LoongServe: efficiently serving long-context large language models with elastic sequence parallelism. In Proc. SOSP, Cited by: §9. H. Wu, J. Deng, M. Yu, Y. Yu, Y. Liu, H. Fan, S. Wu, and W. Wang (2026) Efficient data passing for serverless inference workflows: a gpu-centric approach. In Proc. ACM EuroSys, Cited by: §5.2. Y. Xia, F. Fu, H. Yuan, H. Zhang, X. Miao, Y. Liu, S. Ling, J. Jiang, and B. Cui (2025) TridentServe: a stage-level serving system for diffusion pipelines. External Links: 2510.02838 Cited by: §9. Y. Xu, T. Gu, W. Chen, and A. Chen (2025) OOTDiffusion: outfitting fusion based latent diffusion for controllable virtual try-on. Proc. AAAI. Cited by: §1. L. Yang, Y. Wang, Y. Yu, Q. Weng, J. Dong, K. Liu, C. Zhang, Y. Zi, H. Li, Z. Zhang, N. Wang, Y. Dong, M. Zheng, L. Xi, X. Lu, L. Ye, G. Yang, B. Fu, T. Lan, L. Zhang, L. Qu, and W. Wang (2025) GPU-disaggregated serving for deep learning recommendation models at scale. In Proc. USENIX NSDI, Cited by: §9. Y. Yang, L. Zhao, Y. Li, H. Zhang, J. Li, M. Zhao, X. Chen, and K. Li (2022) INFless: A native serverless system for low-latency, high-throughput inference. In Proc. ACM ASPLOS, Cited by: §9. J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang (2025a) CacheBlend: fast large language model serving for RAG with cached knowledge fusion. In Proc. EuroSys, Cited by: §9. X. Yao, Q. Hu, and A. Klimovic (2025b) DeltaZip: efficient serving of multiple full-model-tuned llms. In Proc. EuroSys, Cited by: §9. H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang (2023) IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721. Cited by: §2.1. G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun (2022) Orca: A distributed serving system for transformer-based generative models. In Proc. USENIX OSDI, Cited by: §9. L. Yu, J. Lin, and J. Li (2025a) Stateful large language model serving with pensieve. In Proc. EuroSys, Cited by: §9. M. Yu, T. Cao, W. Wang, and R. Chen (2023) Following the data, not the function: rethinking function orchestration in serverless computing. In Proc. NSDI, Cited by: §5.2, §5.3. M. Yu, A. Wang, D. Chen, H. Yu, X. Luo, Z. Li, W. Wang, R. Chen, D. Nie, H. Yang, and Y. Ding (2025b) Torpor: gpu-enabled serverless computing for low-latency, resource-efficient inference. In Proc. ATC, Cited by: §5.1, §8.3, §9, footnote *. M. Yu, R. Yang, C. Jia, Z. Su, S. Yao, T. Lan, Y. Yang, Z. Wang, Y. Cheng, W. Wang, A. Wang, and R. Chen (2026) FaaScale: unlocking fast llm scaling for serverless inference. In Proc. MLSys, Cited by: §5.2, §9. Y. Yu, Y. Gan, N. Sarda, L. Tsai, J. Shen, Y. Zhou, A. Krishnamurthy, F. Lai, H. Levy, and D. Culler (2025c) IC-cache: efficient large language model serving via in-context caching. In Proc. SOSP, Cited by: §9. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauly, M. J. Franklin, S. Shenker, and I. Stoica (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In Proc. NSDI, Cited by: §5.3. S. Zeng, M. Xie, S. Gao, Y. Chen, and Y. Lu (2025) Medusa: accelerating serverless llm inference with materialization. In Proc. ASPLOS, Cited by: §9. zeromq (2025) ZeroMQ. Note: https://github.com/zeromq/pyzmq Cited by: §7. C. Zhang, M. Yu, W. Wang, and F. Yan (2019) MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In Proc. USENIX ATC, Cited by: §9. D. Zhang, H. Wang, Y. Liu, X. Wei, Y. Shan, R. Chen, and H. Chen (2025a) Fast and live model auto scaling without caching. In Proc. USENIX OSDI, Cited by: §9. L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In Proc. IEEE/CVF ICCV, Cited by: §1, Figure 1, §2.1, §2.1. L. Zhang, A. Rao, and M. Agrawala (2025b) Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport. In Proc. ICLR, Cited by: §2.1. L. Zhang (2025) Fooocus. Note: https://github.com/lllyasviel/Fooocus Cited by: §2.1. W. Zhang, Z. Wu, Y. Mu, R. Ning, B. Liu, N. Sarda, M. Lee, and F. Lai (2026a) JITServe: slo-aware llm serving with imprecise request information. In Proc. USENIX NSDI, Cited by: §9. Z. Zhang, P. Dash, Y. C. Hu, Q. Xu, J. Li, and H. Guan (2026b) Rethinking dvfs for mobile llms: unified energy-aware scheduling with core. In Proc. MLSys, Cited by: §9. H. Zhu, R. Wu, Y. Diao, S. Ke, H. Li, C. Zhang, J. Xue, L. Ma, Y. Xia, W. Cui, F. Yang, M. Yang, L. Zhou, A. Cidon, and G. Pekhimenko (2022) ROLLER: fast and efficient tensor compilation for deep learning. In Proc. USENIX OSDI, Cited by: §9. N. D. Zone (2026) Peer Device Memory Access. Note: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__PEER.html Cited by: §5.2. Appendix A Caching Workflows in Host Memory This appendix provides additional details on the trace replay described in §5.1. A T2I workflow is large, often totaling tens of GiBs, and loading its weights from external storage can introduce substantial latency. However, production workflow popularity is highly skewed: a small number of workflows account for the vast majority of requests. This skew makes host memory an effective intermediate cache tier. By storing the weights of only the most popular workflows in otherwise underutilized host memory, ServerlessT2I can avoid reading external storage for nearly all model loads, leaving only the much faster host-to-GPU transfer on the critical path. To quantify this effect, we replay our production trace using host memory caches of varying capacities. We use least-recently-used (LRU) replacement and measure the cache miss rate, defined as the fraction of requests whose workflow weights are not present in host memory and therefore must be fetched from remote storage. The miss rate drops quickly as cache capacity increases and then saturates near zero: caching the 1010 most popular workflows reduces the miss rate to 0.37%0.37\%, while caching the top 1515 reduces it further to 0.09%0.09\%. Thus, with only the top 1515 workflows cached in host memory, more than 99%99\% of model loads are served from host memory rather than remote storage. We also evaluate a least-frequently-used (LFU) policy and observe similar results, suggesting that the benefit comes primarily from the inherent popularity skew rather than from a specific eviction policy. Since host memory is largely underutilized in our production setting (Fig. 4), caching these popular workflows imposes little additional resource cost. Based on this observation, ServerlessT2I keeps popular workflow weights in host memory and focuses the remainder of the loading design on optimizing the host-to-GPU transfer path.