Paper deep dive
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
Utopia Meng, Unicornt Zhao, Derek Li, Goalen Gao, Frank Du
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/21/2026, 4:41:35 AM
Summary
The paper introduces Talaria, a session-aware serverless multi-model LLM serving system designed to optimize Session Completion Time (SCT) for tool-using agents. Talaria addresses spatial and temporal mismatches in existing schedulers by implementing a session-aware router that uses soft reservations to preserve KV locality and a Session-Prefill (SP) mechanism for mid-slot admission. It also features an instance-local memory substrate to manage HBM stability and weight staging. Experiments on a TP=8 server show Talaria reduces p50 SCT from 1000s to 189s and p95 from 2296s to 867s compared to a baseline scheduler.
Entities (19)
Relation Signals (15)
Talaria → optimizes → Session Completion Time
confidence 95% · Talaria cuts p50 SCT from 1000 s to 189 s
Talaria → optimizes → Session Completion Time
confidence 95% · Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s
Talaria → uses → Session-Prefill
confidence 95% · Session-prefill (SP) admits budget-eligible continuations before the active model slot closes.
Talaria → evaluatedon → SWE-Bench
confidence 90% · On a single TP=8 server, we replay 30 SWE-Bench model-sessions
Soft Reservation → preserves → KV Cache
confidence 90% · soft reservations account for likely returns in the last serving instance's admission budget... preserves host-restorable KV
Talaria → uses → Session-Prefill
confidence 90% · Session-prefill (SP) admits budget-eligible continuations before the active model slot closes
Talaria → uses → Soft Reservation
confidence 90% · Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns
Talaria → evaluatedon → SWE-Bench
confidence 88% · we replay 30 SWE-Bench model-sessions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.
Tags
Links
- Source: https://arxiv.org/abs/2607.17181v1
- Canonical: https://arxiv.org/abs/2607.17181v1
Trouble viewing inline? Open PDF directly →
Full Text
91,037 characters extracted from source content.
Expand or collapse full text
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs Utopia MengUnicornt ZhaoDerek LiGoalen GaoFrank Du 2197096044@q.comunicornt@unicornt.xyz940619089@q.com94205123@q.com1093782566@q.com Abstract Serverless multi-model LLM systems multiplex popularity- skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only rout- ing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model’s next slot. Both failures are especially costly for hundred-billion- parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi- model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks place- ments by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance’s admission budget. Session-prefill (SP) ad- mits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses sta- ble, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model- sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging dis- abled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3× and 2.6×. 1 Introduction Modern LLM serving platforms host popularity-skewed model catalogs on shared GPU clusters: a few hot models carry stable traffic, while a long tail is invoked only inter- mittently [1–5]. Statically reserving GPUs for every model wastes accelerator capacity on the tail, motivating serverless multi-model serving: demand-driven opening, closing, and time-sharing of models over pooled GPUs. ServerlessLLM optimizes cluster placement and cold-start loading [6]; Al- paServe, MuxServe, and Aegaeon multiplex or fast-switch models on pooled resources [5, 7, 8]. These techniques make model residency a first-class concern, but their placement and scheduling abstractions do not explicitly preserve continuity across calls in an agent session. Large models intensify this gap: when the aggregate working set exceeds HBM capacity, a placement miss can trigger weight loading together with KV restoration or prefix recomputation. Tool-using agents expose why a request alone is an insuffi- cient scheduling unit. A user task alternates between model calls and tool execution, carrying state from one call to the next [9–12]. Across 445 SWE-Bench Verified sessions [13] (7,386 calls), median session depth is 11 model calls (p95: 49). Prompt-length measurements (n = 5,697) have a me- dian of 25K tokens (p95: 105K), and 96.8% of measured continuations (n = 5,270) reuse more than half of the prior call’s prefix. The median tool-use gap is only 0.39 s—over an order of magnitude shorter than the 4.3 s needed to recom- pute a median-length prefix on Qwen3-235B/TP=8. These timescales create a continuity window: continuations return quickly, so preserving their large, mostly reusable prefixes can avoid seconds of recomputation. Treating each return as a fresh request repeatedly incurs avoidable state-recovery cost across a session. Per-call scheduling therefore misaligns with session com- pletion time along two coupled dimensions. Spatial mis- match: a router using only load or queue length may send a continuation to the least-loaded instance, which may hold neither the target weights nor the session KV; for long-context sessions, KV movement or prefix recomputation can exceed the queueing delay it avoids. Temporal mismatch: even when routing is correct, a multi-model scheduler that rotates through models in fixed-length rounds [5] may defer a return- ing session until the next slot for that model—KV locality is preserved spatially but squandered on the time axis. Un- der prefill/decode disaggregation, the same locality problem spans tiers: the returning session enters through a prefill path, while reusable KV is valuable on the decode side. These failure modes compound. KV-aware routing can pre- serve the right state yet still leave a continuation waiting until the target model’s next slot; mid-slot admission can exploit 1 arXiv:2607.17181v1 [cs.DC] 19 Jul 2026 Cluster layer Instance layer Memory substrate Router Hot PoolCold Pool stable hot modelsmulti-model slots + SP Instance 1 Model A KV Cache Instance 2 Model A KV Cache Instance 3 Model C Model D (activate) KV Cache Instance 4 Model E Model F (activate) KV Cache slot PDSPD slot PDSPD dynamic placement KV migration mooncake backend Multi-model Memory Substrate (HKVR · HBM Allocator · D2D Accelerate) requestKV migrationsoft reservation turn n outturn n+1 in w kv cache hittelemetry updates Figure 1: Talaria architecture. The router preserves return affinity across hot and cold pools; SP admits eligible mid-slot returns, while each cold-pool instance preserves restorable KV across model switches. preserved state only when routing has placed the continuation on the right instance. The cluster layer must land a return on a useful instance, and the instance layer must expose a timely execution opportunity. We present Talaria, a serverless multi-model LLM infer- ence system for agentic workloads (Figure 1). Talaria realizes session continuity through coordinated placement, admission, and state restoration. At deployment time, it configures a hot pool of pinned single-model instances and a cold pool whose instances token-level time-share a set of long-tail models. A session-aware router uses soft reservations to account for likely returns in the serving instance’s next eligible model-slot budget (§3.3); calls without a usable reservation are placed by ranking candidate instances according to model residency, KV residency, and pressure. Within each cold-pool slot, co- located prefill/decode execution enables session-prefill (SP) to admit eligible returns mid-slot, reusing device-resident KV or restoring host-restorable KV while the model remains active. A per-instance memory substrate coordinates HBM allocation, host-side KV state, and weight staging to reduce the switching and restoration costs charged to each round. On a single TP=8 server, we replay ten SWE-Bench is- sues across Qwen3-235B, GLM5-nvfp4, and Qwen3.5-122B- A10B, yielding 30 model-sessions and 960 calls. For each model-session, fixed replay holds request bodies, model and session IDs, per-session call order, completion-token counts, and return gaps constant across configurations. This prevents agent-path divergence; global interleaving may still differ be- cause policies change completion times. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria’s instance-local mechanisms cut p50 session completion time from 1000 s to 189 s and p95 from 2296 s to 867 s—speedups of 5.3×and 2.6×. On a separate two-worker testbed, residency-aware routing reduces high-load avoidable model opens from 37 to 1 and TTFT p95 from 8.07 s to 5.28 s versus least-pressure routing. This paper makes three contributions. • Session continuity as a scheduling abstraction. We identify session continuity as the missing unit in server- less multi-model serving for agents. From 445 agent ses- sions, we quantify the depth, return timing, and reusable state that make request-level scheduling costly, then iso- late spatial placement and temporal admission with con- trolled experiments. •Joint session-continuity scheduling. At the cluster layer, soft reservation accounts for likely returns without pinning device KV; at the instance layer, SP provides budgeted mid-slot admission. The two mechanisms ad- dress complementary failure modes along the same re- turn path. •Switch-resilient instance substrate. A stable HBM layout and host-restorable KV preserve usable session state across model switches; opportunistic D2D staging reduces the switching cost charged to each round. 2 0255075100 LLM calls per session 0 25 50 75 100 CDF (%) p50=11 p95=49 (a) Session depth 024 Tool gap (s) p50=0.39s p95=1.3s (b) Inter-call gap 050100150200 Prompt tokens (K) p50=25K p95=105K (c) Context length SWE-Bench Verified · 445 sessions, 7,386 calls · prompt n=5,697 · prefix n=5,270 Figure 2: SWE-Bench Verified sessions are deep (median 11 calls), fast-returning (median gap 0.39 s), and long-context (median 25K tokens). Axes truncate at 100 calls, 5 s, and 200K tokens; the gap CDF excludes five>30s environment- startup intervals. 2 Background and Motivation We first characterize agent sessions in a real agent trace (§2.1), then show how request-level placement loses spatial locality (§2.2) and how round-based multi-model scheduling loses temporal locality (§2.3). We close with the memory- management requirements that make session-aware schedul- ing implementable (§2.4). 2.1 Agent sessions are stateful We define an agent session as a single user task: the agent issues a sequence of model calls interleaved with tool invoca- tions until the task completes. This request pattern follows the agent execution model introduced by tool-use and software- engineering agents [9–11]. We characterize 445 completed agent executions of SWE-Bench Verified tasks [13], total- ing 7,386 LLM calls. Prompt length is available for 5,697 calls and reusable-prefix overlap for 5,270 continuations. We combine these traces with measured Qwen3-235B and GLM5- nvfp4 profiles to quantify recovery cost. Figure 2 summarizes session depth, tool gaps, and prompt lengths. Session depth. The median session issues 11 model calls, with a p95 of 49 and a maximum of 187. A request-level scheduler therefore sees one user task as many unrelated ar- rivals, hiding the prefix reuse and timing dependence between consecutive calls. Short inter-call gaps. The median tool-use gap is only 0.39 s, with a p95 of 1.32 s. This is more than ten times shorter than recomputing the median-length prefix, as mea- sured below. A return therefore often arrives while its prior state remains worth preserving, whether it is still in HBM or restorable from host memory. Continuation inputs. Prompts are long and highly repeti- tive: the median call has 25K prompt tokens, the p95 reaches 105K, and 96.8% of measured continuations reuse more than half of the previous call’s prefix. Recomputing a 25K-token prefix on Qwen3-235B/TP=8 costs 4.3 s TTFT at p50; at 100K tokens, it reaches 10.9 s. A single misplaced return can therefore dominate the latency of an entire session turn. Taken together, these properties mean that agent serving is not scheduling a stream of independent requests. It is schedul- ing a stateful session whose KV state, target model, and return timing persist across calls and must be tracked explicitly by the serving system. 2.2 Spatial mismatch: routing to the wrong in- stance Current serverless routers schedule at request granularity, commonly using load, queue length, or model-load cost as the routing signal [5, 6]. This is a poor proxy for a returning agent call because the fastest instance for the cluster may be the slowest instance for that session. Consider two instances in the same cold pool. Instance A is currently serving the target model and still holds the session’s reusable KV prefix, but has a short queue. Instance B is idle, but may hold neither the target model nor the session KV. A least-load or round-robin router sends the returning call to B because its queue is shorter. For this session, B must first make the target model resident if needed, then reconstruct the prefix. The recomputation alone costs 4.3 s TTFT on Qwen3- 235B/TP=8. Instance A may incur queueing or wait until the target model’s next slot, whereas B must additionally activate the model and recover the prefix. Because recovery cost grows with context length, a shorter queue on B does not imply a faster return. A load-only router cannot see this asymmetry. KV-aware routing [14–16] narrows the gap, but it is still incomplete for multi-model serving. Knowing where the ses- sion KV lives is not enough if the router cannot also tell whether the target model is resident and whether the instance can accept a prefill soon. Good placement therefore requires joint visibility into model residency, KV residency, and in- stance pressure. This visibility must also be session-scoped: the router must remember which instance just served a ses- sion and treat the next call as a likely continuation, not as an unrelated arrival. 2.3Temporal mismatch: admitting at the wrong time Correct placement is not sufficient if the instance scheduler exposes no time window in which the returning session can run. A cold-pool instance that time-shares models in fixed rounds may preserve the session KV in space while failing to use it in time. Consider a round-based token-level scheduler such as Ae- gaeon [5]. The decode side commits to one active model for a slot. After a session finishes a decode step and enters a tool call, the scheduler may move on to another model. When the tool returns 0.39 s later, the session’s KV can still be resident in HBM, but its model window may have closed. 3 The session then waits for that model’s next slot—up to one round—before it can resume. Systems that disaggregate prefill and decode [17–19] ex- pose an analogous cross-tier mismatch: the return enters through a prefill path while its reusable state is valuable on the decode side. Talaria focuses on co-located prefill/decode; extending its admission contract across tiers requires coordi- nated KV ownership (§7). The failure is temporal rather than spatial: the state may be in the right place, but the scheduler cannot admit the session while it is still useful. Request-level autoscaling does not solve this problem because it requeues the returning session as a fresh request, losing the continuity established by the router. What is needed is mid-slot admission: a returning session should be able to re-enter the current model window and reuse preserved KV before the scheduler rotates away. 2.4 Multi-model memory management The router and scheduler above assume that model and KV state can be moved, kept valid, and restored within a bounded time. Existing LLM serving systems already show that KV layout, paging, and virtual memory are central to serving per- formance [20–22]. In large multi-model serving, the require- ment is tighter: freeing HBM for one model can invalidate reusable KV, while slow weight movement consumes the same round budget that mid-slot admission relies on. Memory management becomes part of the scheduling substrate. HBM capacity. Weights are only part of a model’s device footprint. Each model also brings runtime structures such as CUDA graphs, FlashInfer workspaces, and KV pool regions. If each model allocates these regions independently, the foot- prints accumulate even when request load is low, leaving too little HBM for useful KV or for staging the next model. Host-side KV consistency. Cold-pool switching often re- quires session KV to leave HBM and later return. The system must know which model owns each KV block, whether the block is still valid, and whether the current copy is on de- vice or host. Without this metadata, the serving stack can restore stale KV and produce incorrect output, or miss a valid host-side copy and recompute a long prefix unnecessarily. Switch latency. Model switching directly consumes the round budget that the scheduler relies on for TTFT and TPOT. A large-model switch may include weight loading, kernel reinitialization, and KV-pool reorganization, each of which can be second-scale. In our fixed-replay measurements, a logical TP=8 H2D-only switch has a 1.50 s p50 and a 1.52 s mean (§5.6). This consumes a substantial fraction of a 10 s TTFT budget. If switching occupies a large fraction of the round, time-sharing stops being a multiplexing benefit and becomes a throughput tax. These constraints are coupled. Bounded switching requires spare HBM for staging; KV restoration requires host-side consistency; and both must complete quickly enough for mid- slot admission to matter. The memory layer must therefore manage device memory, host KV state, and weight movement as one substrate rather than as independent mechanisms. 3 Design 3.1 System Overview At a high level, Talaria separates multi-model agent serving into a control plane and a data plane. The control plane—a unified router—decides where each session call should land. The data plane—a set of hot and cold inference instances— executes those calls while maintaining the local budget and telemetry contracts that the router’s decisions assume. The router creates a soft reservation when a completed call enters tool execution. It records the serving instance, model, and reusable-prefix handle and charges that instance’s admission budget for a likely return. If the session returns before the lease expires and the target remains healthy, the router sends it back to that instance, where SP can admit it while the KV is device-resident or host-restorable. Requests without a usable reservation take the normal cost path, which weighs instance pressure, KV recovery, and model activation. The router never touches token-level execution. Instances are split into a hot pool and a cold pool. Hot- pool instances obey a single invariant: each pins one model and never switches, eliminating switching tail latency for sta- ble traffic. Cold-pool instances time-share multiple long-tail models using round-based scheduling. Each model slot has the structureP→ D 1 → (SP→ D) ∗ → D fin : queued prefill, initial decode, zero or more interleaved SP (session-prefill) / decode pairs, and final decode. Each SP step batch-admits eligible returns that fit the remaining prefill budget, enabling preserved-prefix reuse without a model switch; excess returns remain queued for a later opportunity. Beneath the instance scheduler, a multi-model memory sub- strate manages the HBM and host memory of each inference instance, addressing the layout, host-KV, and switch-latency constraints identified in §2.4. It provides a stable HBM lay- out, persists session KV across model switches, and reduces switch cost through weight staging. The router observes this substrate through telemetry such as memory pressure and staging slack; the instance scheduler uses it to restore KV and switch models within the round budget. One ownership boundary holds across the design: the router reasons about cluster placement but never executes tokens, while each instance enforces local execution and memory budgets without reconstructing cluster-wide demand. Pinned hot instances avoid the switch path; cold instances expose the telemetry and actuation needed for multiplexing. 4 (a) With soft reservation (b) Without soft reservation Model Call 1 (Instance A) tool execution Model Call 2 (Instance A, KV hit) soft reservation τ Model Call 1 (Instance A) tool execution router sees A as loaded → routes to Instance B state recovery (KV migration / recompute) Model Call 2 (Instance B) session returns state recovery Timeline tool executionmodel call Figure 3: Soft reservation preserves return affinity to an in- stance with device-resident or host-restorable state; it does not pin device KV. 3.2 Configured Hot-Cold Pool Organization Hot-pool instances pin a single model and never switch; this eliminates switching tail latency on stable, high-throughput traffic. Cold-pool instances time-share long-tail models through round-based scheduling (§3.4). Our prototype configures pool membership at deployment time. Within the cold pool, the router prefers instances that already hold the requested model and session prefix, subject to admission and HBM constraints. This concentrates de- mand on useful state and avoids needless model opens; if the preferred instance fails admission, cost ranking selects another feasible cold instance or invokes the configured over- load policy. These are per-call placement decisions and do not reclassify instances. This configured split separates stable high-demand service from the multi-model tail while keeping the continuity mech- anisms independent of a particular pool-sizing controller. The telemetry used for placement could also drive promotion and demotion, but dynamic pool reallocation is outside the current prototype and evaluation. 3.3 Session-Aware Residency Routing The router assigns each session call to an inference instance with two objectives: preserve KV locality from the previous call, and respect model and memory capacity constraints. The two objectives have different time horizons. A session prefix is reusable when a model call completes, but it may move to host or be evicted while the session executes a tool. Preserving or restoring that locality is therefore time-sensitive. Capacity constraints depend on model residency and aggregate round pressure, which evolve over scheduling windows rather than a single routing decision. Soft reservation for returning sessions.The router creates a soft reservation only after a model call completes. From the completion metadata, it records the session id, model, serving instance, reusable-prefix handle, and expiry timet ex p = t done + τ. We chooseτ = 1s from the sensitivity sweep in §5.4; it is near the p90 tool gap in our traces. The reservation is an admission lease, not a GPU-KV pin or an execution guarantee: while live, it charges one configured admission unit against the serving instance’s next eligible model-slot budget. When the next call for the same session and model arrives, the router checks that the lease is live, the instance is healthy, and the prefix remains device-resident or host-restorable. A successful lookup consumes the lease and routes the call to that instance. A successor lease is installed only after this call completes; routing does not refresh the old lease. If any check fails, the router discards the lease and falls back to normal residency-aware placement. The lease reduces the chance that unrelated arrivals con- sume all predicted admission capacity during a tool gap, but it cannot guarantee same-slot admission or a TTFT deadline: the next prompt suffix and concurrent returns are unknown when the lease is created. KV may also move from device to the host registry under memory pressure. Reservation therefore preserves bounded affinity and accounts for likely demand; it does not guarantee device residency. Longer leases preserve more affinity but hold budget longer and can reduce model residency hit rate; §5.4 measures this tradeoff. Placement for non-reserved requests. Requests without an active reservation—new sessions and sessions whose reser- vation has expired—are placed by a cost-ranking policy. The router scores each candidate instance i by Cost(r, i) = ∆ b R req i (r)+ b C KV i (r)+ b C act i (m r ),(1) where all terms are in latency units. The three terms charge in- cremental queue/prefill/decode work, KV recovery, and model activation exactly once.∆ b R req i contains only the incremen- tal queue, prefill, and decode work introduced byr. State recovery and model activation are excluded from this term and charged once by the remaining terms. b C KV i is zero for a device-resident prefix, the measured H2D restore cost for a host-restorable prefix, and the profiled recomputation cost oth- erwise. b C act i is zero whenm r is active, the advertised switch cost when its state is resident or staged, and the measured full- open cost otherwise. Partial staging changes the activation estimate rather than adding a second model-open penalty. The router refreshes its per-model latency profiles from recent round summaries using an exponentially weighted moving average. The in-flight overlay in §4.1 adds calls and reservations forwarded since the latest snapshot. For each can- didate, the router simulates the resulting round load and HBM use, retaining it only if both configured admission checks pass. These are predictive overload controls rather than hard TTFT guarantees under stale telemetry. Miscalibration affects placement quality, while deadline-risk telemetry makes an un- derestimated instance less attractive in subsequent decisions. KV i (s)is valid when instanceireports sessions’s prefix as either device-resident or host-restorable through HKVR with 5 Algorithm 1 Session-aware residency routing. Require:Requestrfor sessions r , modelm r ; instance snap- shots and gateway queue/reservation overlay 1: // Consume a lease created by the previous completion 2: if s r has live lease (s r , m r , h, k, t ex p ) then 3:if h is healthy and KV h (s r ) is valid then 4:consume lease; route r to h 5:return 6:else 7:discard lease 8:end if 9: end if 10: // Cost-based placement 11:C ← / 0 12: for all instances i do 13: simulate request, KV, activation, round, and HBM costs 14:if round-admission and HBM checks pass then 15:if i needs no fresh replica or Eq. (2) passes then 16:C ←C∪i 17:end if 18:end if 19: end for 20: ifC = / 0 then 21:invoke the configured gateway overload policy 22:return 23: end if 24: i ∗ ← arg min i∈C Cost(r, i) 25: route r to i ∗ 26: return an agreed prefix. A new model replica is opened on instancej only when the predicted demand benefit exceeds the opening cost: gain(m, j)> T swa p (m, j)+ μ j ∆W m +ν m ,(2) wheregain(m, j)is the latency-equivalent benefit of opening the replica: the predicted reduction in queued or spillover demand for modelmover the next control window, computed from the router’s observed arrivals and reserved returns.T swa p is the switch latency,∆W m is the added HBM footprint,μ j is the instance-specific HBM pressure price, andν m is a per- replica penalty against unjustified expansion.μ j has units of latency per GiB and converts added HBM footprint into a latency-equivalent cost using instancej’s current memory pressure and staging slack. Algorithm 1 handles call arrival only. The completion hook in §4.1 installs a lease for a possible subsequent call. The lease accounts for a likely return and preserves instance affinity; SP provides a budgeted opportunity to consume that return while the model slot remains open. Neither mechanism alone guarantees same-slot admission. Within the configured cold pool, model-residency preference concentrates demand on P Prefill D DecodeModel AModel B Timeline tool call returns tool call returns (a) With SP admission. next round (b) Without SP (round-level scheduler). slot boundary PDSPDSPDPDSPD switch TTFTTTFT tool call returns PDPDP switchswitch TTFT waits full round TTFT Figure 4: Cold-pool slot structure. SP opens mid-slot prefill windows for eligible continuations before the scheduler ro- tates away. useful instances, while the feasibility filter redirects overflow to nodes with sufficient round budget and staging slack (re- maining capacity for D2D pre-staging, defined in §3.5). 3.4Cold-Pool Scheduler with Session-Prefill (SP) Admission Cold-pool instances must share a GPU group across multi- ple models while still serving agent sessions with short tool gaps. The evaluated design co-locates prefill and decode on the same GPU group, avoiding a cross-tier KV handoff on each turn. A round-based multi-model scheduler then faces a second problem: a session returning from a tool call after its model’s slot has closed must wait until the next full rotation even when its prefix remains reusable. The SP stage resolves both: by staying non-P/D-disaggregated it avoids per-turn KV handoff, and by opening a mid-slot admission window it captures eligible returns before the active model slot closes. Talaria implements a non-P/D-disaggregated token-level scheduler for the cold pool. A cold-pool instance serves sev- eral models on the same tensor-parallel group, executing only one model at any instant. Time is organized into rounds; each active model receives one slot per round, and the instance switches models only at slot boundaries. A slot begins with a queued-prefill phaseP, which admits ordinary work already waiting when the slot opens, and an initial decode phaseD 1 . Zero or more session-prefill/decode pairs may follow before a final decode phase: P → D 1 → SP→ D ∗ → D fin . An SP opportunity occurs only at a decode boundary; it does not preempt an executing kernel. A call is SP-eligible when it targets the active model, is a continuation that arrived after P, and has a device-resident or host-restorable prefix on this instance. At thek-th opportunity, the scheduler forms the eligible setE m,k and selects a bounded batchB m,k ⊆E m,k 6 whose predicted prefill, restore, and HBM costs fit the budget remaining after protected decode. Calls that do not fit remain queued for a later SP opportunity or the next model slot. After the batch completes, decode resumes for all in-flight requests. The final decode phaseD fin is the drain interval after the last SP opportunity in the slot: the scheduler stops admitting new prefill but spends the remaining protected budget on in- flight decode before switching models. New sessions do not bypassPthrough SP. The slot closes when the round budget is exhausted or the scheduler decides to switch models. A round on instance i is therefore R i = ∑ m∈M active i P m + D 1,m + ∑ k (SP m,k + D m,k )+ D m,fin + ∑ (m→m ′ ) S m→m ′ ,(3) whereSP m,k = b T p f (B m,k )+ b T restore (B m,k )is the batched prefill and restore cost at thek-th opportunity, andS m→m ′ is the switch cost exposed by the memory substrate (§3.5). The switch estimate is updated from staging coverage and dirty- chunk telemetry rather than treated as a static model constant. Decode-share budget. The round scheduler controls aver- age token cadence; it does not claim a hard deadline for every inter-token gap. Letd m be the number of decode iterations reserved for modelmin a predicted round, and lett D m be its profiled decode-step time. Define U m = R i − d m t D m as all predicted time in the round not spent onm’s decode iterations, including its ownP/SPphases, other model slots, and switches. For a sequence that remains active throughout the round, the predicted average time per token isR i /d m . To keep this average belowT TPOT , the scheduler requires d m T TPOT − t D m ≥ U m .(4) IfT TPOT ≤ t D m , the target is infeasible for that model profile. The scheduler reserves the correspondingd m decode itera- tions before assigning the remaining slot budget to P and SP. Intuitively, protected decode establishes the target average token cadence; prefill and SP share only the time left after that reservation. The round itself is bounded by the TTFT budget with a safety margin: R i ≤ γT TTFT , γ< 1.(5) This round-length constraint bounds predicted model revisit time; it is not by itself a hard TTFT guarantee because queue- ing across rounds and estimation error can still violate the target. Prefill arrivals and prompt lengths are unknown when a round begins, so prefill is treated as best-effort within the remaining slot budget. The key departure from a standard round-based scheduler is the SP step: whenever an eligible continuation returns during a slot, it is queued for the next SP opportunity rather than immediately deferred until the next slot for that model. At that opportunity, the scheduler admits only the bounded subset that fits the remaining bud- get; it never delays protected decode to admit an unbounded return batch. Decode then resumes for all in-flight sessions, including newly admitted continuations. This interleaving can repeat multiple times within a slot as successive tool calls complete, preserving useful locality and avoiding an unneces- sary next-slot wait. A continuation that arrives after the slot closes, or does not fit before the final decode phase, waits for the next slot of the same model. Best-effort prefill does not mean silently deferring work. When the protected decode budget leaves insufficient prefill time, or when cumulative pressure pushes the round past the TTFT bound, the instance reports the shortfall through struc- tured telemetry—staging slack (remaining capacity for D2D pre-staging, detailed in §3.5) and deadline-risk counters— rather than hiding overload by rolling work into later rounds. The router reacts by spilling traffic to another instance, open- ing a replica, or applying the configured admission-control policy. 3.5 Multi-Model Memory Substrate The router and cold-pool scheduler decide where and when model weights and session KV should reside. They rely on a memory substrate that makes these placement decisions executable within the round budget. The substrate exposes three capabilities to the rest of Talaria. Stable HBM layout. Cold-pool instances switch among models whose weights, KV pools, runtime buffers, and graph state all compete for HBM. The substrate gives each instance a stable multi-model layout: structural regions can be released, reused, and rebound across switches without fragmenting device memory or invalidating the runtime state needed to serve the next slot. Persistent session KV. Switching away from a model may reclaim its device KV region, but a returning session still needs the prefix it produced before the tool call. The substrate therefore maintains a host-side KV registry that stores valid session prefixes across model switches and restores them to device memory when the session returns. The registry is part of the correctness boundary: it must not expose a prefix unless all tensor-parallel ranks agree on the same valid blocks. Reduced switch cost. Model switching consumes the same round budget that SP admission depends on. The substrate stages future model weights while the current model is serv- ing, then uses device-to-device movement for the staged por- tion at switch time and falls back to host-to-device loading only for uncovered chunks. The router sees the remaining staging capacity as staging slack; when that slack is low, the 7 instance becomes a worse target for additional traffic. Staging slack is therefore both a memory signal and a switch-cost signal: lower slack predicts a larger uncovered H2D fraction on the next switch. These capabilities form the contract between scheduling and implementation: the instance reports measured restore and switch estimates, and the router and SP scheduler charge those estimates to their admission budgets. Section 4 de- scribes how Talaria implements the lease and memory sub- strate. 4 Implementation We implement Talaria on SGLang [23] as a stateful Go gate- way (5.5K lines) and an extended inference engine (15K lines) that adds session caching, HKVR, SP scheduling, and staged switching. We additionally adapt 1.4K lines of SGLang’s existing TMS support for per-model memory reclamation. The gateway maintains session leases and routing telemetry; each engine instance implements the instance-local cache, scheduling, and memory contracts. 4.1Router State and Session Cache Integra- tion The gateway extracts a stable session id from the agent run- time and keeps a session table keyed by that id. Each entry records the last serving instance, model, observed KV length, last-use time, and a soft-reservation lease. After a request finishes, the gateway updates the entry from response usage metadata and installs a lease on the serving instance. Before routing the next call, it overlays its local in-flight queue on top of the latest instance snapshot, so recent forwards and reser- vations affect admission immediately instead of waiting for the next telemetry poll. Instance snapshots are versioned and refreshed on the control-plane heartbeat; if a snapshot is stale or the instance misses a heartbeat, the gateway can still honor live reservations only when the instance passes the health check, but it excludes that instance from new non-reserved placements until fresh telemetry arrives. The lease is represented as(s, m, i, k, t ex p ): sessions, model m, reserved instancei, prefix handlek, and expiry timet ex p . A live lease contributes one configured admission-unit charge to the gateway’s local load overlay. On a matching return, the gateway consumes the lease before forwarding the call toi; the completion hook may later install a successor lease. If the lease expires, the instance becomes unhealthy, or the prefix is no longer restorable, the call falls back to normal placement. Because the next suffix length is unknown at reservation time, this charge is capacity accounting rather than a hard latency guarantee. Each SGLang instance makes the prefix handle executable through a session-aware prefix-cache wrapper. When a turn Self-managed vram TMS / VMM CUDA graph pool token-to-slot table CUDA runtime NCCL buffer flashinfer workspace decode input buffer layer first kv cache buffer L0-k L1-k L0-v L1-v Host model weight chunkchunk chunkchunk chunkchunk chunkchunk dispatch kernel H2D chunks can be preempted (valid D2D, dirty H2D) model region kv cache region runtime region other Bump Pytorch pytorch inductor activation fallback H2D gather kernel D2D layer head first layer tail first Switch accelerate Figure 5: Bump-managed HBM layout. Stable regions pre- serve pointers across switches; clean staged chunks move by D2D and dirty chunks reload from host. finishes, the wrapper saves the request-pool slot, commit- ted KV length, radix lock state, and any architecture-specific cache state. On the next turn, it restores that slot before prefix matching, so the existing scheduler sees a prefix hit without a separate token path. The same finish path checkpoints aligned KV blocks to HKVR when host KV is enabled. Instance telemetry reports the fields the gateway needs for routing: loaded models, cached sessions, model queues, round remain- ing time, deadline risk, KV occupancy, and free staging bytes. 4.2 Bump-Managed HBM Layout A cold-pool instance switches among models whose weights, KV tensors, CUDA graphs, runtime workspaces, and book- keeping state all occupy HBM. Rebuilding these objects on every activation would recapture CUDA graphs, reinitialize attention backends, rebuild KV-pool shells, and fragment Py- Torch’s caching allocator. Talaria therefore removes the large structural regions from the caching allocator and places them in one managed HBM buffer. For PyTorch-owned per-model objects, it uses SGLang’s existingtorch_memory_saver (TMS) integration, a VMM-based physical-page manager that preserves virtual addresses while reclaiming physical pages. Talaria adapts that mechanism with per-model tags and switch-time pause/resume calls, so CUDA-graph and run- time allocations can be parked with the model that owns them. TMS is a memory mechanism below the scheduler, not the scheduling abstraction itself. The bump buffer uses a fixed layout: weights and KV data grow upward from the low-address end, runtime buffers are anchored at the high-address end, and the gap between 8 them forms the KV pool. This layout lets buffer-like regions be shared across models without static per-model partitions. TMS covers the PyTorch-managed objects that cannot sim- ply be rebound into the bump buffer, including CUDA-graph pools and per-model runtime metadata. When a model is parked, TMS releases their physical pages while preserving virtual addresses; on restore, the same addresses are remapped, so captured kernel pointers remain valid. Switching from modelmtom ′ then reduces to releasing the activekv_cacheregion, resizing theweightsregion for m ′ , and rebindingm ′ ’s parameter tensors to views in the bump buffer. The engine caches each model’s CUDA graph, atten- tion backend, and KV-pool shell, and recaptures only if a restore-time pointer check detects an address change. The check compares the base addresses of the weight, KV, graph- pool, and runtime buffer regions. Once the bump layout is stable, the common switch path reuses captured kernels di- rectly and avoidscudaFree/cudaMallocchurn. The stable addresses keep cached CUDA graphs and runtime objects valid across model switches. 4.3 Host KV Registry Switching away from a model may reclaim its device KV region, but returning sessions still need the prefix produced before the tool call. HKVR stores this state in a unified pinned- memory pool shared across attention layouts. MHA models allocate paired K/V slabs, MLA models allocate latent-state slabs, and Mamba-style models allocate recurrent-state slabs. All slabs return to the same pool when evicted, so host mem- ory is not statically partitioned by model or architecture. HKVR checkpoints aligned KV blocks asynchronously when a request finishes or a session enters a tool gap. Each block is keyed by the model name and cumulative prefix hash, matching the radix tree’s chunk granularity. On re- turn, the scheduler allocates new device slots, restores the agreed prefix through H2D layer-first/page-first transfer ker- nels, and recomputes only the unconfirmed tail. This path reuses SGLang’s KV movement kernels for MHA. MLA and Mamba share the registry metadata and slab allocator, while architecture-specific completeness checks may decline a re- store rather than publish partial state. The router sees a com- mon device/host/missing state; the instance decides whether the architecture-specific state is restorable. Three invariants keep HKVR safe under concurrent serving. First, transfer pins are separated from radix ownership: DMA pins prevent reclaiming blocks in use, while radix references track whether prefix metadata still points to a block. Second, HKVR exposes a prefix only after TP ranks agree on the same valid common prefix. A disagreement is handled con- servatively: the restore tracker truncates the published prefix to the longest common block range shared by all ranks and marks the inconsistent tail for recomputation. Partial or in- consistent tails remain private until a later checkpoint extends the agreement. Third, every model switch drains outstanding D2H events before freeing the activekv_cacheregion. Since D2H is issued as turns finish rather than batched at switch time, this barrier usually waits only for the final tail blocks. 4.4 D2D-Staged Switching Reloading a multi-hundred-GB model from host at the switch boundary would spend the same round budget that SP relies on. Talaria instead pre-stages the next model’s weights into unused HBM while the current model is serving, then uses D2D movement for the staged fraction at switch time. The staging surface is the high-address tail of each KV layer block; the KV allocator reuses low addresses first, keeping those tails clean under normal allocation pressure. Staging is opportunistic and correctness-preserving. The staging manager writes weights into fixed-size chunks and marks a chunk dirty if a concurrent prefill allocation takes rows that overlap the chunk’s staging range. During an active H2D write, the allocator skips the chunk’s row range to avoid reading partially written data. At switch time, the gather plan copies only clean staged chunks with D2D and reloads dirty or missing chunks from host. This protocol makes the expensive H2D component scale with the uncovered weight fraction; the total switch still in- cludes D2D gathering for staged chunks. It also gives the router a simple signal: staging slack, the remaining HBM space that can be used to pre-stage a future switch. When slack falls, the instance becomes a worse target for more traffic in the residency cost model; Section 5.6 separates the full-D2D, partial, and H2D fallback paths. Together, the router lease, bump layout, HKVR, and D2D staging realize the substrate contract exposed to the design: stable residency, restorable session state, and measured switch-cost telemetry. 5 Evaluation We evaluate Talaria with fixed replay as the primary evidence and targeted mechanism measurements for effects that a single replay cannot separate. Fixed replay holds the application- level call trace constant, so policy differences are not con- founded by the agent choosing a different tool path. We ask five questions: (1) do the mechanisms improve end-to-end session completion, (2) does residency-aware routing avoid misplaced model opens, (3) what is the cost/benefit of soft reservation, (4) does SP capture returns within an active slot, and (5) does the memory substrate reduce switch cost without exceeding HBM capacity? 5.1 Methodology The main replay is captured from Claude Code running ten Astropy issues from SWE-Bench Verified [13]. We replay 9 each issue once on Qwen3-235B, GLM5-nvfp4, and Qwen3.5- 122B-A10B, yielding 30 model-session executions and 960 calls. Reusing the ten task identities across three model pro- files enables paired, within-trace policy comparisons; the per- session plot shows every observation. For each model-session, every configuration replays the same request bodies under the same model and session IDs, in the same per-session call order, and with the same completion- token count for each call. The replay is closed-loop: after a call finishes, the next waits for the same recorded tool gap after a 10 s cap. The cap affects 20 of 960 calls. Global inter- leaving may differ across configurations because completion times differ. The resulting trace targets the long-context, short- return-gap regime for which session continuity matters most. Large-model runs use one TP=8 server with eight 183 GiB GPUs, 2 TB host memory, dual-socket x86 CPUs, and eight 400 Gb/s network adapters. Each run performs the same pre- window warmup; all measurements exclude warmup. We compare Talaria’s instance-local configuration (SP, HKVR, and D2D staging) with reverse ablations. no-SP dis- ables mid-slot session-prefill admission while keeping HKVR and D2D staging. no-HKVR keeps SP and D2D staging but disables host-restorable KV. H2D-only keeps SP and HKVR but disables D2D weight staging, so switches reload from host memory. Round-only disables SP, HKVR, and D2D staging and serves the same trace with the otherwise identical round scheduler and host-to-device model reloads. Round-only is the controlled all-off baseline within the same engine. Router-policy experiments use two TP=4 workers to create real placement choices. They replay a 120-call, 30-model- session subset under three relative offered-load regimes. The trace preserves session ids, return gaps, model order, and reusable-prefix lengths, but maps the large models to Qwen3- 32B, Qwen2.5-32B-Instruct, and Qwen3.5-35B-A3B. This split is intentional: TP=8 evaluates instance-local mechanisms under large-model memory pressure, while TP=4 isolates placement decisions. Together, the two testbeds separate in- stance execution from cluster placement without conflating their effects. The primary metric is session completion time (SCT), mea- sured from a model-session’s first call to its final response. We also report TTFT, switch latency, round composition, and HBM footprint. TTFT is schedule-aware: it starts at the in- tended enqueue time and therefore includes admission delay. For raw replay distributions, percentiles use the nearest-rank empirical definition: forNobservations,p q is the observation at rank⌈qN/100⌉; targeted TP=4 experiments retain their original harness summaries. The main trace intentionally runs at high pressure to expose differences in how the policies pre- serve session state while per-session call order, bodies, and recorded gaps remain fixed. 10 2 10 3 Session completion time (s) 0.0 0.2 0.4 0.6 0.8 1.0 CDF TP=8 fixed replay Talaria no-SP no-HKVR H2D-only Round-only Figure 6: End-to-end SCT on the TP=8 fixed replay. Talaria cuts p50 SCT from 1000 s to 189 s relative to the Round-only ablation. Talariano-SPno-HKVRH2D-onlyRound-only 0 600 1200 1800 2400 Session completion time (s) 189s 623s 486s 194s 1000s Reverse ablations on the same trace Figure 7: SCT for all 30 model-sessions on the same replay; boxes show the interquartile range under the nearest-rank empirical definition. Each point is one of ten issues executed on one model. 5.2 End-to-End Session Completion Figures 6 and 7 show the SCT distribution for the TP=8 fixed- replay configurations. All rows replay the same 960 calls over the same 30 sessions. Relative to Round-only, Talaria’s instance-local configura- tion reduces p50 SCT from 1000 s to 189 s and p95 SCT from 2296 s to 867 s, delivering 5.3×and 2.6×speedups. The p50 TTFT falls from 13.44 s to 0.55 s. The reverse ablations show each mechanism’s marginal effect with the others enabled; these effects are not additive. Round-only combines next-slot waiting with prefix recomputation after reusable KV is lost. SP provides the largest p50 marginal contribution, reducing SCT from 623 s to 189 s by admitting eligible returns within the active model slot. HKVR reduces p50 SCT from 486 s to 189 s and TTFT p95 from 26.33 s to 14.17 s. D2D staging leaves p50 SCT nearly unchanged (194 s to 189 s) but reduces p95 from 933 s to 867 s; §5.6 isolates its switch path. Talaria improves 29 of 30 paired model-sessions. 10 02468 TTFT (s) 0.0 0.2 0.4 0.6 0.8 1.0 CDF (a) High-load TTFT lowmediumhigh 0 10 20 30 40 Avoidable cold opens (b) Cold opens Least-pressureSession-stickyFull router Figure 8: Two-worker TP=4 placement over 120 calls. The full router nearly eliminates avoidable model opens while controlling high-load TTFT. 5.3 Spatial Placement: Router Policy Figure 8 compares three routing stacks on the same 120- call trace. Least-pressure chooses the least-loaded worker for each call. Session-sticky pins a session to its first worker. The full router combines residency cost ranking with soft reservations, matching the prototype policy; §5.4 separately sweeps the reservation timeout. We count an avoidable cold open when the selected worker must open the requested model even though another worker already has that model resident. Least-pressure frequently selects a worker that must reopen the requested model. Session-sticky preserves affinity but cannot trade it against model residency and queue pressure. The full router eliminates avoidable opens at low and medium load and leaves one at high load. At medium load, TTFT p50 is 0.36 s, versus 1.67 s for least-pressure and 2.29 s for session-sticky. At high load, it retains 97.8% of returning calls on the same worker, reduces avoidable opens from 37 (least- pressure) and 5 (session-sticky) to 1, and cuts TTFT p95 from 8.07 s and 6.23 s to 5.28 s. The placement gains translate most clearly once model opens and queue pressure matter. At medium load, full rout- ing improves SCT p50/p95 to 6.28/12.74 s from 11.01/19.59 s for least-pressure and 10.97/19.28 s for session-sticky. At high load, it lowers p95 to 21.31 s from 31.29 s for least- pressure and 23.03 s for session-sticky; its 17.44 s p50 is slightly above sticky’s 16.33 s. At low load, eliminating avoid- able opens does not translate into an SCT gain: full-router p50/p95 is 7.39/15.81 s, versus 6.82/11.31 s for least-pressure and 5.62/13.68 s for session-sticky. Thus residency-aware placement yields its clearest SCT benefit at medium load; at high load it improves p95 but not p50, while at low load it eliminates model opens without an SCT gain. 00.5125 Reservation timeout τ (s) 0 20 40 60 80 Rate (%) (a) Locality tradeoff same instance model-residency hit unused-lease proxy 00.5125 Reservation timeout τ (s) 0.75 1.00 1.25 1.50 1.75 2.00 2.25 Returning TTFT (s) -35% p50 (b) Returning TTFT p50 p95 Figure 9: TP=4 reservation-timeout sweep. A 1 s lease gives the lowest returning-call p95; extending to 5 s trades model residency for higher return locality. 5.4 Soft-Reservation Tradeoff Soft reservation adds a temporal decision to router placement: after a session leaves for a tool call, the gateway leases ca- pacity for a bounded intervalτinstead of making the next call compete as an unrelated arrival. The TP=4 sensitivity run sweepsτ∈0, 0.5, 1, 2, 5s and reports same-instance return rate, model-residency hit rate, returning-call TTFT, model opens, and an unused-lease proxy. The proxy counts reserved returns that did not produce a timely same-instance hit; it does not measure how long a live lease occupied capacity. Withτ = 0, returning sessions still use residency-aware cost ranking, but no capacity is held: 56.2% return to the same worker and 43.8% miss the device-KV proxy. A 1 s lease raises same-worker returns to 67.7% and lowers returning-call TTFT p50/p95 from 1.51/2.11 s to 0.99/1.91 s, the lowest p95 in the sweep. This gain uses additional placement capacity: relative toτ = 0, model-residency hits fall from 72.5% to 60.8%, model opens rise from 33 to 47, and the unused-lease proxy is 50%. Extending the lease to 5 s raises same-worker returns further to 79.2%, but provides similar TTFT while model-residency hits fall to 53.3% and model opens rise to 56. We therefore use 1 s as the operating point that best balances return locality and model residency for this workload;τis a workload-dependent policy knob. 5.5 Temporal Placement: SP Admission SP addresses the temporal mismatch inside a model slot: a re- turning session should not wait for the next full prefill window when the current slot can admit its reusable prefix. Figures 10 and 11 compare Talaria with no-SP on the same fixed replay, keeping request bodies, arrival gaps, HKVR, D2D staging, and the router fixed. With all other mechanisms fixed, SP lowers p50 TTFT from 4.60 s to 0.55 s and p95 from 20.5 s to 14.2 s. Within Talaria’s 200 rounds, SP admits 316 return-handling events into the active slot and parks 345, an admission rate of 47.8%. 11 no-SPTalaria 0 10 20 30 40 50 Within-policy events admitted (%) 0% 48% (a) Admission no-SPTalaria 0 1 2 3 4 Handling events/round (not unique calls) 1.8 1.6 1.7 (b) Handling Admitted Deferred Figure 10: SP handling events on the TP=8 fixed replay. Ta- laria admits 47.8% of its recorded return-handling events within the active model slot. 0.111060 TTFT (s) 0.0 0.2 0.4 0.6 0.8 1.0 CDF p95 Per-call TTFT no-SP (p95=20.5s) Talaria (p95=14.2s) Figure 11: SP TTFT impact on the same replay. Mid-slot admission lowers per-call TTFT. Talaria records none of the 123 host-cache capacity misses and 2,532 architecture-specific restore skips observed with no-SP. These are schedule-dependent diagnostics, not unique- call outcomes; all 960 calls complete in both configurations. A parked request may also appear in multiple rounds, so we normalize handling events within each policy and use TTFT, measured over the common call set, as the direct cross-policy result. 5.6Memory Substrate: Switch and HBM Cost Across the replay, D2D staging reduces aggregate switch time by 38% and switch share from 51.7% to 34.6%. Figure 12 isolates the switch path by comparing Talaria with H2D-only, which keeps SP and HKVR enabled but reloads weights from host memory. Each logical switch emits one record per TP rank; we re- port the maximum rank latency because the scheduler resumes only after all ranks finish. At this granularity, D2D staging im- proves the center of the distribution but exposes a tail tradeoff under high KV pressure: p50 falls from 1497 ms to 984 ms and mean from 1521 ms to 1071 ms, while p95 rises from 2133 ms to 2679 ms. Full D2D switches, 27.7% of the ob- served mix, complete in 106 ms at p50; partial D2D plus H2D accounts for 36.3% at a 1000 ms p50, and the remain- 10 2 10 3 Switch latency (ms) 0.0 0.2 0.4 0.6 0.8 1.0 CDF 984 ms 1497 ms (a) Latency H2D-only Talaria H2D-onlyTalaria 0.0 0.5 1.0 100% 28% 36% 36% (b) Paths D2DPartialH2D Figure 12: Logical TP=8 switch latency, computed as the max- imum across ranks. Each configuration contains 600 logical switches; paths use the worst-rank classification. Staging cuts p50 by 34%. H2D-onlyTalaria 0.0 0.5 1.0 Fraction of wall time 48% 65% 52% 35% (a) Composition H2D-onlyTalaria 0 500 1000 1500 2000 Aggregate round time (s) 1753s 1632s Switch -38% (b) Aggregate ActiveSwitch Figure 13: Round-time composition for Talaria and H2D-only. D2D staging reduces aggregate switch time by 38%. ing 36.0% falls back to H2D at a 2011 ms p50. The staged common path therefore delivers the largest speedup, while partial coverage retains a proportional benefit. Active writes dirty some staged chunks, causing the correctness guard to reload them from host and accounting for the heavier tail. H2D-only also records 39 host-cache capacity misses and 1,704 Mamba restore skips, versus zero for Talaria. These schedule-dependent diagnostics capture downstream cache interactions; the logical-switch distribution is the direct path comparison. Figure 13 shows the same effect at round granularity. H2D- only spends 51.7% of round wall time in switching, while Talaria spends 34.6%; total switch time drops from 907 s to 565 s. Talaria’s aggregate round wall time is also lower (1632 s versus 1753 s). Because SP and HKVR already preserve the common return path, the 38% switch-time reduction has a larger effect on tail SCT than on p50: p50 changes from 194 s to 189 s, while p95 falls from 933 s to 867 s. 12 0255075100125150175 0 10000 20000 30000 Round time (ms) (a) Talaria 050100150200250300 Round index 0 10000 20000 30000 Round time (ms) (b) Round-only ActiveSwitch Figure 14: Per-round composition for Talaria and Round-only. The combined mechanisms reduce switch share and round count. 050100150200250300 Round index 160 170 180 HBM used (GiB) 183 GiB cap peak 177.1 GiB Talaria no-SP no-HKVR H2D-only Round-only 183 GiB cap Figure 15: HBM footprint under the bump allocator. Runs remain below the 183 GiB device cap. Curves show per-round averages; the annotation is the maximum across all configura- tions. Figure 14 shows the combined effect over time. Round-only carries a persistent switch band because every slot-boundary model change reloads from host memory. Talaria’s SP, HKVR, and staging mechanisms jointly reduce switch share and the number of rounds; this comparison does not attribute the difference to staging alone. Curves use an 11-round moving average for visual clarity and retain separate round-index axes because the configurations execute 200 and 339 rounds. Figure 15 shows device footprint across configurations. The peak observed HBM use is 177.1 GiB, below the 183 GiB device cap. This confirms that the managed layout, restorable KV, and weight staging remain within the device-memory budget throughout the replay. SystemUnitModel route KV routeAgent return Aegaeon [5]request/tokenYesNonext model slot Preble [16]promptNoYesrequest arrival Mooncake [15]KV objectNoYesstorage restore ServerlessLLM [6]requestYesNocold start TalariasessionYesYesreservation + SP Table 1: Placement signals and cross-call return handling. 6 Related Work Table 1 distinguishes KV-aware placement (“KV route”) from internal KV synchronization: only the former lets residency change which instance receives a return. Serverless and multi-model LLM serving.Classical prediction-serving systems focus on model lookup, batching, replication, pipeline provisioning, and interference control under request-level latency objectives [24–29]. Serverless systems similarly hide placement behind a request/function abstraction whose state is short-lived relative to model execu- tion [30]. LLM serving makes weights and KV cache persis- tent serving state. ServerlessLLM optimizes checkpoint load- ing and migration for on-demand LLM inference, and Spot- Serve adapts serving to preemptible GPU availability [6, 31]; AlpaServe and MuxServe multiplex multi-model or model- parallel inference on shared GPUs [7, 8]; Aegaeon targets long-tail LLM marketplaces with token-level auto-scaling [5]. These systems multiplex models and requests. Talaria instead treats the session as the scheduling object: a router must preserve model and KV residency across tool gaps, and a cold-pool instance must admit the returning session before the locality expires. Adapter and fine-tuned-model serving. S-LoRA, Punica, and dLoRA make large catalogs of LoRA adapters practical by sharing a resident base model, paging adapter state, or dynamically migrating requests and adapters [3, 4, 32]. These systems reduce the cost of serving many fine-tuned variants of a base model. Talaria targets the complementary full-model setting: the active model itself may switch, and a returning session’s KV must remain usable across tool gaps and model residency changes. Memory management for LLM serving. PagedAttention, vAttention, FlexGen, LoongServe, InfiniGen, and Infinite- LLM improve LLM serving by reducing KV fragmentation, exploiting virtual memory, offloading cache state, or distribut- ing long-context KV across memory tiers [20–22, 33–35]. These systems optimize the memory hierarchy for active re- quests or long contexts. Talaria uses memory management for a different scheduling contract: a model can leave the GPU without losing a returning session’s prefix, and a future model switch can be prepared while the current model is still serving. Execution engines and phase-aware scheduling. Orca, vLLM, Sarathi-Serve, FastServe, DeepSpeed-FastGen, and 13 NanoFlow improve utilization of a loaded model through iteration-level scheduling, KV paging, chunked prefill, pre- emption, dynamic split-fuse scheduling, and intra-device over- lap [20, 36–40]. Splitwise, DistServe, and TetriInfer separate prefill and decode resources, and TokenScale autoscales such deployments using token velocity [17–19, 41]. Talaria is com- plementary: it coordinates model switching, session returns, and memory pressure across many models. Its evaluated co- located design avoids adding a cross-tier KV handoff to each session turn. KV reuse and cache-aware routing. CachedAttention and SGLang reuse conversation or structured-program pre- fixes through KV hierarchy and RadixAttention [14, 23]. Hy- dragen accelerates shared-prefix attention, CacheGen and CacheBlend reduce the cost of moving or composing reusable KV, and LMCache provides a KV-cache layer for enterprise- scale serving [42–45]. Mooncake and MemServe expose KV cache as a disaggregated cluster resource, while Preble, DualMap, and Llumnix use cache-aware routing or migration to balance locality and load [15, 16, 46–48]. Talaria builds on the same principle that KV is schedulable state, but the rout- ing decision also depends on model residency, round pressure, and observed session-return timing. Serving agentic programs. ReAct, Toolformer, ReWOO, Reflexion, and AutoGen popularized tool-interleaved, reflec- tive, and multi-agent language-model execution, and SWE- agent shows that software-engineering agents repeatedly call models while interacting with a repository and test envi- ronment [9–11, 49–51]. SGLang, Parrot, Pie, Autellix, and Continuum then show that such programs expose reusable prefixes, control flow, inter-request dataflow, and KV life- times that should be scheduled above the single-request level [12, 23, 52–54]. Talaria consumes that session-level structure at a complementary layer: placement and execu- tion across a serverless multi-model pool. Here, models may be hot or cold, and the router must jointly decide whether to reuse a model replica, restore session state, or recompute a prefix. 7 Conclusion This paper presented Talaria, a serverless multi-model serv- ing system for agentic workloads. Serving an agent session requires tracking both session state—KV locality and return timing—and instance state—model residency and pressure. Request-level placement and slot-boundary admission lose continuity in different ways. Talaria addresses them with soft reservation at the router and session-prefill (SP) at the cold-pool scheduler. An instance-local substrate preserves restorable KV and stages weights across model switches. On the fixed TP=8 replay, the instance-local mechanisms deliver 5.3×and 2.6×speedups in p50 and p95 SCT relative to the internal Round-only ablation, and improve 29 of 30 paired model-sessions. SP lowers p50 TTFT from 4.60 s to 0.55 s, while D2D staging cuts p50 logical-switch latency by 34% and aggregate switch time by 38%. On the two- worker placement testbed, the router eliminates every avoid- able model open at low and medium load and leaves only one at high load. The design intentionally keeps the router out of the token data path. Weight and KV transfers are executed by instances, while the router uses telemetry to decide when those trans- fers are worth paying. This separation is what lets Talaria combine cluster-level placement with instance-level execu- tion contracts: the router selects an instance using model residency, KV availability, and pressure; that instance restores KV and activates or stages models as needed. Limitations and future work. Our current evidence com- bines controlled large-model execution on one TP=8 server with isolated placement experiments on two workers. Broader workload and cluster-scale evaluation are the next empirical steps. On the mechanism side, workload-adaptive lease and host-cache policies, remote-KV integration, dynamic hot-cold pool reallocation, and P/D-disaggregated admission are nat- ural extensions. D2D staging improves the p50 and mean switch path under high KV pressure; reducing dirty-chunk fallbacks can extend these gains to p95. References [1]Stanford Institute for Human-Centered Artificial Intel- ligence. The 2025 ai index report. Technical report, Stanford University, 2025. [2] Avijit Ghosh, Lucie-Aimée Kaffee, Yacine Jernite, and Irene Solaiman. State of open source on hugging face: Spring 2026. Hugging Face Blog, 2026. [3]Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonza- lez, and Ion Stoica. S-LoRA: Serving thousands of concurrent LoRA adapters. In Proceedings of Machine Learning and Systems (MLSys), 2024. [4]Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xu- anzhe Liu, and Xin Jin. dLoRA: Dynamically orches- trating requests and adapters for LoRA LLM serving. In Proceedings of the 18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI), pages 911–927, 2024. [5] Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. Aegaeon: Effective gpu pooling for concurrent llm serving on the market. In Proceedings 14 of the ACM SIGOPS 31st Symposium on Operating Sys- tems Principles (SOSP), 2025. [6]Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-latency serverless inference for large language models. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. [7] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Al- paServe: Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023. [8] Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. MuxServe: Flexible spatial-temporal multiplex- ing for multiple LLM serving, 2024. [9]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023. [10]Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [11]John Yang, Carlos E. Jimenez, Alexander Wettig, Kil- ian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [12]Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. Autellix: An efficient serving engine for llm agents as general programs, 2025. [13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. [14]Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with CachedAtten- tion. In Proceedings of the 2024 USENIX Annual Tech- nical Conference (USENIX ATC), 2024. [15]Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation – a KVCache-centric architecture for serving LLM chatbot.In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST), 2025. [16] Vikranth Srivatsa, Zijian He, Reyna Abhyankar, Dong- ming Li, and Yiying Zhang. Preble: Efficient distributed prompt scheduling for LLM serving, 2024. [17] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Iñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In Proceedings of the 51st Annual Inter- national Symposium on Computer Architecture (ISCA), 2024. [18]Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. [19] Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Disaggregate LLM in- ference for mixed downstream workloads, 2024. [20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023. [21]Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ra- machandran Ramjee, and Ashish Panwar. vAttention: Dynamic memory management for serving LLMs with- out PagedAttention, 2024. [22]Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th 15 International Conference on Machine Learning (ICML), 2023. [23]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Bar- rett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [24]Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, LiLao, FangweiLi, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. TensorFlow-Serving: Flexible, high-performance ML serving, 2017. [25] Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In Pro- ceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017. [26]Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. Nexus: A GPU cluster engine for accelerating DNN-based video analysis. In Proceed- ings of the 27th ACM Symposium on Operating Systems Principles (SOSP), pages 322–337, 2019. [27]Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar, Xi- angxi Mo, Joseph E. Gonzalez, Ion Stoica, and Alexey Tumanov. InferLine: Latency-aware provisioning and scaling for prediction serving pipelines. In Proceed- ings of the 11th ACM Symposium on Cloud Computing (SoCC), 2020. [28]Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less inference serving. In Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC), pages 397–411, 2021. [29]Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictability from the bottom up. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020. [30]Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Go- har Chaudhry, Paul Batum, Jason Cooke, Eduardo Lau- reano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC), 2020. [31]Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. SpotServe: Serv- ing generative large language models on preemptible instances. In Proceedings of the ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024. [32] Lequn Chen, Zhiqiang Ye, Yilong Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy, and Tianqi Zhang. Punica: Multi-tenant LoRA serving. In Proceedings of Machine Learning and Systems (MLSys), 2024. [33]Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. LoongServe: Efficiently serving long-context large language models with elastic sequence parallelism, 2024. [34]Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In Proceedings of the 18th USENIX Symposium on Op- erating Systems Design and Implementation (OSDI), 2024. [35]Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wen- cong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. Infinite-LLM: Efficient LLM ser- vice for long context with DistAttention and distributed KVCache, 2024. [36]Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative mod- els. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022. [37]Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee.Taming Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In Proceedings of the 18th USENIX Sym- posium on Operating Systems Design and Implementa- tion (OSDI), 2024. [38]Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. FastServe: Fast distributed inference serving for large language models. In Pro- ceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023. [39] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Am- mar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong He. DeepSpeed-FastGen: High-throughput text generation for LLMs via dynamic splitfuse, 2024. 16 [40]Kan Zhu, Yilong Zhao, Liang Zhao, Gan Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. NanoFlow: Towards optimal large language model serving through- put, 2024. [41]Ruiqi Lai, Hongrui Liu, Chengzhi Lu, Zonghao Liu, Siyu Cao, Siyang Shao, Yixin Zhang, Luo Mai, and Dmitrii Ustiugov. TokenScale: Timely and accurate autoscaling for disaggregated LLM serving with token velocity, 2025. [42] Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput LLM inference with shared prefixes, 2024. [43] Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM Conference, 2024. [44]Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yi- hua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion, 2024. [45]Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xi- aokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. LMCache: An efficient KV cache layer for enterprise-scale LLM infer- ence, 2025. [46] Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yun- gang Bao, Ninghui Sun, and Yizhou Shan. MemServe: Context caching for disaggregated LLM serving with elastic memory pool, 2024. [47]Ying Yuan, Pengfei Zuo, Bo Wang, Zhangyu Chen, Zhipeng Tan, and Zhou Yu. DualMap: Enabling both cache affinity and load balancing for distributed LLM serving, 2026. [48] Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In Pro- ceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024. [49]Binfeng Xu, Zhendong Peng, Bowen Lei, Subhabrata Mukherjee, Yu Liu, and Dongkuan Xu. ReWOO: De- coupling reasoning from observations for efficient aug- mented language models, 2023. [50]Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Lan- guage agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [51]Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation, 2023. [52]Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Par- rot: Efficient serving of LLM-based applications with semantic variable. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI), pages 929–945, 2024. [53] In Gim, Zhiyao Ma, Seung-seob Lee, and Lin Zhong. Pie: A programmable serving system for emerging LLM applications. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP), 2025. [54]Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph E. Gonzalez, and Ion Stoica. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live, 2025. A Trace and Replay Details Session boundaries.A session is one agent task. The trace starts at the first model call issued for that task and ends at the final model response before the task terminates. We de- fine the inter-call gap as the wall-clock interval between a model response and the next model enqueue. This interval includes tool execution and agent/runtime orchestration, so it is not a pure tool-execution measurement. Harness startup and teardown fall outside the session boundary. Across the 6,941 possible consecutive-call intervals, 6,294 have a complete timestamp pair; the remaining 647 do not. The characteri- zation plot further excludes five>30s environment-startup intervals, leaving 6,289 gaps for the reported distribution. Prefix reuse.For consecutive calls within a session, prefix reuse is measured after tokenization as the longest common prefix between the new prompt and the prior prompt plus generated response that remains in the session context. The resulting prefix length quantifies the amount of reusable work. Whether a return actually reuses device KV, restores host KV, or recomputes the prefix also depends on cache residency, validity, and capacity at return time. 17 Replay inputs.The 445-session characterization trace sup- ports the distributions and recovery-cost estimates in §2.1. Each available record contains the session id, call index, model id, enqueue timestamp, prompt and output tokens, reusable-prefix tokens, and inter-call gap. The recovery-cost estimates combine these records with model-specific prefill profiles measured on the testbed. The TP=8 fixed replay is a separate 960-call workload. Across configurations, each model-session uses the same re- quest bodies, model and session ids, per-session call order, completion-token counts, and recorded inter-call gaps. The replay caps each gap at 10 s, affecting 20 of 960 calls. It is closed-loop: each next call is released after the preceding call completes and its capped gap elapses, so global interleaving may differ as configurations finish calls at different times. The TP=4 router experiments use a separate 120-call, 30-model- session subset and map its model ids to the models on that testbed (§5.1). Timing telemetry. Replay logs keep request-local TTFT and schedule-aware TTFT in the same schema. The latter starts at the replay scheduler’s intended enqueue time, so it includes any replay-side admission delay. Unless stated otherwise, the paper reports schedule-aware TTFT. Round summaries use a mutually exclusive decomposition,wall_ms = active_ms + switch_ms . Phase counters such as prefill, decode, H2D, and D2H are diagnostic subviews inside active slots; they can overlap and are not summed into wall time. B Correctness Conditions Talaria relies on three implementation-level conditions when moving state across HBM and host memory. Prefix agreement. Each tensor-parallel rank reports its committed prefix length and cumulative prefix hash at page boundaries. HKVR exposes only the longest page-aligned pre- fix on which all ranks agree for the same model. If a rank lags or hashes diverge, HKVR publishes the last agreed boundary and marks the remaining suffix for recomputation. This makes TP disagreement a reuse loss rather than a correctness risk. Lifetime separation. Radix-tree ownership and DMA transfer pins are tracked separately. A block can remain pinned until an asynchronous copy finishes even if the radix tree no longer references it; conversely, radix metadata cannot expose a block whose DMA copy has not completed. Switch barrier. Before a model’s activekv_cacheregion is released, the scheduler drains outstanding D2H events for prefixes that may be restored later. The barrier is scoped to the outgoing model, so unrelated models do not block the switch path. Validation protocol. We exercise these guards at two lev- els. Synthetic switch/restore tests use deterministic prompts whose first generated tokens are known; after a forced model switch and HKVR restore, the client compares the generated prefix with the expected sequence and records whether the cache hit came from device or host state. Large-model stress runs enable the same runtime checks under round scheduling: TP-prefix disagreement truncates the exposed prefix, stale or uncommitted DMA blocks are never published to radix metadata, and staged weight chunks marked dirty are reloaded from host rather than copied through D2D. These tests ex- ercise the guards; performance claims use the workloads in §5. C HKVR Transfer Layouts HKVR separates the scheduler-visible cache state from the physical layout used to move KV bytes. The router only ob- serves whether a prefix is device-resident, host-restorable, or missing. The restore path below that interface packs and unpacks KV pages through layout-aware transfer kernels. Layer-first and page-first movement. SGLang’s KV movement path supports two equivalent views of the same log- ical prefix. In a layer-first (LF) layout, host slabs are grouped by model layer, then by page within that layer; this matches the device-side KV cache layout used by attention kernels and makes a layer’s pages contiguous for DMA. In a page- first (PF) layout, the host representation is grouped by logical prefix page, with the per-layer fragments for that page stored under one page descriptor. PF is useful for registry operations because the prefix can be committed, invalidated, or restored at page granularity. HKVR records the logical prefix as pages and invokes LF/PF transfer kernels to translate between the registry view and the active device layout. LF/PF are data- movement layouts; HKVR is the consistency and naming layer above them. Attention-layout metadata. Each HKVR entry carries a compact layout descriptor: model id, TP rank, prefix hash, committed token length, page size, dtype, attention family, and slab shape. MHA models store paired K/V slabs, while MLA models use latent-state slabs. Supported Mamba-style paths register recurrent and auxiliary state through architecture- specific descriptors rather than treating that state as KV slabs. The router-visible states remain device-resident, host- restorable, or missing across these families, but transfer and completeness checks are architecture-specific. HKVR pub- lishes restorable state only when that state is complete and the agreement rule in §B holds; otherwise the unconfirmed suffix is recomputed. D Switch Timeline Instrumentation The implementation records oneSWITCH_TIMINGline per TP rank for every model switch. Each rank’s timer starts after pre-switch diagnostic probes. The instrumented latency of a 18 logical switch is the maximum across its TP ranks; the phase breakdown below instead reports rank-level records so that per-rank work remains visible. A switch follows five ordered phases. Phase 1: save and teardown. The instance drains outstand- ing HKVR D2H writes for the outgoing model, cancels stale H2D restores, saves the CUDA-graph and KV-pool handles, releases runtime buffers, parks the outgoing radix tree, and pauses the outgoing model’s per-model TMS tags. Phase 2: open target weights. The instance waits for any outstanding pre-staging work, verifies the staging bitmap, updates the active model configuration, resolves the CPU-side model entry, and opens the target weights either through D2D gather, partial D2D plus H2D top-up, or full H2D fallback. Phase 3: restore the KV shell. The scheduler checks whether the target model’s KV-pool shell can be reused under the current weight-region size, page size, and capacity. A hit rebinds the cached shell; a miss rebuilds the pool under the target model’s TMS tag. Phase 4: restore runtime and graph state. The instance restores a cached attention backend and CUDA graph when the KV shell and graph pointers match; otherwise it rebuilds runtime buffers and recaptures graphs. Phase 5: publish references. The instance propagates the new model, allocator, KV, radix, and telemetry references back to the scheduler and worker. This phase is intentionally small: large memory movement has already completed in Phase 2. PhaseD2D fullD2D + H2DH2D-only p50p95p50p95p50p95 P126.234.726.930.921.931.7 P2109.6 244.8995.8 1819.8 2570.6 3142.2 P30.52.00.51.20.81.9 P48.620.38.320.07.716.4 P50.20.30.20.30.20.3 Total149.8 298.9 1041.2 1854.6 2608.6 3170.6 Table 2: Per-rank five-phase switch timing in a separate com- ponent stress run (ms). Table 2 reports a separate TP=8 component stress run, not the 600 logical switches per configuration in Figure 12. D2D- full and D2D+H2D use steady-state rank-level records after excluding warm-up and instrumentation outliers (n = 14,952 andn = 102); H2D-only uses rank-level records from a no- D2D stress run (n = 2,104). The table shows why staging helps: phases other than weight opening are already small under cache hits, so the switch path is dominated by Phase 2. Full D2D staging reduces the p50 Phase 2 cost from 2.57 s to 110 ms, while partial staging lands between the two according to the uncovered H2D fraction. Path classification. Each rank-level record also identifies the load path, staged-byte coverage, dirty staging bytes, and D2D/H2D gather times. In the TP=8 staged run used for the first two path groups of Table 2, the filtered set contains 15,070 rank-level records: 14,952 D2D-full, 102 partial D2D+H2D, and 16 full-H2D fallbacks. These records hit the cached KV shell and CUDA graph, so the tail is driven by uncovered weight bytes rather than runtime recapture. D2D-full switches have a p50 D2D gather-sync time of 18.3 ms; partial switches add H2D synchronization with a p50 of 696.2 ms for uncov- ered chunks. Dirty chunks arise when active KV allocations overlap the staging range, and the guard reloads those chunks from host rather than copying unsafe D2D sources. Model-specific weight postprocessing. Some models ex- pose runtime weights that are not a byte-for-byte copy of the raw checkpoint tensors. In our GLM runs, the quantized post- load path materializes packed and derived tensors on the GPU after the first open. The switch path therefore treats the post- processed physical layout as the stable representation: after finalization, it writes the processed tensors back to the host- side model entry and uses that layout for later H2D or D2D reloads. This keeps region sizing and staged-byte account- ing aligned with the physical parameter and persistent-buffer order used by the active model. E Policy Knobs Reservation timeoutτ. We selectτ = 1s from the sensi- tivity sweep in §5.4; it is near the p90 inter-call gap in the characterization trace. Expired sessions fall back to normal residency-aware placement. Bump memory fraction. The component experiments use mem-fraction-bump=0.8, reserving most HBM for the man- aged layout while leaving headroom for runtime allocations and a safety margin. Admission marginγ. Cold-pool rounds are bounded below the configured TTFT target withγ< 1, after accounting for switching and restore overhead. This knob is an admission- control margin; absolute target attainment still depends on the offered load and provisioning level. Replica penaltyν m . The router uses a per-model replica penalty to avoid opening replicas for transient bursts whose benefit does not cover switch and memory cost. 19