Paper deep dive
MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs
Youssef Ennouri, Soonhoi Ha
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/3/2026, 3:51:17 AM
Summary
This paper introduces a MeanField surrogate modeling approach to enable scalable runtime scheduling of concurrent heterogeneous AI inference workloads (LLMs and vision models) on shared GPUs. By predicting per-model performance based on local configurations and aggregate GPU state rather than joint interactions, the method achieves linear sample complexity growth with the number of co-running models. Integrated into a genetic algorithm scheduler, the solution maintains high predictive accuracy (R² ≈ 0.96) and near-optimal scheduling utility with low latency, significantly outperforming exhaustive search methods.
Entities (10)
Relation Signals (7)
MeanField Surrogate → appliedto → Vision Models
confidence 95% · Experiments on concurrent LLM and vision workloads... show high predictive accuracy
MeanField Surrogate → appliedto → LLM
confidence 95% · Experiments on concurrent LLM and vision workloads... show high predictive accuracy
MeanField Surrogate → integratedinto → Genetic Algorithm Scheduler
confidence 95% · Integrated into a genetic algorithm scheduler, the surrogate scales to an N=5 problem...
MeanField Surrogate → predicts → per-model performance
confidence 95% · We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions.
MeanField Surrogate → runson → GPU
confidence 95% · Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention... We propose a MeanField surrogate...
Genetic Algorithm Scheduler → achieves → Low Latency
confidence 90% · complete online GA decisions take 26 ms median, about 5x faster than exhaustive surrogate search.
MeanField Surrogate → reduces → profiling complexity
confidence 90% · The MeanField formulation avoids this growth: its profiling cost scales as O(N · k) rather than O(k^N)...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.
Tags
Links
- Source: https://arxiv.org/abs/2609.02109v1
- Canonical: https://arxiv.org/abs/2609.02109v1
Trouble viewing inline? Open PDF directly →
Full Text
22,387 characters extracted from source content.
Expand or collapse full text
1 MeanField Surrogate Modeling for Scalable Runtime Scheduling of Concurrent Heterogeneous AI Inference on Shared GPUs Youssef Ennouri and Soonhoi Ha, Fellow, IEEE Abstract—Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across N ∈ 2, 3, 4, 5, 6 show high predictive accuracy (R 2 ≈ 0.96) with an empirical sample budget that grows approximately linearly in N , in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to a N = 5 problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about 5× faster than exhaustive surrogate search. Index Terms—GPU Scheduling, Heterogeneous AI Inference, Performance Modeling, Runtime Systems, Surrogate Modeling I. INTRODUCTION M odern inference deployments increasingly co-locate heterogeneous AI workloads such as large language models (LLMs) and vision models on shared GPUs to improve utilization. Concurrent execution, however, induces non-linear interference over shared resources, including streaming-multiprocessor (SM) occupancy, video memory (VRAM) capacity, and memory bandwidth. This interference depends jointly on model variants, runtime parameters, and workload intensity, making analytical modeling impractical. Efficient scheduling therefore requires a fast surrogate that predicts concurrent performance without live benchmarking. While established serving systems like Nexus [1], Clipper [2], and Clockwork [3] improve utilization through batching, placement, and allocation, their scope is limited to homogeneous or isolated workloads. This limitation, along with interference-aware GPU-sharing systems such as Orion [4], ElasticRoom [5], Miriam [6], and automated runtime-aware scheduling [7] underscores the need for accurate performance prediction under co-execution. Corresponding author: Soonhoi Ha. This work was supported by the National Research Foundation of Korea (NRF), funded by the Korean government (MSIT), under Grant RS-2026-25483479, with research facilities provided by the Institute of Computer Technology (ICT), Seoul National University. The authors are with the Department of Computer Science and Engineering, Seoul National University, Seoul 08826, Republic of Korea (e-mail: youssefennouri@snu.ac.kr; sha@snu.ac.kr). Scalability, however, remains the obstacle: training a surrogate on joint N -model configurations needs a profiling budget that grows combinatorially in N , and existing surrogate-based approaches [8] do not address this for heterogeneous GPU inference. To address this limitation, we propose a MeanField surrogate formulation with approximately linear empirical sample-complexity growth in N and integrate it into a practical runtime scheduler. This paper makes three main contributions: • We introduce the MeanField surrogate, a scalable, contention-aware predictor based on local model configurations and aggregate GPU state. • Under a stratified sampling protocol, we empirically show that the profiling budget required to reach R 2 ≥ 0.95 grows approximately linearly over N ∈2,..., 6, with n ∗ ≈ 20N . • We develop a runtime scheduler by integrating the MeanField surrogate into a genetic algorithm (GA)-based framework, which is subsequently validated at N = 5 under dynamic workload scenarios. Model 1 config λ 1 Model 2 config λ 2 ⋮ Model N config λ N Shared GPU SMs • VRAM • Memory BW Runtime Monitor Workload + GPU state GA + MeanField Runtime scheduler Observed State s t apply λ t * Fig. 1. The runtime monitor samples the observed state s t , which combines workload and aggregate GPU information. The MeanField surrogate guides the GA scheduler in selecting the joint configuration λ ⋆ t under SLA and VRAM constraints. I. RUNTIME SCHEDULING MODEL The system architecture is illustrated in Fig. 1. Because exhaustive online benchmarking is infeasible, the scheduler selects configurations for the co-executing models using a lightweight runtime surrogate to predict their performance. To formalize the scheduling problem, we consider N heterogeneous AI models concurrently executing on a shared GPU. The formulation is agnostic to the specific application types and applies to any heterogeneous workloads exposing This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013 © 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://w.ieee.org/publications/rights/index.html for more information. arXiv:2609.02109v1 [cs.DC] 2 Sep 2026 2 a discrete configuration space; in this work it is instantiated and evaluated on two workload families: LLMs based on the Qwen2.5 series and vision models based on the YOLO11 family. Each model exposes a discrete configuration space Λ i = V i × P i , where V i denotes model variants and P i runtime parameters. For the LLM, V i = FP16, AWQ (16-bit floating-point and Activation-aware Weight Quantization variants, respectively) and P i = request concurrency with values in 1, 2, 4; for YOLO11, V i = n, s, m and P i = skip rate, imgsz with values in1, 2, 4 and320, 480, 640 respectively. Here, request concurrency controls the number of parallel LLM requests and therefore affects KV-cache pressure and SM occupancy. The scheduler must select one configuration for each model, forming the joint configuration space: Λ = N Y i=1 Λ i (1) At each scheduling step, the runtime system observes the state s t = (ρ t ,m t ,u t )(2) where ρ t denotes the workload level, m t the current VRAM usage, and u t the aggregate GPU utilization. The scheduler selects λ ∗ = arg max λ∈Λ J (λ,s t ), where the utility J rewards throughput-weighted quality while penalizing unnecessary reconfigurations and SLA violations: J = X i ω i f i Q i −α X i η i c rel i −β X i max(0,τ i +δ−f i ) 2 (3) The first term P i ω i f i Q i is a throughput-weighted quality score: a configuration is rewarded only when it is both fast and accurate. The normalized throughput f i ∈ [0, 1] uses per-variant min–max scaling, f i = ( ˆ f i − f min v )/(f max v − f min v ), where ˆ f i is the raw predicted throughput (tokens/s for the LLM, FPS for vision) and f min v ,f max v are the extrema observed for variant v during profiling. Per-variant rather than global normalization is essential: a faster variant (AWQ) would otherwise dominate the scale and render every configuration of a slower one (FP16) effectively infeasible. The quality term Q i captures variant accuracy (Q LLM = 1.000, 0.979 for FP16/AWQ from relative perplexity; Q VIS =0.73, 0.96, 1.00 for YOLO11n/s/m from mean Average Precision, mAP50-95), scaled so the best variant scores 1. The priority weights ω i are operator-defined and encode application-specific importance; in our experiments all models are given equal priority ω i = 1 unless a scenario explicitly elevates one. They may further be adapted online as ω i = ̄ω i + γ max(0,τ i − ̄ f i ), with γ = 2.0. Reconfiguration overhead is modeled through η i = 1[v (t) i ̸= v (t−1) i ], which indicates whether model i changes variant and therefore incurs an amortized reload cost c rel i (c LLM = 30 s, c VIS = 5 s). The coefficient α = 1/60 ≈ 0.017 is the inverse of a 60-second amortization horizon, so a reload is accepted only when its predicted gain over the next 60 s exceeds the reload cost itself. SLA compliance is enforced through a hard per-model throughput floor τ i . In the N = 5 evaluation, τ i is set to the empirical P25 of each model’s normalized performance distribution. The quadratic penalty uses a safety margin δ = 0.05 above this floor, discouraging configurations that remain feasible but operate too close to the SLA boundary. We set β = 5.0. Hard constraints enforce both VRAM feasibility and minimum throughput requirements: X i r VRAM i ≤ R, f i ≥ τ i (4) VRAM-infeasibleconfigurationsareremovedwhen constructing Λ. Candidates violating any throughput floor receive J = −∞; infeasible offspring are repaired by replacement with a randomly sampled feasible individual. I. MEANFIELD SURROGATE FOR SCALABLE SCHEDULING A. Offline Profiling Experiments are conducted on a single NVIDIA RTX 3090 (24 GB GDDR6X) running CUDA 12.6 and Ubuntu 20.04. LLM workloads are served through vLLM [9] with FP16 and AWQ variants. Vision workloads use the Ultralytics engine. Offline profiling systematically explores the configuration space to generate training data for surrogate learning. The baseline N = 2 configuration space contains 162 concurrent configurations (6 LLM × 27 Vision), evaluated under four workload regimes (low, medium, high, burst), yielding 648 profiling measurements. For larger systems (N > 2), additional LLM and vision models were instantiated and profiled under the same workload regimes. Each cell is measured 10 times (two warmup passes discarded) under fixed GPU clocks, thermal stabilization below 70 ◦ C, and barrier-synchronized launches. A profiling sample denotes one sampled joint configuration evaluated under one workload regime. The ten repeated measurements are averaged into a single training row and are not counted as distinct samples. For N > 2, joint configurations are sampled stratified by model type, variant, and workload level rather than enumerating the full Cartesian product. The interference factor for model i is defined as: ∆ i = f isolated i − f concurrent i (5) Profiling reveals substantial interference: AWQ LLM variants lose 8.4% throughput on average under co-execution, while vision models lose 17–24% FPS depending on model size. Analysis of variance (ANOVA) [10] indicates workload intensity as the dominant factor for vision degradation (p < 0.01), whereas the vision-model skip rate dominates LLM slowdown by reducing shared SM and memory-bandwidth contention. These results suggest that interference is primarily driven by aggregate resource pressure rather than by specific model pairings, motivating the MeanField approximation proposed in this work. This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013 © 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://w.ieee.org/publications/rights/index.html for more information. 3 B. MeanField Surrogate The name follows the mean-field approximation of statistical physics, in which the many-body interactions acting on a particle are replaced by a single aggregate field rather than modeled pairwise. By analogy, each model is predicted to respond not to the identities of its co-runners but to an aggregate field, the observed GPU state, that summarizes their combined resource pressure. The surrogate therefore trains one lightweight predictor per model from the model’s local configuration and aggregate GPU state: ˆ f i = g i (λ i ,m t ,u t ,ρ t ).(6) The observed state affects scheduling through the surrogate prediction ˆ f i = g i (λ i ,s t ): raw throughput ˆ f i is normalized into f i , which then enters both the throughput-quality reward and the SLA penalty in Eq. (3). During scheduling, (m t ,u t ,ρ t ) are observed context variables obtained from runtime monitoring or simulation traces and are held fixed during GA search. The surrogate thus ranks candidate configurations conditioned on the current workload state rather than predicting future GPU-state evolution. Our measurements support this approximation: aggregate utilization, memory pressure, and workload level provide sufficient predictive information to retain high accuracy across the evaluated model mixtures. The surrogate can therefore be trained directly from concurrent N -model executions, and its profiling cost grows as O(N · k) in the number k of sampled workload states rather than combinatorially. Each predictor g i is a 64–32 MLP with BatchNorm, ReLU, Dropout(0.1), and Sigmoid output. Categorical features are ordinally encoded and targets are min–max normalized per variant. This architecture was selected by ablation at N = 3 and n = 80, providing the best accuracy–latency trade-off and sub-millisecond local inference (Table I). C. Scalability Challenge The main limitation of direct surrogate modeling is the growth of the joint configuration space. A fully joint predictor requires profiling complexity of O(k N ), where k denotes the number of configurations per model, and its input dimensionality grows with N . The Joint MLP baseline in Table I therefore serves only as an N = 2 reference, as both profiling requirements and input dimensionality become prohibitive beyond small deployments. The MeanField formulation avoids this growth: its profiling cost scales as O(N · k) rather than O(k N ), while prediction accuracy remains high (Section IV). To characterize this trade-off, the MeanField surrogate is trained from scratch and evaluated using stratified cross-validation across N ∈2, 3, 4, 5, 6, measuring both its asymptotic accuracy and the number of samples required to reach convergence. IV. EXPERIMENTAL RESULTS A. Sample-Complexity Scaling We evaluate surrogate sample efficiency under identical training protocols across N ∈ 2, 3, 4, 5, 6. Performance is measured using the mean R 2 across targets, and convergence is defined as the minimum training set size n ∗ required for the mean R 2 to reach 0.95. Results are averaged over 25 evaluations (5 random seeds × 5-fold stratified cross-validation), capturing both data-sampling and optimization variance. 0 50 100 150 200 250 0.6 0.8 1.0 Mean Accuracy ( R 2 ) (a) N=3 0 50 100 150 200 250 (b) N=4 0 50 100 150 200 250 Profiling Samples (n) 0.6 0.8 1.0 Mean Accuracy ( R 2 ) (c) N=5 0 50 100 150 200 250 Profiling Samples (n) (d) N=6 Fig. 2. Mean R 2 vs. training sample count n for the MeanField surrogate at N ∈ 3, 4, 5, 6. At every N , the surrogate reaches the R 2 ≥ 0.95 threshold within a modest sample budget, with the required budget increasing approximately linearly as N grows. Fig. 2 shows the MeanField learning curves under increasing training budgets. At every value of N the surrogate reaches the target accuracy with a small number of samples and then plateaus near R 2 ≈ 0.95–0.97, with low inter-seed and inter-fold variability. 234 5 6 System Size (N) 40 60 80 100 120 Required samples n ¤ n ¤ =20N measured Fig. 3. Empirical convergence sample count n ∗ for the MeanField surrogate. The observed budget grows approximately linearly, n ∗ ≈ 20N , over N ∈ 2, 3, 4, 5, 6. Fig. 3 summarizes this behavior through the empirical convergence sample count n ∗ . MeanField reaches the target This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013 © 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://w.ieee.org/publications/rights/index.html for more information. 4 accuracy after approximately (40, 60, 80, 100, 120) samples for N ∈ 2, 3, 4, 5, 6, consistent with a near-linear trend (n ∗ ≈ 20N ) and in sharp contrast to the combinatorial O(k N ) cost of a joint surrogate. B. Prediction Accuracy Table I reports the mean R 2 of the MeanField surrogate at full training budget. Accuracy remains high (R 2 > 0.95) up to N = 6 and stays within a narrow band of the N = 2 Joint MLP reference, confirming that the scalable MeanField formulation sacrifices little asymptotic accuracy while removing the joint surrogate’s combinatorial profiling cost. TABLE I ASYMPTOTIC PREDICTION ACCURACY (MEAN R 2 ± STD OVER 25 EVALUATIONS) OF THE MEANFIELD SURROGATE ACROSS N , WITH THE JOINT MLP SHOWN AS AN N = 2 REFERENCE. NJoint MLPMeanField 2 0.9826± 0.002 0.9860± 0.003 3–0.9779± 0.006 4–0.9689± 0.006 5–0.9645± 0.014 6–0.9588± 0.013 C. Runtime Scheduling Validation The MeanField surrogate drives a GA scheduler at N = 5, over a VRAM-feasible space of 78,732 configurations spanning three LLMs and two YOLO-family models. The 7B LLM is AWQ-only due to the 24 GB memory limit, and per-model SLA floors are set to the empirical P25. The GA uses 100 candidates, crossover probability 0.7, mutation probability 0.1, tournament selection (k = 3), elitism, and 30 generations. MeanField evaluates all local configurations in a median of 8.5 ms, followed by 17 ms for GA search, yielding a 26 ms end-to-end decision latency. Exhaustive surrogate search requires 131 ms, making the GA about 5× faster. TABLE I SCHEDULING RESULTS AT N = 5 OVER EIGHT DYNAMIC SCENARIOS. SchedulerMean JAvg. Gap Worst Gap SLA (%)(%)(%) Exhaustive Search2.7440.000.000.00 StaticFull2.7430.050.410.00 GA (ours)2.7420.100.380.00 StaticBudget (K = 5000)2.6005.227.360.00 StaticBudget (K = 500)2.44310.5117.160.00 Random1.42849.1358.713.45 Table I reports mean utility over eight dynamic scenarios and three seeds. Exhaustive denotes a per-step argmax over the same surrogate-predicted landscape, so the reported gap measures search quality only. We compare against StaticFull, StaticBudgetK, and Random. StaticFull is an adaptivity ablation with full-space access, whereas StaticBudgetK selects one fixed configuration from only K offline samples. The GA remains within 0.10% of exhaustive search with zero SLA violations. StaticFull is also near-optimal, indicating limited adaptation benefit in these scenarios. Under realistic calibration budgets, however, StaticBudgetK loses 10.51% utility at K = 500 and 5.22% at K = 5000. The main benefit is therefore profiling-efficient near-optimal search rather than adaptation alone. V. CONCLUSION This paper proposed a MeanField surrogate for scalable runtime scheduling of concurrent heterogeneous AI workloads on a shared GPU. Across N ∈2, 3, 4, 5, 6, it achieved high prediction accuracy (R 2 ≈ 0.96) with approximately linear empirical sample-complexity growth. Integrated into a GA scheduler, it scaled to an N = 5 problem with 78,732 feasible configurations, remaining within 0.10% of exhaustive search with zero SLA violations across eight dynamic scenarios, with a median end-to-end decision latency of 26 ms. Future work will extend the LLM workload model to capture prefill/decode phases and KV-cache occupancy. REFERENCES [1] H. Shen, L. Chen, Y. Jin, L. Zhao, B. Kong, M. Philipose, A. Krishnamurthy, and R. Sundaram, “Nexus: A GPU cluster engine for accelerating DNN-based video analysis,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19), 2019, p. 322–337. [2] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system,” in 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017, p. 613–627. [3] A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y. Vigfusson, and J. Mace, “Serving DNNs like clockwork: Performance predictability from the bottom up,” in 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’20), 2020, p. 443–462. [4] F. Strati, X. Ma, and A. Klimovic, “Orion: Interference-aware, fine-grained GPU sharing for ML applications,” in Proceedings of the 19th European Conference on Computer Systems (EuroSys ’24), 2024, p. 1075–1092. [5] L. Ma, H. Chen, E. Shao, L. Wang, Q. Chen, and G. Tan, “Elasticroom: Multi-tenant DNN inference engine via co-design with resource-constrained compilation and strong priority scheduling,” in Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’24), 2024, p. 1–14. [6] Z. Zhao, N. Ling, N. Guan, and G. Xing, “Miriam: Exploiting elastic kernels for real-time multi-DNN inference on edge GPU,” arXiv preprint arXiv:2307.04339, 2023. [7] F. Yu, S. Bray, D. Wang, L. Shangguan, X. Tang, C. Liu, and X. Chen, “Automated runtime-aware scheduling for multi-tenant DNN inference on GPU,” in Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2021. [8] Y. Han, Z. Nan, S. Zhou, and Z. Niu, “DVFS-aware DNN inference on GPUs: Latency modeling and performance analysis,” arXiv preprint arXiv:2502.06295, 2025. [9] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” in Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23), 2023, p. 611–626. [10] D. C. Montgomery, Design and Analysis of Experiments, 9th ed. Wiley, 2017. This article has been accepted for publication in IEEE Embedded Systems Letters. This is the author’s version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/LES.2026.3724013 © 2026 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://w.ieee.org/publications/rights/index.html for more information.