Paper deep dive
User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling
Alfreds Lapkovskis, Ali Beikmohammadi, Sindri Magnússon, Praveen Kumar Donta
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Growing demand for artificial intelligence (AI) inference services requires scalable infrastructure, yet centralized serving costs rise with demand. We propose a collaborative distributed inference system combining dedicated infrastructure with resources contributed by service users. Dedicated resources provide baseline capacity for maintaining quality of service (QoS), while volunteered resources absorb increasing demand without proportional growth in centralized infrastructure. To capture stochastic and dynamic interactions among users, resources, tasks, and policies, we develop a high-dimensional generative Markov model with structured temporal factorization. The model supports simulation and provides a foundation for task scheduling and QoS-aware resource allocation optimization. We evaluate the system across user populations, resource capacities, and centralized and distributed scheduling policies. Simulations show that distributed scheduling becomes increasingly advantageous as the user population grows, improving request completion and P99 latency while substantially reducing dedicated resource consumption. These results demonstrate the feasibility of user-assisted collaborative inference for infrastructure-efficient autoscaling.
Tags
Links
- Source: https://arxiv.org/abs/2608.11840v1
- Canonical: https://arxiv.org/abs/2608.11840v1
Trouble viewing inline? Open PDF directly →
Full Text
51,519 characters extracted from source content.
Expand or collapse full text
User-Assisted Collaborative Distributed Inference for Efficient QoS-Aware Autoscaling Alfreds Lapkovskis Affiliation: Department of Computer Systems and Sciences, Stockholm University Ali Beikmohammadi Affiliation: SE-106 91 Stockholm, Sweden Affiliation: E-mail alfreds.lapkovskis, beikmohammadi, sindri.magnusson, praveen@dsv.su.se Sindri Magnússon Praveen Kumar Donta Abstract Growing demand for artificial intelligence (AI) inference services requires scalable infrastructure, yet centralized serving costs rise with demand. We propose a collaborative distributed inference system combining dedicated infrastructure with resources contributed by service users. Dedicated resources provide baseline capacity for maintaining quality of service (QoS), while volunteered resources absorb increasing demand without proportional growth in centralized infrastructure. To capture stochastic and dynamic interactions among users, resources, tasks, and policies, we develop a high-dimensional generative Markov model with structured temporal factorization. The model supports simulation and provides a foundation for task scheduling and QoS-aware resource allocation optimization. We evaluate the system across user populations, resource capacities, and centralized and distributed scheduling policies. Simulations show that distributed scheduling becomes increasingly advantageous as the user population grows, improving request completion and P99 latency while substantially reducing dedicated resource consumption. These results demonstrate the feasibility of user-assisted collaborative inference for infrastructure-efficient autoscaling. Keywords: Distributed AI Inference AI Inference Services Generative Markov Model Quality of Service QoS-Aware Scheduling Resource Autoscaling Volunteer Computing 1 Introduction Artificial intelligence (AI) inference has become a core service in modern computing systems. According to the AI Index Report 2026, corporate AI adoption has reached 88%, while global investments surged to $581.69 billion in 2025 [21]. As AI inference scales to meet this demand, cumulative inference costs increasingly outpace one-time training expenses, making operational efficiency a growing concern. These trends highlight the importance of the computing paradigms underlying AI inference. Central to these paradigms is cloud computing, the de facto standard for Internet applications [18], including AI inference. Its key advantages include on-demand resource provisioning, rapid elasticity, standardized access interfaces, utilization monitoring, and pay-as-you-go pricing [17, 15]. However, centralized clouds are also subject to significant challenges. The costs associated with the setup and management of cloud infrastructure are prohibitively high [19, 18]. Furthermore, expenses grow proportionally with demand, making cloud computing increasingly costly at scale. This scaling also carries a substantial environmental burden. Training Grok 4 alone emitted more CO2 than an average car produces over its entire lifetime, and the annual inference cost of GPT-4o exceeds the drinking water demand of 12 million people [21]. More broadly, cloud data centers consume vast amounts of energy and freshwater, with AI workloads emerging as a primary driver [13], and these impacts are projected to worsen as AI demand continues to rise. Edge and fog computing partially address these limitations. By distributing computation closer to end users and data sources, they reduce latency and bandwidth consumption, and decrease reliance on centralized cloud resources [7]. The computing continuum extends this approach by integrating cloud, fog, and edge tiers into a unified, synergistic architecture [8]. Recent work has improved inference within these environments through heterogeneous on-device co-execution [26], task-aware DNN partitioning in mobile edge computing [9], and joint batching and transformer partitioning for distributed LLM inference [28]. Other studies address on-device inference techniques [24], disaggregated inference deployment across heterogeneous infrastructure [27], inference-data distribution for model monitoring [16], and cloud inference autoscaling [11]. These approaches improve inference within predefined deployment environments, but generally rely on dedicated infrastructure rather than resources that become available dynamically as users join the service. Moreover, computing-continuum systems still face architectural and economic barriers to widespread adoption [6, 20]. Yet deploying dedicated edge and fog infrastructure is not the only path forward. A vast number of commodity devices owned by individuals and institutions worldwide, remain largely underutilized [4, 19]. Harnessing these idle resources toward a shared computational objective is known as volunteer computing [15, 18], a paradigm that offers access to cheaper, distributed compute without requiring energy-hungry data center infrastructure. However, existing systems provide limited support for stable QoS guarantees due to resource heterogeneity and intermittent availability [18]. Hence, this model has limited applicability to consumer-oriented AI inference applications. Yet the underlying concept remains compelling in the context of AI inference, where operational costs are a growing concern. Therefore, we propose a collaborative distributed inference system that partitions inference computations between dedicated resources, such as centralized servers, and volunteered resources, such as users’ devices (as shown in Fig. 1). Unlike centralized cloud inference or approaches distributed exclusively across dedicated computing continuum infrastructure, our system does not require dedicated computational capacity to scale in proportion to demand. In contrast to conventional volunteer computing, however, it retains dedicated infrastructure to provide a reliable baseline capacity and support QoS guarantees despite the intermittent availability and heterogeneity of volunteered resources. Given an appropriate scheduler, the server can process all requests when few users are online; as the number of available users grows, the demand increases, but so does the pool of volunteered resources, allowing the system to progressively offload computation to user devices. Determining how to schedule tasks across these dynamic resources and how much dedicated capacity is necessary to satisfy QoS requirements requires a model that captures the complex interactions among users, resources, tasks, and policies. We therefore develop such a model to enable system analysis, scheduling optimization, and QoS-aware allocation of dedicated resources. Our contributions are threefold: 1. We propose a novel collaborative distributed AI inference system that combines the dedicated and volunteered user resources for automatic scaling while maintaining QoS and reducing the need for centralized resources. 2. We model the system as a generative Markov model whose temporal factorization captures sparse dependencies among structured state variables and their elements, enabling tractable simulation, inference, and policy learning. 3. We evaluate and compare various scheduling policies by simulating the proposed system under diverse configurations, varying the number of users, dedicated, and volunteered resources. Figure 1: Overview of the proposed distributed inference system. The server receives requests, schedules subtasks across dedicated and volunteered resources according to the scheduling policy, and aggregates the results into responses. 2 Proposed Work We propose a collaborative distributed inference system that partitions inference computations between dedicated resources, such as centralized servers, and volunteered resources, such as users’ devices. Ensuring QoS in such a system requires not only provisioning additional dedicated resources but also optimizing computation scheduling strategies. In principle, QoS can be maintained either by increasing server capacity and executing more computations on dedicated infrastructure or by improving the scheduler to make more effective use of volunteered resources. However, volunteered resources are inherently heterogeneous, highly dynamic, and intermittently available. Scheduling decisions must therefore be adaptive and capable of anticipating future system states to prevent QoS degradation while efficiently utilizing the available resource pool. Optimizing such scheduling strategies directly on a live system is impractical: it would be costly, time-consuming, and require extensive trial-and-error exploration in a complex, dynamic environment. Consequently, a system model is needed to enable efficient evaluation and optimization of scheduling strategies. In the following section, we propose our model of the system. 2.1 System Model We model the system using the generative Markov framework introduced in [12], where states ts_t evolve over time according to the Markov property. The framework provides a unified representation of system components and allows their distributions to be modeled, modified, and extended independently. Its sparse factorization makes the high-dimensional model tractable for simulation, inference, and policy learning. We consider a system consisting of a dedicated server v0v_0, a set of users i∈ℐi , and a set of nodes v∈V=v0∪ℐv∈ V=\v_0\ , each providing resources r∈ℛr . The system supports |||K| types of inference tasks, where each task type k∈k consists of a set of subtasks p∈kp _k. Users can both request inference tasks and contribute resources to execute subtasks. The system is controlled by two decision-making policies: a scheduler policy π and an executor policy ς . Together, these policies generate the joint action t=⟨tπ,tς⟩u_t= _t^π,u_t . We assume a fixed interval of 1 s between timesteps t. We define the system state ts_t as a tuple of high-dimensional variables: t=⟨t,t,t,t,t,t,t,t⟩,s_t= _t,q_t,a_t,x_t,y_t,c_t,d_t,u_t , (1) where each variable captures a specific aspect of the system state. Each variable is indexed by the system entities to which it applies, with one dimension assigned to each entity type. Following the Markov property, variables may depend on other variables within the same or previous time step. These dependencies are generally sparse, since many entities are independent in various state variables. In the following sections, we describe the system variables and the implementations of the decision-making policies. 2.2 System Variables Availability states (to_t). We represent user availability with the online indicator t∈0,1|ℐ|o_t∈\0,1\^|I| and the state-duration variable to∈ℕ|ℐ|d^o_t ^|I|, where ot,i=1o_t,i=1 indicates that user i∈ℐi is online and dt,iod^o_t,i denotes the number of time steps since the last online/offline transition. These variables evolve as to∼P(⋅∣t−1o,t−1)d^o_t P(· ^o_t-1,o_t-1) and t∼P(⋅∣t−1,to)o_t P(· _t-1,d^o_t), factorized over users under independent availability processes. The duration dependence captures the fact that transition probabilities change with the time spent in the current state, while t−1o_t-1 selects the corresponding online or offline duration model. For each user, dt,iod^o_t,i increments if the state persists and resets to 00 if it changes; ot,io_t,i then remains equal to ot−1,io_t-1,i when dt,io>0d^o_t,i>0 and switches to 1−ot−1,i1-o_t-1,i otherwise. We model availability durations with parametric survival distributions, following findings from [10], and sample transitions from their discrete-time hazards. We use a Weibull survival function S(t)≜exp(−(t/λ)k)S(t) (-(t/λ)^k) for online durations, with λ=1133.1λ=1133.1 and k=0.5764k=0.5764, and a log-normal survival function S(t)≜1−Φ((logt−μ)/σ)S(t) 1- (( t-μ)/σ) for offline durations, with μ=log(5259)μ= (5259), σ=1.5σ=1.5, and standard normal CDF Φ(⋅) (·). Given duration d−1d-1, the current state persists with probability S(d)/S(d−1)S(d)/S(d-1) and changes otherwise. These parameters yield median online and offline durations of approximately 1010 minutes and 1.51.5 hours, respectively, and a steady-state user availability of approximately 10%10\%. Inference requests (tq_t). We represent inference requests with t∈|ℐ|q_t ^|I|, where =∪k0Q=K∪\k_0\ and k0k_0 denotes no active request, and with the request-state duration tq∈ℕ|ℐ|d^q_t ^|I|, where dt,iqd^q_t,i counts the time steps since the last request-state change of user i. The variables evolve as tq∼P(⋅∣t−1q,t−1,t,t−1)d^q_t P(· ^q_t-1,q_t-1,o_t,y_t-1) and t∼P(⋅∣t−1,t,tq)q_t P(· _t-1,o_t,d^q_t), factorized over users. Users can submit requests only while online; hence, if ot,i=0o_t,i=0, we set qt,i=k0q_t,i=k_0 and dt,iq=0d^q_t,i=0. Each user has at most one active request, so requests alternate between k0k_0 and some k∈k . The duration dt,iqd^q_t,i increments while the request state persists and resets when a request starts, an active request completes according to t−1y_t-1, or the user abandons it. In the implementation, we use a Pareto survival function S(t)≜1S(t) 1 for t<κt<κ and S(t)≜(κ/t)αS(t) (κ/t)^α for t≥κt≥κ, following heavy-tailed inter-request times observed in interactive systems [2, 5]. We use this distribution both for user think time, when qt−1,i=k0q_t-1,i=k_0, and patience, when qt−1,i≠k0q_t-1,i≠ k_0 and the request has not completed. Given duration d−1d-1, the current request state persists with probability S(d)/S(d−1)S(d)/S(d-1) and changes otherwise; when a new request starts, we sample its type k from Categorical(p1,…,p||)Categorical(p_1,…,p_|K|). We set κ=30sκ=30\,s and α=1.5α=1.5, yielding a minimum think time and patience of 30s30\,s, a median of approximately 48s48\,s, and equal steady-state fractions of time with and without an active request. Allocated resources (ta_t). We represent allocated resources with t∈ℝ≥0|ℛ|×|V|a_t _≥ 0^|R|×|V|, where at,r,va_t,r,v denotes the amount of resource r∈ℛr allocated to the system on node v∈Vv∈ V. The variable evolves as t∼P(⋅∣~t−1,t)a_t P(· a_t-1,o_t), where ~t−1=[t−1,…,t−H] a_t-1=[a_t-1,…,a_t-H] captures the autocorrelation of resource trajectories. We factorize the transition over resources and nodes, condition user resources on availability, and treat the server v0v_0 as always online. Thus, we set at,r,i=0a_t,r,i=0 for offline users and sample allocations for online users and the server. Each at,r,va_t,r,v represents the total resource capacity allocated to the system, including both used and unused capacity. In the implementation, we use ℛ=band_in,band_out,cpu,memory,storageR=\ band\_in, band\_out, cpu, memory, storage\ and fit one generative ARMA model [3] per resource. We collect workstation measurements every 3s3\,s for 44 hours, resample them to 1s1\,s, standardize each resource, and apply a Yeo–Johnson transformation [25]. For each r, we fit ARMA models with parameters p∈1,2,5,10p∈\1,2,5,10\ and q∈0,1,2,5,10q∈\0,1,2,5,10\ by maximum likelihood, selecting ARMA∗A,rARMA^A,r_* with the lowest AIC [1]. During simulation, we sample from ARMA∗A,rARMA^A,r_*, invert the transformation, rescale using node-specific mean μr,vAμ^A_r,v and standard deviation σr,vAσ^A_r,v, and clip to [a¯r,vo,a¯r,vo][ a^o_r,v, a^o_r,v], where a¯r,vo=0 a^o_r,v=0 and a¯r,vo a^o_r,v denote the minimum and maximum permitted allocations. We assume homogeneous user nodes but allow different server parameters. Subtask readiness states (tx_t). We represent subtask-data readiness with an auxiliary state ^t∈|V|×||×Pmax x_t ^|V|×|K|× P_ , a duration variable tx∈ℕ|V|×||×Pmaxd^x_t ^|V|×|K|× P_ , and the actual readiness state t∈|V|×||×Pmaxx_t ^|V|×|K|× P_ , where =empty,download,pause,readyX=\ empty, download, pause, ready\, Pmax=maxk∈|k|P_ = _k |P_k|, and p∈kp _k indexes subtasks of task type k. These variables evolve as ^t∼P(⋅∣t−1,t,t−1x) x_t P(· _t-1,o_t,u^x_t-1), tx∼P(⋅∣t−1x,^t,tx)d^x_t P(· ^x_t-1, x_t,c^x_t), and t∼P(⋅∣^t,tx)x_t P(· x_t,d^x_t), factorized over nodes, task types, and subtasks. The auxiliary state first applies scheduler and executor actions t−1x=⟨t−1x,π,t−1x,ς⟩u^x_t-1= ^x,π_t-1,u^x, _t-1 : the scheduler can start or cancel downloads, while the executor can pause, resume, or abort them. We suppress transitions for offline users and treat the server v0v_0 as always online; moreover, the server stores all subtask data a priori. The duration dt,v,k,pxd^x_t,v,k,p matters only when x^t,v,k,p=download x_t,v,k,p= download. For all other states, we set it to 00. During an active download, dt,v,k,pxd^x_t,v,k,p increments while the accumulated storage consumption ct,storage,v,k,pxc^x_t, storage,v,k,p remains below the required subtask-data size ck,pc_k,p, and resets to 00 once ct,storage,v,k,px=ck,pc^x_t, storage,v,k,p=c_k,p. The actual state tx_t copies ^t x_t, except that x^t,v,k,p=download x_t,v,k,p= download becomes xt,v,k,p=readyx_t,v,k,p= ready when dt,v,k,px=0d^x_t,v,k,p=0, indicating download completion. In the implementation, paused downloads retain their accumulated storage, ready subtasks keep storage fixed at ck,pc_k,p, and all non-storage download consumptions vanish outside the download state. Subtask execution states (ty_t). We represent subtask execution with an auxiliary state ^t∈|ℐ|×|V|×||×Pmax y_t ^|I|×|V|×|K|× P_ , a duration variable ty∈ℕ|ℐ|×|V|×||×Pmaxd^y_t ^|I|×|V|×|K|× P_ , and the actual execution state t∈|ℐ|×|V|×||×Pmaxy_t ^|I|×|V|×|K|× P_ , where =idle,download,execution,upload,doneY=\ idle, download, execution, upload, done\. The entry yt,i,v,k,py_t,i,v,k,p describes the execution state of subtask p∈kp _k of user i’s request of type k on node v. These variables evolve as ^t∼P(⋅∣t−1,t,t−1y) y_t P(· _t-1,o_t,u^y_t-1), ty∼P(⋅∣t−1y,^t,ty)d^y_t P(· ^y_t-1, y_t,c^y_t), and t∼P(⋅∣^t,ty)y_t P(· y_t,d^y_t), factorized over users, nodes, task types, and subtasks. The scheduler action t−1y,πu^y,π_t-1 assigns, cancels, or finalizes subtasks, while the executor action t−1y,ςu^y, _t-1 can abort running subtasks. Assigning a subtask starts with download on user nodes and directly with execution on the server v0v_0; offline user nodes reset non-completed subtasks to idle, whereas done subtasks can still be finalized because the server already has their outputs. The duration dt,i,v,k,pyd^y_t,i,v,k,p tracks the current active phase y∈~=download,execution,uploady∈ Y=\ download, execution, upload\. It increments while the phase continues and resets to 00 when the phase completes, is interrupted, or reaches done. The actual state ty_t copies ^t y_t, except that duration resets advance active phases as download→execution download→ execution, execution→upload execution→ upload and upload→done upload→ done on user nodes, and execution→done execution→ done on the server. In the implementation, we model phase completion with discrete-time hazard models [23] fitted separately for each y∈~y∈ Y. We fit these models from measured executions of representative computer-vision subtasks under different bandwidth levels, batch sizes 1,2,5,10,20,50\1,2,5,10,20,50\, and subtask types. The covariates include standardized resource consumptions tyc^y_t, batch size, logdt,i,v,k,py d^y_t,i,v,k,p, polynomial interactions up to degree q∈1,2,3q∈\1,2,3\, and one-hot subtask-type indicators; we select a common degree q by average AIC across phases. During simulation, subtasks with the same node, subtask type, and execution age form a batch, share one sampled completion event, and either advance by resetting dyd^y to 00 or continue with dy+1d^y+1. Resource consumption (tc_t). We define total resource consumption as t=⟨tx,ty⟩c_t= ^x_t,c^y_t , where tx∈ℝ≥0|ℛx|×||×||×Pmaxc^x_t _≥ 0^|R^x|×|V|×|K|× P_ captures subtask-data preparation and ty∈ℝ≥0|ℛy|×|ℐ|×||×||×Pmaxc^y_t _≥ 0^|R^y|×|I|×|V|×|K|× P_ captures request execution. We use ℛx=ℛ∖band_outR^x=R \ band\_out\ and ℛy=ℛ∖storageR^y=R \ storage\, with execution-stage subsets for downloading, computation, and uploading. The variables evolve jointly as (tx,ty)∼P(⋅∣~t−1x,~t−1y,t,^t,^t,t−1x,t−1y)(c^x_t,c^y_t) P(· c^x_t-1, c^y_t-1,a_t, x_t, y_t,d^x_t-1,d^y_t-1), where the history windows ~t−1x c^x_t-1 and ~t−1y c^y_t-1 capture autocorrelation. We condition on allocated resources because they bound feasible consumption, on auxiliary states because inactive subtasks consume no resources, and on durations because subtasks with the same phase age may share resources through batching. Server and user consumptions are coupled: user transfers consume server bandwidth, while limited server bandwidth constrains user-side transfer rates. In the implementation, we fit generative ARMA models to measured consumption traces after resampling to 1s1\,s, standardization, and Yeo–Johnson transformation. For txc^x_t, we collect traces by downloading subtask data in 1Kb1\,Kb chunks under representative inbound bandwidth levels obtained by applying Lloyd–Max quantization [14] to the inbound-bandwidth traces collected for allocated-resource modeling. We model storage as a delta and accumulate it over time. For tyc^y_t, we collect traces from representative subtasks under different bandwidth levels, batch sizes 1,2,5,10,20,50\1,2,5,10,20,50\, and subtask types. For each resource and stage, we fit ARMA candidates with p∈1,2,5,10p∈\1,2,5,10\ and q∈0,1,2,5,10q∈\0,1,2,5,10\, selecting the model with the lowest AIC. During simulation, we sample node-level consumption, interpolate scaling statistics over the relevant bandwidth or batch-size levels using inverse-distance weighting [22], clip transfer consumption to the effective bandwidth shared with the server, proportionally downscale CPU-dependent throughput when CPU is overcommitted, and divide node-level consumption among concurrent transfers or same-age execution batches. 2.3 Decision-Making Policies The system uses two types of policies. The server-side scheduler π issues readiness and execution actions, tx,πu^x,π_t and ty,πu^y,π_t, while the node-side executor ς issues constraint-enforcement actions, tx,ςu^x, _t and ty,ςu^y, _t. Their action domains are x,π=none,download,cancelU_x,π=\ none, download, cancel\, y,π=none,execute,finish,cancelU_y,π=\ none, execute, finish, cancel\, x,ς=none,abort,pause,resumeU_x, =\ none, abort, pause, resume\, and y,ς=none,abortU_y, =\ none, abort\. Both policies condition on the current state excluding the actions being generated, i.e., tπ∼π(⋅∣t∖t)u^π_t π(· _t _t) and tς∼ς(⋅∣t∖t)u _t (· _t _t). Executor actions take precedence over scheduler actions. Executor policy. The executor policy ς enforces local feasibility by comparing total consumption from txc^x_t and tyc^y_t with allocated resources ta_t. Under non-storage overflow, it pauses all active subtask-data downloads on the affected node, since they share a single download channel and can later resume. Under storage overflow, it aborts downloads with the largest storage footprints until feasibility is restored. It also aborts executions whose required subtask data are no longer available or have been aborted. If overflow remains, it aborts executions in the order download→execution→upload download→ execution→ upload; for execution, it groups subtasks by node, subtask, and execution age, and greedily aborts the heaviest normalized-load batches. Paused downloads resume once the node has no active executions and no non-storage overflow. Centralized scheduler policy. The centralized scheduler represents fully centralized inference baseline. Since all executions run on the server v0v_0, which stores all subtask data a priori, it schedules no user-node downloads. For each active request qt,i=kq_t,i=k, it cancels obsolete executions, finishes requests whose required subtasks have reached done, and assigns missing subtasks p∈kp _k only to v0v_0, subject to available server memory estimated by m^k,p,ny m^y_k,p,n. Uniform scheduler policy. The uniform scheduler uses online volunteered nodes opportunistically. For each online node, the policy schedules downloading at most one missing subtask data whose storage requirement and conservative download-memory estimate m^x m^x fit within the node’s free capacity; the server is excluded because it already stores all subtask data. The policy cancels obsolete executions, finishes completed requests, identifies requested but unplaced user–subtask pairs, and assigns them uniformly at random among feasible nodes. A node is feasible if it is online, has xt,v,k,p=readyx_t,v,k,p= ready, and has enough free memory for the marginal execution-memory estimate m^k,p,ny m^y_k,p,n. We precompute m^x m^x and m^k,p,ny m^y_k,p,n from learned memory models using a μ+2σμ+2σ safety margin, interpolating m^k,p,ny m^y_k,p,n across measured batch sizes. Memory estimates are needed because memory, unlike CPU and bandwidth, is a hard constraint. Affinity-based scheduler policy. The affinity-based scheduler uses the same download, cancellation, and finishing logic as the uniform policy, but assigns executions to improve batching. For each required subtask (k,p)(k,p), it prioritizes eligible nodes by the number of active executions of the same subtask already placed on them, excluding offline nodes and nodes without ready subtask data. Using m^k,p,ny m^y_k,p,n, it computes how many additional executions fit in each eligible node’s free memory and fills the resulting assignment slots with randomly shuffled users requiring (k,p)(k,p). This greedily co-locates same-subtask executions to increase batch sizes while aiming to preserve memory feasibility. 2.4 Optimization Problems Given the proposed system model, two optimization problems naturally arise. First, for a fixed executor policy ς , one can optimize the scheduler policy π to maximize QoS, e.g., by minimizing long-term average request latency: maxπlim infT→∞[1T∑t=1Tr(t)∣0,0:t−1π∼π], _π\; _T→∞\,E [ 1T _t=1^Tr(s_t) _0,u^π_0:t-1 π ], (2) where r(t)≜−⊤(t,tq)r(s_t) -1 δ(q_t,d^q_t) is a reward function, and (t,tq)∈0,1|ℐ| δ(q_t,d^q_t)∈\0,1\^|I| denotes per-user latency increments. Second, for a fixed scheduler π and executor policy ς , one can optimize the allocation of dedicated server resources. In this case, the objective is to determine the amount of dedicated capacity required to satisfy a target QoS constraint under the given model and policies; e.g., one may require request latency to remain below 10s10\,s for 99%99\% of requests, corresponding to P99 latency. We define this problem as min† _a_ ∥†∥ _ _w (3) s.t. .t. ℙ(dqt,i≤τk∣qt+1,i=k0,qt,i=k)≥1−ϵk,∀k∈, \! (d^q_t,i≤ _k q_t+1,i=k_0,\,q_t,i=k )≥ 1- _k, ∀ k , where †a_ is a nominal, i.e., maximum, server resource capacity that parametrizes transition probabilities of ta_t, and ∥⋅∥ · _w is a weighted ℓ1 ^1 norm with per-resource cost weights w, τk _k are per-request-type latency thresholds, and ϵk∈(0,1) _k∈(0,1) are significance levels. Formal definitions of these problems are provided in [12]. It is noteworthy to mention that this paper does not focus on designing a scheduling algorithm or resource-allocation optimizer. Instead, our primary focus is to define the collaborative distributed inference system and develop a model of its dynamics. We use this model to simulate fixed policies under different configurations and compare collaborative distributed inference with centralized inference. Future work will use this model to address the optimization problems. 3 Evaluation 3.1 Experimental Setup We evaluate the proposed system by simulating trajectories of the generative model, sampling t∼P(⋅∣t−1)s_t P(· _t-1). We parameterize ARMA resource-consumption and hazard-based duration models from measurements collected on a MacBook M4 Pro with 24GB24\,GB RAM and 512GB512\,GB storage. We consider four task types: light and heavy, each with 55 or 1010 subtasks, sampled with probabilities 0.4,0.3,0.2,0.1\0.4, 0.3, 0.2, 0.1\ to reflect that simpler tasks are typically requested more often. Subtasks are instantiated using established computer-vision models. Each simulation runs for 100,000100,000 time steps, or approximately 2828 hours. We vary user count 100,1,000,5,000,10,000\100,1,000,5,000,10,000\, user capacity 1×,2×\1×,2×\ of base resource profile, server capacity 10×,30×,50×,80×,100×\10×, 30×, 50×, 80×, 100×\, and scheduler policy. These dimensions test how the system behaves as demand and volunteered resources grow together, how offloading depends on user-device capacity, how much dedicated capacity is needed to sustain QoS, and how scheduling affects system behavior. We define the base resource profile as a moderate, non-intrusive contribution from a user device: 100Mbit100\,Mbit inbound/outbound bandwidth, 3GB3\,GB storage, 2.5GB2.5\,GB memory, and 200%200\% CPU. The bandwidth represents neither a weak nor exceptional connection; the storage can hold several small subtasks or one large subtask with a few smaller ones. We set memory and CPU to rounded values slightly above the largest measured average subtask consumption at batch size 22, making small batches of demanding subtasks feasible with high probability and larger lightweight batches more likely. These values define the maximum allocation thresholds a¯r,vo a^o_r,v, while realized capacities ta_t still vary over time. We report request completions and cancellations, P99 latency as the primary QoS metric, dedicated resource consumption, and server/user subtask completion rates. We focus on P99 latency because other latency summaries, such as mean and P95 latency, showed similar trends and are omitted due to space constraints. For full reproducibility, we published our implementation code on GitHub.11 1 https://anonymous.4open.science/r/AutoscalingArchitecture-C41F 3.2 Results Completed vs. canceled tasks. Fig. 2 compares completed and canceled requests. The centralized policy is unaffected by user capacity because all subtasks run on the server. Under the centralized policy, at 10×10× capacity, completions grow only through 1,0001,000 users, then plateau and decline as cancellations surge. Higher capacities delay saturation, but gains diminish beyond 30×30×; at large user counts, cancellations still equal roughly one third to one half of completions. (a) Server 10×10× (b) Server 30×30× (c) Server 50×50× (d) Server 80×80× (e) Server 100×100× Figure 2: Completed vs. canceled requests under different scheduler policies and server capacities. Each subfigure corresponds to a different server capacity multiplier. The x-axis varies the user configuration, i.e., the number of users (|ℐ||I|) and the per-user resource-capacity multiplier. The uniform policy matches the centralized through 1,0001,000 users and improves with 2×2× user capacity. At 5,0005,000 users, it performs better at 10×10× and 30×30× server capacity and comparably above them. At 10,00010,000, it yields more completions and fewer cancellations at every server capacity, including 100×100×. Thus, the growing volunteered resource pool offsets demand more effectively than centralized scaling, whose gains from 50×50× to 100×100× are incremental. Centralized execution may also form larger batches that save resources but delay individual subtasks, whereas uniform distribution creates smaller batches across more nodes. The affinity-based policy is competitive through 1,0001,000 users and beats the centralized policy at 10×10× capacity, but otherwise generally yields the worst completion-to-cancellation balance. Its larger batches may extend execution and increase cancellation risk; node departures or executor aborts can also interrupt several batched subtasks simultaneously. Overall, distributed inference is effective and feasible, but batching efficiency must be balanced against latency and failure risk, motivating scheduler optimization using the proposed model. Task latencies. Fig. 3 reports P99 request latency. The centralized policy performs best or comparably through 1,0001,000 users, and at 5,0005,000 as server capacity grows. At 10,00010,000, the uniform policy remains substantially better. Centralized latency varies more across request types under higher demand. Besides differences in task complexity, unequal request probabilities lead to frequent subtasks forming larger batches that run longer and reduce the resources available to other requests. More server capacity mitigates but does not eliminate this effect. (a) Server 10×10× (b) Server 30×30× (c) Server 50×50× (d) Server 80×80× (e) Server 100×100× Figure 3: P99 request latency under different scheduler policies and server capacities. Each subfigure corresponds to a different server capacity. The x-axis shows user configurations. The curves represent average P99 latencies across request types. The faded regions represent the range of P99 latencies across request types, from the lowest to the highest. The uniform policy matches the centralized policy through 1,0001,000 users, outperforms it at 5,0005,000 with up to 30×30× server capacity, and remains comparable above that. At 10,00010,000, it consistently achieves much lower P99 latency. Its variation under constrained configurations likely arises because weaker users store fewer models, particularly for heavy tasks, forcing more execution onto the server. Increasing user and server capacities distributes more subtasks and largely removes this variation. At 10×10× server capacity, the affinity-based policy is slightly worse than the uniform policy but better than the centralized policy at large user counts. Beyond 30×30×, its latency changes little and generally remains above the uniform policy, likely because larger batches delay and couple multiple requests. Overall, both distributed policies show little improvement beyond 30×30×–50×50× server capacity, i.e., this range provides sufficient dedicated capacity to complement volunteered resources. Centralized scaling improves slowly and does not surpass the uniform policy at a large scale. Thus, distributed scheduling provides better tail latency and under lower server capacity than centralized scheduling. Dedicated resource consumption. Fig. 4 shows that the centralized policy consistently uses more dedicated resources. CPU consumption reaches every allocation limit, indicating persistent saturation. Since consumption is recorded before the executor resolves hard-constraint violations, mean memory demand reaches nearly twice the allocated 25GB25\,GB at 10×10× capacity and slightly exceeds 75GB75\,GB at 30×30×, causing the subtask aborts. At higher capacities, memory remains within its allocation and grows slowly, suggesting that demand approaches its required level while additional CPU accelerates processing, causing slight further increases in memory usage. (a) Server 10×10× (b) Server 30×30× (c) Server 50×50× (d) Server 80×80× (e) Server 100×100× (f) Server 10×10× (g) Server 30×30× (h) Server 50×50× (i) Server 80×80× (j) Server 100×100× Figure 4: Server resource consumption under different scheduler policies and server capacities. The first row reports CPU utilization, where 100%100\% corresponds to full utilization of one CPU core, and the second row reports memory usage. The curves represent average resource consumptions over time. The faded regions represent the corresponding standard deviations. Both distributed policies use substantially fewer dedicated resources and never exceed their allocations. Consumption changes little between 30×30× and 100×100×, indicating that approximately 30×30×–50×50× capacity sufficiently complements volunteered resources. The affinity-based policy is slightly more efficient than the uniform policy, likely because it considers node loads and exploits batching. Overall, distributed scheduling accommodates growing demand without proportional expansion of dedicated infrastructure. Subtask completion. Fig. 5 reports subtask completion rates on server and user nodes; the centralized policy is omitted from user-node plots. Under distributed policies, user-side rates are largely independent of server capacity. The uniform policy achieves approximately 0.750.75–0.950.95, improving with user capacity because stronger nodes better tolerate resource fluctuations. Affinity-based scheduling performs better at 100100 and 1,0001,000 users with 1×1× capacity, but degrades as user count grows, especially on weaker nodes. Larger batches may exceed expected resource demands, leading to the simultaneous abort of multiple subtasks due to node departures or resource-constraint violations. (a) Server 10×10× (b) Server 30×30× (c) Server 50×50× (d) Server 80×80× (e) Server 100×100× (f) Server 10×10× (g) Server 30×30× (h) Server 50×50× (i) Server 80×80× (j) Server 100×100× Figure 5: Subtask completion rates under different scheduler policies and server capacities. Completion rate denotes the fraction of scheduled subtasks completed successfully. The first and second rows show completion rates on the server and user nodes, respectively. Centralized server-side rates at 10×10× capacity fall from 0.90.9–1.01.0 for up to 1,0001,000 users to approximately 0.20.2 at 10,00010,000. At 30×30×, they remain near 1.01.0 through 1,0001,000 users but decline to approximately 0.60.6 at 10,00010,000; only 100×100× capacity approaches consistently complete execution. Distributed policies reach near-perfect server-side rates with considerably less capacity, particularly for 2×2× users, because volunteered resources reduce server overcommitment. Some redundant work remains because rule-based schedulers cannot anticipate availability and resource fluctuations. Overall, distribution reduces latency and dedicated capacity needs, showing the value of scheduler optimization. 3.3 Limitations and Future Work This paper focuses on the collaborative distributed inference architecture and its modeling framework, instead of a production-ready implementation. Accordingly, we assume homogeneous users with identical availability and resource distributions, which also allows us to omit explicit fairness mechanisms. A real deployment could condition user behavior and resources on temporal and user-specific context and incorporate contribution-aware request prioritization. We further use 1s1\,s time steps from the server’s perspective; irregular observations can be aligned with this representation through interpolation. Although the conceptual execution state grows quadratically with the user count, our implementation uses limited slots to bound the tasks maintained per node and scale linearly. Alternative request-centric representations could further reduce this space cost, making model scalability and decentralization promising research directions. The framework can also naturally extend to multiple dedicated servers, DAG-based workflows, and autoregressive workloads such as LLM inference. Finally, optimizing the scheduler and dedicated resource allocation formulated in Section 2.4, as well as addressing deployment-level security and privacy, remains future work. These promising extensions build directly on the proposed model, whose modular factorization allows individual assumptions and components to be refined without redesigning the overall framework. 4 Conclusion We proposed a collaborative distributed inference system that combines dedicated server infrastructure with resources contributed by users’ devices, enabling demand to scale without proportional growth in dedicated capacity. To capture the system’s stochastic and dynamic behavior, we developed a generative Markov model that supports simulation, analysis, and policy optimization. Our evaluation showed that distributed scheduling increasingly outperforms centralized inference as the user population grows, improving request completion and tail latency while substantially reducing dedicated resource consumption. These findings establish the feasibility of user-assisted inference and provide a foundation for future work on optimized scheduling, QoS-aware resource allocation, and extensions to richer workloads and deployment environments. Acknowledgements The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725 and also supported by MSCA’s TALENTS (101299722) project. References [1] D. Anderson and K. Burnham (2004) Model selection and multi-model inference. Second. NY: Springer-Verlag 63 (2020), p. 10. Cited by: §2.2. [2] F. Benevenuto et al. (2009) Characterizing user behavior in online social networks. In Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, IMC ’09, New York, NY, USA, p. 49–62. External Links: ISBN 9781605587714, Document Cited by: §2.2. [3] G. E. Box et al. (2015) Time series analysis: forecasting and control. John Wiley & Sons. Cited by: §2.2. [4] D. Che and W. Hou (2011) A novel “credit union“ model of cloud computing. In Digital Information and Communication Technology and Its Applications, H. Cherifi, J. M. Zain, and E. El-Qawasmeh (Eds.), Berlin, Heidelberg, p. 714–727. External Links: ISBN 978-3-642-21984-9 Cited by: §1. [5] M.E. Crovella and A. Bestavros (1997) Self-similarity in world wide web traffic: evidence and possible causes. IEEE/ACM Transactions on Networking 5 (6), p. 835–846. External Links: Document Cited by: §2.2. [6] A. Y. Ding et al. (2022) Roadmap for edge ai: a dagstuhl perspective. 52 (1), p. 28–33. External Links: ISSN 0146-4833, Document Cited by: §1. [7] P. K. Donta et al. (2023) Exploring the potential of distributed computing continuum systems. Computers 12 (10). External Links: ISSN 2073-431X, Document Cited by: §1. [8] S. Dustdar et al. (2023) On distributed computing continuum systems. IEEE Transactions on Knowledge and Data Engineering 35 (4), p. 4092–4105. External Links: Document Cited by: §1. [9] Y. Gao et al. (2026) Optimizing multi-dnn inference on mobile devices through heterogeneous processor co-execution. IEEE Transactions on Mobile Computing 25 (6), p. 7766–7781. Cited by: §1. [10] B. Javadi et al. (2011) Discovering statistical models of availability in large distributed systems: an empirical study of seti@home. IEEE Transactions on Parallel and Distributed Systems 22 (11), p. 1896–1903. External Links: Document Cited by: §2.2. [11] Y. Jin and Z. Yang (2025) Scalability optimization in cloud-based ai inference services: strategies for real-time load balancing and automated scaling. In Proceedings of the 2025 4th International Conference on Big Data, Information and Computer Network, BDICN ’25, New York, NY, USA, p. 266–270. External Links: ISBN 9798400712425, Document Cited by: §1. [12] A. Lapkovskis et al. (2026) Brief announcement: generative markov model for distributed computing systems. arXiv preprint arXiv:2606.03061. Cited by: §2.1, §2.4. [13] P. Li et al. (2025) Making ai less ’thirsty’. Commun. ACM 68 (7), p. 54–61. External Links: ISSN 0001-0782, Document Cited by: §1. [14] S. Lloyd (1982) Least squares quantization in pcm. IEEE Transactions on Information Theory 28 (2), p. 129–137. External Links: Document Cited by: §2.2. [15] P. Ma et al. (2024) Research allocation in mobile volunteer computing system: taxonomy, challenges and future work. Future Generation Computer Systems 154, p. 251–265. External Links: ISSN 0167-739X, Document Cited by: §1, §1. [16] M. F. A. MARZBAN et al. (2025) Inference data distribution criteria for artificial intelligence or machine learning model monitoring. Google Patents. Note: US Patent App. 18/544,782 Cited by: §1. [17] P. M. Mell and T. Grance (2011) SP 800-145. the nist definition of cloud computing. Technical report National Institute of Standards & Technology, Gaithersburg, MD, USA. Cited by: §1. [18] T. M. Mengistu and D. Che (2019) Survey and taxonomy of volunteer computing. ACM Comput. Surv. 52 (3). External Links: ISSN 0360-0300, Document Cited by: §1, §1, §1. [19] T. M. Mengistu et al. (2018) CuCloud: volunteer computing as a service (vcaas) system. In Cloud Computing – CLOUD 2018, M. Luo and L. Zhang (Eds.), Cham, p. 251–264. External Links: ISBN 978-3-319-94295-7 Cited by: §1, §1. [20] T. Meuser et al. (2024) Revisiting edge ai: opportunities and challenges. IEEE Internet Computing 28 (4), p. 49–59. External Links: Document Cited by: §1. [21] S. Sajadieh et al. (2026) The AI index 2026 annual report. Technical report AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA. Cited by: §1, §1. [22] M. M. Shawky (2025) A comparative study of interpolation methods for the development of ore distribution maps. Discover Geoscience 3 (1), p. 2. Cited by: §2.2. [23] J. D. Singer and J. B. Willett (2003) Applied longitudinal data analysis: modeling change and event occurrence. Oxford university press. Cited by: §2.2. [24] W. Wang et al. (2025) A survey of ai inference technologies for on-device systems. IEEE Internet of Things Journal 12 (24), p. 51927–51950. External Links: Document Cited by: §1. [25] I. Yeo and R. A. Johnson (2000) A new family of power transformations to improve normality or symmetry. Biometrika 87 (4), p. 954–959. External Links: ISSN 0006-3444 Cited by: §2.2. [26] G. Zhang et al. (2026) Task-aware collaborative inference and fine-grained dnn partitioning in mec networks. IEEE Transactions on Mobile Computing 25 (6), p. 8911–8927. External Links: Document Cited by: §1. [27] T. Zhang et al. (2026) DisHelis: optimizing deployment of disaggregated llms inference serving over heterogeneous environments via hierarchical max-flow. IEEE Transactions on Cognitive Communications and Networking 12 (), p. 5473–5488. External Links: Document Cited by: §1. [28] T. Zheng et al. (2026) Joint optimization of dynamic batching and adaptive partitioning for distributed llms inference in mobile edge computing. IEEE Transactions on Mobile Computing 25 (6), p. 8747–8763. External Links: Document Cited by: §1.