Paper deep dive
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks
Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias, Iakovos S. Venieris
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multi-path tail architecture, enabling a dynamic accuracy--throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6pp higher than on-device alternatives, across up to 40 concurrent devices.
Tags
Links
- Source: https://arxiv.org/abs/2609.14660v1
- Canonical: https://arxiv.org/abs/2609.14660v1
Trouble viewing inline? Open PDF directly →
Full Text
34,438 characters extracted from source content.
Expand or collapse full text
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks Sokratis Nikolaidis2 Stylianos I. Venieris3 Leonidas Malachias2 and Iakovos S. Venieris2 Affiliation: 2National Technical University of Athens, Athens, Greece, 3Samsung AI Center, Cambridge, UK Affiliation: Email: sokratisnikolaidis@mail.ntua.gr, s.venieris@samsung.com, malachiasnleonidas@gmail.com, venieris@cs.ece.ntua.gr Affiliation: Abstract Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multi-path tail architecture, enabling a dynamic accuracy–throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6p higher than on-device alternatives, across up to 40 concurrent devices. I Introduction The proliferation of mobile and Internet-of-Things (IoT) devices has driven a growing demand for deploying sophisticated deep neural networks (DNNs) in smart environments [1]. Conventionally, to alleviate the heavy computational demands of state-of-the-art DNNs, developers have resorted to cloud or edge offloading, pushing as much computation as possible to a remote server. However, the emergence of modern System-on-Chips (SoCs), equipped with progressively more powerful processors, has expanded the capabilities of embedded devices. This shift has led to the new distributed inference paradigm of onloading [2]. Contrary to offloading, onloading allows server-based DNN applications to deliberately push computation onto the edge devices in order to exploit their growing local compute. In the context of split computing, the server onloads the early stages of computation onto the device, reducing both the server’s processing burden and the dimensionality of the intermediate feature maps transmitted over the network. While existing split-computing approaches have demonstrated latency and communication benefits, the vast majority assume a static environment where a single device has exclusive access to the server. In realistic smart environments, a single edge server must provide inference assistance to multiple devices concurrently [3]. As the number of assisted devices grows, the aggregate arrival rate of inference requests inevitably surpasses the processing throughput of the server’s tail model. In a naive deployment, this contention leads to unboundedly growing queues, causing severe latency service-level objective (SLO) violations. Without a dynamic mechanism to manage the server’s load, the system is eventually forced into an indiscriminate fallback to on-device execution, negating the accuracy benefits of server-assisted inference. To address the limitations of static onloading, we propose MANE,11 1 Multi-Path Adaptive Network for Edge Onloading. a novel distributed inference framework designed specifically for multi-tenant edge servers. MANE overcomes rigid throughput constraints by transforming the server-side architecture into a multi-path network, enabling a dynamic trade-off between accuracy and speed. MANE onloads a highly compressed, path-agnostic head to the client devices. During inference, these devices transmit intermediate feature maps to the server, which dynamically routes them through varying depths of the pretrained backbone model based on instantaneous load. To manage this architecture, we introduce a server-side scheduler that continuously monitors request queues, scaling down to faster execution paths during traffic surges, and enforcing a targeted, equitable fallback to on-device execution when maximum capacity is exceeded. The main contributions of the paper are the following: • A novel split-computing architecture that pairs a highly compressed, path-agnostic on-device head with a multi-path server-side tail, introducing a dynamic accuracy–throughput trade-off to edge onloading. • A three-stage training methodology featuring a novel Joint Head Network Distillation loss that mathematically ensures that the onloaded head learns a universally compatible intermediate representation for all server paths. • A dynamic scheduler that employs hysteresis-based path switching and an equitable device-fallback policy to maximise system-wide accuracy while meeting latency SLOs. I Related Work Split Computing & Onloading. The distribution of DNN inference between mobile devices and edge servers has been extensively studied within the scope of split computing [4]. After Neurosurgeon [5] introduced the partitioning of DNN computation at layer granularity, a large body of work followed, introducing onloading [2] and tackling challenges including dynamic split-point selection [6, 7], model partitioning and resource allocation [8, 9, 10], and energy-aware inference and feature compression [11, 12, 13]. Knowledge Distillation for Edge Inference. Knowledge distillation (KD) [14] transfers knowledge from a large teacher model to a compact student by training on the teacher’s soft output distributions rather than hard labels alone. Tailored to split computing, Head Network Distillation (HND) [15] distills the on-device head of a bottleneck-injected DNN, minimising both on-device computation and the size of intermediate feature maps transmitted to the server while preserving accuracy. In MANE, we enhance HND in our three-stage training scheme. Subsequent work has further advanced bottleneck-based split computing through improved training strategies, compression benchmarking, and multi-task extensions [16, 17, 18]. Ladon [18] proposed a single shared encoder compatible with multiple task-specific heads, comprising a parallel to MANE’s path-agnostic head design. Multi-Device Scheduling. The majority of split computing works assume a single device with exclusive access to a server, an assumption that does not hold in real-world, multi-device smart environments. Several works have addressed multi-user inference scheduling at the edge through batching, early exiting, and latency-aware model switching [19, 20, 21, 22]. More related to MANE, MultiTASC [3] proposed a multi-tenancy-aware scheduler for cascade architectures that adaptively controls per-device forwarding decisions to maximise throughput while satisfying latency SLOs. MANE focuses on the split-computing setting with a richer accuracy–throughput trade-off enabled through a server-side, multi-path architecture. To the best of our knowledge, this paper is the first to jointly address multi-path split computing, bottleneck distillation, and multi-tenant scheduling in a unified framework. I System Model & Problem Formulation Fig. 1: Multi-path adaptive architecture for edge onloading. In this section, we define the system architecture of the Multi-Device Onloading setting. An edge server seeks to provide DNN inference assistance to a set of M IoT devices =d1,…,dMD=\d_1,…,d_M\, all performing the same task. Rather than processing raw inputs, the server onloads the early stages of computation onto the devices, delegating execution of a lightweight head model to each device and reserving its resources for the more computationally demanding tail. Single-Device Onloading. Let x∈x be a raw input sample on an IoT device. The server onloads a lightweight head model fh:→f_h:X onto the device, which executes it locally to produce a compressed intermediate representation z=fh(x)∈z=f_h(x) . This representation is transmitted to the server, where the tail model ft:→[0,1]Kf_t:Z→[0,1]^K completes the inference, producing a probability distribution over K classes, with predicted label y^=argmaxft(z) y= f_t(z). Formally: onloadfh,ft(x)=ft(fh(x))onload_f_h,f_t(x)=f_t(f_h(x)) (1) From the server’s perspective, onloading reduces the volume and dimensionality of incoming data — the server receives compressed representations z∈z rather than raw inputs x∈x — alleviating both communication overhead and the server’s preprocessing burden. From the device’s perspective, the head model is substantially lighter than the full DNN, and raw input data never leaves the device, preserving privacy. Multi-Device Onloading. To capture the more realistic setting of multi-device smart environments, we extend the single-device model to M devices, each running the same head model fhf_h and transmitting compressed representations concurrently. Let xm∈x_m denote the input sample of device dmd_m. The inference result for device dmd_m is: onloadfh,ft(xm)=ft(fh(xm)),∀dm∈onload_f_h,f_t(x_m)=f_t(f_h(x_m)), ∀\,d_m (2) All intermediate representations are placed in a shared request queue Q, from which the server draws batches of size ℬB and processes them using the shared tail model ftf_t at a throughput of μℬ _B (samples/sec). Each device dmd_m generates requests at an arrival rate λm _m, yielding an aggregate arrival rate Λ=∑m=1Mλm = _m=1^M _m. As the number of assisted devices grows, Λ increases and the server’s ability to process requests within a given latency budget degrades. Problem Formulation. The server’s objective is to maximise accuracy α across all assisted devices while satisfying a latency SLO L∗L^* for each processed sample. Formally: maxαs.t.lhead,m+ltransfer,m+ltail(ℬ)≤L∗,∀dm∈ split α\;s.t.\;&l_head,m+l_transfer,m+l_tail(B)≤ L^*,\;∀\,d_m split (3) where lhead,ml_head,m is the on-device head execution time for device dmd_m, ltransfer,ml_transfer,m is the network transmission time of the compressed representation from device dmd_m, and ltail(ℬ)l_tail(B) is the server-side tail inference time as a function of batch size ℬB, including queueing delay. While lhead,ml_head,m and ltransfer,ml_transfer,m remain approximately constant regardless of the number of assisted devices, ltail(ℬ)l_tail(B) grows with Λ as requests accumulate in Q and queueing delay increases. IV Proposed System The multi-device onloading setting demands a server-side architecture capable of dynamically adapting its accuracy–throughput operating point in response to fluctuating demand. To this end, we propose MANE, whose system architecture is illustrated in Fig. 1. Building upon the Head Network Distillation scheme comprising a custom lightweight head with a bottleneck, MANE enhances it by introducing multiple tail adapter paths on the server side, offering multiple options to balance the accuracy–latency trade-off during inference. To ensure stability, high accuracy, low latency and scalable system throughput, MANE uses a three-stage training process and a runtime scheduler that dynamically chooses the execution path and whether to selectively apply on-device execution. IV-A Training Scheme Fig. 2 depicts MANE’s carefully orchestrated training process for multi-path model splitting, comprising three stages. First Stage. The aim of the first stage is for the student model to learn to produce intermediate feature maps that satisfy the requirements of all paths simultaneously. Rather than training each path to independently mimic its respective teacher, we introduce the Joint Head Network Distillation (JHND) loss. JHND jointly optimises the shared head and all path adapters via a weighted combination of per-path HND losses [15]: ℒJHND=∑i=1Nwi⋅‖hsi−hti‖2L_JHND= _i=1^Nw_i·\|h_s^i-h_t^i\|^2 (4) where hsih_s^i and htih_t^i denote the feature maps produced by the student and the respective teacher for path i, and wiw_i is a scalar weight controlling each path’s contribution to the total loss. This joint optimisation encourages the shared head to learn a path-agnostic intermediate representation. Training utilises early stopping on the validation set. Second Stage. With the universally compatible representation established, the shared head is now frozen and the pretrained tail remains detached. Each path adapter is trained individually using standard HND loss against its respective teacher head: ℒHNDi=‖hsi−hti‖2L_HND^i=\|h_s^i-h_t^i\|^2 (5) By isolating each path and freezing the shared representation learned during Stage 1, this second stage allows each adapter to specialise towards its respective teacher signal without interfering with the learned weights of other paths. Third Stage. In the third stage, the shared head remains frozen and the pretrained tail is attached to the pipeline. Each path is then fine-tuned individually using a standard KD loss [14], combining KL divergence between the student and teacher output distributions with a cross-entropy term against the ground-truth labels, balanced by a hyperparameter b∈[0,1]b∈[0,1] and scaled by temperature T. Following this fine-tuning, each path reaches its peak accuracy, matching the performance that would be achieved if trained independently from the outset. Fig. 2: MANE’s three-stage training process. IV-B Scheduler To effectively utilise MANE’s multi-path capabilities, we design a server-side scheduler that dynamically adapts the accuracy–throughput trade-off by continuously monitoring system load and via two novel policies: path switching and device fallback. Upon initialisation, the scheduler calculates the largest batch size that can be processed by each path p within the latency SLO, defining this as the path’s capacity (CpC_p). At run time, the scheduler evaluates the system state by comparing the current request queue length (QLcurrentQL_current) against the capacity of the active path (CcurrentC_current). Path Switching. Path switching enables the server to dynamically adapt to fluctuating arrival rates. To prevent rapid oscillation between paths due to instantaneous load spikes, the scheduler employs a hysteresis mechanism by means of a growth counter GCGC and a drain counter DCDC. The GCGC counter tracks sustained queue growth. When the queue length exceeds the current path’s capacity and has grown since the last batch was processed, GCGC is incremented by one; otherwise it is reset to zero: GC=GC+1ifQLcurrent>CcurrentandQLcurrent>QLlast0otherwiseGC= casesGC+1&if QL_current>C_current\\ &and QL_current>QL_last\\ ~~0&otherwise cases (6) where QLcurrentQL_current is queue length at scheduling time, CcurrentC_current the active path’s capacity, and QLlastQL_last queue length at the previous batch’s time of processing. When GCGC≥ LGL, where GLGL is the growth limit, the scheduler switches to the next fastest path. Conversely, the drain counter DCDC tracks sustained queue draining. When the queue length falls below the capacity of the next more accurate path (CaccurateC_accurate) and has decreased since the last batch, DCDC is incremented; otherwise it is reset to zero: =+1ifcurrent<Caccurateandcurrent<last0otherwise DC= casesDC+1&if 10.22217ptQL_current<C_accurate\\ &and 10.22217ptQL_current<QL_last\\ ~~0&otherwise cases (7) where CaccurateC_accurate is the capacity of the next more accurate path. When DC≥DLDC≥ DL, where DLDL is the drain limit, the scheduler switches to the more accurate path to maximise accuracy. Device Fallback. Device fallback is triggered when server demand surpasses the processing capacity of even the fastest path, using the same growth counter logic as path switching with the additional condition that the current path must already be the fastest. When GCGC≥ LGL, a fraction ϕφ of devices — prioritised by sample submission volume to ensure fair resource allocation — is instructed to process their workloads locally using a lightweight fallback model, directly reducing the total arrival rate Λ and restoring the server’s ability to meet the latency SLO. Fallback devices are periodically rotated to maintain equitable quality of service under prolonged load, and are reintroduced to server-side onloading one at a time once the queue falls below the fastest path’s capacity and exhibits sustained draining, avoiding sudden load oscillations. Unlike path switching, fallback enforcement is subject to a minimum cooldown period TcooldownT_cooldown between successive engagements. V Evaluation V-A Experimental Setup TABLE I: Compression Analysis Model Path Param. Reduction Feature Map Reduction ResNet152 Accurate 213× 1213× 42.7×42.7× Balanced 1550×1550× 21.3×21.3× Fast 1922×1922× 21.3×21.3× ConvNeXt Small Accurate 102× 1102× 16×16× Balanced 1329×1329× 8× 18× Fast 1930×1930× 8× 18× vs. Raw Input (all paths) — 16×16× MANE’s multi-path model architecture was implemented using PyTorch 2.11.0. We target an edge server equipped with a 10GB NVIDIA RTX 3080 GPU and an AMD Ryzen 5 3600 3.6GHz CPU, for both training and deployment. For the client devices, we target Samsung S20 FE, a dated flagship smartphone to simulate a median-capability user device, with device-side models deployed using PyTorch Mobile targeting the device’s NPU. For device–server communication, we employ the AMQP protocol, following standard practice for IoT device messaging. We target 200-class image classification using the Tiny ImageNet dataset, comprising 100k samples with an 80%–10%–10% training–validation–test split. All deployment results are reported on the held-out test set. Hyperparameters. All stages use the AdamW optimiser with early stopping on the validation set with a patience of 3. The learning rate is set to 10−310^-3 for Stages 1 and 2, and reduced to 10−410^-4 for Stage 3. Stage-specific hyperparameters are set as follows: wi=1Nw_i= 1N for Stage 1’s JHND, b=0.9b=0.9 and T=4T=4 for Stage 3’s KD. The scheduler growth GLGL and drain limits DLDL are set to 3 and 5, respectively, and ϕφ to 0.2. Evaluation Protocol. We evaluate MANE on ResNet152 [23] and ConvNeXt Small [24] adaptations, targeting latency budgets ranging from 100ms to 200ms with a varying number of devices. A three-path structure was implemented for each backbone model. The initial split point was chosen to preserve full accuracy, while the two additional tail injection points were selected based on the accuracy–throughput curve of candidate injection points to realise an accurate–balanced–fast path scheme. For ResNet152, the balanced and fast injection points were placed 14 and 18 blocks after the initial split point, respectively, while for ConvNeXt Small at 12 and 18 blocks. Each device is assigned the same subset of 1000 images, randomly sampled from the test set using a fixed seed. Each experiment is repeated across three different seeds, reporting average, minimum, and maximum values. The sample order is shuffled before each run, and inter-arrival delays are sampled from a Poisson distribution with a mean of 20 ms to emulate realistic device usage patterns. Device inference latency measurements use a batch size of 1, averaged across 500 runs. The evaluation metrics are system throughput, average accuracy across devices, and SLO satisfaction rate, which is the percentage of samples processed within a latency target. TABLE I: Path Specifications Model Path Accuracy Accel. ResNet152 Accurate 83.98% 1.23×1.23× Balanced 80.79% 1.85×1.85× Fast 78.03% 2.11×2.11× ConvNeXt Small Accurate 89.39% 1.35×1.35× Balanced 85.43% 2.12×2.12× Fast 80.39% 2.43×2.43× Baselines. We compare MANE against two baselines: i) server-only execution, where samples are processed by the original pretrained model without any path switching or fallback, and i) on-device execution, where all inference is performed locally using MobileNetV2 [25]. We also use two ablated versions of MANE to showcase the importance of each component: 1) MANE-NPS, where we disable both path switching and fallback from the scheduler, allowing only the accurate path to be used, and 2) MANE-NF, where we enable path switching and remove fallback. V-B Model Evaluation Compression Analysis. Table I shows the compression achieved by MANE’s shared head relative to the original backbone models. The head model comprises approximately 12.6k parameters, representing a reduction of 213×213× and 102×102× over the accurate-path split point of ResNet152 and ConvNeXt Small, respectively. The intermediate feature maps produced by the head have a shape of [12×28×28][12× 28× 28], yielding a transmission cost reduction of ≈16×≈ 16× relative to the raw input and up to ≈42.7×≈ 42.7× relative to the naive split point of ResNet152’s fast path. As a result, MANE significantly reduces both the on-device compute cost and the communication overhead, independently of which server-side path is active. Per-Path Performance. Table I reports the accuracy and acceleration of each path over the pretrained backbone, using a batch size of 128. The three paths offer a consistent accuracy–throughput trade-off across both model families. The three-stage training scheme manages to preserve accuracy across all paths. Notably, the accurate path matches or slightly exceeds the pretrained backbone’s accuracy, demonstrating that the JHND-based training in Stage 1 successfully guides the shared head towards a representation that serves all paths without compromising accurate. Balanced and fast modestly trade accuracy for substantial speedup, with fast delivering up to 2.11×2.11× for ResNet152 and 2.43×2.43× for ConvNeXt Small. End-to-end Model Performance. Table I compares MANE against the original pretrained models in terms of accuracy and inference latency, measured with a batch size of 1. MANE’s accurate path matches or exceeds the accuracy of the original backbone in both cases, while also achieving lower latency. This is attributed to the bottleneck injection and path adapter design, which allows the tail to begin inference at a deeper point in the network, reducing the total computation performed on the server. The MANE shared head adds only 5 ms of on-device latency, which is substantially lower than the 25 ms required by the MobileNetV2 fallback model, demonstrating that the onloaded head induces minimal on-device overhead. TABLE I: End-to-end Model Performance Model Loc. Accuracy Latency MANE Shared Head Device — 15ms MobileNetV2 Device 73.13% 25ms ResNet152 Server 82.64% 15ms MANE ResNet152 Server 83.98% 13ms ConvNeXt Small Server 89.35% 13ms MANE ConvNeXt Small Server 89.39% 11ms V-C Full System Evaluation SLO Satisfaction Rate & Accuracy. Fig. 3 evaluates MANE against our baselines on SLO satisfaction rate and accuracy under a challenging latency target of 100 ms. With the exception of device-only execution, all baselines eventually fail as the number of devices increases. Server-only execution using the pretrained ConvNeXt Small model collapses to a satisfaction rate of 0% at 15 devices, with MANE-NPS and MANE-NF following at 20 and 30 devices, respectively. The difference in failure point correlates directly with the throughput of each pipeline: the original ConvNeXt Small has the lowest throughput, followed by the accurate path used in MANE-NPS, and finally the fast path which MANE-NF eventually settles on before failing. In contrast, MANE maintains a satisfaction rate of around 80% even when all other baselines have completely failed, showcasing its adaptability under demanding conditions. On-device execution achieves 100% satisfaction rate throughout, as each sample is processed in approximately 25 ms, but at a substantially lower accuracy. On accuracy, MANE and MANE-NF begin trading off accuracy to sustain the satisfaction rate at around 20 devices. At 30 devices, MANE continues to trade accuracy via the fallback policy, while MANE-NF has exhausted its ability to do so, having already switched to the fastest path. Even under the challenging 100 ms target with 40 devices, MANE achieves a mean accuracy of 78% with low inter-device deviation, a direct consequence of the democratic fallback rotation policy. Similar behaviour is observed in Fig. 4, where MANE uses ResNet152 as the server-side model. The baselines fail at 15, 20, and 35 devices, respectively, consistent with the ConvNeXt results. A more pronounced satisfaction rate drop is visible at intermediate device counts due to ResNet152’s lower throughput compared to ConvNeXt Small. Despite this, MANE maintains a satisfaction rate above 75% across all tested device counts up to 40. On accuracy, the same trade-off pattern is observed, with MANE trading more aggressively once the fast path is insufficient and fallback is engaged. Even in the most demanding scenario, MANE achieves an accuracy of 76%, substantially higher than on-device execution. Fig. 3: SLO Satisfaction Rate and Accuracy for ConvNeXt Small. Fig. 4: SLO Satisfaction Rate and Accuracy for ResNet152. System Throughput. Fig. 5 presents system throughput as the number of devices grows. Since each device contributes 1000 samples to the system, the ideal behaviour is a linear increase in throughput with the number of devices. Full server onloading and MANE-NPS plateau at around 20 devices, indicating the system has reached its maximum processing capacity. MANE-NF sustains linear throughput growth up to 30 devices before plateauing, demonstrating the benefit of the multi-path architecture in extending the system’s effective capacity. MANE is the only server-assisted method that continues to increase throughput beyond this point without plateauing, reflecting the combined effect of path switching and the fallback policy in avoiding server overload. The comparison with on-device execution is particularly instructive. Despite displaying linearly increasing throughput, on-device execution achieves substantially lower throughput values than server-assisted schemes at low device counts, as the server can process samples more efficiently by leveraging batched inference. This explains why even the more conservative baselines, i.e. server-only execution and MANE-NPS, outperform on-device execution in throughput when the device count is small. As device count grows enough that MANE begins reverting a significant portion to fallback, its throughput growth rate converges towards that of on-device execution, with the two curves becoming approximately parallel. Fig. 5: Throughput for ResNet152 and ConvNeXt Small. Relaxed Latency Target. Fig. 6 shows SLO satisfaction rates and accuracy for all methods under a relaxed latency target, where the scheduler can sustain a higher capacity per path and therefore trade off accuracy less aggressively. On the satisfaction rate side, server-only execution and MANE-NPS fail completely at 15 and 20 devices, respectively, consistent with the 100 ms experiment. MANE begins trading off accuracy at 25 devices and maintains satisfaction rate above 80% throughout, mirroring its behaviour under the stricter SLO. A subtle but notable difference is observed in MANE-NF, which retains satisfaction rate of approximately 20% at 30 devices rather than collapsing to 0% as in the 100 ms setting, reflecting the additional headroom due to the relaxed latency budget. The most significant difference between the two setups is visible on accuracy. MANE leverages the relaxed latency target to preserve accuracy above 80% across a wider range of device counts, compared to the 78% floor observed under the 100 ms target. In this setting, MANE also consistently achieves higher accuracy than MANE-NF across all device counts. VI Conclusion This paper presents MANE, a novel distributed inference framework that enables split-computing systems to operate effectively in demanding multi-device edge environments. By equipping the server with a multi-path tail architecture and a Joint Head Network Distillation scheme, we introduce a dynamic accuracy–throughput trade-off that existing static onloading approaches cannot provide. MANE continuously adapts the active inference path in response to server load, while its equitable fallback policy ensures that no single device monopolises server resources, sustaining high satisfaction rates and accuracy across a growing number of concurrent devices. MANE’s adaptability and scalability represent a step towards making split computing a viable and robust solution for the intelligent indoor environments of the future. Fig. 6: SLO Satisfaction Rate and Accuracy with 200ms time target. References [1] S. Laskaridis, S. I. Venieris, A. Kouris, R. Li, and N. D. Lane (2024) The Future of Consumer Edge-AI Computing. IEEE Pervasive Computing 23 (3). Cited by: §I. [2] M. Almeida, S. Laskaridis, S. I. Venieris, I. Leontiadis, and N. D. Lane (2022) DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device. ACM Trans. Embed. Comput. Syst.. Cited by: §I, §I. [3] S. Nikolaidis, S. I. Venieris, and I. S. Venieris (2023) MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge. In ISCC, Cited by: §I, §I. [4] Y. Matsubara, M. Levorato, and F. Restuccia (2022) Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges. ACM Comput. Surv.. Cited by: §I. [5] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang (2017) Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. In ASPLOS, Cited by: §I. [6] A. Bakhtiarnia, N. Milošević, Q. Zhang, D. Bajović, and A. Iosifidis (2023) Dynamic Split Computing for Efficient Deep EDGE Intelligence. In ICASSP, Cited by: §I. [7] L. Zhang, L. Chen, and J. Xu (2021) Autodidactic Neurosurgeon: Collaborative Deep Inference for Mobile Edge Intelligence via Online Learning. In W, Cited by: §I. [8] J. Yan, S. Bi, and Y. A. Zhang (2022) Optimal model placement and online model splitting for device-edge co-inference. IEEE Transactions on Wireless Communications. Cited by: §I. [9] J. Huang, C. Samplawski, D. Ganesan, B. Marlin, and H. Kwon (2020) CLIO: Enabling Automatic Compilation of Deep Learning Pipelines across IoT and Cloud. In MobiCom, Cited by: §I. [10] X. Hou, Y. Guan, T. Han, and N. Zhang (2022) DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices. In IPDPS, Cited by: §I. [11] Y. G. Kim and C. Wu (2020) AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning. In MICRO, Cited by: §I. [12] M. Sung, S. Im, D. Bang, I. Kim, S. Yun, and J. Kang (2025) Why should the server do it all?: a scalable, versatile, and model-agnostic framework for server-light DNN inference over massively distributed clients via training-free intermediate feature compression. arXiv. Cited by: §I. [13] D. Cha, J. Lee, D. Jung, and S. Pack (2025) Fast and fair split computing for accelerating deep neural network (DNN) inference. ICT Express. Cited by: §I. [14] G. Hinton, O. Vinyals, and J. Dean (2014) Distilling the Knowledge in a Neural Network. In NeurIPS, Cited by: §I, §IV-A. [15] Y. Matsubara, D. Callegaro, S. Baidya, M. Levorato, and S. Singh (2020) Head Network Distillation: splitting distilled deep neural networks for resource-constrained edge computing systems. IEEE Access. Cited by: §I, §IV-A. [16] Y. Matsubara, D. Callegaro, S. Singh, M. Levorato, and F. Restuccia (2022) BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing. In WoWMoM, Cited by: §I. [17] Y. Matsubara, R. Yang, M. Levorato, and S. Mandt (2023) SC2 benchmark: supervised compression for split computing. Transactions on Machine Learning Research. Cited by: §I. [18] Y. Matsubara, M. Mendula, and M. Levorato (2025) A Multi-Task Supervised Compression Model for Split Computing. In WACV, Cited by: §I. [19] W. Shi, S. Zhou, Z. Niu, M. Jiang, and L. Geng (2023) Multiuser Co-Inference With Batch Processing Capable Edge Server. IEEE Transactions on Wireless Communications. Cited by: §I. [20] Z. Liu, Q. Lan, and K. Huang (2023) Resource Allocation for Multiuser Edge Inference With Batching and Early Exiting. IEEE Journal on Selected Areas in Communications. Cited by: §I. [21] A. Kouris, S. I. Venieris, S. Laskaridis, and N. D. Lane (2023) Fluid Batching: Exit-Aware Preemptive Serving of Early-Exit Neural Networks on Edge NPUs. In ICCAD, Cited by: §I. [22] V. Paramanayakam, A. Karatzas, D. Stamoulis, and I. Anagnostopoulos (2025) Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers. IEEE Transactions on Computers. Cited by: §I. [23] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep Residual Learning for Image Recognition. In CVPR, Cited by: §V-A. [24] Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie (2022) A ConvNet for the 2020s. In CVPR, Cited by: §V-A. [25] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen (2018) MobileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR, Cited by: §V-A.