Paper deep dive
Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones
Moritz Nottebaum, Matteo Dunnhofer, Christian Micheloni
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 1:56:56 AM
Summary
The paper introduces LowFormer, a novel vision backbone family designed to optimize hardware efficiency by prioritizing actual execution time (latency) over traditional MAC count metrics. The authors identify that MACs are an unreliable predictor of performance on edge devices due to memory access costs and parallelism. LowFormer incorporates 'Lowtention', a lightweight alternative to Multi-Head Self-Attention, and provides specialized edge GPU variants, demonstrating superior speed-accuracy trade-offs across various hardware platforms and downstream tasks.
Entities (5)
Relation Signals (3)
LowFormer → includes → Lowtention
confidence 100% · LowFormer features a streamlined macro and micro design that includes Lowtention
LowFormer → evaluatedon → ImageNet-1K
confidence 95% · The backbone family includes five models (B0, B1, B1.5, B2, B3), achieving top-1 accuracy on the ImageNet-1K dataset
LowFormer → optimizedfor → Nvidia Jetson TX2
confidence 90% · LowFormer models consistently achieve remarkable speed-ups across various hardware platforms
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision backbone networks play a central role in modern computer vision. Enhancing their efficiency directly benefits a wide range of downstream applications. To measure efficiency, many publications rely on MACs (Multiply Accumulate operations) as a predictor of execution time. In this paper, we experimentally demonstrate the shortcomings of such a metric, especially in the context of edge devices. By contrasting the MAC count and execution time of common architectural design elements, we identify key factors for efficient execution and provide insights to optimize backbone design. Based on these insights, we present LowFormer, a novel vision backbone family. LowFormer features a streamlined macro and micro design that includes Lowtention, a lightweight alternative to Multi-Head Self-Attention. Lowtention not only proves more efficient, but also enables superior results on ImageNet. Additionally, we present an edge GPU version of LowFormer, that can further improve upon its baseline's speed on edge GPU and desktop GPU. We demonstrate LowFormer's wide applicability by evaluating it on smaller image classification datasets, as well as adapting it to several downstream tasks, such as object detection, semantic segmentation, image retrieval, and visual object tracking. LowFormer models consistently achieve remarkable speed-ups across various hardware platforms compared to recent state-of-the-art backbones. Code and models are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.26551v1
- Canonical: https://arxiv.org/abs/2603.26551v1
Trouble viewing inline? Open PDF directly →
Full Text
103,515 characters extracted from source content.
Expand or collapse full text
Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones Moritz Nottebaum 1* , Matteo Dunnhofer 1,2 and Christian Micheloni 1 1 Machine Learning and Perception Lab, University of Udine Via delle Scienze 206, Udine, 33100, UD, Italy. 2 Centre for Vision Research, York University 4700 Keele St, Toronto, M3J 1P3, ON, Canada. *Corresponding author(s). E-mail(s): nottebaum.moritz@spes.uniud.it; Contributing authors: matteo.dunnhofer@uniud.it; christian.micheloni@uniud.it; Abstract Vision backbone networks play a central role in modern computer vision. Enhancing their efficiency directly ben- efits a wide range of downstream applications. To measure efficiency, many publications rely on MACs (Multiply Accumulate operations) as a predictor of execution time. In this paper, we experimentally demonstrate the short- comings of such a metric, especially in the context of edge devices. By contrasting the MAC count and execution time of common architectural design elements, we identify key factors for efficient execution and provide insights to optimize backbone design. Based on these insights, we present LowFormer, a novel vision backbone family. LowFormer features a streamlined macro and micro design that includes Lowtention, a lightweight alternative to Multi-Head Self-Attention. Lowtention not only proves more efficient, but also enables superior results on Ima- geNet. Additionally, we present an edge GPU version of LowFormer, that can further improve upon its baseline’s speed on edge GPU and desktop GPU. We demonstrate the LowFormer’s wide applicability by evaluating it on smaller image classification datasets, as well as adapting it to several downstream tasks, such as object detec- tion, semantic segmentation, image retrieval, and visual object tracking. LowFormer models consistently achieve remarkable speed-ups across various hardware platforms compared to recent state-of-the-art backbones. Code and models are available at https://github.com/altair199797/LowFormer/blob/main/Beyond_MACs.md. Keywords: vision backbones, efficient attention, hardware efficiency, edge devices 1 Introduction In many computer vision applications, it is critical to achieve accurate predictions in the shortest time possible. This is important in real-time domains such as robotics (Wen et al, 2024; Dunnhofer et al, 2021; Matthies et al, 2007), autonomous driving (Gopalkr- ishnan et al, 2024; Jiang et al, 2023), video surveil- lance (Suzuki and Aoki, 2024; Upmanyu et al, 2009), and user assistance (Mahendran et al, 2021; Tan et al, 2023; Leo et al, 2017), and especially when these sys- tems have to be deployed on mobile and edge devices (Vasu et al, 2023b; Ganesh et al, 2022). Nowadays, vision backbone networks are critical components of such computer vision systems. They are used to generate representations that support a wide range of high-level tasks for image (Carion et al, 2020; Kirillov et al, 2023; Liu et al, 2025; Ma et al, 2021) and video understanding (Yan et al, 2021; 1 arXiv:2603.26551v1 [cs.CV] 27 Mar 2026 Fig. 1 Comparison of hardware efficiency of different vision back- bone architecture families on the Nvidia Jetson TX2. Models in the top-left offer the best hardware efficiency on the Jetson TX2. Both axes are in logarithmic scale. LowFormer base models (B0-B3) out- perform all architectures in hardware efficiency, with edge variants (E1-E3) further enhancing efficiency Dunnhofer et al, 2023; Kong and Fu, 2022). Improv- ing the efficiency of vision backbones is, therefore, a crucial step towards enhancing the running time of many computer vision pipelines. Since the introduction of deep convolutional net- works (LeCun et al, 1989), vision backbones have evolved to balance accuracy and efficiency. Recent architectures (Vasu et al, 2023a; Ma et al, 2024; Zhu et al, 2023; Chen et al, 2022a) combine convolutional (Sandler et al, 2018; He et al, 2016) and attention lay- ers (Bolya et al, 2022; Vaswani et al, 2017; Wang et al, 2020): convolutions extract local image features, while attention captures global relationships by aggre- gating information across the image. To develop com- putationally efficient deep learning models, includ- ing vision backbones, researchers commonly count and minimize Multiply and Accumulate operations (MACs) (Zhu et al, 2023). Simply put, this metric counts the number of multiplications and additions performed by a neural network to compute the out- put from input data. In other words, MACs can be viewed as a measure of the “tasks” a model must per- form to produce an output. Generally, the fewer tasks required, the faster and more efficient the model will be (Tan and Le, 2019). But research has also shown a strong correlation between a model’s increased MAC count and its accuracy, with networks having higher MAC counts achieving better prediction accu- racy (Tan and Le, 2019; Zhai et al, 2022). In light of this evidence, the research community has increas- ingly focused on developing backbone architectures that maximize accuracy and minimize the number of MACs (Zhu et al, 2023; Chen et al, 2022a; Tan and Le, 2019). By doing so, it is claimed that the accuracy-speed trade-off is optimized. We argue that MAC count is not always the best measure of backbone efficiency, and the reduction of MACs does not necessarily translate to a backbone that achieves reduced execution time. This dispro- portion stems from factors like memory access costs and the degree of parallelism (Dehghani et al, 2021; Liu et al, 2023; Vasu et al, 2023b; Ma et al, 2018a). The former identifies the delay caused by an opera- tion that must wait for the retrieval of its operands from the memory, leading to idle periods that hin- der overall execution time (Chen et al, 2023). The latter refers to the parallel execution of core oper- ations like multiplications and additions on modern hardware. In these settings, the number of MACs still counts all the tasks required to produce the output, but fails to account for time saved by performing multiple operations simultaneously. As a result, models vary in their hardware effi- ciency, leading to differences in execution time for the same amount of MAC operations. In Figure 1, we analyze the hardware efficiency of different vision backbone families, by relating MAC count and actual execution time – referred to as latency – on an Nvidia Jetson TX2. The figure depicts significant differences in hardware efficiency between the compared vision backbone architecture families. The MAC count of models within an architecture family (e.g. Low- Former) correlates well with execution time (TX2 latency), meaning they feature a similar hardware efficiency. However, the hardware efficiency differs significantly between the architecture families. The EfficientMod (Ma et al, 2024) models (xxs, xs and s) for example have a similar latency as LowFormer- B1.5, B2 and B3, but a fraction of their MAC oper- ations. This emphasizes the importance of a sound architecture design, as a measure to improve hardware efficiency. Determining the extent to which a model’s archi- tecture influences hardware efficiency, due to memory access cost and level of parallelism, is complex and heavily dependent on hardware-specific implementa- tion details (Qin et al, 2025). We believe that latency is a more effective metric for evaluating vision backbone efficiency, as model efficiency cannot be determined by MAC count alone but must be tested across various execution devices. Many recent publications have contributed to improve the understanding of the relationship between 2 Fig. 2 Comparison of Nvidia Jetson TX2 latency and top-1 accu- racy for state-of-the-art vision backbones with LowFormer. Models in the top-left offer the best speed-accuracy trade-off. LowFormer consistently achieves lower latency for similar accuracy. Its edge variants (E1/E2/E3) further enhance this trade-off over the base models (B0-B3) MACs and execution time on real devices (Vasu et al, 2023b; Chen et al, 2023; Ma et al, 2018b; Wang et al, 2024), highlighting architectural configurations that deteriorate model speed. In this paper, we add to this knowledge by pointing out additional factors, like: the operating resolution for convolutions; the short- comings of the frequently applied mobile inverted bottleneck block (Sandler et al, 2018); and the con- siderable inefficiency of Multi-Head Self-Attention (MHSA) (Vaswani et al, 2017). All of these factors can slow down model execution, however in what magnitude depends on the hardware device, where- fore we exploit a variety of different devices for a new hardware efficiency study. We rely on the outcomes of such a study to develop LowFormer, a new class of vision backbone networks that mitigates the impact of the aforemen- tioned factors. By minimizing execution time instead of MAC count, LowFormer is able to achieve a new position in the state-of-the-art speed-accuracy trade- off (see Figure 2), exceeding previous approaches on a variety of hardware devices. A key compo- nent in LowFormer’s architecture is Lowtention, a new and lightweight adaptation of MHSA (Vaswani et al, 2017). LowFormer features a simple micro and macro design, enabling scalability from low model size (LowFormer-B0) to higher (LowFormer-B3). The backbone family includes five models (B0, B1, B1.5, B2, B3), achieving top-1 accuracy on the ImageNet- 1K (Deng et al, 2009) dataset ranging from 78.4% to 83.64%. We further extend the architecture family by three models, namely LowFormer-E1/E2/E3, which are specifically targeted for edge GPU devices and are derivations of the original LowFormer design. To confirm the applicability of LowFormer backbones in downstream tasks, we evaluated LowFormer’s trans- fer learning capabilities on several image classifica- tion datasets. Furthermore, we integrated our back- bones in object detection and semantic segmentation frameworks. Additionally, we utilize the embeddings of LowFormer backbones for image retrieval, and present a LowFormer-based visual object tracking architecture. In all of these applications, LowFormer contributes significantly to improve hardware effi- ciency while maintaining or even increasing accuracy. The contributions of this paper can be summarized as follows: • We carry out a new exhaustive hardware effi- ciency analysis of key design elements in vision backbone architectures. We show how the MAC count of those elements translates to execution time on several devices, and highlight their dif- ferences for overall model efficiency. • We present LowFormer, a new family of vision backbones that features a hardware-efficient macro design and a new lightweight attention operation. These backbone models are faster in terms of latency and throughput compared to models with similar accuracy. • We further propose three edge GPU variants of LowFormer, that improve upon the base models in terms of edge GPU efficiency. • We show that LowFormer models generalize well to several downstream tasks, such as image classification, object detection, semantic seg- mentation, image retrieval, and single object tracking. 2 Related Work 2.1 Hardware-Efficient Model Design Achieving the highest accuracy at all computational cost has long since ceased to be the only goal in vision backbone networks (Tan and Le, 2019). An ever increasing share of research focuses on developing the most efficient architecture, subsequently achiev- ing the best accuracy-speed trade-off (Chen et al, 2022b; Munir et al, 2023; Howard et al, 2019). Earlier approaches equated speed with the minimal amount of MACs (Tang et al, 2022; Han et al, 2020; Chen et al, 2022a), while more recent research increasingly aims for models that reduce latency or throughput – 3 i.e. number of images processed in a second – on sev- eral different types of hardware such as desktop GPU and CPU (Ma et al, 2018b; Yun and Ro, 2024; Cai et al, 2023), mobile NPU and GPU (Vasu et al, 2023b), or microcontroller CPU (Paissan et al, 2022). Within these approaches, the cost of memory access and the degree of hardware parallelism have become impor- tant factors for efficient model design, as they can have a significant impact on model speed (Chen et al, 2023; Qin et al, 2025). The design of EfficientViT (Liu et al, 2023) demonstrates that MHSA intro- duces a higher memory access cost compared to the Feed-Forward Network (FFN) within the transformer block. Therefore, this architecture proposes increasing the proportion of FFN operations relative to MHSA, resulting in improved efficiency without compromis- ing accuracy. MobileOne (Vasu et al, 2023b), on the other hand, is based on an analysis of how activa- tion functions and multi-branch architectures impact latency on mobile devices. ShuffleNetV2 (Ma et al, 2018b) and FasterNet (Chen et al, 2023) were pro- posed on the observation that grouped convolutions are executed inefficiently on GPUs due to their high memory access costs (Chen et al, 2023). In this paper, we draw inspiration from these insights. However, rather than ungrouping a portion of the convolutions (Ma et al, 2018b) or introduc- ing a new micro design (Chen et al, 2023), we study the impact of fusing depthwise and pointwise convo- lutions on execution time. Additionally, we examine how the operating resolution of convolutional lay- ers impacts latency, and further explore strategies to effectively mitigate the significant efficiency drop caused by increasing input resolution for MHSA. To the best of our knowledge, we are the first to assemble a diverse set of execution devices to per- form these efficiency experiments and analyze how different hardware platforms compare. This paper extends (Nottebaum et al, 2025), where the LowFormer architecture was initially presented. In this version, we provide additional contributions, tai- lored to the domain of edge computing. We expand the execution time analysis to cover several edge devices and examine the efficiency of MHSA for increased input resolution, exploring ways to improve it. We also consistently compare the efficiency of LowFormer on edge devices with the top competing models, across most benchmarks. We further extend the ablation study to provide a stronger rationale for our design choices. Additionally, we propose three edge GPU variants of LowFormer and demonstrate empirically their viability. We evaluate LowFormer on several new downstream tasks, including image classification and image retrieval. Lastly, we present LowFormer-Track, which improves performance of the SMAT (Gopal and Amer, 2024) single object tracking (SOT) framework by integrating LowFormer’s design principles. 2.2 Convolutions, Attention and MLP in Architecture Design Most modern backbone architectures consist entirely or partially of three main building blocks: convolu- tions, attention mechanisms, and multi-layer percep- trons (MLPs) (Yun and Ro, 2024; Vasu et al, 2023a). Some approaches, called hybrid models, combine all three in their design (Liu et al, 2023; Qin et al, 2025; Yun and Ro, 2024; Vasu et al, 2023a), some relieve of the MLP (Cai et al, 2023), while others solely rely on convolutions (Wang et al, 2024; Vasu et al, 2023b). The latter focus towards efficient mobile execution, where convolutions have been shown to achieve supe- rior latency results (Qin et al, 2025). The work of (Brock et al, 2021) demonstrates that purely convolu- tional backbones can be on par with hybrid models in terms of accuracy, however (Dai et al, 2021) exper- imentally show that the attention operation provides higher model capacity if incorporated in an archi- tecture. On the other side, convolutions exhibit an improved generalization ability compared to atten- tion modules, wherefore a combination of both on a macro design level is beneficial (Dai et al, 2021). Other works went further by joining convolutions and attention operations on a micro design level (Wu et al, 2021), relieving of the need for hand-crafted positional encoding as a result. In the design of the LowFormer architecture, we integrate all three building blocks (attention, convo- lution, MLP) in a straightforward manner, resulting in a robust and versatile architecture. Unlike other approaches that depend on neural architecture search (Qin et al, 2025; Tan and Le, 2019) resulting in irreg- ular macro designs, enforce a fixed micro design (Liu et al, 2023), or entirely exclude MLPs and/or atten- tion mechanisms to improve performance (Vasu et al, 2023b; Wang et al, 2024; Cai et al, 2023), LowFormer takes a more flexible approach. Its architecture allows the removal of any building block without compro- mising its core design principles. In Section 4.4, we present a variant of the original LowFormer architec- ture, where we remove a portion of said components to further boost efficiency on edge GPU devices. 4 2.3 Efficient Attention The landscape of attention mechanisms is vast, with many alternatives proposed to replace MHSA (Vaswani et al, 2017). A lot of effort has been spent (Bolya et al, 2022; Cai et al, 2023; Tu et al, 2022) to reduce its quadratic complexity by variations of lin- ear attention (Wang et al, 2020). On the other side, the work (Yu et al, 2022) shows that attention can be replaced by a simple pooling operation. Subsequent research (Li et al, 2022) takes that idea further and uses the efficient pooling operation for the first three backbone stages and the traditional attention for the last two stages (Li et al, 2022). Other works (Wu et al, 2021; Wang et al, 2021; Fan et al, 2021) downsample the key and value vectors before the attention opera- tion, either with convolutions or pooling. Pooling is also used to downsample all query, key, and value vec- tors in order to make attention completely operate on a lower resolution (Si et al, 2022). In contrast to previous works, we harness the learning capability of convolutions to downscale the resolution of all input vectors for the Scaled Dot- Product Attention (SDA), effectively serving as condi- tional position embeddings (Chu et al, 2021). Unlike others, we further reduce the channel dimension before the SDA. Both reductions – of resolution and channel dimension – have a significant effect on effi- ciency but a minimal effect on accuracy. 3 Optimizing Vision Backbone Design by Contrasting MACs and Latency To create efficient vision backbone architectures, it is not sufficient to assess them only by the amount of their MAC operations (Vasu et al, 2023b; Chen et al, 2023). Hardware efficiency needs to be taken into account, as it is a crucial factor for the execu- tion time of a model (Qin et al, 2025). For example, a model can be considered more hardware efficient than a compared model, when it executes the same amount of MAC operations in less time. Modern backbone architectures widely utilize convolutions as a core component (Vasu et al, 2023a; Tan and Le, 2019; Cai et al, 2023; Zhu et al, 2023), yet their impact on hardware efficiency remains largely unexamined. Therefore, we will investigate the hard- ware efficiency of convolutions in different scenarios. We will show that grouping (e.g. depthwise convo- lutions) and operating resolution of convolutions can have a substantial effect on the hardware efficiency. We will further analyze under which configurations (resolution and channel dimension) it is beneficial to replace the mobile inverted bottleneck (MBConv) block (Sandler et al, 2018), which is a popular com- ponent of many backbone architectures (Tan and Le, 2019; Cai et al, 2023), with the fused MBConv (Gupta and Tan, 2019). At last, we analyze the efficiency of different adaptations of the original MHSA (Vaswani et al, 2017) under different input resolutions. To quantify hardware efficiency, we measured latency and image throughput on a diverse set of computing devices, including: a server machine with an Nvidia A40 GPU for GPU throughput; a desk- top machine with a Nvidia Titan RTX GPU for GPU latency; a GPU-accelerated embedding device Nvidia Jetson TX2; an iPhone 13 smartphone with an Apple A15 Bionic; and a Raspberry Pi5 with an ARM CPU. Following previous practices (Yun and Ro, 2024; Cai et al, 2023), we run a model with a batch size of 200 for GPU throughput, while to measure latency we use a batch size of 1. We always use the median time of all iterations for latency and also to calculate through- put. As input images, we generate random tensors beforehand. 3.1 Depthwise Convolutions have low Hardware Efficiency Previous research demonstrated that when optimizing backbone architectures for mobile-friendly design and efficiency, depthwise convolutions serve as a promi- nent alternative to standard convolutions (Howard et al, 2019; Tan and Le, 2019; Qin et al, 2025). Grouped Convolutions. Depthwise convolutions group their channels for computation. They divide them into as many groups as input channels. As a result, each feature map (channel) is processed inde- pendently by the depthwise convolution. Therefore each output feature map is only computed by pro- cessing its corresponding input feature map. In con- trast, standard convolutions (Ma et al, 2018b) are ungrouped, meaning all input channels contribute to every output channel (groups = 1) during the convolu- tional operation. Motivation. Evaluating models solely based on their MACs often encourages incorporating as many 5 Table 1 Contrasting MAC count and execution time of depthwise convolutions and ungrouped convolutions Model Channel Depthwise MACsGPU Throughput↑Latency(ms)↓ [C 0 , C 1 , C 2 , C 3 , C 4 ](M)(images/s)TX2GPUMobileARM CPU #1[15, 30, 60, 120, 240]✗46312,7221.830.780.6312.58 #2[30, 60, 120, 240, 480]✓4210,5263.550.240.5317.71 #3[30, 50, 100, 160, 160]✗9567,1422.900.780.6424.29 #4[60,120,240,480,480]✓825,4226.160.390.6736.59 #5[30, 60, 150, 240, 240]✗17105,3504.270.991.0048.19 #6[60, 180, 360, 720, 720]✓1044,2448.030.430.8647.29 The six toy models (#1-6) are divided into three groups by similar throughput. Each model differs in the number channels (see second column), if its convolutions are depthwise or not (see third column) and the number of layers in each stage. The table shows, that ungrouped convolutions that have a higher MAC count (more than 10× the amount) can still be similarly fast or faster as depthwise convolutions. Bold entries refer to the best value in each group and column depthwise convolutions as possible into the architec- ture, without considering their actual execution time speed-up. The efficiency of depthwise convolutions in terms of MACs does not always translate to equivalent gains in execution time. To illustrate the discrepancy between theoretical MAC efficiency and practical exe- cution efficiency, we conduct an experiment using a simplified toy architecture. Setting. In Table 1, we examine the effect of using depthwise convolutions instead of ungrouped ones on throughput and latency. We created three models with only depthwise convolutions (#2,#4,#6) and three with only ungrouped ones (#1,#3,#5). Each model features five stages, mimicking common backbone architec- ture design. Consequently, with an input of resolution of 224×224, the output is of resolution 7×7. For the depthwise models, we doubled the amount of layers for each stage and increased the channel dimension to close up on the ungrouped models in terms of MACs. The depthwise models still have no more than a 10th of the MACs of the ungrouped convolu- tional models. Matching MACs – e.g. by repeating the depthwise convolutions 20 additional times – would result in an unusually high number of layers, deviating significantly from typical architectural designs. Results. Even though model #1 and #2 in Table 1 have a similar GPU throughput, their latency dif- fers greatly. On the Jetson TX2 and ARM CPU the ungrouped convolutions are considerably faster in most scenarios, while on desktop GPU the oppo- site is true. This is due to higher memory bandwidth on the desktop GPU, which is particularly important for efficient execution of depthwise convolutions (Qin et al, 2025). This relationships also mostly holds true for the other models (#3-#6). Regarding mobile exe- cution, the difference in latency is not significant. Even in the best-case scenario, depthwise convolu- tions require at least three times the processing time for the same number of MAC operations as their ungrouped counterparts. In the worst scenario (#5 and #6), the ungrouped convolutions execute MACs 34× faster on the Jetson TX2, highlighting the substantial hardware inefficiency of depthwise convolutions. Conclusion. In summary, ungrouped convolutions are more hardware efficient than depthwise convolu- tions, processing the same number of MACs in less time. However, the extent of this difference varies depending on the device. 3.2 Fusing the MBConv Block can speed up Models The mobile inverted bottleneck block (MBConv) (Sandler et al, 2018) has a successful history in effi- cient backbones (Tan and Le, 2019; Sandler et al, 2018; Howard et al, 2019) and is still used within many new architectures (Cai et al, 2023; Dai et al, 2021; Tan and Le, 2021). It consists of two pointwise convolutions (PWConv) and a depthwise (DWConv) in between (see Figure 4 for a depiction of it). The PWConvs increase and decrease the channel dimen- sion by the attributed expansion factor. An alternative to the MBConv block is the fused MBConv block (Gupta and Tan, 2019). The latter merges the first PWConv with the DWConv into an ungrouped convo- lution, thus removing any depthwise convolution. Due to the depthwise convolution, the original MBConv block usually has a lower amount of MACs than the fused one, although it features one layer more. Motivation. In Section 3.1, we showed that depth- wise convolutions are hardware inefficient. Here, we extend that experiment to obtain insights into efficient backbone macro-design. We measure under which operating resolution and channel dimension the fused 6 a)b) c)d) Fig. 3The left figures a) and c) depict the average execution time (a) and latency (c) of the fused mobile inverted bottleneck (MBConv) relative to the unfused one, while the right figures b) and d) depict the relative amount of MACs. The blue areas in the left figures a) and c) correspond to configurations (number of channels and resolution) where fused MBConv is faster, while red corre- sponds to the opposite. For the figures b) and d), the red areas correspond to configurations where the fused MBConv has a higher amount of MACs. Bold numbers refer to entries with a particularly unequal ratio. Even though the fused MBConv always has more MACs, it is faster for many configurations MBConv block (Conv+PWConv) is faster than the original MBConv (PWConv+DWConv+PWConv), allowing us to determine the optimal choice for differ- ent parts of the architecture. Setting. For this experiment, we measure latency on the Nvidia Jetson TX2 and average execution time on a Nvidia A40 (see Figure 3). To calculate the average execution time, we run 100 iterations with a batch size of 200, take the median execution time across these iterations, and divide it by 200. This is the inverse of the throughput metric and ensures consistency in the presentation of the sub-figures in Figure 3. We apply an expansion factor of 4 for the fused and original MBConv in Figure 3. For this, we cre- ated toy models that just consist out of the same layer repeated after another. Relative execution time. In Figure 3, each metric is a relative metric, meaning it is always the value of the fused MBConv divided by the unfused one. For exam- ple in a) of Figure 3, we depict the relative average execution time, meaning the average execution time of the fused MBConv divided by the unfused one. In b) and d), we depict their relative MAC count (fused divided by unfused) and in c) their relative latency on the TX2. We apply these metrics for various res- olutions and input channel dimensions. We omit the scenarios where channel dimension range from 16 to 64 and resolutions from 7 to 28, as GPU utilization is too low for the results to have significance. Results. It can be noticed that resolution and chan- nel dimension have a big influence on the relative latency c) of Figure 3 and average execution time a). Even though the fused MBConv always has more MACs –values over one in right part b) and d) –, it is faster in many scenarios – i.e., values smaller than one in a) and c) – and is always more hardware effi- cient, i.e. the relative latency/average execution time is smaller than relative MAC count. Only for an high number of channels (>=256) and an high operat- ing resolution (>=56), the fused MBConv presents slower execution. For channel dimension 512, and res- olutions 112 and 256, the fused MBConv is faster than the unfused one. However, for resolution 224, it is the other way around. This inconsistency occurs because the computational load and memory requirements for these configurations exceed the Jetson TX2’s capacity, leading to unpredictable behavior. Conclusion. The efficiency of the fused MBConv heavily depends on the channel dimension and to a lesser extent on the operating resolution. Conse- quently, it is advantageous to apply the fused MBConv in the early stages of an architecture, where the chan- nel dimension is typically low (<256). In later stages that feature a higher channel dimension, the original MBConv proves more efficient. 7 Table 2 Experiment on the impact of resolution on hardware efficiency Scenario ResolutionChannelsRelativeRelative Throughput↑Relative Latency (ms)↓ (pixel)(#)MACsGPU (images/s)TX2MobileARM CPU #1 224241.00.31.711.261.42 281961.03.30.580.790.70 #2 224481.00.51.601.221.16 112961.01.90.630.820.86 #3 224961.00.51.061.021.09 563841.01.90.940.980.92 #4 224481.00.51.161.171.02 561961.02.00.860.850.98 #5 112241.00.31.620.891.12 141961.03.30.621.120.89 #6 56961.00.51.00.780.74 143841.02.21.01.281.35 #7 112961.00.60.880.850.99 283841.01.81.141.181.01 Each scenario contains two configurations of convolutions (first row and second row in each scenario), that approximately feature the same amount of MACs (see 4th column), but strongly differ in operating resolution and number of channels. We set their throughput and latency in relation to each other(see 5th-8th column). The table shows that, for the same number of MACs, using convolutions with lower operating resolution and higher channel dimension (highlighted in gray) is more hardware efficient than using a higher resolution with lower channel dimension. Bold values refer to the best value for each scenario and column 3.3 High Resolution vs. High Channel Motivation. A key aspect of architecture design is determining the distribution of layers across differ- ent stages of the model. In backbone architectures, early stages typically have a high operating resolution and a low channel dimension, whereas later stages the opposite trend (Tan and Le, 2019; Cai et al, 2023; Qin et al, 2025). Therefore, understanding whether layers with a high channel dimension and low oper- ating resolution are more efficient than those with a low channel dimension and high operating resolu- tion is crucial for efficient architecture design. This insight helps optimize the allocation of layers across stages. In the following, we will analyze this factor for convolutional layers. Setting. For this experiment, we create toy mod- els that consist of 20 times the same convolution stacked after another, where each toy model has a different configuration regarding operating resolution and channel dimension. In Table 2, we put at test seven scenarios, each contrasting two convolutions with the same amount of MACs, but differing in chan- nel dimension and operating resolution. The models only contain standard convolutions (ungrouped). The upper row in each scenario is always the model oper- ating on a higher resolution, while the lower row features a higher channel dimension. For improved readability we state relative throughput, latency and MAC count in Table 2. In each scenario, the metrics (throughput, latency, MAC count) of the low resolu- tion model are expressed relative to those of the high resolution model and vice versa for the high resolution model. Results. In scenario #1 the first model has a third of the throughput of the second one and almost twice the latency. It operates on eight times the resolution, however features less channels and its MACs equal the second layer. The same effect also occurs with a smaller resolution difference, as scenario #2 shows, where the first model runs on twice the resolution, but still fails to execute its MACs as fast as the second one in terms of throughput and latency. On the other side in scenario #7 the model with a higher operat- ing resolution, has a slightly lower latency. Regarding GPU throughput however, the lower resolution models always have a considerably higher throughput, rang- ing from a factor of 1.8 to 3.3. In scenario #5 the lower resolution model achieves lower latencies for the TX2 and ARM CPU, while for mobile the higher resolu- tion model prevails, showing the impact that different hardware can have on model execution. Nevertheless a clear trend is visible. Models with a high operat- ing resolution tend to be slower in most scenarios in terms of latency and throughput, than their MAC 8 count might suggest. In (Nottebaum et al, 2025) this is also shown for GPU latency. Conclusion. We can conclude that in most cases it proves more hardware efficient to apply more layers in later stages of the backbone architecture and have only few layers in the high resolution stages. Based on these experiments, we emphasize that model scal- ing (increasing width, depth and input resolution (Zhai et al, 2022)) should be guided by actual measured execution time (Tan and Le, 2019), as MACs can be misleading, when scaling models by higher input res- olution (Yun and Ro, 2024; Cai et al, 2023), as our results indicate. 3.4 Optimizing MHSA Efficiency for Higher Input Resolutions The MHSA introduced by (Vaswani et al, 2017) is a crucial building block in many recent computer vision frameworks (Kirillov et al, 2023; Yan et al, 2021; Carion et al, 2020; Dunnhofer et al, 2022; Khan et al, 2024; Bansal et al, 2022). The actual atten- tion operation takes place within the SDA module of MHSA. Motivation. A key limitation of MHSA is that the computational complexity of SDA scales quadrati- cally with spatial dimensions and linearly with the number of channels. To address this issue, we propose adaptations that reduce the input dimensions for SDA, enhancing efficiency without sacrificing performance. We evaluate the efficiency of both the original MHSA and our adapted versions across input resolutions ranging from 8×8 to 64×64. This is motivated from the fact, that many frameworks incorporating MHSA typically feature a high input resolution (>512×512) (Kirillov et al, 2023; Zheng et al, 2023; Zhang et al, 2021; Carion et al, 2020), consequently leading to SDA being executed at resolutions above 16×16. This is particularly true when the backbone itself includes MHSA. For instance, in the ViT-B/16 vision back- bone (Dosovitskiy et al, 2020), an input resolution of 1024×1024 (Kirillov et al, 2023) leads to SDA being computed at resolution 64×64. Setting. To conduct this experiment, we employ toy models that consist of 4 times the same attention layer after another. We feature an input and output channel dimension of 128 to prevent overloading the edge devices, as MHSA at increased input resolutions imposes a high computational burden. We measure latency on the Jetson TX2, the iPhone 13, the ARM CPU of the Raspberry Pi5 and the Nvidia TITAN RTX GPU. Efficient Adaptations. In total, we feature two modifications of MHSA: "chcompr." (channel com- pression) stands for halving the channel dimension before the SDA by the input projection and restoring the input channel dimension by the output projection of the MHSA. "conv+low" means we add a convolu- tion before the input projection and after the output projection, which downsample and upsample the res- olution of the feature maps. In Table 3, we compare how these two modifications, taken together and sep- arately, influence the execution time under different input resolution (from 8×8 to 64×64). Results. In Table 3 we compare MHSA with the two adaptations we propose. Table 3 illustrates the impact of the quadratic explosion of MHSA. On edge devices, latency is at least 442 times higher for a res- olution of 64×64 compared to 8×8, and even 2232 times higher on an ARM CPU. In contrast, GPU execution shows only a 28-fold increase in latency, highlighting the immense parallelization capabilities of GPUs compared to edge devices. Regarding our adaptations, we can see that "chcompr." leads to a reduction in latency of between 30% and 63% on edge devices, while "conv+low" leads to an even higher efficiency gain. The performance improve- ment for both adaptations increases significantly with higher input resolution. On GPU with input resolu- tions 8×8 and 16×16, the efficiency remains similar to MHSA due to the potential of parallelization a desktop GPU has or slightly declines, because of the layers added by "conv+low" adaptation. Nevertheless for input resolutions above 16×16, both adaptations lead to a considerable reduction in latency by up to 90% compared to MHSA. Combining both meth- ods ("conv+low+chcompr") maximizes efficiency. For input resolution 64×64 its latency reduction ranges between 92% and 98% across all devices. Conclusion. While "chcompr." can lead to a reduc- tion of up to half of the latency, "conv+low" can reduce latency to a smaller fraction of it. Overall, the combination of the two optimizations ("chcompr." and "conv+low") provides significant benefits for perfor- mance, particularly for edge devices. At high input resolutions, the adaptations also lead to substantial performance gains for GPUs. Since many downstream tasks rely on high input resolution, these optimizations 9 Table 3 Efficiency comparison between original MHSA and three adaptations of it Attention Resolution 8×816×1632×3264×64avg. diff. MHSA0.592.7919.70260.97 TX2 Latencychcompr.0.38 (-37%)1.63 (-41%)10.29 (-47%)130.73 (-50%)-43% (ms)conv+low0.53 (-10%)1.39 (-50%)4.87 (-75%)28.09 (-89%)-56% conv+low+chcompr.0.39 (-35%)1.21 (-56%)3.62 (-81%)18.56 (-92%)-66% MHSA0.201.605.48179.75 Mobile Latencychcompr.0.15 (-25%)0.79 (-50%)2.62 (-52%)65.10 (-63%)-47% (ms)conv+low0.24 (+20%)0.45 (-71%)1.34 (-75%)5.60 (-97%)-56% conv+low+chcompr.0.21 (+5%)0.33 (-79%)0.71 (-87%)2.71 (-98%)-65% MHSA0.816.1695.321808.16 ARM CPU Latencychcompr.0.57 (-30%)2.42 (-60%)46.73 (-50%)915.41 (-49%)-47% (ms)conv+low0.85 (+5%)2.21 (-64%)10.97 (-88%)126.48 (-93%)-60% conv+low+chcompr.0.75 (-7%)1.88 (-69%)7.23 (-92%)75.79 (-95%)-66% MHSA0.490.491.2413.85 GPU Latencychcompr.0.49 (-0%)0.50 (+2%)0.85 (-31%)7.76 (-44%)-18% (ms)conv+low0.62 (+26%)0.63 (+28%)0.63 (-49%)1.42 (-90%)-22% conv+low+chcompr.0.63 (+28%)0.64 (+30%)0.64 (-48%)1.02 (-92%)-20% "conv+low" refers to encapsulating MHSA with convolutions that reduce the resolution of the attention by a factor of 2, "chcompr." refers to a reduced channel dimension for SDA and "conv+low+chcompr." refers to the combination of the latter two. We evaluate execution time under different input resolutions. The highest efficiency is achieved by applying the "conv+low+chcompr." adaptation on MHSA. Results for original MHSA are highlighted in gray are highly relevant (Kirillov et al, 2023; Carion et al, 2020; Liu et al, 2025). In Section 5.2, we will further demonstrate that the combination of both optimiza- tions has a positive impact on ImageNet (Deng et al, 2009) accuracy. 4 LowFormer The aim of the LowFormer architecture design is to improve hardware efficiency, allowing vision back- bones to execute faster while maintaining a high model size that enables reaching high accuracy. To achieve this goal, we follow the insights of the analysis presented in Section 3. In this section, we will first introduce Lowtention, a lightweight adap- tation of the original MHSA (Vaswani et al, 2017) (Section 4.1), and then outline the key principles of LowFormer’s macro design and provide both an overview (Section 4.2) and detailed description of the overall architecture (Section 4.3). Finally, we will pro- pose adaptations to the main LowFormer architecture aimed specifically at further enhancing performance on edge devices (Section 4.4). 4.1 Micro Design - Lowtention In LowFormer’s architecture, we employ a lightweight adaptation of the original MHSA (Vaswani et al, 2017), which we call Lowtention(see Figure 5). Lowtention encapsulates the SDA by two depthwise convolutions and two pointwise convolutions. The lat- ter perform the input and output projections of the queries (Q), keys (K) and values (V), just as in the original MHSA. Channel compression. However, in Lowtention the input projection (before the SDA) reduces the channel dimension of Q, K and V by half (see Figure 5). The output projection (following the SDA) then restores the original channel dimension, which is required for the residual connection after the Lowtention (see Figure 5). In Section 3.4, we demonstrated that compressing the channel dimension can significantly improve latency on a variety of devices, especially at high input resolutions. Lower resolution. As depicted in Figure 5 we down- and upsample the resolution of the feature maps in Lowtention around the SDA, such that the SDA is executed on half the resolution. This motivates from the experiments in Section 3.4, where we have shown that this can improve latency significantly, similar to the channel compression in Lowtention. In Section 5.2 we further show that the combi- nation of both strategies (channel compression and lower resolution) improves top-1 accuracy on Ima- geNet (Deng et al, 2009). MLP following Attention. Following (Vaswani et al, 2017), we append layer normalization and a multi-layer perceptron (MLP) after the Lowtention. 10 Fig. 4 Structure of the fused and unfused MBConv block. C refers to the channel dimension. Both have an expansion factor of 4 Fig. 5 Lowtention block design. LN refers to layer normalization. In contrast to the traditional MHSA, we encapsulate the SDA with two depthwise convolutions (the second is a transposed depth- wise convolution). The projections for MHSA are realized with pointwise convolutions. The DW ↓ n means that the resolution is downscaled by the factor n and DW ↑ n that it is upscaled by n Fig. 6 Architecture of LowFormer. The resolutions refer to a 224x224 sized input. Lowtention block can be seen in Figure 5. "Conv" refers to convolution and "Cls head" refers to the image classification head (see Figure 7). Specification of C 0 −C 4 and L 0 − L 4 depend of the version of LowFormer and can be found in Table 4 This is motivated by Liu et al (2023), who pointed out the significance of MLPs for improving accuracy of a backbone. 4.2 Macro Design The LowFormer architecture features five stages that adapt the architectural macro design of EfficientViT (Cai et al, 2023) and MobileViT (Mehta and Rastegari, 2021) according to the insights gained from the hard- ware efficiency analysis presented in Section 3. The whole architecture is depicted in Figure 6. In total we present five different base versions of LowFormer, namely B0, B1, B1.5, B2, B3. The versions differ in the number of layers and channel dimension employed in each stage (see Table 4). We chose to feature five base versions of LowFormer to demonstrate that its design principles are adaptable to various model sizes and accuracy levels. LowFormer- B0 represents the model with the lowest model size, while LowFormer-B3 has the highest. Consequently a base version with a higher model size also achieves superior results compared to a lower size variant, as shown in Section 5. Table 4 Specification of LowFormer architecture versions B0-B3 ModelL 0 , L 1 , L 2 , L 3 , L 4 C 0 ,C 1 ,C 2 ,C 3 ,C 4 LowFormer-B00, 0, 0, 3, 416, 32, 64, 128, 256 LowFormer-B10, 0, 0, 5, 516, 32, 64, 128, 256 LowFormer-B1.5 0, 0, 0, 6, 620, 40, 80, 160, 320 LowFormer-B2 0, 0, 0, 6, 624, 48, 96, 192, 384 LowFormer-B3 1, 1, 2, 6, 632, 64, 128, 256, 512 The number of layers (L 0 − L 4 ) and channels (C 0 − C 4 ) relates to Figure 6 Lowtention. We include Lowtention in the last two stages and keep the first three stages purely convolu- tional. Additionally, we only downsample the feature maps of Lowtention (as mentioned in Section 4.1) in the forelast stage. Fusing Depthwise and Pointwise Convolutions. In Section 3.1 we showed that depthwise convolutions are not as hardware-efficient as standard convolution and in Section 3.2 we came to the conclusion that the fused MBConv (see Figure 4) can be faster than the unfused one, even though it usually has a higher MAC count. This effect diminishes however with increasing number of channels. We therefore fused the MBConv in our architecture, wherever the number of input channels reach at most 256, except for the strided MBConv blocks at the beginning of the last two stages 11 (see Figure 6). We additionally fuse the depthwise and pointwise convolutions after the SDA in the Lowten- tion (see Figure 5), as their input channel dimension does not exceed 256 for any LowFormer model, due to the channel compression. We confirm the effect of this approach by reverting the fusion of the MBConv block for LowFormer-B1 in the ablation in Section 5.2. Less layers in the first stages. From the insights in Section 3.3 we conclude that a minimal amount of lay- ers in the first stages is more hardware-efficient (see Table 4). It proved optimal to apply the reduction for the first three stages. Most computation is therefore concentrated in the last two stages, where for an input size of 224×224, the operating resolutions are 14×14 and 7×7. 4.3 Additional Architectural Details The details of LowFormer’s architecture mainly fol- low design principles of previous publications (Cai et al, 2023; Vasu et al, 2023a). As activation function, we utilize HardSwish (Howard et al, 2019), except for the MLP, where we use GeLU (Hendrycks and Gimpel, 2016) in between its two linear layers (Vaswani et al, 2017). MBConv Micro Design Details. LowFormer’s micro design for MBConv blocks, regarding the combination of batch normaliation (bn) (Ioffe, 2015) and activation functions (act), differs between fused and original (unfused) MBConv. For the fused MBConv, each block follows the design of "conv,bn,act,pwconv,bn", while the original MBConvs design is "pwconv,act,dwconv,act,pwconv,norm". MBConv Expansion Factor. The expansion factor in MBConvs controls by which factor the channel dimension is increased by the first convolution and decreased by the last convolution (see Figure 4). We set it to 6 for all MBConv blocks (fused and unfused) that reduce resolution and 4 otherwise. Residual Connections. Besides the residual con- nections in the Lowtention block (see Figure 5), every MBConv block that does not reduce resolution, has a residual connection. We only reduce resolution by strided MBConv blocks at the beginning of each stage. Classification Head. In Figure 7 the design of our classification head is depicted, which we apply for all versions of LowFormer and only differs in the input channel dimension, given by the last layer of the final Fig. 7 Design of classification head that is used for all variants of LowFormer stage. It is based on EfficientViTs (Cai et al, 2023) classification head. 4.4 LowFormer for Edge GPU We further present three edge GPU variants of LowFormer, namely LowFormer-E1/E2/E3. They are derivations of the so far described architecture. The changes only focus on the last two stages, as most of the computational load is concentrated there (see Table 4). We feature three changes to the original LowFormer Architecture: a) Reduction of the depth by 2 b) Removal of the MLP c) Removal of the Lowtention All changes follow the intuition that Edge GPUs (like the Jetson TX2) usually consist of many cores 1 and focus on high parallelization, similar to GPUs 2 . This is in contrast to CPUs for example, who usually feature much less cores, like the ARM CPU in the Raspberry Pi5, who operates on 4 cores 3 . Change a) therefore originates from the intuition, that depth is a crucial factor in optimizing efficiency on edge GPU devices, as the compulsory sequential execution of high amount of layers, prohibits paral- lelization, thus it would be beneficial to rather have fewer layers, but with high amount of computation, that is parallelizable. Change b) motivates from the observation of (Qin et al, 2025), that MLP executions are usually memory bound and therefore are not efficient on hardware with high compute ability (e.g. GPUs and edge GPUs). Additionally MLPs account for a significant portion of the whole models computational load (MACs). For LowFormer-B1 for example, the MLPs make up 17% of the total MACs. 1 https://developer.nvidia.com/embedded/jetson-tx2 2 https://w.nvidia.com/content/dam/en- z/Solutions/titan/documents/ti tan-rtx-for-creators-us-nvidia-1011126-r6-web.pdf 3 https://w.raspberrypi.com/products/raspberry-pi-5/ 12 Table 5 Architecture changes for LowFormer edge variant, versions E1/E2/E3 ModelChangesBase model LowFormer-E1a), b), c)LowFormer-B1.5 LowFormer-E2a), b), c)LowFormer-B3 LowFormer-E3 b)LowFormer-B3 The changes a), b) and c) refer to the enumeration in Section 4.4 Change c) on the other side motivates from our observation in Section 3.4, where the attention mecha- nisms show an immense computational burden, espe- cially with increased input resolution. Furthermore, (Liu et al, 2023) demonstrated that MHSA operations are heavily memory bound, even more so than the MLP. They recommend allocating a lower portion of the model to MHSA to yield the best speed accuracy trade-off. We combine the three changes a), b) and c) into the models LowFormer-E1/E2/E3 as depicted in Table 5. In Section 5.3.1, we substantiate our reasoning by comparing how each mentioned change impacts accu- racy and efficiency for edge GPU and GPU. 5 Experiments In this section, we present and the discuss the exper- imental results achieved by the LowFormer family of vision backbones (LowFormer-B0/-/B3), includ- ing the edge variants (LowFormer-E1/E2/E3). For the experiments, we examine model efficiency by measur- ing GPU throughput, GPU latency and edge device latency. Additionally measuring throughput is moti- vated by its ability to assess how efficiently big quan- tities of data can be processed. A high throughput is particularly beneficial for tasks such as image retrieval and batched offline processing of video data by detec- tion and segmentation algorithms. It is also crucial to reduce training time (Tan and Le, 2021). Protocols for Measuring Execution Time. In Table 6 specifics of the measuring protocols are listed , that we apply throughout the paper. We always take the median time per input instance for latency and throughput measurements (Vasu et al, 2023a; Yun and Ro, 2024). For latency measurements we always use a batch size of 1, while for GPU throughput we fea- ture a batch size of 200. As depicted in Table 6, we measure latency on three edge devices, namely the Nvidia Jetson TX2 8GB developer kit, the iPhone 13 and the ARM CPU of the Raspberry Pi5 8GB (Arm Table 6 Specifics of efficiency measurements. Further details are given in the supplementary document. DeviceMetricIterationsFramework Nvidia A40 GPUthroughput100PyTorch Nvidia TITAN RTX GPUlatency4000TorchScript 5 ARM Cortex A76 CPUlatency400ONNX 6 iPhone13 latency-CoreML Nvidia Jetson TX2 8GB latency200ONNX 6 +TensorRT 7 5 https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.h tml 6 https://github.com/microsoft/onnxruntime 7 https://docs.nvidia.com/deeplearning/tensorrt/ Cortex A76 processor @ 2.4GHz). The amount of iter- ations differ, because some devices require a higher amount of iterations to retrieve stable results. However we always feature 5 warm-up iterations. Regarding the iPhone 13 we utilize the CoreML 4 performance tool to retrieve latency results, wherefore we do not have specifics about the amount of iterations. 5.1 ImageNet-1K Classification Settings. We conduct image classification experi- ments on ImageNet-1K (Deng et al, 2009), which includes 1.28M training and 50K validation images for 1000 categories. All models were trained from scratch using a similar setting as (Cai et al, 2023) and featuring an input resolution of 224. We also trained for a total of 320 epochs using AdamW (Loshchilov and Hutter, 2017) optimizer and a learning rate of 10 −3 , however we use a batch size of 512. As learn- ing rate scheduler we use cosine decay (Loshchilov and Hutter, 2016) and 20 warm-up epochs with a lin- ear schedule. We also feature the multi-scale learning from (Cai et al, 2023). We trained LowFormer-B3 with a batch size of 2400 and a base learning rate of 3× 10 −3 . For LowFormer-B2 we had a batch size of 850 and a base learning rate of 8.3× 10 −4 . Results. In Table 7, we evaluate the speed accu- racy trade-off of the compared models. For this we measure ImageNet (Deng et al, 2009) top-1 accuracy and assess efficiency based on GPU throughput, Jet- son TX2 latency and ARM CPU latency. The base models of LowFormer (LowFormer-B0 to B3) achieve a superior speed accuracy trade-off, outperforming most compared models in all three efficiency metrics. LowFormer-B0, has a slightly higher top-1 accuracy than EfficientMod-xs (Ma et al, 2024), twice the throughput, half of its TX2 latency and executes 38 % faster on ARM CPU. Although BiFormer-T (Zhu 4 https://apple.github.io/coremltools/docs-guides/ 13 Table 7 Performance on ImageNet-1K validation set ModelVenue ParamsMACsGPU Throughput↑TX2 Latency↓ARM CPU↓ResolutionTop-1 (M)(M)(images/s)(ms)(ms)(pixel)(%) MobileViG-Ti* (Munir et al, 2023)CVPRW 20235.366125008.548.222475.7 FastViT-T8 (Vasu et al, 2023a)CVPR 20233.6690169410.165.925675.6 EfficientMod-xxs (Ma et al, 2024)ICLR 20244.7579285715.047.522476.0 RepViT-M0.9 (Wang et al, 2024)CVPR 20245.1816251211.440.822477.4 MobileOne-S2 (Vasu et al, 2023b)CVPR 20237.8129829679.153.822477.4 EdgeViT-XS (Chen et al, 2022b)ECCV 20226.81127212716.058.922477.5 MobileOne-S3 (Vasu et al, 2023b)CVPR 202310.11895243311.874.222478.1 MobileViG-S* (Munir et al, 2023)CVPRW 20237.3983172412.373.922478.2 EfficientMod-xs (Ma et al, 2024)ICLR 20246.6773235217.753.822478.3 LowFormer-B0 (ours)14.194459888.539.122478.4 FastViT-T12 (Vasu et al, 2023a)CVPR 20236.81400205414.5110.425679.1 RepViT-M1.1 (Wang et al, 2024)CVPR 20248.21338194113.563.522479.4 MobileOne-S4 (Vasu et al, 2023b)CVPR 202314.82978155018.6122.922479.4 LowFormer-B1 (ours)17.91410423711.759.122479.9 EfficientFormerV2-S2 (Li et al, 2023)ICCV 202312.6125046819.9102.322480.4 FastViT-SA12 (Vasu et al, 2023a)CVPR 202310.91943107517.5136.425680.6 EdgeViT-S (Chen et al, 2022b)ECCV 202211.11910144924.699.222481.0 EfficientMod-s (Ma et al, 2024)ICLR 202412.91402138130.5105.622481.0 RepViT-M1.5 (Wang et al, 2024)CVPR 202414.02276114623.0113.222481.2 LowFormer-B1.5 (ours)33.92573273918.1111.622481.2 FFNet-1 Yun et al (2024)arXiv 202413.83000109030.4242.125681.3 BiFormer-T Zhu et al (2023)CVPR202313.1220072961.3523.922481.4 LowFormer-B2 (ours)45.03689222721.6144.222481.6 SMT-T (Lin et al, 2023)ICCV202311.5240077050.3195.622482.2 RepViT-M2.3 (Wang et al, 2024)CVPR 202422.9452064240.6227.022482.5 FastViT-SA24 (Vasu et al, 2023a)CVPR 202320.6376960630.9273.725682.6 LowFormer-B3 r192 (ours)57.14479156230.0198.619282.7 iFormer-S (Si et al, 2022)NeurIPS 202219.9482555551.2270.622483.4 FastViT-SA36 (Vasu et al, 2023a)CVPR 202330.4559542944.4399.825683.6 SMT-S (Lin et al, 2023)ICCV202320.5470041896.4397.822483.7 BiFormer-S (Zhu et al, 2023)CVPR202326.04500348130.21134.122483.8 LowFormer-B3 (ours)57.16098116232.5273.822483.6 The table is divided into different groups, determined by similar top-1 accuracy (bold horizontal lines separate groups). Values in bold are the best results for each group and column, while underlined results refer to the second best. For models marked with *, only distilled model results are publicly available. LowFormer models (highlighted in gray) achieve superior speed accuracy trade-offs in terms of GPU throughput, TX2 latency and ARM CPU latency Fig. 8 Impact of input resolution on Jetson TX2 latency for LowFormer-B1 (ours), LowFormer-B1 without downsampling in the Lowtention layers (see Section 5.2), MobileOne-S4 (Vasu et al, 2023b), RepViT-M1.1 (Wang et al, 2024), EdgeViT-XS (Chen et al, 2022b), EfficientMod-xs (Ma et al, 2024) and FastViT-T12 (Vasu et al, 2023a). "Acc" refers to ImageNet top-1 accuracy. LowFormer- B1 demonstrates remarkable efficiency when operating at a higher input resolution, outperforming compared state-of-the-art architec- tures et al, 2023) has 40% less MACs than LowFormer- B2 r224 , our LowFormer has a 305% higher throughput and only a third of its TX2 and ARM CPU latency. At the same time it slightly outperforms BiFormer- T by 0.2% top-1 accuracy. The largest base model of LowFormer, LowFormer-B3, surpasses FastViT-SA36 (Vasu et al, 2023a) in efficiency, achieving nearly three times the GPU throughput and running 36% faster on the TX2, and 46% faster on an ARM CPU. Both models share the same top-1 accuracy. The hardware efficient design of LowFormer base models not only execute MACs more efficiently but also leads to higher top-1 accuracies, while achieving improved efficiency. ResolutionScaling. Amodel’sefficiencyat increased input resolution is critical, as many down- stream tasks run the backbone on high-resolution inputs (Carion et al, 2020; Kirillov et al, 2023). As shown in Table 2, model efficiency can vary in response to higher input resolutions, depending on the architecture. Therefore, we examine in Figure 8 14 Table 8 Ablation study of LowFormer-B1, featuring singular changes to the original model Model version ParamsMACsGPU Throughput↑TX2 Latency↓ARM CPU Latency↓Top-1 (M)(M)(images/s)(ms)(ms)(%) unfused MBConv12.47163558 (-16%)12.8 (+9%)55.5 (-6%)79.1 (-0.8) relu-linear att14.1512103367 (-20%)12.7 (+8%)64.8 (+9%)79.6 (-0.3) original MHSA16.814603590 (-15%)12.5 (+7%)61.2 (+4%)79.8 (-0.1) high-res attention17.6514943759 (-11%)13.4 (+14%)64.0 (+8%)79.9 (+0.0) no channel compr.20.6816503921 (-7%)13.4 (+14%)63.1 (+7%)79.9 (+0.0) Baseline (B1)17.941410423711.759.179.9 Bold entries mark the best in its column. The LowFormer architecture design elements consistently demonstrate superior efficiency while achieving higher ImageNet accuracy how increasing input resolution affects latency on the Jetson TX2 for LowFormer-B1 and other approaches. LowFormer-B1 outperforms depicted models in top- 1 accuracy on ImageNet-1K and at the same time remains considerably faster, independent of the input resolution. The model “Stride 1 Attention” refers to an ablation of LowFormer-B1, discussed in Section 5.2. 5.2 Ablation Study of LowFormer Base Models In Table 8 we ablate our model design decisions. We revert a singular design decision of LowFormer-B1 to demonstrate the impact of that change on accu- racy, GPU throughput and TX2 latency. The featured ablations are the following: • We replace all fused MBConv blocks with the unfused version. • We replace our Lowtention with ReLU linear attention from (Cai et al, 2023) in order to com- pare our attention approach with other recent adaptations. • We omit the downscaling of the feature maps for the Lowtention (high-res attention). • We remove the channel compression done during the projection in Lowtention. • We replace Lowtention with the original MHSA proposed by (Vaswani et al, 2017). 5.2.1 Ablation Results Unfused MBConv. Replacing the fused MBConv with the unfused one results in a 16% lower GPU throughput and 10% higher TX2 latency (see Table 8). On the other side ARM CPU latency improves by 6%, however top-1 accuracy drops significantly by 0.8% As we can see, next to a mostly improved execution time, fusing the MBConv can increase performance significantly. ReLU Linear Attention. Applying ReLU linear attention from (Cai et al, 2023) on the other side results in 0.3% reduced top-1 accuracy, as well as ̃ 8% higher latency and 20% lower GPU throughput, showing the benefit of Lowtention. Downsampling in Lowtention. When we remove the downsampling for the Lowtention, top-1 accu- racy stays the same, but GPU throughput and latency worsen significantly. In Figure 8 we can see that for higher input resolutions the latency difference multi- plies. For input resolution 1024×1024 for example, latency is increased by 220% on the Jetson TX2. Channel Compression. Removing the channel compression of Lowtentionduring the projection phase does not improve accuracy, even though the SDA operates on a lower channel dimension with compression. However throughput and latency worsen significantly without channel compression. MHSA. When removing channel compression and convolutions from Lowtention (no downsampling), we revert to the original MHSA. As shown in Table 8, this not only reduces top-1 accuracy, but also increases model latency significantly. This difference multiplies, when input resolution is increased as shown in Table 3. Summary. The reduced dimensions on which we apply the attention operation (channel compression and resolution reduction) have no effect on top-1 accu- racy, but improve the efficiency of the model signifi- cantly. The additional convolutions improve accuracy (original MHSA fairs worse in Table 8), but only mildly worsen execution time (see Table 3). In con- clusion Lowtention is both more efficient and effective than the original MHSA (Vaswani et al, 2017), with significantly better scalability for increased input res- olution. 15 Table 9 Efficiency comparison of LowFormer on several computing devices with modified versions of the original architecture, where the multi-layer-perceptron (MLP) or the Lowtention (Att) is removed, or the depth of the last two stages is reduced Model MLPOriginalAttGPU ThroughputGPU LatencyTX2 LatencyiPhone 13Top-1 Depth(images/s)(ms)(ms)(ms)(%) LowFormer-B0✓59882.98.51.578.4 LowFormer-B1✓42374.011.71.879.9 B1_mlpless✗✓60672.88.51.578.8 B1_mlpless_shallow✗✓82541.85.91.377.2 LowFormer-B1.5✓27394.82.8111.681.2 B1.5_mlpless✗✓40193.313.22.480.7 B1.5_mlpless_shallow✗✓52682.49.72.079.7 B1.5_conv_shallow (E1)✗63371.06.21.678.8 LowFormer-B3✓11625.232.54.583.6 B3_mlpless (E3)✗✓15663.625.03.683.0 B3_mlpless_shallow✗✓18482.719.62.882.2 B3_conv_shallow (E2)✗20701.514.72.581.6 The highest efficiency increase can be achieved by removing the attention operation. The connotations E1,E2,E3 in braces refer to the proposed edge GPU variants. LowFormer base models are highlighted in gray 5.3 Comparison of LowFormer Edge GPU Variants with the Base Models In Section 4.4 we presented the three derivations LowFormer-E1/E2/E3 from the original LowFormer base models (B1.5 and B3). In the following part we will first experimentally justify why these specific changes were chosen by analyzing the effect of each change on efficiency and accuracy. Then we will put the LowFormer edge variants into perspective with the best competing models of Table 7, based on their speed accuracy trade-off on the Jetson TX2. 5.3.1 Efficiency Analysis of Attention, MLP and Depth In Section 4.4 we presented three possible changes to the LowFormer architecture, namely removing the MLP, reducing the depth and removing the Low- tention. In Table 9 we show the effect of each of these changes cumulatively. First we remove the MLP ("mlpless"), then we additionally reduce the depth ("shallow") and at last we also remove the Lowtention ("conv"), leaving only convolutions in the architec- ture. We use LowFormer B1, B1.5 and B3 as baselines for the modifications. All derivations have the same hyperparameter setting during training as their corre- sponding base version. RemovingAttention. Fromthemodel B1.5_conv_shallow we can see the enourmous penalty attention can have on latency. It has 42% of the GPU latency and 64% of the TX2 latency of B1.5_mlpless_shallow, but loses less than 1% top-1 accuracy. Both models only differ in that B1.5_conv_shallow does not feature the Lowtention. The difference becomes more apparent if you com- pare B1.5_conv_shallow to b1_mlpless, which has the same top-1 accuracy, but fares far worst regarding GPU and TX2 latency. Removing MLP. On the other side, omitting the MLP can also be beneficial, although not as pivotally. B1_mlpless has a 0.4% higher top-1 accuracy than LowFormer-B0, while being slightly faster in terms of latency and throughput. Reducing Model Depth. Reduction of the model depth especially improves latency, while its impact on throughput is less pronounced. For example B1.5_mlpless_shallow achieves a 0.9% higher top-1 accuracy compared to B1_mlpless, while having an improved GPU latency and a slightly worse through- put. Summary. Removing Lowtention, the MLP and reducing the model depth can significantly improve the speed accuracy trade-off, especially in terms of latency on GPU and edge GPU (Jetson TX2). These three modifications are combined in LowFormer- E1 and LowFormer-E2, yielding a substantial effi- ciency gain with only a minimal accuracy drop. In LowFormer-E3, only the MLP is removed, allowing it to achieve a higher accuracy of 83.0% without relying on model-scaling strategies such as width scal- ing, which become less efficient as model capacity increases (Tan and Le, 2019). 16 Table 10 Efficiency comparison between the LowFormer edge GPU variants and the best competing models of Table 7 Model MACsGPU ThroughputGPU latencyTX2 LatencyiPhone 13Top-1 (M)(images/s)(ms)(ms)(ms)(%) EfficientMod-xxs (Ma et al, 2024)57928572.115.01.7176.0 EdgeViT-XS (Chen et al, 2022b)112721272.716.01.577.5 MobileOne-S3 (Vasu et al, 2023b)189524331.0 11.81.278.1 EfficientMod-xs (Ma et al, 2024)77323522.517.72.278.3 LowFormer-B0 (ours)94459882.98.51.578.4 LowFormer-E1 (ours)135063371.06.21.778.8 FastViT-SA12 (Vasu et al, 2023a)194310751.717.51.680.6 EfficientMod-s (Ma et al, 2024)140213813.830.52.681.0 RepViT-M1.5 (Wang et al, 2024)227611464.223.01.581.2 LowFormer-B1.5 (ours)257327394.818.12.881.2 LowFormer-B2 (ours)368922274.821.63.581.6 LowFormer-E2 (ours)380020701.514.72.581.6 FastViT-SA24 (Vasu et al, 2023a)37696063.030.92.682.6 RepViT-M2.3 (Wang et al, 2024)45206425.540.62.482.5 LowFormer-B3 r192 (ours)44791562 5.530.04.582.7 LowFormer-E3 (ours)535015663.625.03.683.0 LowFormer-E1/E2/E3 (highlighted in gray) consistently rank among the most efficient models in terms of GPU throughput, GPU latency, and TX2 latency 5.3.2 Evaluation of Edge Optimization To put our LowFormer edge GPU variants in perspec- tive, we compare them in Table 10 to the highest com- peting models from Table 7 based on their respective Jetson TX2 latency. In the supplementary material, we further analyzed the power consumption of models in Table 10. GPU Throughput & TX2 Latency . LowFormer- E1/E2/E3 consistently achieve a better throughput and TX2 latency than all compared state-of-the-art models with a similar or lower top-1 accuracy, includ- ing the LowFormer base models (B0/B1.5/B2/B3). LowFormer-E1 for example has half of the TX2 latency of MobileOne-S3 (Vasu et al, 2023b), while having 0.7% higher accuracy. LowFormer-E2 has a similar latency compared to FastViT-T12 (Vasu et al, 2023a), but scores 2.5% higher in ImageNet top-1 accuracy. GPU Latency. LowFormer-E1/E2 similarly outper- form all compared models in GPU latency, how- ever LowFormer-E3 fares considerably worse in that regard. This is mainly to the fact, that it still makes use of the attention operation, which has a negative impact on GPU latency (as mentioned in Section 5.3.1). Nevertheless it is considerably more efficient when compared to LowFormer-B3 r192 . Mobile Latency. Regarding mobile execution on the iPhone 13, the edge variant do not give a consis- tent speed-up, when compared with the LowFormer base versions. Moreover, architectures like RepViT and FastViT (Wang et al, 2024; Vasu et al, 2023a) achieve superior performance on iPhone 13. This is in part because the edge variants and base models feature a higher amount of MACs, which the mobile compute hardware cannot effectively parallelize. Summary. LowFormer-E1, E2 and E3 lead the table (Table 10) in terms of GPU throughput, GPU latency and Jetson TX2 latency. However, the optimizations from Section 4.4 do not translate well to mobile exe- cution on the iPhone 13 NPU and GPU. Despite this, the edge GPU variants achieve a speed-up of up to 3×, compared to the LowFormer base models. 5.4 Application to Downstream Tasks For a fair comparison, we compare models of similar size with each other and select those that achieve the best performance in the respective benchmark. 5.4.1 Image Classification We assess LowFormer’s transfer learning capabili- ties by evaluating its performance when finetuned on smaller image classification datasets. We feature three datasets, namely Oxford-IIIT-Pets (Parkhi et al, 2012), Stanford Cars (Krause et al, 2013) and Oxford-102 Flowers (Nilsback and Zisserman, 2008), following (Tan and Le, 2021; Dosovitskiy et al, 2020; Han et al, 2021). Settings. For finetuning LowFormer on the classifi- cation datasets, we maintained a setup similar to that 17 Table 11 Evaluation on transfer learning classification datasets Model GPU Throughput↑FlowersCarsPets (images/sec)Top-1 (%) Top-1 (%) Top-1 (%) ViT-L/16 (Dosovitskiy et al, 2020)3689.7-93.6 ViT-B/16 (Dosovitskiy et al, 2020)11789.5-93.8 TNT-S (Han et al, 2021)14198.8-94.7 DeiT-B (Touvron et al, 2021)11498.993.9- EfficientNetV2-M (Tan and Le, 2021)27798.594.6- CeiT-S (Yuan et al, 2021)26098.694.194.9 LowFormer-B3 (ours)42498.994.495.0 All models are finetuned and evaluated on resolution 384×384. Best Results for each column are marked bold. Low- Former model is highlighted in gray used for ImageNet training (see Section 5.1). How- ever we increased training and evaluation resolution to 384×384, applied a batch size of 512, a base learning rate of 2.5× 10 4 and removed weight decay, follow- ing previous procedures for transfer learning datasets (Tan and Le, 2021; Dosovitskiy et al, 2020). We train for 360 steps on the train splits of Oxford- IIIT-Pets (Parkhi et al, 2012), 800 steps on Oxford-102 Flowers (Nilsback and Zisserman, 2008), and 3200 steps on Stanford Cars (Krause et al, 2013). Results. In Table 11, we compare the evaluation results of LowFormer-B3 against both convolutional and transformer-based approaches. GPU throughput is measured at a resolution of 384×384. The Low- Former models mostly achieve superior results across all three datasets (Parkhi et al, 2012; Nilsback and Zisserman, 2008; Krause et al, 2013), while main- taining equal or lower GPU throughput. The ViT (Dosovitskiy et al, 2020) models, for instance, fall significantly behind in both efficiency and accuracy, whereas CeiT-S (Yuan et al, 2021) achieves accuracy results closer to LowFormer-B3 but with only half the GPU throughput. 5.4.2 Object Detection We show the applicability of the LowFormer architec- ture for object detection. Backbone GPU throughput and Jetson TX2 latency measurement in Table 12 are conducted using an input resolution of 512×512 (Yun and Ro, 2024; Vasu et al, 2023a; Fan et al, 2023). Settings. We plug the pretrained LowFormer base models (B0/B1/B2/B3) into the RetinaNet framework (Lin et al, 2017) and utilize COCO 2017 (Lin et al, 2014) for training and evaluation. We train the Low- Former base models for 12 epochs (1x schedule) and following (Cai et al, 2023; Fan et al, 2023) regard- ing all hyperparameters. As evaluation metric we use mean average precision (mAP) 8 (Chen et al, 2022b; Fan et al, 2023). Results. In Table 12, we compare the performance of LowFormer base models in object detection against recent vision backbones. LowFormer-B2 for example outperforms FAT-B0 (Fan et al, 2023) by +1.0 AP, while having 93% higher backbone throughput on res- olution 512×512 and only 67% of its latency. On the other side, LowFormer-B0 with a smaller model capacity is able to achieve an increase in AP of +1.4 compared to PVTv2-B0 (Wang et al, 2022), while being 4× faster in terms of TX2 latency. In summary, LowFormer base models are able to outperform all compared vision backbones in terms of speed accuracy trade-off, when plugged into the RetinaNet framework (Lin et al, 2017). 5.4.3 Semantic Segmentation We further demonstrate LowFormer’s applicability to semantic segmentation in a similar fashion as object detection. GPU throughput and TX2 latency is again measured with an input resolution of 512×512 (Yun and Ro, 2024; Vasu et al, 2023a; Fan et al, 2023). Settings. We plug the pretrained LowFormer base models (B1,B2,B3) into the Semantic FPN framework (Kirillov et al, 2019) and use the ADE20K dataset (Zhou et al, 2017) for training and evaluation. We train the models for 40K iterations with a batch size of 32, following (Fan et al, 2023; Wang et al, 2024; Vasu et al, 2023a; Ma et al, 2024). We use AdamW opti- mizer (Loshchilov and Hutter, 2017), cosine annealing for the learning rate (Loshchilov and Hutter, 2016) with a base learning rate of 2× 10 −3 and 1K warm- up steps with linear increase. As evaluation metric we 8 Mean average precision (mAP) is commonly abbreviated as AP in many publications that evaluate on COCO. Within the context of COCO evaluation, AP always refers to mAP. 18 Table 12 Comparison results on object detection on COCO 2017 (Lin et al, 2014) using RetinaNet (Lin et al, 2017) head Backbone GPU ThroughputTX2 LatencymAPmAP 50 mAP 75 mAP s mAP m mAP l (images/s)(ms)(%)(%)(%)(%)(%)(%) MobileNetV3 (Howard et al, 2019)86219.729.949.330.814.933.341.1 MobileNetV4-Conv-M (Qin et al, 2025)51727.432.6----- EfficientViT-M4 (Liu et al, 2023)170017.632.752.234.117.635.346.0 PVTv2-B0 (Wang et al, 2022)35596.037.257.239.523.140.449.7 LowFormer-B0 (ours)119022.438.659.140.921.841.851.7 EdgeViT-XXS (Chen et al, 2022b)51851.838.759.041.022.442.051.6 LowFormer-B1 (ours)84031.639.459.841.722.442.952.4 FAT-B0 (Fan et al, 2023)23294.240.461.642.724.044.353.1 EdgeViT-XS (Chen et al, 2022b)40068.140.661.343.325.243.954.6 PVTv2-B1 (Wang et al, 2022)215268.841.261.943.925.444.554.3 LowFormer-B2 (ours)45063.341.462.244.124.545.155.5 FAT-B1 (Fan et al, 2023)174125.042.564.045.126.946.056.7 LowFormer-B3 (ours)245109.043.164.545.927.147.156.7 LowFormer base models (B0, B1, B2, B3) are able to outperform all compared models in speed accuracy trade-off. Backbone GPU throughput and TX2 latency are measured under resolution of 512×512. LowFormer base models are highlighted in gray Table 13 Results on semantic segmentation, using Semantic FPN (Kirillov et al, 2019) Backbone GPU Throug.TX2 Lat.mIoU (images/s)(ms)(%) ResNet50 (He et al, 2016)27145.936.7 PVTv2-B0 (Wang et al, 2022) 35596.037.2 FastViT-SA12 (Vasu et al, 2023a)26562.038.0 EdgeViT-XXS (Chen et al, 2022b)51851.839.7 LowFormer-B1 (ours)84031.639.7 RepViT-M1.1 (Wang et al, 2024)40455.040.6 FastViT-SA24 (Vasu et al, 2023a)151109.941.0 EdgeViT-XS (Chen et al, 2022b) 40068.141.4 FAT-B0 (Fan et al, 2023)23294.241.5 EfficientFormerV2-S2 (Li et al, 2023) 18285.342.4 PVTv2-B1 (Wang et al, 2022)215268.842.5 LowFormer-B2 (ours)45063.342.8 FAT-B1 (Fan et al, 2023)174125.042.9 RepViT-M1.5 (Wang et al, 2024) 238217.443.6 FastViT-MA36 (Vasu et al, 2023a)86208.444.6 LowFormer-B3 (ours)245109.044.6 LowFormer models achieve superior speed mIoU trade-offs. Backbone GPU throughput and TX2 latency are measured under resolution of 512×512. Results are grouped by mIoU. Bold marks the best results in each group and column, underline refers to the second best. LowFormer models are highlighted in gray Table 14 Image retrieval results on GPR1200 (Schall et al, 2022) benchmark Backbone Resol.GPU Throug.mAP (pixel)(images/s)(%) EfficientViT-M5 (Liu et al, 2023)224568131.9 SHViT-S4 (Yun and Ro, 2024) 256425535.7 FastViT-T8 (Vasu et al, 2023a)256169442.1 ResNet-101* (He et al, 2016)22486842.8 LowFormer-B0 (ours)224598844.0 EfficientViT-B1 (Cai et al, 2023)224273944.6 MNv4-Conv-M (Qin et al, 2025)224274145.3 LowFormer-B1 (ours)224423745.8 CoaT-Lite Tiny (Xu et al, 2021)224115346.4 EfficientViT-B2 (Cai et al, 2023)224129847.3 LowFormer-B1.5 (ours)224273947.6 FastViT-SA12 (Vasu et al, 2023a)256107548.0 CoaT-Lite Mini (Xu et al, 2021) 224106548.3 EfficientViT-L1 (Cai et al, 2023)224102048.4 FastViT-SA36 (Vasu et al, 2023a) 25642949.0 EfficientNetV2-S (Tan and Le, 2021)30069049.0 LowFormer-B3 (ours)224116249.0 LowFormer models achieve superior speed accuracy trade-offs. Results marked with * are taken from (Schall et al, 2022). Models are grouped by mAP. Entries marked as bold, refer to the best results in the respective group of the table. LowFormer models are highlighted in gray use mean intersection over union (mIoU) (Vasu et al, 2023a; Wang et al, 2024). Results. In Table 13, we compare the performance of LowFormer models in semantic segmentation against recent vision backbones. LowFormer-B2 for example has 2.4× the throughput and a 25% lower latency than EfficientFormerV2-S2(Li et al, 2023), but achieves +0.4 mIoU when plugged into Seman- tic FPN. FastViT-MA36 (Vasu et al, 2023a) achieves a similar mIoU as LowFormer-B3, but has approxi- mately twice the TX2 latency and 35% of its GPU throughput. In summary, LowFormer models show significant efficiency gains compared to previous approaches, while maintaining a similar or superior mIoU. 5.4.4 Image Retrieval In order to evaluate the quality of the image embed- ding of LowFormer, we compare it on the GPR1200 (General-Purpose Image Retrieval) benchmark (Schall et al, 2022). Settings. The GPR1200 benchmark data is selected from several datasets, namely Google Landmarks V2, ImageNet Sketch, INat, INSTRE, SOP and IMDB Faces. The combination ensures that the data spans 19 Table 15 Evaluation of LowFormer-Track with its baseline tracker SMAT (Gopal and Amer, 2024) Model TX2GPUGOT10K-valLaSOT-TestTREK-150NfS30AVisTUAV123 fpsfpsAUCPAUCPAUCPAUCPAUCPAUCP SMAT (Gopal and Amer, 2024)539077.066.360.462.839.623.262.474.046.041.564.183.7 LowFormer-Track (ours)519278.969.361.764.639.722.463.175.047.042.065.285.2 LowFormer-Track is an adaptation of the SMAT architecture that replaces the backbone with a LowFormer-B1.5 and changes the attention layers in the head to Lowtention layers. LowFormer-Track is similarly efficient as SMAT, but consistently outperforms SMAT in terms of AUC and Precision, showing the benefits of the hardware efficient design of LowFormer and Lowtention. Bold values mark the best in each column a variety of domains. In total GPR1200 features 12k images and 1200 different classes. For evaluation we follow the protocol of (Schall et al, 2022) and mea- sure performance by mean-Average-Precision (mAP). All models are executed on the input image resolu- tion they are executed on for ImageNet evaluation. As image embedding we take the output after the final pooling, that reduces the resolution to 1×1. We com- pare models by GPU throughput, as this efficiency measure directly reflects a model’s ability to process large amounts of data, which is crucial for retrieving image embeddings from large datasets. Results. Results of the evaluation are depicted in Table 14. LowFormer architecture variants are able to clearly outperform all compared models in through- put accuracy trade-off (see Table 14). Compared to the recently published MobileNetv4-Conv-Medium (Qin et al, 2025), LowFormer-B1 achieves a 0.5% higher mAP and processes 54% more images in the same time. Though FastViT-SA36 (Vasu et al, 2023a) for example achieves a similar mAP score as LowFormer- B3, it has less than half of the GPU throughput. 5.4.5 Visual Object Tracking Besides tasks that process images separately, we applied the LowFormer architecture to the video task of single object tracking (Wu et al, 2015; Kristan et al, 2020). We use the SMAT architecture (Gopal and Amer, 2024) as a baseline, replacing its backbone (Mehta and Rastegari, 2022) with LowFormer-B1.5 and substituting the attention layers in its Separable Self-Attention Head with Lowtention layers. We refer to this adapted model as LowFormer-Track. Settings. We train LowFormer-Track and the base- line SMAT on the train splits of LaSOT (Fan et al, 2019), GOT10K (Huang et al, 2019), and COCO (Lin et al, 2014). For the latter dataset, we use data augmen- tations to generate image pairs from the still images, following (Gopal and Amer, 2024). To best com- pare LowFormer-Track and SMAT, we assimilate their efficiency by using LowFormer-B1.5 as a backbone, as well as train and evaluate it on a slightly lower resolution than SMAT. For search images we resize them to 224×224 instead of 256×256, for template images we resize to 112×112 instead of 128×128. Besides that, we adopt the hyperparameter setting of SMAT for both models. We assess both models perfor- mance across six diverse and widely used single object tracking benchmarks: the validation set of GOT10K (Huang et al, 2019), the test set of LaSOT (Fan et al, 2019), the TREK-150 first person vision bench- mark (Dunnhofer et al, 2023), the NfS benchmark (Galoogahi et al, 2017) that predominantly contains fast-moving objects, the AVisT (Noman et al, 2022) benchmark featuring diverse scenarios with reduced object visibility, and the UAV123 benchmark (Mueller et al, 2016) consisting of sequences captured from an aerial perspective. Results. In Table 15, we compare LowFormer- Track and SMAT on the aforementioned single object tracking benchmarks using the Area-Under-the-Curve (AUC) and Precision (P) metric, following Gopal and Amer (2024). Both achieve similar fps (frames per second) on the Jetson TX2 and GPU. However, LowFormer-Track surpasses SMAT in AUC across all benchmarks. In terms of Precision, SMAT achieves a higher score on the TREK-150 benchmark but lags behind by a large margin in all other cases. The supe- rior results achieved by our adaptation of the SMAT framework highlight the significance of efficient back- bone design as well as the versatility and effectiveness of our proposed Lowtention for computer vision tasks beyond image understanding. 6 Conclusion In this paper, we have examined the relationship of MACs and execution time for several architectural design choices in vision backbones, such as depth- wise convolutions, operating resolution of layers, fus- ing the MBConv block, and attention mechanisms. We have particularly shown how the execution time differs between several different devices and that 20 MACs can be an insufficient predictor of it. The analysis further gave us guidance to create a new hardware efficient vision backbone architecture fam- ily, named LowFormer, that features Lowtention, a lightweight adaptation of the original MHSA. The LowFormer base models (LowFormer-B0-B3) surpass competing approaches in terms of speed accuracy trade-off on GPU, the Jetson TX2, and ARM CPU. Additionally, we presented three edge GPU variants of LowFormer (LowFormer-E1/E2/E3) that further enhance efficiency on the Nvidia Jetson TX2 and GPU. We have shown that using LowFormer as a backbone improves efficiency across several down- stream computer vision tasks, including various trans- fer learning image classification datasets, object detec- tion, semantic segmentation, and image retrieval. We also presented LowFormer-Track, an adaptation of a recently published tracking framework, where we apply LowFormer-B1.5 as a backbone and incorpo- rate our proposed Lowtention, clearly outperforming the baseline. Overall, the results achieved demonstrate the wide applicability of our LowFormer architecture for the implementation of efficient computer vision pipelines. Acknowledgements. This research has been funded by the European Union, NextGenerationEU – PNRR M4 C2 I1.1, RS Micheloni. Progetto PRIN 2022 EXTRA-EYE CUP G53D23002920006, PRIN 2022 PNRR TEAM CUP G53D23006680001. MD received funding from the European Union’s Horizon Europe research and innovation programme under the Marie Skłodowska-Curie grant agreement n. 101151834 PRINNEVOT (CUP G23C24000910006). Data Availability. ImageNet (Deng et al, 2009) dataset is available at https://w.image- net.org/. GOT10K (Huang et al, 2019), LaSOT (Fan et al, 2019), TREK-150 (Dunnhofer et al, 2023), NfS30 (Galoogahi et al, 2017), AVisT (Noman et al, 2022), GPR1200 (Schall et al, 2022), ADE20K (Zhou et al, 2017), COCO 2017 (Lin et al, 2014), Oxford-IIIT-Pets (Parkhi et al, 2012), Oxford-102 Flowers (Nilsback and Zisserman, 2008) and Stanford Cars (Krause et al, 2013) are publicly available and can be found under the references. Supplementary Material A Comparison of Hardware Specifications for used devices In Table 17 the most relevant hardware specifications for execution efficiency are listed for the Jetson TX2 8GB, the Nvidia A40 GPU, the Nvidia TITAN RTX GPU, the Raspberry Pi5 8GB and the iPhone 13. The respective specification sheets are listed under Table 17. B Evaluation of Power Consumption on the Jetson TX2 In Table 16 we compare the peak power consumption of different models, when executed on the GPU of the Jetson TX2 8GB developer kit. We execute those models in MAXP_CORE_ARM mode, which is the default setting. Table 16 shows that LowFormer mod- els achieve competitive efficiency in power consump- tion. LowFormer-B1 for example only has a Peak Power Consumption of 7.7 W, compared to 9.3 W for MobileOne-S3 Vasu et al (2023b), while achieving a 1.8% higher top-1 accuracy. Table 16 Power Consumption and Latency on the Jetson TX2 ModelLatencyTop-1Power Consumption (ms)(%)(W) EdgeViT-XXS Chen et al (2022b)12.774.47.5 EfficientMod-xxs Ma et al (2024) 15.076.07.7 EdgeViT-XS Chen et al (2022b)16.077.58.1 MobileOne-S3 Vasu et al (2023b) 11.878.19.3 EfficientMod-xs Ma et al (2024)17.778.37.8 LowFormer-E1 (ours)6.278.89.5 MobileOne-S4 Vasu et al (2023b) 18.679.49.3 LowFormer-B1 (ours)11.779.97.7 FastViT-SA12 Vasu et al (2023a)17.580.68.8 EdgeViT-S Chen et al (2022b)24.681.08.0 EfficientMod-s Ma et al (2024) 30.581.07.9 LowFormer-B1.5 (ours)18.181.28.3 LowFormer-E2 (ours)14.781.69.3 FastViT-SA24 Vasu et al (2023a)30.982.69.0 LowFormer-E3 (ours) 25.083.09.1 FastViT-SA36 Vasu et al (2023a)44.483.69.0 LowFormer-B3 (ours) 32.583.69.2 The models are sorted by top-1 accuracy. When idle, the Jetson TX2 consumes 1.3 W. References Bansal V, Micheloni C, Foresti G, et al (2022) Spatio-temporal attention for cloth-changing reid 21 Table 17 Hardware Specifications of devices used throughout the paper Nvidia A40 GPU 1 Nvidia TITAN RTX GPU 2 Raspberry Pi5 3 Jetson TX2 8GB 4 iPhone 13 5 Execution Memory48GB GDDR624GB GDDR68GB SDRAM8GB LPDDR44GB Number of Cores 10,752 CUDA & 336 Tensor4,608 CUDA & 576 Tensor4 CPU2+4 CPU & 256 GPU6 CPU & 4 GPU & 16 NPU Memory Bandwidth696 GB/s672 GB/s-59.7 GB/s- Tensor FLOPS74.8 TFLOPS130 TFLOPS-1.33 TFLOPS- Max Power Consump.300 W280 W10 W15 W- Execution memory refers to the respective memory of each device, where the model is loaded to during execution. While the Nvidia GPUs have a GPU- memory, distinct from the RAM of the desktop/server machine, the embedded devices have a shared memory. 1 Nvidia A40 Datasheet, 2 Nvidia TITAN RTX Datasheet, 3 Raspberry Pi 5 Datasheet, 4 Nvidia Jetson TX2, 5 Apple iPhone 13 Specs in videos. In: European Conference on Computer Vision, Springer, p 353–368 Bolya D, Fu CY, Dai X, et al (2022) Hydra attention: Efficient attention with many heads. In: European Conference on Computer Vision, Springer, p 35– 49 Brock A, De S, Smith SL, et al (2021) High- performance large-scale image recognition with- out normalization. In: International conference on machine learning, PMLR, p 1059–1071 Cai H, Li J, Hu M, et al (2023) Efficientvit: Lightweightmulti-scaleattentionforhigh- resolution dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 17302–17313 Carion N, Massa F, Synnaeve G, et al (2020) End- to-end object detection with transformers. In: Euro- pean conference on computer vision, Springer, p 213–229 Chen J, Kao Sh, He H, et al (2023) Run, don’t walk: Chasing higher flops for faster neural net- works. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, p 12021–12031 Chen Y, Dai X, Chen D, et al (2022a) Mobile-former: Bridging mobilenet and transformer. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, p 5270–5279 Chen Z, Zhong F, Luo Q, et al (2022b) Edgevit: Efficient visual modeling for edge computing. In: International Conference on Wireless Algorithms, Systems, and Applications, Springer, p 393–405 Chu X, Tian Z, Zhang B, et al (2021) Conditional positional encodings for vision transformers. arXiv preprint arXiv:210210882 Dai Z, Liu H, Le QV, et al (2021) Coatnet: Mar- rying convolution and attention for all data sizes. Advances in neural information processing systems 34:3965–3977 Dehghani M, Arnab A, Beyer L, et al (2021) The effi- ciency misnomer. arXiv preprint arXiv:211012894 Deng J, Dong W, Socher R, et al (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee, p 248–255 Dosovitskiy A, Beyer L, Kolesnikov A, et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929 Dunnhofer M, Martinel N, Micheloni C (2021) Weakly-supervised domain adaptation of deep regression trackers via reinforced knowledge dis- tillation. IEEE Robotics and Automation Letters 6(3):5016–5023 Dunnhofer M, Simonato K, Micheloni C (2022) Com- bining complementary trackers for enhanced long- term visual object tracking. Image and Vision Com- puting 122:104448 Dunnhofer M, Furnari A, Farinella GM, et al (2023) Visual object tracking in first person vision. Interna- tional Journal of Computer Vision 131(1):259–283 Fan H, Lin L, Yang F, et al (2019) Lasot: A high- quality benchmark for large-scale single object tracking. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, p 5374–5383 Fan H, Xiong B, Mangalam K, et al (2021) Multi- scale vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, p 6824–6835 22 Fan Q, Huang H, Zhou X, et al (2023) Lightweight vision transformer with bidirectional interaction. Advances in Neural Information Processing Sys- tems 36 Galoogahi HK, Fagg A, Huang C, et al (2017) Need for Speed: A Benchmark for Higher Frame Rate Object Tracking. In: ICCV Ganesh P, Chen Y, Yang Y, et al (2022) Yolo-ret: Towards high accuracy real-time object detection on edge gpus. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p 3267–3277 Gopal GY, Amer MA (2024) Separable self and mixed attention transformers for efficient object tracking. In: Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, p 6708–6717 Gopalkrishnan A, Greer R, Trivedi M (2024) Multi- frame, lightweight & efficient vision-language models for question answering in autonomous driv- ing. arXiv preprint arXiv:240319838 Gupta S, Tan M (2019) Efficientnet-edgetpu: Creating accelerator-optimized neural networks with automl. https://aigoogleblogcom/2019/08/efficientnetedgetpu- creatinghtml Han K, Wang Y, Tian Q, et al (2020) Ghostnet: More features from cheap operations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 1580–1589 Han K, Xiao A, Wu E, et al (2021) Transformer in transformer. Advances in neural information pro- cessing systems 34:15908–15919 He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, p 770–778 Hendrycks D, Gimpel K (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:160608415 Howard A, Sandler M, Chu G, et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision, p 1314–1324 Huang L, Zhao X, Huang K (2019) GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild. IEEE TPAMI Ioffe S (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:150203167 Jiang B, Chen S, Xu Q, et al (2023) Vad: Vectorized scene representation for efficient autonomous driv- ing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 8340–8350 Khan AH, Micheloni C, Martinel N (2024) Idenet: Implicit degradation estimation network for effi- cient blind super resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p 6065–6075 Kirillov A, Girshick R, He K, et al (2019) Panop- tic feature pyramid networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 6399–6408 Kirillov A, Mintun E, Ravi N, et al (2023) Seg- ment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 4015–4026 Kong Y, Fu Y (2022) Human action recognition and prediction: A survey. International Journal of Com- puter Vision 130(5):1366–1401 Krause J, Stark M, Deng J, et al (2013) 3d object representations for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops Kristan M, Leonardis A, Matas J, et al (2020) The eighth visual object tracking vot2020 challenge results LeCun Y, Boser B, Denker JS, et al (1989) Backpropa- gation applied to handwritten zip code recognition. Neural computation 1(4):541–551 Leo M, Medioni G, Trivedi M, et al (2017) Computer vision for assistive technologies. Computer Vision and Image Understanding 154:1–15 Li Y, Yuan G, Wen Y, et al (2022) Efficient- former: Vision transformers at mobilenet speed. 23 Advances in Neural Information Processing Sys- tems 35:12934–12949 Li Y, Hu J, Wen Y, et al (2023) Rethinking vision transformers for mobilenet size and speed. In: Pro- ceedings of the IEEE/CVF International Confer- ence on Computer Vision, p 16889–16900 Lin TY, Maire M, Belongie S, et al (2014) Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part V 13, Springer, p 740–755 Lin TY, Goyal P, Girshick R, et al (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, p 2980–2988 Lin W, Wu Z, Chen J, et al (2023) Scale-aware mod- ulation meet transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 6015–6026 Liu S, Zeng Z, Ren T, et al (2025) Grounding dino: Marrying dino with grounded pre-training for open- set object detection. In: European Conference on Computer Vision, Springer, p 38–55 Liu X, Peng H, Zheng N, et al (2023) Efficientvit: Memory efficient vision transformer with cascaded group attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, p 14420–14430 Loshchilov I, Hutter F (2016) Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint arXiv:160803983 LoshchilovI,HutterF(2017)Decoupled weightdecayregularization.arXivpreprint arXiv:171105101 Ma J, Jiang X, Fan A, et al (2021) Image matching from handcrafted to deep features: A survey. Inter- national Journal of Computer Vision 129(1):23–79 Ma N, Zhang X, Zheng HT, et al (2018a) Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In: Proceedings of the European con- ference on computer vision (ECCV), p 116–131 Ma N, Zhang X, Zheng HT, et al (2018b) Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In: Proceedings of the European con- ference on computer vision (ECCV), p 116–131 Ma X, Dai X, Yang J, et al (2024) Efficient modulation for vision networks. arXiv preprint arXiv:240319963 Mahendran JK, Barry DT, Nivedha AK, et al (2021) Computer vision-based assistance system for the visually impaired using mobile edge artificial intel- ligence. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, p 2418–2427 Matthies L, Maimone M, Johnson A, et al (2007) Computer vision on mars. International Journal of Computer Vision 75:67–92 Mehta S, Rastegari M (2021) Mobilevit: light-weight, general-purpose, and mobile-friendly vision trans- former. arXiv preprint arXiv:211002178 Mehta S, Rastegari M (2022) Separable self-attention for mobile vision transformers. arXiv preprint arXiv:220602680 Mueller M, Smith N, Ghanem B (2016) A Benchmark and Simulator for UAV Tracking. In: ECCV Munir M, Avery W, Marculescu R (2023) Mobile- vig: Graph-based sparse attention for mobile vision applications. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR) Workshops, p 2211–2219 Nilsback ME, Zisserman A (2008) Automated flower classification over a large number of classes. In: Indian Conference on Computer Vision, Graphics and Image Processing Noman M, Ghallabi WA, Najiha D, et al (2022) Avist: A benchmark for visual object tracking in adverse visibility. arXiv preprint arXiv:220806888 Nottebaum M, Dunnhofer M, Micheloni C (2025) Lowformer: Hardware efficient design for convo- lutional transformer backbones. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 24 Paissan F, Ancilotto A, Farella E (2022) Phinets: a scalable backbone for low-power ai at the edge. ACM Transactions on Embedded Computing Sys- tems 21(5):1–18 Parkhi OM, Vedaldi A, Zisserman A, et al (2012) Cats and dogs. In: 2012 IEEE conference on computer vision and pattern recognition, IEEE, p 3498–3505 Qin D, Leichner C, Delakis M, et al (2025) Mobilenetv4: Universal models for the mobile ecosystem. In: European Conference on Computer Vision, Springer, p 78–96 Sandler M, Howard A, Zhu M, et al (2018) Mobilenetv2: Inverted residuals and linear bottle- necks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, p 4510– 4520 Schall K, Barthel KU, Hezel N, et al (2022) Gpr1200: a benchmark for general-purpose content-based image retrieval. In: International Conference on Multimedia Modeling, Springer, p 205–216 Si C, Yu W, Zhou P, et al (2022) Inception trans- former. Advances in Neural Information Processing Systems 35:23495–23509 Suzuki T, Aoki Y (2024) Retinavit: Efficient visual backbone for online video streams. Sensors 24(17):5457 Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: Inter- national conference on machine learning, PMLR, p 6105–6114 Tan M, Le Q (2021) Efficientnetv2: Smaller models and faster training. In: International conference on machine learning, PMLR, p 10096–10106 Tan S, Nagarajan T, Grauman K (2023) Egodistill: Egocentric head motion distillation for efficient video understanding. Advances in Neural Informa- tion Processing Systems 36:33485–33498 Tang Y, Han K, Guo J, et al (2022) Ghostnetv2: Enhance cheap operation with long-range atten- tion. Advances in Neural Information Processing Systems 35:9969–9982 Touvron H, Cord M, Douze M, et al (2021) Train- ing data-efficient image transformers & distillation through attention. In: International conference on machine learning, PMLR, p 10347–10357 Tu Z, Talebi H, Zhang H, et al (2022) Maxvit: Multi- axis vision transformer. In: European conference on computer vision, Springer, p 459–479 Upmanyu M, Namboodiri AM, Srinathan K, et al (2009) Efficient privacy preserving video surveil- lance. In: 2009 IEEE 12th international conference on computer vision, IEEE, p 1639–1646 Vasu PKA, Gabriel J, Zhu J, et al (2023a) Fastvit: A fast hybrid vision transformer using structural repa- rameterization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, p 5785–5795 Vasu PKA, Gabriel J, Zhu J, et al (2023b) Mobileone: An improved one millisecond mobile backbone. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, p 7907–7917 Vaswani A, Shazeer N, Parmar N, et al (2017) Atten- tion is all you need. Advances in neural information processing systems 30 Wang A, Chen H, Lin Z, et al (2024) Repvit: Revis- iting mobile cnn from vit perspective. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p 15909– 15920 Wang S, Li BZ, Khabsa M, et al (2020) Linformer: Self-attention with linear complexity. arXiv preprint arXiv:200604768 Wang W, Xie E, Li X, et al (2021) Pyramid vision transformer: A versatile backbone for dense predic- tion without convolutions. In: Proceedings of the IEEE/CVF international conference on computer vision, p 568–578 Wang W, Xie E, Li X, et al (2022) Pvt v2: Improved baselines with pyramid vision transformer. Compu- tational Visual Media 8(3):415–424 Wen J, Zhu Y, Li J, et al (2024) Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation. arXiv preprint 25 arXiv:240912514 Wu H, Xiao B, Codella N, et al (2021) Cvt: Introduc- ing convolutions to vision transformers. 2103.158 08 Wu Y, Lim J, Yang MH (2015) Object tracking bench- mark. IEEE TPAMI Xu W, Xu Y, Chang T, et al (2021) Co-scale conv- attentional image transformers. In: Proceedings of the IEEE/CVF international conference on com- puter vision, p 9981–9990 Yan B, Peng H, Fu J, et al (2021) Learning spatio- temporal transformer for visual tracking. In: Pro- ceedings of the IEEE/CVF international conference on computer vision, p 10448–10457 Yu W, Luo M, Zhou P, et al (2022) Metaformer is actu- ally what you need for vision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 10819–10829 Yuan K, Guo S, Liu Z, et al (2021) Incorporating con- volution designs into visual transformers. In: Pro- ceedings of the IEEE/CVF international conference on computer vision, p 579–588 Yun S, Ro Y (2024) Shvit: Single-head vision trans- former with memory efficient macro design. arXiv preprint arXiv:240116456 Yun S, Lee D, Ro Y (2024) Metamixer is all you need. arXiv preprint arXiv:240602021 Zhai X, Kolesnikov A, Houlsby N, et al (2022) Scal- ing vision transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, p 12104–12113 Zhang Z, Lu X, Cao G, et al (2021) Vit-yolo: Transformer-based yolo for object detection. In: Proceedings of the IEEE/CVF international confer- ence on computer vision, p 2799–2808 Zheng C, Wu W, Chen C, et al (2023) Deep learning- based human pose estimation: A survey. ACM Computing Surveys 56(1):1–37 Zhou B, Zhao H, Puig X, et al (2017) Scene pars- ing through ade20k dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Zhu L, Wang X, Ke Z, et al (2023) Biformer: Vision transformer with bi-level routing attention. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, p 10323–10333 26