Paper deep dive
Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations
Wenxuan Miao, Haosong Liu, Weiming Hu, Zihan Liu, Aiyue Chen, Jianlin Yu, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, Yu Feng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/16/2026, 7:10:08 AM
Summary
The paper introduces Kaleido, an algorithm-hardware co-design for accelerating Video Diffusion Transformers (vDiTs). It proposes a lightweight channel-wise reuse algorithm that exploits spatiotemporal correlations in latent space (induced by Rotary Positional Encoding) to skip redundant self-attention computations. The accompanying hardware features a reconfigurable systolic array accelerator and a data dispatcher to handle irregular sparsity, achieving up to 5.9x speedup and 16.0x energy savings over state-of-the-art accelerators.
Entities (10)
Relation Signals (10)
Kaleido → accelerates → Video Diffusion Transformers (vDiTs)
confidence 98% · Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs
Kaleido → achieves → 5.9x speedup
confidence 98% · Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators
Kaleido → implements → Channel-wise reuse algorithm
confidence 97% · we propose a lightweight channelwise reuse algorithm that skips redundant computations
Channel-wise reuse algorithm → optimizes → Self-attention
confidence 97% · accelerates not only self-attention but also other operations in vDiTs
Channel-wise reuse algorithm → exploits → Spatiotemporal correlation
confidence 96% · exploiting channel-wise spatiotemporal correlations in latent space
Kaleido → implements → Systolic array accelerator
confidence 96% · we design a systolic array like accelerator with reconfigurable processing elements
Systolic array accelerator → contains → Data dispatcher
confidence 95% · systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher
Systolic array accelerator → contains → Reconfigurable processing element
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
Tags
Links
- Source: https://arxiv.org/abs/2607.13770v1
- Canonical: https://arxiv.org/abs/2607.13770v1
Trouble viewing inline? Open PDF directly →
Full Text
102,035 characters extracted from source content.
Expand or collapse full text
Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations Wenxuan Miao Shanghai Jiao Tong University Shanghai, China miaowenxuan@sjtu.edu.cn Haosong Liu Shanghai Jiao Tong University Shanghai, China 2436824987@sjtu.edu.cn Weiming Hu Shanghai Jiao Tong University Shanghai, China weiminghu@sjtu.edu.cn Zihan Liu Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China altair.liu@sjtu.edu.cn Aiyue Chen Huawei Technologies Shanghai, China chenaiyue@huawei.com Jianlin Yu Huawei Technologies Shanghai, China yujianlin1@huawei.com Yiwu Yao Huawei Technologies Shanghai, China yiwuyao@pku.edu.cn Yiming Gan ICT, Chinese Academy of Sciences Beijing, China ganyiming@ict.ac.cn Jieru Zhao Shanghai Jiao Tong University Shanghai, China zhao-jieru@sjtu.edu.cn Jingwen Leng Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China leng-jw@sjtu.edu.cn Minyi Guo Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China guo-my@cs.sjtu.edu.cn Yu Feng ∗ Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China y-feng@sjtu.edu.cn Abstract Video diffusion transformers (vDiTs) generate high-quality video but introduce extremely high compute cost due to the long diffusion timesteps and self-attention computation. As diffusion timesteps are reduced, the computation cost of self-attention becomes the domi- nant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatio-temporal correlation of video data. This paper presents Kaleido, an algorithm–hardware co-design that accelerates all operations in vDiTs by exploiting channel-wise spatio-temporal correlations in latent space. Based on this insight, we propose a lightweight channel-wise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic-array-like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three main- stream vDiT models show that Kaleido achieves up to 5.9×speedup and 16.0× energy savings over state-of-the-art accelerators. Keywords Video Diffusion Acceleration, Algorithm-Hardware Co-Design 1 Introduction Large video generative models [16,23,36,39,57,61,63,64,69,72, 73,79,90,97] have drawn tremendous attention from both academia ∗ Corresponding Author. and industry in the last two years. Their ability to generate high- fidelity videos has led to rapid adoption in movie editing [17,80], advertisement [1,32], virtual world creation [8,27], and more. Be- yond their substantial economic potential, these models also offer a unique lens to understand the physical world by learning geometric and causal relationships in natural scenes. Thus, they are regarded as a foundation for general-purpose artificial intelligence [60]. Today’s mainstream large video generative models are predom- inantly based on video diffusion transformers (vDiTs). Similar to other diffusion models, a vDiT progressively reverses a forward noise-adding process, iteratively removing latent noises until a video is clean. However, this iterative denoising process is compute- intensive. For example, generating even a short 5-second 720p video clip with a state-of-the-art vDiT, HunyuanVideo [35], can take over 30 minutes on a single Nvidia H100 GPU with 80 GB of memory. This inefficiency stems from two fundamental factors. First, the diffusion process consists of many denoising timesteps. Each re- quires a full forward inference, resulting in long inference latency. Second, each inference performs compute-intensive self-attention over all spatio-temporal tokens, making attention the dominant bottleneck as video resolutions or lengths grow. Recently, the com- munity has made tremendous progress on reducing the number of timesteps via distillation [18,38,62,91], caching [5,43,50,75, 96,100], etc. As the number of timesteps decreases, self-attention computation becomes the primary bottleneck (Fig. 4). In this paper, we address the compute-intensive self-attention in vDiTs via an algorithm–hardware co-design, Kaleido. Unlike prior work [21,77,83,87,88], which largely inherits sparse self-attention optimizations from large language models (LLMs), we introduce a 1 arXiv:2607.13770v1 [cs.AR] 15 Jul 2026 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. key insight that exploits the spatio-temporal similarity among latent tokens to dramatically reduce attention computation by up to 85%. The key is to dissect and understand the various patterns of self- attention scores that exist across all vDiT models [18,35,73,84,91, 97]. Our experiments show that the cause of these patterns is from the channel-wise position encoding used in individual token channels (Sec. 3.2). This encoding mechanism naturally induces spatial and temporal correlations, which accumulate across channels and lead to various patterns in attention scores. With our key insight, we introduce our channel-wise reuse al- gorithm in Sec. 4, which accelerates not only self-attention but also other operations in vDiTs. Our algorithm simply serves as a plug-in that identifies the correlations between tokens before each operation, e.g., self-attention, and skips unnecessary computations by reusing previous partial results. Specifically, our algorithm first evaluates the similarity of adjacent tokens at the token channel level to identify values that can safely share partial computations. During actual operation, channels with high similarity skip explicit computation and reuse previously computed partial results instead. Evaluations in Sec. 7 show that our algorithm saves significant com- putations while retaining much higher generative quality (>17dB) compared to prior algorithm-hardware co-designs [21, 34, 87]. While our algorithm significantly reduces overall computation, it introduces two key challenges. First, its reuse patterns are inher- ently incompatible with the dataflow of existing accelerators, e.g., systolic arrays. Second, reuse further induces irregular sparsity in the input data. To address these challenges, we co-design a reconfig- urable processing element (PE) for systolic arrays, as described in Sec. 5.2. Our PE can be reconfigured into multiple execution modes and supports various dataflows, so that PEs can reuse the previously computed results without repeatedly loading them from on-chip memory. In addition, it incorporates mixed-precision computation to support computations with partially reused sparse inputs. To further mitigate the irregular data access patterns introduced by our reuse algorithm, we propose a data dispatcher in Sec. 5.3. The dispatcher applies a lightweight online clustering to group tokens with similar reuse patterns. Meanwhile, a runtime lookahead buffer is designed to merge compatible channels before feeding them into the PE array. Together, we show that our data dispatcher further improves PE utilization and overall accelerator efficiency (Sec. 5.3). We implement our algorithm-hardware co-design, Kaleido. The hardware builds on top of a classic systolic DNN accelerator [31] implemented in 16 nm technology. The Kaleido hardware aug- ments the baseline accelerator with 7.9% area overhead. We evalu- ate Kaleido on four widely-adopted vDiTs: HunyuanVideo [35], Wan [73], CogVideoX[84]and TurboDiffusion [91]. In addition to the GPU baseline, we also include four recent diffusion accelera- tors [21,33,34,87]. Kaleido achieves up to 5.9×speedup and 16.0× energy savings. The contributions of this paper are as follows: • We are the first to systematically characterize the large vDiT models and explain the spatio-temporal correlations in the latent space of large vDiTs. •We propose a lightweight channel-wise reuse algorithm that significantly reduces computation while achieving higher generative quality than prior methods. Encode Φ(x’T, T) Decode Tokens, x’T、 @Timestep T Noisy Video, Vin Denoised Video, Vout Φ(x’1, 1) Tokens x’0 @Timestep 0 ... Reverse Markov Process - z’T- z’1 ... () 0.80.1 ... () 0.60.2 Fig. 1: An example of a diffusion process. The random-noised video input,푉 푖푛 , is first encoded into the latent space and converted into a sequence of tokens,푥 ′ 푇 . These tokens are then processed by the same diffusion model multiple times to predict the noises,푧 ′ 푡 , at each timestep,푡. After completing 푇timesteps, the final denoised tokens,푥 ′ 0 , are decoded back into a final output video,푉 표푢푡 , in RGB color space. Wan2.1 Hunyuan Opensora OpensoraPlan LTX-Video CogVideoX Wan2.2 Longcat Average 0 20 40 60 80 100 Normalized Exe. Time (%) Self-attnCross-attn (a) Nvidia H100 with HBM [56]. Wan2.1 Hunyuan Opensora OpensoraPlan LTX-Video CogVideoX Wan2.2 Longcat Average 0 20 40 60 80 100 Normalized Exe. Time (%) MLP Others (b) Nvidia RTX PRO 6000 [54]. Fig. 2: The execution breakdown of eight mainstream vDiTs [18,23,35,39,70,73,97] on two recent Nvidia GPUs [54, 56]. Self-attention dominates the overall execution. •We introduce a reconfigurable accelerator architecture to support our reuse algorithm and a tailored data dispatcher to mitigate irregular sparsity and improve PE utilization. •Our architecture achieves up to 5.9×speedup and 16.0× energy savings over existing accelerator designs. 2 Background Diffusion Models. The most widely adopted video generation paradigm is the diffusion model, which learns to generate data, i.e., images or videos, from random Gaussian noise through a reverse Markov process [9, 22, 81], as shown in Fig. 1. In this process, the initial input is a random Gaussian noise,푉 푖푛 , which is encoded into tokens,푥 ′ 푇 , in latent space. Here,푇is the total number of denoising timesteps. Given noisy tokens푥 ′ 푇 , the diffusion model restores the original data by gradually predicting and removing noise푧 ′ 푡 from푥 ′ 푡 at each timestep푡. After a fixed number of denoising timesteps, the final result of the diffusion model,푥 ′ 0 , would be close enough to the original data푥 0 in latent space. The mathematical expression of each denoising timestep is, 푥 ′ 푡−1 = 훼 푡 (푥 ′ 푡 − 훽 푡 푧 ′ 푡 )+ 휎 푡 푛 ′ 푡 , and 푧 ′ 푡 =Φ(푥 ′ 푡 ,푡),(1) whereΦis the prediction function, i.e., the diffusion model, that predicts the noise푧 ′ 푡 . Both훼 푡 and훽 푡 are hyper-parameters.휎 푡 푛 ′ 푡 is a renoising term to add randomness to the denoising process. Lastly, we decode푥 ′ 0 from the latent space to obtain the output video,푉 표푢푡 . Current mainstream diffusion models in video generation often require 30 to 100 denoising steps to generate a short video [23,35, 39,69,70,73,84,97]. Recent studies also reduce to shorter timesteps, 4-10, via distillation [44, 91], finetuning [18, 41, 42], etc. Video Diffusion Transformer. A typical vDiT architecture first encodes input and prompt tokens into latent space with푑channels. 2 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA 10 0 10 1 10 2 10 3 10 4 Arithmetic Intensity 10 0 10 1 10 2 Performance (TFlops) H100 Roofline Peak Performance (204.9 TFLOPs) Ridge Point (100.44 FLOPs/B) Wan2.1 HunyuanVideo Opensora OpensoraPlan LTX-Video CogvideoX Wan2.2 Longcat-video (a) Nvidia H100 with HBM [56]. 10 0 10 1 10 2 10 3 10 4 Arithmetic Intensity 10 0 10 1 10 2 Performance (TFlops) RTX PRO 6000 Roofline Peak Performance (126.0 TFLOPs) Ridge Point (70.39 FLOPs/B) Wan2.1 HunyuanVideo Opensora OpensoraPlan LTX-Video CogvideoX Wan2.2 Longcat-video (b) Nvidia RTX PRO 6000 [54]. Fig. 3: The roofline analyses of eight vDiT models [18,23,35, 39,70,73,97] on two recent Nvidia GPUs [54,56]. All vDiT models are primarily compute-bound. Then, for each timestep, those encoded tokens go through a stack of computational blocks. Each block is a combination of a self- attention, a cross-attention, and a MLP layer. Fig. 2 profiles the execution time of eight popular vDiT models [18,23,35,39,70,73, 97]. Overall, self-attention accounts for 68% of total execution time. Self-Attention. The attention mechanism is the main reason that self-attention is the major computational bottleneck in vDiTs. Given input tokens 푋 in ∈R 푁×푑 , self-attention computes, Attention(푄,퐾,푉)= Softmax 푄퐾 푇 √ 푑 푘 푉,(2) where푄,퐾,푉are query, key, and value, respectively.푁is the token sequence and푑is the channel dimension. These three token se- quences have the same shape as the input tokens and are obtained by performing linear projections on푋 in .푃= 푄퐾 푇 ∈R 푁×푁 is the atten- tion map. Overall, the computational complexity of self-attention is proportional to푁 2 ×푑. Thus, it is compute-intensive. Fig. 3 shows the roofline analyses of vDiTs on Nvidia H100 [56] and Nvidia RTX PRO 6000 [54]. Under two different memory technologies, HBM2e and GDDR7, vDiT models are still compute-bound. Here, Open- sora [97] is not compute-bound because it uses temporal-spatial separate attention, which is no longer used in recent vDiTs. Position Encoding. Next, we explain one of the key concepts in vDiT. The reason that vDiT models can capture spatial-temporal information is that all vDiTs apply rotary positional embedding (RoPE) [23, 39, 72, 73] to encode relative token positions, RoPE([푥,푦])=(cos휃 · 푥 − sin휃 ·푦, sin휃 · 푥 + cos휃 ·푦), (3) where휃 푝,푖 = 휑/10000 2푖/푑 . Importantly, the푑channels are often partitioned into three segments: initial channels encode temporal information, while the remaining channels capture spatial informa- tion along푥and푦axes. This structured encoding causes different channel groups to exhibit unique spatio-temporal patterns. 3 Motivation 3.1 Trends and Challenges of vDiT Models As mentioned in Sec. 2, the inference cost of a vDiT is primarily dominated by two factors: the number of denoising timesteps and the computation of self-attention. While both factors have histori- cally posed major bottlenecks, recent studies [5,7,18,41,42,91,96, 100] exploit various techniques to leverage the similarity between timesteps and reduce the number of timesteps by approximating OpenSora v1.2 OpenSoraPlan v1.2 CogVideoX Mochi HunyuanVideo Wan2.1 Step-Video LTX-Video Wan2.2 TurboDiffusion Fig. 4: The ratio between token sequence length and the num- ber of diffusion timesteps in vDiTs [18,35,39,49,69,73,84, 91,97] has steadily increased over the years. It means that attention computation grows to be the only bottleneck. intermediate results. As these techniques become widely adopted, the diffusion timestep is no longer a dominant bottleneck. In contrast, self-attention remains a key bottleneck, especially as recent vDiTs target longer video durations and higher resolutions [4, 25,44,70,71,91]. Both trends translate into longer token sequences, causing a quadratic increase in self-attention cost. As shown in Fig. 4, we show that the ratio between the token sequence and the number of timesteps across mainstream vDiTs increases over time. This means that accelerating self-attention will be the key to improving the performance of future vDiTs. While a substantial body of prior studies [10,11,20,48,74,76, 85,92] have studied sparse computation to accelerate self-attention in LLMs, these approaches primarily exploit attention sparsity to eliminate insignificant computation. Several recent works on image and video diffusion models [21,77,87,88] have adopted similar ideas. However, as we show in Sec. 7.1, directly transferring LLM- based sparse attention techniques to vDiTs is both ineffective and fundamentally mismatched to the characteristics of vDiT models. Sparse attention is intuitive for LLMs because text sequences naturally exhibit discrete, hierarchical semantics. Thus, many long- range irrelevant dependencies can be safely pruned without any impact on generation quality. Unlike text tokens, video tokens are dense and carry continuous visual semantics. Every token con- tributes to spatial coherence and temporal consistency, especially in early denoising steps where global information must be retained to achieve consistent results [5,77,96]. Our experiment in Sec. 7.1 shows that naively removing attention correlations leads to motion discontinuities and flickering artifacts. Thus, rather than relying on sparsity patterns in LLMs, effective acceleration of self-attention in vDiTs should exploit their unique spatio-temporal correlations. 3.2 Spatio-Temporal Correlations in vDiTs In this subsection, we first present the attention patterns that com- monly exist in vDiTs, and then explain the underlying causes that induce these spatio-temporal correlations in their latent space. Spatio-Temporal Patterns. In Fig. 5, we show two representa- tive self-attention patterns from HunyuanVideo [35], other vDiT models have similar patterns. Also, the patterns in Fig. 5 are consis- tent with patterns reported in prior studies [12,77,78,89]. Overall, the patterns in self-attention maps of vDiTs can be categorized into two types: spatial patterns and temporal patterns. An example of a spatial pattern is shown on the left side of Fig. 5. Spatial patterns consist of small, seemingly “repetitive” tiles spanning across the entire attention map,푃. Each tile captures the spatial correlations among tokens within a single frame, while 3 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. Spatially Vary Temporally Uniform Temporally Vary Spatially Uniform Channel Key One Frame Query Fig. 5: Example of various attention patterns, which can be classified into spatial patterns and temporal patterns. Spa- tial patterns capture the spatial correlations within a sin- gle frame by repetitive “tiles”, whereas temporal patterns capture the temporal correlations across frames via strong diagonal correlations. Both patterns arise from the RoPE encoding and the dominant channels in query푄and key퐾. Table 1: The channel partitioning of different vDiT mod- els [35,39,70,73,84,91]. vDiT models apply RoPE encoding and partition the token channels into t-, x-, and y-dimension. HunyuanVideoWan2.1TurboDiffusionCogVideoXLongCatOpenSoraPlan T-dimension164444324432 X-dimension564242484232 Y-dimension564242484232 the patterns across different tiles remain largely unchanged. Thus, spatial patterns mainly capture spatial relationships within a frame. In contrast, temporal patterns, shown on the right side of Fig. 5, emphasize correlations across frames. In these patterns, the varia- tions of attention scores within a single frame are less significant, whereas inter-frame correlations are more important. In particular, temporally adjacent frames tend to exhibit strong mutual attention, resulting in a diagonal correlation in the attention map. Underneath Mechanism. While prior studies [77,83] also ob- served these patterns in vDiTs, they do not further investigate the underlying cause of these patterns. Existing approaches simply leverage these patterns to accelerate vDiTs via sparse computations of attention maps using the techniques from LLMs. However, with- out a fundamental understanding of the underlying mechanisms, these heuristics often lead to low generative quality. In this work, we take the first step to explain the causes of various attention patterns and propose a principled acceleration technique tailored for vDiTs. Our analysis shows that the domi- nant driver of those patterns is the channel-wise encoding of tokens, i.e., RoPE. As introduced in Sec. 2, RoPE is the primary positional encoding technique in vDiTs, where different channel groups en- code positional information at different frequencies. Specifically, vDiT models partition their channel dimensions into three distinct groups: the temporal (t) dimension, the horizontal (x) dimension, and the vertical (y) dimension, as shown in Fig. 6. Each channel group captures information along a specific direction in latent video data. Tbl. 1 lists the channel partitioning of vDiT models. Meanwhile, we find that the dominant channels with higher values have a higher chance to contribute more to the attention scores in Fig. 5. Here, the bounding boxes on푄and퐾highlight individual frames. Because RoPE causes the dominant channels to Frame 0Frame 1 01 2 3 4 6 5 7 t-dimension x-dimension y-dimension reuse t-dimension groups along t-dimension reuse x-dimension groups along x-dimension reuse y-dimension groups along y-dimension One Row Token Fig. 6: An example of how different channel groups gov- ern the final generation quality. The upper part illustrates how we reuse different channel groups shown by red arrows. Three channels represent t-, x-, and y-dimension channel groups. The lower part shows results after reusing. 1520253035404550 Diffusion Timestep 10 3 10 4 10 5 MSE Loss Reuse along Temporal (a) T-channel group. 1520253035404550 Diffusion Timestep 10 3 10 4 10 5 MSE Loss Reuse along Horizontal (b) X-channel group. 1520253035404550 Diffusion Timestep 10 3 10 4 10 5 MSE Loss Reuse along Vertical (c) Y-channel group. Fig. 7: The MSE loss introduced by reusing different direc- tions on different channel groups. For a given channel group, reuse along the other two dimensions leads to lower MSE loss than reuse along its own direction. have different frequencies, leading to various patterns in 푄 and 퐾 . Different combinations of channels in푄and퐾form various pat- terns in attention maps. As shown in Fig. 5, when spatial channels dominate, attention maps exhibit “spatial-like” patterns. Otherwise, attention maps exhibit “temporal-like” patterns. Theory Verification. To verify our theory, we design an ex- periment that purposely manipulates different channel groups and examines their impact on the final video quality. As shown in Fig. 6, for each channel group, we group pairs of adjacent tokens along the corresponding dimension and force the second token in each pair to reuse the value of the first token across all denoising steps. For example, for the temporal channel group, we allow the t-dimension channels of every second frame to reuse those of the first frame at the same coordinate, while leaving the x- and y-channel groups unchanged, and vice versa. Fig. 6 illustrates how we reuse differ- ent channel groups with a toy example of eight input tokens (two frames with 2× 2 tokens per frame). The bottom part of Fig. 6 shows some results after applying our reuse method. As shown, reusing t-dimension channels introduces temporal distortions, whereas reusing x- or y-dimension channels produces stripe-like artifacts aligned with the corresponding direc- tion. This shows that different channel groups govern different di- mensions of information. Overall, all used frames are much blurrier than the baseline because the self-attention mechanism integrates global contributions across all tokens. 4 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA Kin Calculate Similarity Frame 0 Frame 1 1.3 5.4 2.1 3.2 0.1 0.2 2.4 3.2 2.2 3.6 1.2 6.4 1.3 5.3 4.4 0.6 1.1 2.3 3.4 4.3 0.7 1.5 5.3 Qin Channels 5.6 2.5 3.4 0.1 3.2 3.3 3.2 2.6 3.2 0.2 3.2 4.3 5.4 2.4 3.3 2.4 1.3 1.1 2.5 3.1 4.5 1.4 1.2 2.3 txy 2.4 00.10.10.2 1.35.42.43.2 1.35.32.33.4 y: reuse in t x: reuse in y t: reuse in x 2.13.2 2.23.6 4.40.6 4.30.7 0.10.40.10.1 05.20.40.3 0.10.10.10.1 0.102.10.1 0.11.10.10.1 0.11.21.15.3 0.26.41.55.6 2.53.4 0.13.2 3.3 3.2 2.63.2 0.23.2 4.3 5.4 2.41.3 1.1 4.51.4 1.2 2.3 2.4 2.43.32.53.1 .4 3.2 2.4 5.4 1.3 0.6 4.4 .6 3.2 2.1 .6 5.3 .5 1.1 6.4 1.2 0.1 0.10.21.26.41.11.55.35.6 2.13.22.23.64.40.64.30.7 1.35.42.43.21.35.32.33.4 2.53.42.63.22.43.32.53.1 0.13.20.23.22.41.34.51.4 3.33.24.35.41.11.22.32.4 y-channel: 0.13.22.41.34.5 3.44.35.41.12.3 2.53.42.63.2 t x y mask: mask: mask: x-channel: t-channel: y-channel: mask: mask: mask: x-channel: t-channel: Compute Partial Reuse Reuse Partial Attention Scores Mask GenerationCompute Partial Attention Full-bit Computation Half-bit Computation Skipped Computation .X Lower-bit Data Reuse & Aggregate Final Attention Scores 100% Compute 100% Reuse Frame 0 Frame 1 y: reuse in t x: reuse in y t: reuse in x 1 Reshaped Q and K Similarity Scores 234 Row Fig. 8: An overview of our channel-wise reuse algorithm, which consists of four steps. We first compute the similarity between adjacent tokens along a selected axis. Based on this similarity, a subset of tokens could reuse previously computed partial attention scores, thus reducing the overall attention computation.휃 th1 and휃 th2 are set to be 0.1 and 1 for illustration purposes. Fig. 7 further quantifies the mean square error (MSE) loss be- tween the baseline results and the results generated with different channel reuses. The x-axis shows the diffusion timesteps at which we apply the reusing strategy. For every channel group, we evaluate reuse along three different dimensions: t, x, and y. The result shows that, for a given channel group, reuse is less harmful when applied along the other two dimensions, rather than along the dimension that the channel group encodes. For instance, for the t-dimension channel group, applying reuse along the x- or y-dimension results in lower MSE than reusing along the t-dimension. This result fur- ther confirms that each channel group predominantly captures information specific to its corresponding dimension. 4 Channel-Wise Reuse Algorithm With the major finding of the spatio-temporal correlations in Sec. 3, we introduce our channel-wise reuse algorithm to accelerate all computation blocks in vDiTs. Here, we first provide an overall idea of our algorithm in Sec. 4.1 and then explain the rationale and design decisions behind our algorithm in Sec. 4.2. 4.1 Idea Sec. 3 shows that the spatio-temporal patterns in attention maps are governed by dominant token channels, and each channel has a unique frequency governed by RoPE encoding (see Fig. 5). Here, we show how to leverage this unique insight to accelerate the operations in vDiTs via reusing intermediate results. Our algorithm assumes an 8-bit fixed-point numerical representation, which is widely adopted in diffusion accelerators [33,34]. Here, we use self- attention to demonstrate our idea first. Other compute blocks, i.e., across-attention and MLP, can be accelerated in a similar fashion. Self-Attention. Our overall process to accelerate self-attention is shown in Fig. 8, which consists of four steps. 1Given the mathematical expression of self-attention in Eqn. 2, we first compute the similarity of adjacent tokens in the query푄 and key퐾. Later, these results will guide our reuse algorithm. As shown in Fig. 7, for a given channel group, it is better to reuse the channel values along the other two directions. For example, chan- nels belonging to the t-channel group have lower MSE loss when reusing tokens along x- or y-directions. To better co-design with our architecture, we further restrict each channel group to reuse along one fixed direction. E.g., in Fig. 7, t-, x-, and y-channels are restricted to reuse x-, y-, and t-directions, respectively. At runtime, the token similarity is computed along the predefined direction. The similarityΔ is measured using the absolute difference, Δ(푥 푎 ,푥 푏 )=|푥 푎 − 푥 푏 |,(4) where푥 푎 and푥 푏 are single-channel values of two adjacent tokens along a specific direction. For instance, along the t-direction, we compute the similarityΔ between every two adjacent frames. 2We then check whether the token similarityΔis below our predefined thresholds,휃 th1 and휃 th2 . Here,휃 th1 < 휃 th2 . IfΔis below 휃 th1 , then the second token푥 푏 is marked as “reuse”, i.e., it can com- pletely reuse the partial attention score of the first token푥 푎 . IfΔis below휃 th2 , the second token푥 푏 is then marked as “partial reuse”, i.e., it can partially reuse the attention score of 푥 푎 . 3Once we identify reusable tokens in each channel group, we start to compute the partial attention scores. Here,푄and퐾perform matrix multiplication channel by channel. If any tokens in푄and 퐾are marked as “reuse”, they will not perform computation and instead reuse previously computed results, as the white blocks shown in Fig. 8. If any tokens in푄and퐾are marked as “partial reuse”, they would perform half multiplication. Consider a fixed- point representation, we could reuse the higherℎbits multiplication from the previous token and only compute the subsequent 4 bits, as the half-colored blocks shown in Fig. 8. 4Finally, all partial attention scores from all channels are ag- gregated to calculate the final attention scores푃. Once the final attention scores are obtained, operations, e.g., Softmax, are per- formed the same as in the canonical self-attention. Lastly,푃 ×푉is performed similarly to푄× 퐾 푇 . However, we only perform a reuse 5 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. check on푉, not푃. Because only푉has spatio-temporal correlations among its tokens. The reuse process of푉 is similar to 푄 and 퐾 . Other Operations. In addition to self-attention, other operations in vDiTs can also benefit from a similar reuse strategy. However, unlike self-attention, for other operations such as MLPs, reuse can only be applied to activations rather than weights. This is because only activations inherit the spatio-temporal correlations, whereas weights do not. Thus, for these operations, we modify step1of our reuse algorithm; other steps remain largely unchanged. In step1, we first compute the similarity of activations on a per–channel-group basis. Similar to self-attention, all channels in each channel group are restricted to reuse along one direction. However, since only one input, e.g., activation, has spatial-temporal correlations, we propose a window-based comparison, rather than comparing two adjacent frames, to increase the reuse ratio. The similarityΔ is measured as, Δ(푎)= v u t 푤−1 ∑︁ 푖=0 (푎 푖 − ̄ 푎) 2 /푤, ̄ 푎= 푤−1 ∑︁ 푖=0 푎 푖 /푤, (5) where푎is a window of activation elements along one channel.푤 is the window size.Δ(푎)determines whether all elements within this window can be reused or partially reused. In Sec. 5.2, we will show that this window-based design can also map naturally onto our co-designed PE array. 4.2 Design Decisions Next, we explain several key design rationales in our algorithm. Why Reuse? We first explain why our reuse technique is better than conventional sparse attention techniques [21,77,83]. Recall, sparse attention exploits the insignificance of푄퐾 푇 values by di- rectly skipping their attention computations. In Fig. 9, we compare the MSE loss of our method against two sparse attention base- lines under the same token-saving ratio,휃 ∗ =85%, on Hunyuan- Video [72]. The first baseline skips 85% of tokens with the lowest values during the attention computation. The second baseline uses the same selection criteria as our reuse method to identify tokens, but instead of reusing previously computed results, it skips their attention computations. The results show the output MSE between the original model and different compute-saving techniques. Re- sults show that our technique achieves an order of magnitude lower MSE loss compared to these two baselines. This shows that our technique is more effective than sparse computation methods. Why Channel? Next, we explain why our reuse strategy oper- ates at the channel level rather than the token level, as proposed by prior works [2,87]. Prior works target high-level tasks, like classifi- cation. For those tasks, aggregating token information is acceptable. However, video generation requires preserving dense and smooth content at the pixel level. Merging or reusing at the token level often leads to noticeable quality degradation. Thus, it is intuitive to reuse at the root cause, the channel level. Moreover, different chan- nel groups encode distinct spatio-temporal information. Reusing them at the token level inevitably loses certain information. Sec. 7.1 shows that our channel-level reuse preserves much higher quality against other token-level approaches [2, 87]. Thresholds. The next question is how to determine the ap- propriate thresholds,휃 th1 and휃 th2 . Our key observation is that Ours Sparse Baseline1 Sparse Baseline2 10 3 10 4 10 5 10 6 MSE Loss Fig. 9: MSE comparison be- tween our reusing method and two skipping techniques. We pick 20 random prompts. 01020304050 Diffusion Timestep 10 3 10 4 MSE Loss IQR Fig. 10: The sensitivity of our reuse technique accuracy to different prompts. Here, we show 20 random prompts. vDiT AcceleratorDRAM Input Tokens Linear Layer Weights Global Buffer Data Dispatcher Activa‐ tions 4x4 PE Array Data Dispatcher 4x4 PE Array ... Vector Unit DMA I/O Accumulation Buffer Accumulation Buffer Global Buffer 4x4 PE Array 4x4 PE Array ... ... Fig. 11: The overall architecture design, which comprises a set of data dispatchers, multiple PE arrays, and a vector unit. the threshold impact on the final output is insensitive to the in- put prompts, as shown in Fig. 10. Thus, we can predetermine the threshold values offline and apply them to all inputs. To further simplify hardware implementation, we restrict all thresholds to powers of 2. In our design, we set휃 th1 and휃 th2 to preserve the most significant 5 bits and 3 bits, assuming an 8-bit fixed-point representation, respectively. Specifically, if the first 5 bits of푥 푎 and푥 푏 are the same (computed via a bitwise XOR), then푥 푏 fully reuses the partial results of푥 푎 . If the first 3 bits of푥 푎 and푥 푏 are the same, the most significant 3 bits of the partial result for푥 푏 are reused from푥 푎 , while only the subsequent 4 bits of푥 푏 are explicitly computed. Sec. 7.4 further the impact of different thresholds on model performance and generative quality. 5 Architectural Design Although our channel-wise reuse algorithm significantly reduces the total number of operations across all computational blocks, its reuse-based data access patterns are inherently incompatible with off-the-shelf accelerators, e.g., a systolic array. To address this, we introduce a co-designed architecture that natively supports our algorithm. We first present the overall architectural design in Sec. 5.1, and then explain two key components: a PE array with a tailored dataflow in Sec. 5.2, and a dedicated data dispatcher in Sec. 5.3 to guarantee high PE utilization during the computation. 5.1 Overview Motivation. As shown inFig. 8, the proposed algorithm has var- ious reuse patterns, which make it difficult to exploit efficiently using only GPU kernels due to its fine-grained, data-dependent execution. Directly implementing our idea on a GPU would lead to severe warp divergence, irregular memory accesses, and low 6 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA Frame ... ... ... ... ... ... ... ... Q K ... ... ... ... ... ... ... ... 0 23 1 Frame Fig. 12: The computation order of self-attention computation when reusing along the t-axis. Each PE array is responsible for a group of tiles, as highlighted in colors. The numbers denote the computation order. 8-bit W1 4-bit A14-bit A2 <<4 ➕ ✖ ➕ M M ➕ \ 4 \ 4 \ \ 8 8 8-bit W2 ✖ M 00 Fig. 13: The design of one PE unit, which is built upon a canonical 8- bit MAC unit. The yellow parts highlight our aug- mented logics. tensor core utilization. These challenges motivate us to propose a dedicated architectural support for our algorithm. Overview. Fig. 11 illustrates the overall architecture of our vDiT accelerator. Our design comprises a set of PE arrays and a vector unit. A subset of PE arrays is also coupled with a data dispatcher. The PE arrays are responsible for compute-intensive matrix–matrix operations in vDiTs, e.g., self-attention, while the vector unit han- dles element-wise and vector operations, including Softmax. Each PE array is built upon a classic systolic array design with 4×4 PEs and is augmented with additional support to enable our reuse- aware datapath, as shown in Fig. 13. The data dispatchers are used to feed input data into the PE arrays; their scheduling algorithm is explained in Sec. 5.3. Meanwhile, to support the pipelining be- tween computation and data fetching, both the global buffer and the accumulation buffer are designed to be double-buffered. 5.2 PE Array We first illustrate the overall computation order of our reuse algo- rithm. We then describe our augmentation to an 8-bit fixed-point multiply-and-accumulate (MAC) unit. Lastly, we show how the PE is configured to support various data computation patterns. Computational Order. We first describe our computation order, which minimizes on-chip data movement, and how the workload is distributed across multiple PE arrays. Here, we use self-attention as an example. As shown in Fig. 12, we organize every 2×2 frames in the attention map as a group and process them together. Specifically, we map the same푁 × 푁tile across the 2×2 frames to a single PE array. In Fig. 12,푁is 4. Here, we compute the partial accumulations on a channel group basis, since the reuse direction is the same within one channel group. For instance, when the reuse direction is along the t-dimension, the computational order is shown in Fig. 12. We first process the 4×4 attention elements in frame 0, and then we sequentially compute the same elements for frames 1, 2, and 3. Other reuse directions can be done similarly. Once we finish computing the partial attention scores within a channel group, we store the intermediate results in the accumulation buffer. PE Design. Fig. 13 shows our augmentation of a canonical 8-bit fixed-point MAC unit, with the added logic highlighted in yellow. Specifically, we add two additional registers in each PE to store more intermediate results, along with extra multiplexers to select data from different sources. The control signal, denoted as M, configures the PE to operate in different execution modes. The following paragraphs explain how these configurations enable PEs to support various data reuse patterns. Reuse-Aware Dataflow. Based on our channel-wise reuse al- gorithm, reuse patterns vary across different frames within a 2×2 group. In Fig. 14, each frame has 3×3 tiles in the attention map. To minimize the data traffic between PEs and the accumulation buffer, each PE is configurable to support different reuse patterns. Our overall dataflow adopts an output-stationary pattern in a classic systolic array, with some changes to accommodate reuse. We show- case four execution modes using reuse along the t-dimension as an example; reuse along the other two dimensions works similarly. Additional modes are the variants of the four modes. Mode 0. This mode is used when computing tiles such as Tile 0, as annotated in Fig. 14. When processing Tile 0, the partial attention score could be reused by tiles in subsequent frames, i.e., Tile 1 and Tile 2. The first example in Fig. 14 shows this scenario, where the blue element in푄and the pink element in퐾are reused. Three registers in PE are used to store the accumulated partial attention scores corresponding to the elements in Tile 0, Tile 1, and Tile 2. During computation, the input element of K is read from the column input port, while the input element of Q is read through the first row input port; the second row input port is disabled. After the partial attention score is computed, the result is accumulated into three corresponding registers. The multiplexers푀 2 and푀 3 control whether the partial score should be accumulated into registers푅 1 and푅 2 , respectively, based on the reuse mask described in Sec. 4.1. After computing all channels in one channel group, the current accumulated attention score of Tile 0 is temporarily read out from all 푅 0 of the PE array to the accumulation buffer. Mode 1. The second mode is used when computing tiles such as Tile 1 or Tile 2. When reuse is applied along the t-dimension, elements in Tile 1 or Tile 2 can only be reused by Tile 3, as shown by the second case in Fig. 14. In this example, we show how elements in 푄 are reused, i.e., Tile 3 reuses the values in Tile 1. In this mode, the partially accumulated attention score of Tile 2 is temporarily stored in푅 2 to avoid reloading it from the accumula- tion buffer. Registers푅 0 and푅 1 hold the accumulated partial sums for Tile 1 and Tile 3, respectively. During computation, the input values of푄and퐾are read from the row and column input ports, respectively. The resulting partial sum is accumulated into푅 0 for Tile 1. Meanwhile, it is also selectively accumulated into푅 1 based on the reuse criterion. Note that, if the resulting partial sum is con- sidered “half-reuse”, i.e., the leading bits are reused, the multiplexer 푀 2 is configured to accumulate only the leading ℎ bits into 푅 1 . Mode 2.The third mode is also used when computing tiles such as Tile 1 or Tile 2. However, in this case, Tile 1 and Tile 2 have already partially reused the results from Tile 0, i.e., the leading ℎ-bit partial sums. Thus, Tile 1 and Tile 2 still need to calculate the remaining partial sum of the subsequent 4 bits after the ℎth bit. Specifically, in the third example in Fig. 14, the values in푄are reused, while the values in퐾are partially reused. To use our PE, 7 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. ✖ <<4 ➕ ✖ ➕ ➕ 01 23 Reuse Reuse += += += * * * Higher Bits Lower Bits ✖ <<4 ➕ ✖ ➕ ➕ += * + * ❌ ✖ <<4 ➕ ✖ ➕ ➕ += * += ❌ R0R1R2R0R1R2R0R1R2 ✖ <<4 ➕ ✖ ➕ += * ❌ ➕ R0R1R2 Q K Q K Q K M0M1 M2M3 M0M1 M2M3 M0M1 M2M3 M0M1 M2M3 00000000 += * + * 01 23 Reuse * 01 23 Reuse Half Reuse 01 23 Q K Mode 0: Computing Tile , reusing on Tile and . 0 12 Mode 1: Computing Tile , reusing on Tile . 1 3 Mode 2: Computing Tile after half-reusing on Tile , and reusing on Tile . 1 3 Mode 3: Computing Tile , no reusing. 3 Frame 0 01213133 Lower Bits Higher Bits Lower Bits Higher Bits Lower Bits Lower Bits M4M4M4M4 Fig. 14: An overview of our reuse-aware dataflow for a single PE. In this example, 3 blocks in푄and퐾stand for one frame, i.e., 3×3 tiles are a frame in the attention map. We show four representative cases to show the flexibility of our PE. The control logics are omitted in PEs. The gray color indicates the parts that are disabled during the computation. the lower bits of two퐾elements are fed into two 4-bit column input ports, while two distinct 8-bit푄values are read from the two row input ports. The control logic configures the datapath such that the two푄values are processed by separate 4×8 MAC units. The resulting products are then directly added without performing the shift operation. The accumulated result is stored in registers푅 0 and푅 1 . Note that, if reuse is not applied to푄, the corresponding multiplexer can simply be configured to not accumulate into 푅 1 . Mode 3.The final mode is used when computing tiles in the last tile in a 2×2 group. In this mode, input values from푄and 퐾are fed into the PE through the row and column input ports, respectively.푅 1 and푅 2 are always disabled, as no need to reuse elements from Tile 3, as shown in the last example of Fig. 14. The computed partial sum is then accumulated into register푅 0 , which stores the accumulated results of Tile 3. Note that, the datapath of the last example shows accumula- tion for a single full 8-bit multiplication result. When values in K are partially reused, the PE can be configured, similar toMode 2. Two lower-bit퐾values are fed into two 4-bit column ports, so that we can compute two partial sums simultaneously. The only difference is that multiplexer푀 2 is configured to always disable the accumulation of the partial sum into register 푅 1 . Other Operations. Self-attention has the most complex reuse patterns, whereas other operations are simpler because reuse ap- plies to only one input. As a result, our PE can also accommodate reuse patterns in other vDiT operations, such as MLPs. Configur- ing the PE to support these operations is straightforward: register 푅 0 is used to compute the current value, while the remaining two registers are used to accumulate partial results when reuse is ap- plied. The overall configuration is similar toMode 1. Meanwhile, we also support “half-reuse” in those operations. The configuration is similar to Mode 2in Fig. 14. 5.3 Data Dispatcher Issues. With our reconfigurable PE array design, we are able to support various reuse data access patterns. However, given the fixed dataflow of the PE array, i.e., an output-stationary dataflow, sparse and irregular input patterns often lead to low PE utilization. Fig. 15a illustrates such a case. Here, we show only one side of the input data, i.e., keys퐾, while assuming that the other input, PE PE PE PE 01234567 Complete Reuse Partial Reuse Full Compute Row Col. 0 Col. 1 Col. 2 Col. 3 Row: (a) Low PE utilization schedule. 012345 Col. 0 67 Complete Reuse Partial Reuse Full Compute PE PE PE PE Row Col. 1 Col. 2 Col. 3 Row: (b) High PE utilization schedule. Fig. 15: Examples of two scheduling cases with identical data sparsity. The left example shows low PE utilization because many PEs perform no effective computation. In contrast, the right example achieves high PE utilization because two adjacent inputs can be merged and processed together. The red blocks highlight the rows that are merged. queries푄, is fully dense. Note that,퐾is streamed from the column ports of the PE array. Each column port of the PE array accepts 8-bit data, whereas each row port can read at most 16-bit data, corresponding to two input elements. In principle, if two rows of 퐾are sufficiently sparse, they could be merged and processed at once to improve PE utilization. However, the sparsity pattern of 퐾is highly irregular, as shown in Fig. 15a. The adjacent two rows rarely exactly match and allow merging; thus, many PEs remain idle since fully reused elements skip computation entirely. In contrast, an ideal case is shown in Fig. 15b, where every pair of adjacent rows can be merged. For instance, the first and second rows can be combined and fed into the PE array within a single cycle. Note that, the only difference between Fig. 15a and Fig. 15b is the computation order. Thus, in this subsection, we design a data dispatcher to reorder computations and improve PE utilization. Design. Fig. 16 illustrates the overall design of our data dis- patcher, which consists of a scheduler and a matcher. The scheduler groups tokens such that those with similar reuse patterns are dis- patched to the PE array together. The matcher then further pairs channels within each token group to combine compatible input pairs, similar to the example in Fig. 15b, further improving PE uti- lization. The following paragraphs explain these two components. The left side of Fig. 16 illustrates our scheduler. The reuse pattern of each token is represented as a binary string, where “0” denotes full reuse and “1” denotes half-reuse or no reuse. These token strings 8 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA String Queue (Double-buffered) 0110010101...01 0110010101...01 0110010101...01 ... 0110010101...01 16 channels 00: 01: 02: 15: Argmin ... 060201 ... 070300 ... 111008 ... 141209 Feature Buffer Feature Buffer PE PE PE PE Row Lookahead Buffer Merge Unit Clustered Queue Global Buffer Clustering Unit Pivot Strings 011011010...01 11100111001 011010011...01 011001010...01 XOR ➕ ➕ ➕ XOR ➕ ➕ ➕ XOR ➕ ➕ ➕ XOR ➕ ➕ ➕ Matched SchedulerMatcher Fig. 16: An overview of our data dispatcher design, which consists of a scheduler and a matcher. The scheduler groups tokens with similar reuse patterns together to improve the PE utilization. The matcher then pairs channels within each token group to combine compatible input pairs. are stored in a string queue with 64 entries. Each entry contains a 16-bit string, i.e., 16 channels are processed as a batch. To cluster these 64 strings into four groups, the clustering unit first randomly picks four strings as pivot strings. It then computes the Hamming distance between each input string and each pivot string, and uses an adder tree to accumulate the total differences. Based on the results, each input string is assigned to the cluster corresponding to the pivot string with the minimum distance and is enqueued into the corresponding clustered queue. Note that, each clustered queue has a fixed maximum capacity, i.e., 16 strings in this case. If the preferred queue is already full, the input string is assigned to the queue with the next smallest distance instead. Through this process, each clustered queue ultimately contains exactly 16 input strings for subsequent computing. Once a batch of input strings is clustered, each cluster is mapped to one PE array. We design one matcher for one PE array. Each matcher reads the corresponding token features from the global buffer into the feature buffer. In each cycle, the merge unit examines one channel value from each of the four input tokens and checks whether these values can be merged with any other inputs. If a match is found, the merge unit combines two input values and feeds them into the PE array. Otherwise, the single input values are fed into the PE array. Note that, a scheduler is required for both sides of the data, i.e.,푄and퐾. However, only one side requires a matcher. The other side follows the same merging policy implicitly. 6 Experimental Setup Experimental Methodology. We develop validated RTL imple- mentations for the Kaleido hardware. The hardware is based on a systolic array architecture, consisting of 128×128 PE arrays, each with 4×4 PEs clocked at 1 GHz. The PEs are designed for 8-bit fixed-point representation. The hardware also has a scalar unit, which consists of 512 parallel lanes, each capable of performing the Softmax and other activation functions. Both the global buffer and the accumulation buffer are implemented by SRAM. The global buffer is 64 MB to store input data and model weights. The Accu- mulation buffer is 32 MB in size to store output data. Both buffers are double-buffered. While we primarily evaluate Kaleido using this configuration, we will later show the sensitivity of Kaleido performance to different hardware resource configurations. Table 2: Area breakdown of Kaleido architecture. 4× 4 PE Array Dispatcher On-Chip Buffer Total Configuration128× 12864× 3264 MB+32 MB– Area (m 2 )25.11.368.494.8 Simulation Methodology. The RTL is synthesized using Syn- opsys tools and laid out using Cadence tools in TSMC 16nm FinFET technology, with SRAMs generated by an ARM compiler. Power is simulated using Synopsys PrimeTimePX, with full annotated switching activity. The off-chip DRAM is modeled as 16 DDR5-6400 channels based on Micron specifications [53], and DRAM energy is estimated using Micron’s System Power Calculator [26]. We build a cycle-level simulator of the architecture with the latency and energy of each component parameterized from post-synthesis results. Area. Tbl. 2 shows that the overall area of Kaleido is 94.8 m 2 at 16 nm. Compared to a baseline systolic array with 96 MB on-chip buffer, our augmentation on PE arrays and our data dispatcher introduce area overheads of 6.6% and 1.3%, respectively. Models and Metrics. We evaluateKaleidoon four widely adopted vDiT models: HunyuanVideo[35], Wan[73], CogVideoX[84] and TurboDiffusion[91]. Without further notice, we generate 5- second videos with 480×540 resolution. Note that, TurboDiffusion is a distilled model with only 4 timesteps. To evaluate the gener- ative quality, we use the VBench[24]as the video quality metric, which consists of 950 benchmark prompts. The generated videos are evaluated across 16 aspects from VBench. For image quality evaluation, we use PSNR, SSIM, and LPIPS, and compare against the videos generated by the baseline models. Hardware Baselines. We compare six hardware baselines: • A100: a Nvidia A100 GPU with 80 GB HBM [55]. • H100 : a Nvidia H100 GPU with 80 GB HBM [56]. • Cambricon-D[34]: approximates the computation between adjacent timesteps to accelerate the diffusion process. • AdapTiV[87]: leverages the value similarity among tokens to reduce the effective computation for image classification. • Exion[21]: exploits inter- and intra-timstep sparsity among intermediate results to skip computation. • Ditto [33]: also leverages the value similarity between ad- jacent timesteps to reduce the effective computation. 9 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. All accelerator baselines are implemented using 8-bit fixed-point precision and scaled to the same number of PEs, 512×512, at 1 GHz. The hardware simulation is modeled under 16 nm technology. The energy and performance ofA100andH100are measured using the built-in power sensing circuitry on Nvidia A100 and H100. Software Baselines. In addition to six hardware baselines, we also compareKaleidoagainst four algorithmic baselines. Specifi- cally, we compare one block caching technique,PAB[96]. We also compared three sparse attention acceleration methods, including MInference[30],SVG[77]andSVG2[82]. The performance num- bers of all software baselines are measured on a Nvidia A100 GPU. Variants. We evaluate three variants ofKaleidoto separate the contributions in our paper: • Kaleido-pe: a variant which only consists of our modified PE arrays without our data dispatcher. • Kaleido-pe-sh: a variant which only consists of our modi- fied PE arrays and the scheduler in our data dispatcher, but without the matcher. • Kaleido-pe-m: a variant which only consists of our modified PE arrays and the matcher in our data dispatcher, but without the scheduler. • Kaleido : our full-fledged design with all optimizations. For both variants, we set휃 th1 and휃 th2 to preserve the first 5 bits and first 3 bits of accuracy, respectively. We do not show a GPU implementation of our algorithm because its reuse-based data access patterns are fundamentally incompatible with existing atten- tion acceleration frameworks[11, 37], e.g., FlashAttention. Thus, a direct comparison of GPU implementations is not meaningful. 7 Evaluation 7.1 Generative Quality Here, we primarily compare against three approximation base- lines:Cambricon-D,AdapTiV, andExion. We also include four software-optimized baselines here. Since the algorithm proposed in Dittois lossless, we do not include it in the quality comparison. Fig. 17compares the video generation quality ofKaleidowith the three approximation methods shown above. Across all four met- rics,Kaleidoconsistently delivers the best quality. For instance, Kaleidoimproves reconstruction fidelity by a large margin in terms of PSNR.Kaleidoachieves 29.9 dB, 24.5 dB, 24.4 dB, and 26.5 dB on HunyuanVideo, Wan, TurboDiffusion, and CogVideoX, respec- tively. In particular,Kaleidoachieves 17.0 dB overCambricon-D on HunyuanVideo. Across all models,Kaleidoachieves over 6 dB higher than all prior accelerator methods. Even compared with the software-optimized baselines,Kaleidoconsistently achieves the highest results over the image-based metrics. This shows that the channel-wise reuse algorithm is the key to preserving high pixel-level precision. Similar trends can be seen in SSIM (Fig. 17b) and LPIPS (Fig. 17c). For instance,Kaleidoachieves 0.87 to 0.90 in SSIM, while others only achieve 0.50 to 0.78 in SSIM. Finally, on the VBench metric (Fig. 17d), which captures high- level video quality, Kaleido maintains the same scores as the base- line algorithms (around 0.81), exceeding the best prior methods. Fig. 17e further showcases the qualitative results of all methods. Vi- sually,Kaleidopreserves much higher quality compared to others. HunyuanVideoWanTurboDiffusion CogVideoX 0 10 20 30 40 PSNR (dB) SVG SVG2 Minference PAB Cambricon-D AdapTiV EXION Kaleido (a) PSNR. Higher is better. HunyuanVideoWanTurboDiffusion CogVideoX 0.2 0.4 0.6 0.8 1.0 SSIM (b) SSIM. Higher is better. HunyuanVideoWanTurboDiffusion CogVideoX 0.0 0.2 0.4 0.6 0.8 1.0 LPIPS (c) LPIPS. Lower is better. HunyuanVideoWanTurboDiffusion CogVideoX 0.5 0.6 0.7 0.8 0.9 VBench Score (d) VBench score. Higher is better. Ground TruthCambricon-DAdapTiVEXIONKaleido (e) Qualitative comparison on HunyuanVideo. Please zoom in to check details. Fig. 17: Video generation quality comparison. 7.2 Performance Comparison Performance.Fig. 18acompares the end-to-end performance com- parison of different accelerators. Here, we compareKaleidoagainst four accelerators:Cambricon-D,AdapTiV,ExionandDitto. Mean- while, we also compare four GPU-based optimizations:SVG,SVG2, Minference,PAB, Across all four vDiT models,Kaleidoachieves the highest speedup. Specifically,Kaleidodelivers 6.6×, 5.6×, 5.2×, and 6.1×speedup overA100on HunyuanVideo, Wan, TurboDiffu- sion, and CogVideoX, respectively. While methods likeAdapTiValso achieve relatively high perfor- mance, over 5×speedup,AdapTiVachieves much lower generative quality compared toKaleido. Because its acceleration technique, token merging, is designed for image classification and fundamen- tally ill-suited for generative tasks.Cambricon-Dyields marginal speedups because its architecture is primarily designed for CNN- like models and cannot accelerate self-attention, which is the pri- mary computation bottleneck in modern vDiT models. Finally, both ExionandDittostruggle to achieve high speedups due to the ir- regularity of sparse computation in video generation, especially for long token sequences. Both designs have to deal with the workload imbalance between PEs. In contrast,Kaleidoco-designs a data dispatcher to tame the workload imbalance and improve the PE utilization. Sec. 7.3 further dissects the impact of data dispatch. Energy.Fig. 18bshows the energy savings of all evaluated hardware baselines normalized toA100. Here, energy saving is defined as the ratio between the energy consumption of the baseline, 10 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA HunyuanVideoWanTurboDiffusion CogVideoX 0 2 4 6 8 Speedup A100 SVG SVG2 Minference PAB H100 Ditto Cambricon-D AdapTiV EXION Kaleido (a) Speedup evaluation. Higher is better. HunyuanVideoWanTurboDiffusion CogVideoX 0 5 10 15 20 25 Energy Savings (b) Energy savings. Higher is better. HunyuanVideoWanTurboDiffusion CogVideoX 10 0 10 1 10 2 10 3 Ops Count ( × 10 15 ) A100H100DittoCambricon-DAdapTiVEXIONKaleidoOracle (c) Overall operation reduction. Lower is better. Fig. 18: Overall performance evaluation against prior work. i.e.,A100, and that of the corresponding accelerator design. Overall, Kaleidoachieves 14.8×, 15.6×, 15.2×, and 18.4×energy savings on HunyuanVideo, Wan, TurboDiffusion, and CogVideoX, respectively. The latency overhead of the data dispatcher is negligible. Note that, the scheduler in the data dispatcher can be pipelined and overlapped with the subsequent matrix computation. WhileAdapTiVexhibits higher energy savings thanKaleido, its energy savings primarily come from its aggressive merge ratio. By merging tokens,AdapTiVcan directly reduce the overall token se- quence length and reduce a large amount of off-chip DRAM accesses. However, any sparsity computation techniques, e.g.,Kaleidoor Exion, still have to maintain the full token sequence and cannot eliminate the off-chip data traffic. Nevertheless, as we mentioned earlier, the token merging technique inAdapTiVintroduces unac- ceptable accuracy loss and is practically unacceptable. Remaining baselines all achieve lower energy savings compared toKaleido. Note that, the data dispatcher contributes 0.1% of the total energy. Operation Reduction. Fig. 18c shows the total operation count for each hardware baseline. Here, we show two numbers of Kaleido: KaleidoandOracle.Kaleidois the effective operation reduction achieved during actual execution under our hardware limitations. Oracle, on the other hand, is the ideal operation reduction with- out any hardware limitations. For other hardware baselines, we show their operation reduction based on their algorithms without considering their hardware constraints. We show that, by exploiting spatio-temporal correlations and reusing intermediate channel-wise results,Oraclecan achieve up to 85% of operation reduction, eliminating a massive amount of computations. However, with hardware constraints, the operation reduction decreases to around 60%. There are two main reasons leading to the gap betweenOracleandKaleido. First is an algo- rithmic constraint. InKaleido, each channel group is restricted to reuse along a single fixed direction to simplify scheduling and hard- ware mapping. In contrast,Oracleallows each channel to reuse in any direction. Second is hardware-aware mapping overhead. Due to PE-array granularity, some operations marked as reusable may HunyuanVideoWanTurboDiffusion CogVideoX 0.80 0.90 1.00 1.10 1.20 1.30 Speedup 0.80 0.90 1.00 1.10 1.20 1.30 Normalized Energy KALEIDO-PEKALEIDO-PE-SHKALEIDO-PE-MKALEIDO SpeedupNormalized Energy Fig. 19: Ablation study of the different hardware components in Kaleido. KALEIDO-ATTN Attention Only KALEIDO-FULL Attention + Linear 5.8 6.0 6.2 6.4 6.6 6.8 Speedup SpeedupPSNR 29.8 29.9 30.0 30.1 30.2 30.3 PSNR (dB) Fig. 20: Ablation study of per- formance and quality on Hun- yuanVideo, with and without applyingKaleidoto linear projection layers. 2x24x48x816x1632x3264x64 # of PE 0.85 0.90 0.95 1.00 1.05 Speedup HunyuanVideoWanTurboDiffusion Fig. 21: Sensitivity of perfor- mance to per-array PE size. The numbers are normalized to the default configuration with 4×4 PEs per PE array. still be executed as part of a regular PE tile.Dittoachieves the second-best theoretical operation reduction because it aggressively skips 0-value computations. However,Ditto’s adder tree design cannot efficiently process these irregular operations, leading to severe pipeline stalling and poor PE utilization. 7.3 Ablation Study Hardware Ablation. Fig. 19 shows an ablation study of the four variants inSec. 6:Kaleido-pe, which includes only the reconfig- urable PE array without the data dispatcher;Kaleido-pe-sh, which adds only the scheduler in the data dispatcher;Kaleido-pe-m, which adds only the matcher in the data dispatcher; and the fullKaleido design. Both speedup and energy are normalized toKaleido-pe. Across three vDiT models, we show that introducing our data dispatcher achieves 1.2×speedup and 9.0% of energy reduction. This is because, without the dispatcher, our channel-wise reuse algorithm would introduce irregular sparsity in computation and lead to PE under-utilization. The variant with the matcher achieves higher speedup than the variant with only the scheduler. With our data dispatcher, we improve the PE utilization from 69.4% to 85.2% by clustering tokens with similar reuse patterns and taming the ir- regularity during computation. The scheduler in the data dispatcher is pipelined and overlapped with the subsequent computation. The latency overhead of our data dispatcher is negligible. Algorithmic Ablation.Fig. 20compares the performance and generative quality of applying our algorithm either to attention layers only or to all layers. We report the results on Hunyuan- Video. Compared with applyingKaleidoonly to attention layers, the full design achieves higher speedup by exploiting additional channel-wise reuse opportunities in the linear projection layers. Meanwhile, the PSNR remains nearly unchanged. It shows that extending reuse to linear layers preserves generation quality while providing additional acceleration. 11 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. 12244896192 Buffer Size(MB) 32x32 64x64 128x128 256x256 512x512 #PE Arrays 0.060.060.060.060.06 0.180.260.260.260.26 0.230.430.711.001.03 0.240.470.891.622.63 0.240.470.901.662.88 0.5 1.0 1.5 2.0 2.5 (a) Speedup. 12244896192 Buffer Size(MB) 32x32 64x64 128x128 256x256 512x512 #PE Arrays 4.492.681.771.301.07 4.292.441.531.060.83 4.252.401.471.000.77 4.252.391.460.990.75 4.242.391.460.990.75 1.0 1.5 2.0 2.5 3.0 3.5 4.0 (b) Normalized energy. Fig. 22: Sensitivity of performance and energy efficiency to the number of PE arrays and buffer size. Data are normalized to the configuration with 128×128 PE arrays and 96 MB buffer. (3, 1)(4, 2)(5, 3)(6, 4) Reuse Thresholds (θ th1 , θ th2 ) 4.80 5.60 6.40 7.20 Speedup SpeedupPSNR 26 28 30 32 PSNR Fig. 23: Sensitivity of speedup and generative quality to reuse thresholds,휃 th1 and휃 th2 . GeneralFast-Motion 0.0 2.0 4.0 6.0 8.0 Speedup SpeedupPSNR 15 25 35 PSNR (dB) Fig. 24: Comparison of perfor- mance and generative quality under fast-motion prompts. 7.4 Sensitivity Study PE Size. Fig. 21 shows the sensitivity of speedup to the per-array PE size, where we vary the number of PEs per array while retaining the total number of PEs constant. All numbers are normalized to the default 4×4 configuration. Overall, speedup degrades slightly as the per-array PE size increases. This is because larger PE arrays are less flexible in handling irregular reuse patterns, thus reducing PE utilization. Nevertheless, Kaleido remains robust even with larger PE arrays. Meanwhile, further reducing the array size to 2×2 achieves marginal benefit. Thus, we choose the 4×4 configuration. PE Array & Buffer Size. Fig. 22 shows the sensitivity of perfor- mance and energy efficiency to the number of PE arrays and buffer size. All results are normalized to our default configuration, i.e., 128×128 PE arrays and a 96 MB buffer. Each PE array consists of 4×4 PEs. As shown in Fig. 22a, increasing the number of PE arrays generally improves the overall performance. However, if the on- chip buffer is not large enough, off-chip data communication would impact the performance. Larger buffers reduce off-chip memory traffic, often leading to higher performance. Fig. 22 further shows the normalized energy consumption under different PE and buffer configurations. Increasing the number of PE arrays slightly reduces overall energy because it reduces execution time. Meanwhile, a larger buffer size can better improve energy effi- ciency because it minimizes off-chip data traffic overhead. Overall, both Fig. 22a and Fig. 22 show that the number of PE arrays and the total size of on-chip buffer need to be balanced to achieve optimal performance and energy efficiency. Thresholds. In our algorithm design, our default threshold set- ting is to set휃 th1 and휃 th2 to preserve the most significant 5 bits and 3 bits of accuracy, respectively. In Fig. 23, we show the sensitivity of performance and accuracy to different threshold combinations. The results show that preserving fewer bits leads to noticeable degradation in visual quality, whereas preserving more bits slows 0246810 Speedup 10 15 20 25 30 35 PSNR (dB) Cambricon-D AdaptiV EXION Kaleido 480P (Empty) 720P (Solid) Better Fig. 25: Scalability analysis on generating 720p high- resolution videos. 45678 Speedup 15 20 25 30 PSNR (dB) Q th = 15 Kaleido EXION AdapTiV 75% 50% 25% 50% 65% 70% Better Fig. 26: Comparison against AdapTiVandExionunder different configurations. inference. We find that setting휃 th1 and휃 th2 to 5 and 3 provides a good trade-off between performance and quality. Fast Motion.Fig. 24compares the performance and genera- tive quality ofKaleidounder general prompts and fast-motion prompts on HunyuanVideo. We extract the prompts with fast mo- tions from VBench. Compared with general prompts, fast-motion prompts achieve slightly lower speedup because rapid temporal changes reduce the amount of reusable spatio-temporal correlation. However,Kaleidostill maintains substantial acceleration while preserving similar PSNR. It means that our channel-wise reuse remains effective even for videos with fast motions. 7.5 Scalability Analysis Fig. 25 shows the scalability of different methods when generating higher-resolution (720p) videos. As the resolution increases from 480p to 720p,Kaleidoconsistently achieves both higher speedup and better visual quality compared to prior approaches. This is because our approach inherently exploits the spatio-temporal cor- relations across frames in the latent space. These correlations are stronger as the resolution increases. As a result,Kaleidonot only scales efficiently with resolution but also further widens the gap in both performance and quality over prior approximation methods. 7.6 Comparison with AdapTiVand Exion. Fig. 26comparesKaleidowithAdapTiVandExionunder differ- ent configurations. The configurations are annotated inFig. 26. AlthoughAdapTiVandExionachieve higher speedup with more aggressive token merging or sparsity, their generative quality de- grades substantially as the compression ratio increases. In con- trast,Kaleidoachieves higher speedup while maintaining much higher PSNR, demonstrating that channel-wise reuse better pre- serves video quality while still providing strong acceleration. 8 Related Work Diffusion Acceleration Techniques. Algorithmically, early diffu- sion acceleration methods primarily focus on reducing the number of denoising timesteps, using techniques such as DDIM[65], DPM- Solver[46, 47], and flow matching[40]. Subsequent work further exploits similarity across timesteps to skip less important denois- ing steps, e.g., PAB [96], Astraea [42], and ToCa [100]. In contrast, Kaleidooperates at the channel level, allowing different channel groups to be reused differently according to the spatio-temporal dimensions they encode. This finer granularity can preserve better visual fidelity. Meanwhile, another line of research explores sparse patterns in attention scores[30, 52, 77, 83]to reduce self-attention 12 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA overhead. Instead,Kaleidoreuses previously computed partial at- tention results for correlated channel values. Thus, rather than dis- carding attention computations,Kaleidopreserves them through reuse. Diffusion caching methods[5, 41]exploit redundancy across denoising timesteps by reusing intermediate activations.Kaleido targets a different source of redundancy. Therefore,Kaleidois complementary to diffusion caching methods. Diffusion Accelerators. More recently, the hardware commu- nity has also focused on diffusion acceleration[21, 33, 34, 87, 88, 101], proposing a range of accelerator designs that exploit computation sparsity and timestep similarity. For example, AdapTiV[87]merges similar tokens during execution to reduce inference computation, while EXION[21]exploits the similarity of attention sparsity across denoising steps to reduce masking overhead. Other works, such as Cambricon-D[34]and Ditto[33], leverage mixed precision to reduce computation overhead. However, none of these studies ex- plores the unique channel-wise spatio-temporal correlations in the video latent space. In contrast, we are the first work that identi- fies the root cause of pattern diversity and proposes a principled reuse-based strategy to accelerate all vDiTs. Sparse Acceleration. A broad body of prior work has explored sparse computation techniques [6,15,19,28,29,45,48,58,59,68, 74,76,93,98]. Early efforts primarily focused on accelerating sparse matrix–matrix multiplication. E.g., OuterSPACE [58], SpArch [93], and MatRaptor [68] explore different dataflows to improve computa- tional efficiency, while SIGMA [59] and STC [98] address irregular sparsity. As DNNs became the dominant workload, subsequent work focused on exploiting sparsity in DNNs. For instance, Eye- riss [6] improves flexibility for compact and sparse models, while SparTen [15] and S2TA [45] target different sparsity patterns. High- Light [76] further addresses workload imbalance in sparse computa- tion. More recently, research has focused on accelerating attention mechanisms. For example, A 3 [19], SpAtten [74], and Sanger [48] re- duce attention overhead through techniques such as approximation, token pruning, and quantization. In contrast, our work targets a fundamentally different opportunity and exploits the channel-wise spatio-temporal correlations unique to vDiTs. Spatio-Temporal Similarity. There is a long history of exploit- ing spatio-temporal correlations to improve the efficiency of contin- uous vision [3,14,51,67,86,94,95,99]. For example, both EVA 2 [3] and Euphrates [99] observe the continuity in videos and propose motion-guided reuse techniques to reduce CNN inference cost. Sub- sequent work, e.g., Diffy [51] and ASV [14], extends this idea to other vision tasks. VR-DANN [67] and CMC [66] further exploit codec metadata to guide efficient video understanding. Meanwhile, Deja View [94] and Cicero [13] show that spatio-temporal correla- tions can also be used to improve the efficiency of VR applications. 9 Conclusion As video generation continues to advance rapidly, the next frontier of generative AI will be vDiT models that can fundamentally un- derstand the physical world. This paper introduced Kaleido, an al- gorithm–hardware co-design that rethinks how to accelerate video diffusion transformers. By exploiting channel-wise spatio-temporal correlations in the latent space, we reveal the fundamental relation- ships between the token channels and their attention scores. By leveraging this insight, we proposed a lightweight reuse algorithm with a co-designed accelerator to efficiently address the irregular sparsity. Our principled design achieves up to 5.9×speedup and 16.0×energy savings over state-of-the-art GPUs and accelerators. Acknowledgments This work was supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB2025XDXM113), the National Natural Science Founda- tion of China (NSFC) Grants (62532006 and 62402312), Shanghai Pujiang Talent Program (24PJA044), and Shanghai Qi Zhi Institute Innovation Program (SQZ202316). References [1]Akshay Kulkarni. 2026. Predis.ai Announces Launch of New State-of-the-Art (SOTA) AI Models for Ad Videos. https://w.manilatimes.net/2026/01/09/tmt- newswire/globenewswire/predisai-announces-launch-of-new-state-of-the- art-sota-ai-models-for-ad-videos/2256039 [2] Daniel Bolya and Judy Hoffman. 2023. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4599–4603. [3] Mark Buckler, Philip Bedoukian, Suren Jayasuriya, and Adrian Sampson. 2018. EVA 2 : Exploiting temporal redundancy in live computer vision. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 533–546. [4] Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang, Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, et al.2025. SANA- Video: Efficient Video Generation with Block Linear Diffusion Transformer. (2025). arXiv:2509.24695 [cs.CV] https://arxiv.org/abs/2509.24695 [5]Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos- Savvas Bouganis, Yiren Zhao, and Tao Chen. 2024. Delta-DiT: A Training- Free Acceleration Method Tailored for Diffusion Transformers. arXiv preprint arXiv:2406.01125 (2024). [6]Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze. 2019. Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 2 (2019), 292–308. [7] Zheng Chen, Zichen Zou, Kewei Zhang, Xiongfei Su, Xin Yuan, Yong Guo, and Yulun Zhang. 2025. DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution. In NeurIPS. [8]Chris McKay. [n. d.]. World Labs’ AI System Can Generate a 3D World from an Image.https://w.maginative.com/article/world-labs-ai-system-can- generate-a-3d-world-from-an-image/ [9]Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 9 (2023), 10850–10869. [10]Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023). [11]Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359. [12]Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, Ion Sto- ica, and Hao Zhang. 2025. Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile. arXiv preprint arXiv:2502.06155 (2025). [13]Yu Feng, Zihan Liu, Jingwen Leng, Minyi Guo, and Yuhao Zhu. 2025. Cicero: Addressing Algorithmic and Architectural Bottlenecks in Neural Rendering by Radiance Warping and Memory Optimizations. In Proceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina) (ISCA ’24). IEEE Press, 1293–1308. https://doi.org/10.1109/ISCA59077.2024. 00096 [14]Yu Feng, Paul Whatmough, and Yuhao Zhu. 2019. Asv: Accelerated stereo vision system. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture. 643–656. [15] Ashish Gondimalla, Noah Chesnut, Mithuna Thottethodi, and T. N. Vijaykumar. 2019. SparTen: A Sparse Tensor Accelerator for Convolutional Neural Networks. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microar- chitecture (Columbus, OH, USA) (MICRO-52). Association for Computing Ma- chinery, New York, NY, USA, 151–165. https://doi.org/10.1145/3352460.3358291 [16]Google. 2024. Veo 2: Our state-of-the-art video generation model.https: //deepmind.google/technologies/veo/veo-2/ [17]Google DeepMind. [n. d.]. Veo3: Our state-of-the-art video generation model. https://deepmind.google/models/veo/ 13 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. [18]Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. 2024. LTX-Video: Realtime Video Latent Diffusion. arXiv preprint arXiv:2501.00103 (2024). [19]Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al.2020. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 328–341. [20]Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W. Lee. 2021. ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural Networks. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). 692–705. https://doi.org/10.1109/ISCA52012.2021.00060 [21]Jaehoon Heo, Adiwena Putra, Jieon Yoon, Sungwoong Yune, Hangyeol Lee, Ji-Hoon Kim, and Joo-Young Kim. 2025. EXION: Exploiting Inter-and Intra- Iteration Output Sparsity for Diffusion Models. In 2025 IEEE International Sym- posium on High Performance Computer Architecture (HPCA). 324–337. https: //doi.org/10.1109/HPCA61900.2025.00034 [22]Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion proba- bilistic models. Advances in neural information processing systems 33 (2020), 6840–6851. [23]Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. 2022. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 (2022). [24]Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al.2024. Vbench: Comprehensive benchmark suite for video generative models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21807–21818. [25] Team HunyuanWorld. 2025. HunyuanWorld 1.0: Generating Immersive, Ex- plorable, and Interactive 3D Worlds from Words or Pixels. arXiv preprint (2025). [26]Micron Technology Inc. 2025. Micron System Power Calculators. https://w. micron.com/support/tools-and-utilities/power-calc. [27]Jack Parker-Holder, Shlomi Fruchter. [n. d.]. Genie 3: A new frontier for world models.https://deepmind.google/blog/genie-3-a-new-frontier-for-world- models/ [28] Hanhwi Jang, Joonsung Kim, Jae-Eon Jo, Jaewon Lee, and Jangwoo Kim. 2019. MnnFast: a fast and scalable system architecture for memory-augmented neural networks. In Proceedings of the 46th International Symposium on Computer Ar- chitecture (Phoenix, Arizona) (ISCA ’19). Association for Computing Machinery, New York, NY, USA, 250–263. https://doi.org/10.1145/3307650.3322214 [29] Jun-Woo Jang, Sehwan Lee, Dongyoung Kim, Hyunsun Park, Ali Shafiee Ardestani, Yeongjae Choi, Channoh Kim, Yoojin Kim, Hyeongseok Yu, Hamzah Abdel-Aziz, Jun-Seok Park, Heonsoo Lee, Dongwoo Lee, Myeong Woo Kim, Hanwoong Jung, Heewoo Nam, Dongguen Lim, Seungwon Lee, Joon-Ho Song, Suknam Kwon, Joseph Hassoun, SukHwan Lim, and Changkyu Choi. 2021. Sparsity-Aware and Re-configurable NPU Architecture for Samsung Flagship Mobile SoC. In 2021 ACM/IEEE 48th Annual International Symposium on Com- puter Architecture (ISCA). 15–28. https://doi.org/10.1109/ISCA52012.2021.00011 [30]Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al.2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems 37 (2024), 52481– 52515. [31]Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al.2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual ACM/IEEE International Symposium on Computer Architecture. [32]Katie Deighton. [n. d.]. Coca-Cola Injects ‘Holidays Are Coming’ Ads With an Upgraded Dose of AI.https://w.wsj.com/articles/coca-cola-injects- holidays-are-coming-ads-with-an-upgraded-dose-of-ai-bc8921e2 [33]Sungbin Kim, Hyunwuk Lee, Wonho Cho, Mincheol Park, and Won Woo Ro. 2025. Ditto: Accelerating Diffusion Model via Temporal Value Similarity. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 338–352. [34] Weihao Kong, Yifan Hao, Qi Guo, Yongwei Zhao, Xinkai Song, Xiaqing Li, Mo Zou, Zidong Du, Rui Zhang, Chang Liu, et al.2024. Cambricon-d: Full-network differential acceleration for diffusion models. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 903–914. [35]Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al.2024. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 (2024). [36]Kuaishou. 2024. Kuaishou Unveils Proprietary Video Generation Model ‘Kling’; Testing Now Available. https://ir.kuaishou.com/news-releases/news-release- details/kuaishou-unveils-proprietary-video-generation-model-kling [37]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [38]Jianze Li, Jiezhang Cao, Zichen Zou, Xiongfei Su, Xin Yuan, Yulun Zhang, Yong Guo, and Xiaokang Yang. 2025. Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator. (2025). arXiv:2410.04224 [cs.CV] https://arxiv.org/abs/2410.04224 [39] Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. 2024. Open-Sora Plan: Open-Source Large Video Generation Model. arXiv preprint arXiv:2412.00131 (2024). [40] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022). [41]Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. 2024. Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model. (2024). arXiv:2411.19108 [cs.CV] https://arxiv.org/abs/2411.19108 [42]Haosong Liu, Yuge Cheng, Wenxuan Miao, Zihan Liu, Aiyue Chen, Jing Lin, Yiwu Yao, Chen Chen, Jingwen Leng, Yu Feng, and Minyi Guo. 2025. Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformers. (2025). arXiv:2506.05096 [cs.CV] https://arxiv.org/abs/2506.05096 [43]Joseph Liu, Joshua Geddes, Ziyu Guo, Haomiao Jiang, and Mahesh Kumar Nandwana. 2024. SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers. arXiv preprint arXiv:2411.10510 (2024). [44]Jinlai Liu, Jian Han, Bin Yan, Hui Wu, Fengda Zhu, Xing Wang, Yi Jiang, Bingyue Peng, and Zehuan Yuan. 2025. InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation. (2025). arXiv:2511.04675 [cs.CV] https://arxiv. org/abs/2511.04675 [45]Zhi-Gang Liu, Paul N Whatmough, Yuhao Zhu, and Matthew Mattina. 2022. S2ta: Exploiting structured sparsity for energy-efficient mobile cnn acceleration. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 573–586. [46] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35 (2022), 5775–5787. [47] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. 2022. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 (2022). [48]Liqiang Lu, Yicheng Jin, Hangrui Bi, Zizhang Luo, Peng Li, Tao Wang, and Yun Liang. 2021. Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture. In MICRO-54: 54th Annual IEEE/ACM In- ternational Symposium on Microarchitecture (Virtual Event, Greece) (MICRO ’21). Association for Computing Machinery, New York, NY, USA, 977–991. https://doi.org/10.1145/3466752.3480125 [49]Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung-Yeung Shum, and Daxin Jiang. 2025. Step- Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model. arXiv:2502.10248 [cs.CV] https://arxiv.org/abs/2502.10248 [50]Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2024. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15762–15772. [51]Mostafa Mahmoud, Kevin Siu, and Andreas Moshovos. 2018. Diffy: a déjà vu- free differential deep neural network accelerator. In Proceedings of the 51st An- nual IEEE/ACM International Symposium on Microarchitecture (Fukuoka, Japan) (MICRO-51). IEEE Press, 134–147. https://doi.org/10.1109/MICRO.2018.00020 [52]Wenxuan Miao, Yulin Sun, Aiyue Chen, Jing Lin, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, and Yu Feng. 2026. TimeRipples: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space. In 14 Kaleido : Algorithm-Hardware Co-Design for Video Diffusion TransformersConference’17, July 2017, Washington, DC, USA Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR). 25688–25698. [53]Micron Technology, Inc. 2023.Micron DDR5 128GB RDIMM. https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3A6ffd17ac- e709-469d-9473-a0a904681d9/renditions/original/as/128gb-ddr5-rdimm- product-brief.pdf Product Brief, Rev. A. [54] Nvidia. [n. d.]. NVIDIA RTX PRO 6000 Blackwell. https://w.nvidia.com/en- us/products/workstations/professional-desktop-gpus/rtx-pro-6000/ [55] Nvidia. 2021. NVIDIA A100 PCIe 80 GB. https://w.nvidia.com/en-us/data- center/a100/ [56]Nvidia. 2022. NVIDIA H100 PCIe 80 GB. https://w.nvidia.com/en-us/data- center/h100/ [57] OpenAI. 2024. Sora: Bring your imagination to life with text, image, or video. https://openai.com/sora/ [58]Subhankar Pal, Jonathan Beaumont, Dong-Hyeon Park, Aporva Amarnath, Siying Feng, Chaitali Chakrabarti, Hun-Seok Kim, David Blaauw, Trevor Mudge, and Ronald Dreslinski. 2018. OuterSPACE: An Outer Product Based Sparse Matrix Multiplication Accelerator. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA). 724–736. https://doi.org/10. 1109/HPCA.2018.00067 [59]Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudarshan Srini- vasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. 2020. SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training. In 2020 IEEE International Symposium on High Performance Computer Architec- ture (HPCA). 58–70. https://doi.org/10.1109/HPCA47549.2020.00015 [60] Lenny Rachitsky. 2025. The Godmother of AI on jobs, robots and why world models are next. https://w.lennysnewsletter.com/p/the-godmother-of-ai [61] Runway. 2024.Runway Gen 2: The Next Step Forward for Generative AI. https://medium.com/@dbhatt245/runway-gen-2-the-next-step-forward- for-generative-ai-an-introduction-b85bc90d3e45 [62]Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022). [63]Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al.2025. Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model. arXiv preprint arXiv:2504.08685 (2025). [64] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.2022.Make- a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022). [65] Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020). [66]Zhuoran Song, Chunyu Qi, Fangxin Liu, Naifeng Jing, and Xiaoyao Liang. 2024. CMC: Video Transformer Acceleration via CODEC Assisted Matrix Condensing. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (La Jolla, CA, USA) (ASPLOS ’24). Association for Computing Machinery, New York, NY, USA, 201–215. https://doi.org/10.1145/3620665.3640393 [67] Zhuoran Song, Feiyang Wu, Xueyuan Liu, Jing Ke, Naifeng Jing, and Xiaoyao Liang. 2020. Vr-dann: Real-time video recognition via decoder-assisted neural network acceleration. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 698–710. [68]Nitish Srivastava, Hanchen Jin, Jie Liu, David Albonesi, and Zhiru Zhang. 2020. MatRaptor: A Sparse-Sparse Matrix Multiplication Accelerator Based on Row- Wise Product. In 2020 53rd Annual IEEE/ACM International Symposium on Mi- croarchitecture (MICRO). 766–780. https://doi.org/10.1109/MICRO50266.2020. 00068 [69] Genmo Team. 2024. Mochi 1. https://github.com/genmoai/models. [70]Meituan LongCat Team, Xunliang Cai, Qilong Huang, Zhuoliang Kang, Hongyu Li, Shijun Liang, Liya Ma, Siyu Ren, Xiaoming Wei, Rixu Xie, and Tong Zhang. 2025. LongCat-Video Technical Report. (2025). arXiv:2510.22200 [cs.CV] https://arxiv.org/abs/2510.22200 [71]Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, Yihang Chen, Jie Liu, Yansong Cheng, Yao Yao, Jiayi Zhu, Yihao Meng, Kecheng Zheng, Qingyan Bai, Jingye Chen, Zehong Shen, Yue Yu, Xing Zhu, Yujun Shen, and Hao Ouyang. 2026. Advancing Open-source World Models. arXiv preprint arXiv:2601.20540 (2026). [72] Tencent. 2024. Tencent launches and open-sources Hunyuan video-generation model. https://technode.com/2024/12/04/tencent-launches-and-open-sources- hunyuan-video-generation-model/ [73]Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jin- gren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. 2025. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv preprint arXiv:2503.20314 (2025). [74]Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 97–110. https://doi.org/10.1109/HPCA51647.2021.00018 [75]Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al.2024. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6211– 6220. [76]Yannan Nellie Wu, Po-An Tsai, Saurav Muralidharan, Angshuman Parashar, Vivienne Sze, and Joel Emer. 2023. HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO ’23). Association for Computing Machinery, New York, NY, USA, 1106–1120. https: //doi.org/10.1145/3613424.3623786 [77]Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al.2025. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. arXiv preprint arXiv:2502.01776 (2025). [78]Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. 2025. Training-free and Adaptive Sparse Attention for Efficient Long Video Generation. arXiv preprint arXiv:2502.21079 (2025). [79] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. 2024. Easyanimate: A high-performance long video genera- tion method based on transformer architecture. arXiv preprint arXiv:2405.18991 (2024). [80] Lu Xu, Sijie Zhu, Chunyuan Li, Chia-Wen Kuo, Fan Chen, Xinyao Wang, Guang Chen, Dawei Du, Ye Yuan, and Longyin Wen. 2025. Beyond raw videos: Un- derstanding edited videos with large multimodal model. In Proceedings of the Computer Vision and Pattern Recognition Conference. 503–512. [81]Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. 2023. Diffusion models: A comprehensive survey of methods and applications. Comput. Surveys 56, 4 (2023), 1–39. [82]Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al.2025. Sparse VideoGen2: Acceler- ate Video Generation with Sparse Attention via Semantic-Aware Permutation. arXiv preprint arXiv:2505.18875 (2025). [83] Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, Jianfei Chen, Song Han, Kurt Keutzer, and Ion Stoica. 2025. Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation. (2025). arXiv:2505.18875 [cs.CV] https://arxiv.org/abs/2505.18875 [84] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al.2024. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 (2024). [85]Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al.2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005 (2025). [86]Ziyu Ying, Shulin Zhao, Haibo Zhang, Cyan Subhra Mishra, Sandeepa Bhuyan, Mahmut T Kandemir, Anand Sivasubramaniam, and Chita R Das. 2022. Ex- ploiting Frame Similarity for Efficient Inference on Edge Devices. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS). IEEE, 1073–1084. [87]Seungjae Yoo, Hangyeol Kim, and Joo-Young Kim. 2024. AdapTiV: Sign- Similarity Based Image-Adaptive Token Merging for Vision Transformer Accel- eration. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). 64–77. https://doi.org/10.1109/MICRO61859.2024.00015 [88]Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, and Yingyan Lin. 2023. ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 273–286. https://doi.org/10.1109/HPCA56546.2023.10071027 [89]Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. 2024. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems 37 (2024), 1196–1219. [90]Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. 2024. Make pixels dance: High-dynamic video generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8850–8860. 15 Conference’17, July 2017, Washington, DC, USAWenxuan Miao et al. [91]Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. 2025. TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times. arXiv preprint arXiv:2512.16093 (2025). [92] Zhekai Zhang, Hanrui Wang, Song Han, and William J. Dally. 2020. SpArch: Efficient Architecture for Sparse Matrix Multiplication. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 261–274. https: //doi.org/10.1109/HPCA47549.2020.00030 [93]Zhekai Zhang, Hanrui Wang, Song Han, and William J. Dally. 2020. SpArch: Efficient Architecture for Sparse Matrix Multiplication. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). 261–274. https: //doi.org/10.1109/HPCA47549.2020.00030 [94]Shulin Zhao, Haibo Zhang, Sandeepa Bhuyan, Cyan Subhra Mishra, Ziyu Ying, Mahmut T Kandemir, Anand Sivasubramaniam, and Chita R Das. 2020. Déja view: Spatio-temporal compute reuse for ‘energy-efficient 360 vr video stream- ing. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Ar- chitecture (ISCA). IEEE, 241–253. [95]Shulin Zhao, Haibo Zhang, Cyan Subhra Mishra, Sandeepa Bhuyan, Ziyu Ying, Mahmut Taylan Kandemir, Anand Sivasubramaniam, and Chita Das. 2021. HoloAR: On-the-fly optimization of 3D holographic processing for augmented reality. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Mi- croarchitecture. 494–506. [96] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. 2024. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408.12588 (2024). [97] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. 2024. Open-Sora: Democratiz- ing Efficient Video Production for All. https://github.com/hpcaitech/Open-Sora [98]Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie. 2019. Sparse Tensor Core: Algorithm and Hardware Co-Design for Vector-wise Sparse Neural Networks on Modern GPUs. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (Columbus, OH, USA) (MICRO-52). Association for Computing Machinery, New York, NY, USA, 359–371. https://doi.org/10. 1145/3352460.3358269 [99] Yuhao Zhu, Anand Samajdar, Matthew Mattina, and Paul Whatmough. 2018. Euphrates: Algorithm-soc co-design for low-power mobile continuous vision. arXiv preprint arXiv:1803.11232 (2018). [100]Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang. 2024. Accelerating diffusion transformers with token-wise feature caching. arXiv preprint arXiv:2410.05317 (2024). [101]Zihan Zou, Xinming Yan, Shun Zhang, Peng Zheng, Guang Yang, Hao Cai, and Bo Liu. 2025. S-DMA: Sparse Diffusion Models Acceleration via Spatiality- Aware Prediction and Dimension-Adaptive Dataflow. In Proceedings of the 58th IEEE/ACM International Symposium on Microarchitecture. 432–444. 16