Paper deep dive
Parameterized Stripe Attention for Efficient Video Generation
Xingyu Jia, Baole Ai, Ang Wang, Kang Zhao, Yong Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion Transformers (DiTs) enable high-quality video generation but suffer from substantial inference latency, primarily attributable to the computationally expensive full spatio-temporal attention. While sparse attention methods offer potential solutions, existing approaches face an inherent flexibility--efficiency dilemma: predefined masks lack the flexibility to capture diverse attention patterns, while runtime-determined masks introduce overheads and sacrifice hardware efficiency. We identify the lack of a unified structural characterization of DiT attention as a key limitation of existing methods, and establish that video DiT attention exhibits \textbf{periodic diagonal stripe structures} along both temporal and spatial dimensions. To formally encode these structured patterns within a single efficient kernel, we present {\bf PSA}, a parameterized stripe attention that formalizes the observed stripe regularity, unifying diverse attention patterns for efficient mask generation. This unified representation enables a single hardware-efficient CUDA kernel to process all sparse patterns, achieving FlashAttention-3-level Model FLOPs Utilization. To determine optimal sparsity configurations, we propose a training-free offline search algorithm that automatically maximizes sparsity under a specified error tolerance for each attention head. Experiments on HunyuanVideo and Wan~2.1 demonstrate that PSA achieves 1.57$\times$ and 1.37$\times$ end-to-end speedups over FlashAttention-3 baselines, with acceptable visual quality degradation.
Tags
Links
- Source: https://arxiv.org/abs/2609.37001v1
- Canonical: https://arxiv.org/abs/2609.37001v1
Trouble viewing inline? Open PDF directly →
Full Text
79,229 characters extracted from source content.
Expand or collapse full text
Parameterized Stripe Attention for Efficient Video Generation Xingyu Jia Baole Ai Ang Wang Kang Zhao Yong Li Affiliation: Alibaba Group Abstract Diffusion Transformers (DiTs) enable high-quality video generation but suffer from substantial inference latency, primarily attributable to the computationally expensive full spatio-temporal attention. While sparse attention methods offer potential solutions, existing approaches face an inherent flexibility–efficiency dilemma: predefined masks lack the flexibility to capture diverse attention patterns, while runtime-determined masks introduce overheads and sacrifice hardware efficiency. We identify the lack of a unified structural characterization of DiT attention as a key limitation of existing methods, and establish that video DiT attention exhibits periodic diagonal stripe structures along both temporal and spatial dimensions. To formally encode these structured patterns within a single efficient kernel, we present PSA, a parameterized stripe attention that formalizes the observed stripe regularity, unifying diverse attention patterns for efficient mask generation. This unified representation enables a single hardware-efficient CUDA kernel to process all sparse patterns, achieving FlashAttention-3-level Model FLOPs Utilization. To determine optimal sparsity configurations, we propose a training-free offline search algorithm that automatically maximizes sparsity under a specified error tolerance for each attention head. Experiments on HunyuanVideo and Wan 2.1 demonstrate that PSA achieves 1.57× and 1.37× end-to-end speedups over FlashAttention-3 baselines, with acceptable visual quality degradation. 1 Introduction Diffusion Transformers (DiTs) [18] have rapidly emerged as a foundational architecture for image and video generation. DiTs employ Transformers [24] as the backbone and leverage algorithms such as Diffusion [20, 14, 15, 18] or Flow Matching [11, 5] to iteratively generate images and videos from noise. Prominent open-source models such as Wan 2.1 [25] and HunyuanVideo [8] leverage the DiT architecture for tasks like text-to-video and image-to-video synthesis, producing high-quality, highly dynamic video content. However, DiT inference requires multi-step denoising with enormous computational costs and prolonged latency. For instance, generating a 5-second, 720p video with Wan 2.1 on an NVIDIA H100 GPU takes nearly 30 minutes [25], severely limiting practical applications. Since these models utilize full spatio-temporal 3D attention [32], the attention computation constitutes the primary bottleneck, accounting for over 60% [8] of the total inference time. Consequently, designing more efficient attention mechanisms is a critical optimization target. Sparse attention—selectively computing only the most informative token interactions—offers a promising avenue, provided the attention distribution exhibits exploitable structure. Figure 1: Heat maps in Wan 2.1 and HunyuanVideo. Figure 2: Visualization of the PSA parameterization that captures various attention sparse patterns. Existing sparse attention methods for DiTs lack a principled structural characterization of attention distributions, leading to an inherent trade-off between flexibility and efficiency. The first category relies on predefined masks designed from empirical observations rather than principled analysis. SVG [28] identifies only two fixed patterns—temporal and spatial heads—without analyzing why only these two arise or where the observed periodicity originates; consequently, it cannot capture the diverse stride, offset, and hybrid patterns revealed by our analysis. STA [35] achieves efficient local computation via tile-wise sliding windows, but assumes only local attention and cannot represent long-range, strided, or hybrid structures as shown in fig. 1. Because these methods are not grounded in a comprehensive characterization of attention patterns, their fixed masks inevitably fail to cover the full diversity of sparse structures. The second category pursues more flexible pattern representations by determining the sparse mask dynamically at runtime, effectively treating attention distributions as arbitrary. For example, SVG2 [31] clusters Q and K tokens separately to enable flexible sparse patterns, overlooking the structural regularity revealed by our analysis. This abandonment of structure comes at a cost: the irregular cluster-based kernel’s reliance on FlashInfer [33] precludes Tensor Memory Accelerator (TMA) utilization and introduces non-trivial padding overhead (∼ 20% at 70% sparsity) despite token permutation; and the clustering procedure itself incurs additional computational cost (∼ 10% at 70% sparsity). Consequently, the flexibility–efficiency trade-off in existing methods stems from the lack of a principled, unified framework to characterize sparse attention: methods with fixed masks fail to cover the full diversity of DiT-specific sparse structures, while methods pursuing flexibility sacrifice efficiency by treating attention distributions as arbitrary rather than structured. This motivates the need for a more mechanistic understanding of DiT attention patterns that can support both broad pattern coverage and high-performance implementation. Concretely, sparse attention for video DiTs involves three main challenges: (1) A principled characterization of the DiT attention structure space is needed, providing a unified and sufficiently expressive description of the diverse sparse patterns that emerge across heads, layers, and denoising timesteps. (2) A hardware-aware implementation must be developed that supports various mask configurations while maximizing GPU utilization. (3) An efficient search method is required to determine per-head sparse configurations without incurring additional inference-time overhead. To address these challenges, we propose PSA, a parameterized stripe sparse attention framework. First, we reveal that video DiT attention exhibits periodic diagonal stripe structures along both temporal and spatial dimensions, through theoretical derivation (via RoPE-3D logit decomposition and video spatio-temporal redundancy) and large-scale empirical validation across all attention heads in HunyuanVideo and Wan 2.1 (fig. 1), identifying four primary patterns: intra-frame, inter-frame, hybrid, and uniform (see section 3.1 for details). Second, based on this structural understanding, we design a parameterized mask ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) that provides a unified representation of periodic stripe sparse patterns across attention heads, with quantitative evaluation demonstrating high-fidelity approximation of full attention outputs (average cosine similarity of 0.9831). Leveraging this unified representation, we support different patterns with a single kernel, eliminating multi-kernel launch overhead. Our implementation adopts block-wise partitioning as the fundamental granularity to fully exploit modern GPU hardware resources, achieving 61% Model FLOPs Utilization (MFU) on HunyuanVideo—close to FlashAttention-3 levels—thereby resolving the flexibility–efficiency tension. Finally, observing strong similarity in attention masks across different prompts, we design an offline parallel search algorithm that requires no training and introduces no runtime search overhead, assigning the mask with the highest sparsity to each head while maintaining error bounds. Our parametric pattern representation and offline search achieve both objectives: head-adapted patterns are determined prior to inference, thereby eliminating runtime overhead while preserving adaptability. We validate our method on HunyuanVideo and Wan 2.1, achieving 1.57× and 1.37× end-to-end speedups over FlashAttention-3, respectively, with acceptable quality degradation (PSNR >> 23). In summary, our main contributions are as follows. 1. We reveal that video DiT attention exhibits periodic diagonal stripe structures along both temporal and spatial dimensions, and provide both theoretical derivation and large-scale empirical validation to establish this structural property. 2. We propose a parameterized mask ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) that unifies diverse sparse patterns, enabling a single hardware-efficient CUDA kernel to process all supported patterns with FlashAttention-3-level MFU, resolving the flexibility–efficiency tension. We further introduce 3D-padding to support arbitrary video resolutions with minimal overhead (≈ 5% for 720p). 3. We develop a training-free offline search algorithm that maximizes sparsity under a constrained error threshold for each attention head, with support for distributed parallel search to accelerate the process. 4. The PSA achieves substantial acceleration without compromising effectiveness, outperforming state-of-the-art methods such as SVG2 and STA in both accuracy and efficiency. 2 Related Work Attention mechanisms are pivotal to the success of Transformers [24], yet their quadratic complexity poses a fundamental efficiency bottleneck. Existing efficient attention methods for video generation primarily construct sparse masks by exploiting the distributional characteristics of attention heatmaps, falling into two paradigms. The first paradigm relies on predefined sparse mask formats. SVG [28] classifies attention heads into spatial and temporal types and employs online profiling to generate sparse masks. VORTA [23] trains a lightweight router to dynamically select among full attention, sliding window, and coreset attention patterns. Sparse-vDiT [3] selects from a set of predefined structural patterns, including diagonal, multi-diagonal, and vertical-stripe sparsity. Additionally, other window-based attention methods exploit the locality of attention. NATTEN [6] implements sliding window attention for vision tasks, localizing self-attention to nearest neighborhoods. STA [35] employs tile-wise window attention for improved memory access and parallelism. The second paradigm determines sparse masks dynamically during inference based on attention scores. VMoBA [27] extends Moba [16] to adapt spatio-temporal attention within DiTs. MOD-DiT [12] avoids the sampling overhead of runtime mask determination by linearly extrapolating pattern intensities from early denoising steps. RoPeSLR [13] combines block-sparse attention with a trained low-rank compensator to restore dropped background contributions. SVG2 [31] improves efficiency through k-means clustering of Q and K vectors, followed by token permutation and top-k cluster selection to identify the most relevant attention regions. However, existing methods suffer from notable limitations: predefined pattern approaches are constrained by a limited pattern set, while dynamic mask generation leads to non-trivial overhead and poor efficiency, yielding limited speedup over optimized dense implementations such as FlashAttention [4, 19]. In contrast, PSA uniformly parameterizes diverse sparse patterns with hardware-aware kernels, achieving superior flexibility and efficiency. 3 Method 3.1 Structural Analysis of Attention Patterns We begin by formally characterizing the structural properties of attention maps in video DiTs, aiming to move beyond empirical observation toward a mechanistic understanding of the patterns governing these distributions. Our analysis reveals that attention in video DiTs is neither arbitrary nor purely local; rather, it consistently manifests as periodic diagonal stripe structures, parameterized by their widths, offsets, and strides. We verify this structural analysis through a comprehensive examination of attention maps in HunyuanVideo and Wan 2.1, identifying four primary attention patterns as shown in fig. 1: 1. Intra-frame [Fig. (a)–(c)]: diagonal stripes with varying widths, offsets, and strides within each frame, repeating identically across frames. 2. Inter-frame [Fig. (d)–(g)]: relatively uniform intra-frame weights, with diagonal band structures of varying widths, offsets, and strides emerging between frames. 3. Hybrid [Fig. (h)]: a combination of intra-frame and inter-frame patterns. 4. Uniform [Fig. (i)]: attention weights uniformly distributed across all positions. We show that this structure arises from two complementary properties: (1) video spatio-temporal redundancy, which concentrates high attention scores within local neighborhoods, and (2) RoPE-3D cosine periodicity, which causes attention scores to repeat at axis-dependent intervals. Property 1: Video spatio-temporal redundancy. Videos exhibit rich spatio-temporal redundancy at multiple scales [17, 26]. Since attention is the only component in a DiT block that computes pairwise token interactions, this redundancy is directly reflected in the attention score matrix A. Concretely, fixing any two axes and varying the third, high attention scores concentrate within locality radii εt _t, εh _h, εw _w along the temporal, height, and width axes, respectively: |Δt|<εt(h,w fixed),|Δh|<εh(t,w fixed),|Δw|<εw(t,h fixed),| t|< _t\;\;(h,w fixed), | h|< _h\;\;(t,w fixed), | w|< _w\;\;(t,h fixed), (1) where Δt=t−t′ t=t-t , Δh=h−h′ h=h-h , Δw=w−w′ w=w-w denote relative offsets along each axis. In the 2D heatmap, the locality conditions in eq. 1 concentrate high attention along diagonal bands—forming the stripe width (εt _t inter-frame, εw _w intra-frame)—while column-wise locality (εh _h) produces multiple parallel diagonals with layout-determined stride (see Appendix D). Property 2: RoPE-3D cosine periodicity. We show that RoPE-3D [22, 8, 25] introduces structured periodicity by partitioning embedding channels into three disjoint subsets MtM_t, MhM_h, MwM_w encoding temporal, height, and width positions. For query qiq_i at position (t,h,w)(t,h,w) and key kjk_j at (t′,h′,w′)(t ,h ,w ), the attention logit decomposes as a(t,h,w),(t′,h′,w′)= a_(t,h,w),(t ,h ,w )=\; ∑m∈Mt‖qi(m)‖‖kj(m)‖cos(ϕ(m)+(t−t′)θm) _m∈ M_t \|q_i^(m) \|\, \|k_j^(m) \| (φ^(m)+(t-t )\, _m ) + +\; ∑m∈Mh‖qi(m)‖‖kj(m)‖cos(ϕ(m)+(h−h′)θm) _m∈ M_h \|q_i^(m) \|\, \|k_j^(m) \| (φ^(m)+(h-h )\, _m ) + +\; ∑m∈Mw‖qi(m)‖‖kj(m)‖cos(ϕ(m)+(w−w′)θm), _m∈ M_w \|q_i^(m) \|\, \|k_j^(m) \| (φ^(m)+(w-w )\, _m ), (2) where qi(m),kj(m)∈ℝ2q_i^(m),k_j^(m) ^2 are the content query and key slices for channel m; ϕ(m)φ^(m) is the angle between qi(m)q_i^(m) and kj(m)k_j^(m); and θm=c−2m/d _m=c^-2m/d is the angular frequency of channel m, with d the total embedding dimension and c the RoPE base. The periodicity becomes most pronounced when a single channel dominates its group [30]. Suppose a single channel mλ∗∈Mλm^*_λ∈ M_λ dominates its group, i.e., ‖qi(mλ∗)‖‖kj(mλ∗)‖≫‖qi(m)‖‖kj(m)‖∀m∈Mλ,m≠mλ∗. \|q_i^(m^*_λ) \|\, \|k_j^(m^*_λ) \|\; \; \|q_i^(m) \|\, \|k_j^(m) \| ∀\,m∈ M_λ,\;m≠ m^*_λ. (3) We empirically verify this dominance assumption on Wan 2.1, where 87.55%87.55\% of attention heads exhibit a dominant channel (see Appendix E for details). Then the contribution of group λ∈t,h,wλ∈\t,h,w\ to eq. 2 reduces to a single cosine term. Taking λ=tλ=t as a representative example, a(t,h,w),(t′,h′,w′)≈‖qi(mt∗)‖‖kj(mt∗)‖cos(ϕ(mt∗)+(t−t′)θmt∗),a_(t,h,w),(t ,h ,w )\;≈\; \|q_i^(m^*_t) \|\, \|k_j^(m^*_t) \| (φ^(m^*_t)+(t-t )\, _m^*_t ), (4) and analogously for λ∈h,wλ∈\h,w\ with (t−t′)(t-t ) replaced by (h−h′)(h-h ) or (w−w′)(w-w ), respectively. The logit is thus a cosine of the relative offset along axis λ, yielding an axis-wise attention period of λ=2πθmλ∗= 2πc 2mλ∗/d,λ∈t,h,w.T_λ\;=\; 2π _m^*_λ\;=\;2π\,c^\,2m^*_λ/d, λ∈\t,h,w\. (5) In the heatmap visualization, this cosine periodicity means that high attention weights recur at fixed intervals λT_λ—producing the stride of the observed stripes. Moreover, the content-dependent phase ϕ(mλ∗)φ^(m^*_λ) can shift the cosine peak away from the main diagonal, producing the offset. Joint effect. When A is visualized as a 2D heatmap under the flattened 1D token ordering, the two properties identified above—video redundancy and axis-wise cosine periodicity—jointly produce the observed periodic diagonal stripes: (1) Inter-frame stripes are driven by temporal locality in eq. 1: the stripe width is governed by εt _t, and the stripe period is governed by tT_t. (2) Intra-frame stripes are driven by spatial locality within each frame block: the stripe width is governed by εw _w (row-wise locality), and the stripe period is governed by hT_h and wT_w (eq. 5), with column-wise locality (εh _h) inducing repeated off-diagonals at intervals of W under row-major flattening. Figure 3: An overview of the PSA framework. (a) Unified Representation: The parametric model ℳM captures diverse per-head sparse attention patterns, covering both intra-frame and inter-frame structures with varying widths, offsets, and strides. (b) Efficient Kernel: A dedicated CUDA kernel enables high-performance, hardware-aware computation across all supported sparse patterns. (c) Automatic Search: PSA-Search automatically identifies the optimal sparse configuration for each head by maximizing sparsity subject to a predefined output degradation threshold. 3.2 Efficient and Flexible Kernel Unified Representation. To efficiently encode the sparse diagonal-stripe patterns identified above and enable a single, hardware-efficient kernel, we propose a parametric formulation ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s), where six integer parameters govern inter-frame and intra-frame diagonal bands via their widths, offsets, and strides, respectively, as illustrated in fig. 2. SVG and STA are special cases: SVG corresponds to ℳ(Tw,0,∞,Sw,0,∞)M(T_w,0,∞,S_w,0,∞), and STA’s sliding window to ℳ(Tw,0,∞,Sw,0,Ss)M(T_w,0,∞,S_w,0,S_s), where Ts=∞T_s=∞ or Ss=∞S_s=∞ reduces the corresponding axis to a single diagonal. These restricted configurations fail to capture multi-diagonal, hybrid, and uniform patterns, which account for over 60% of attention heads in HunyuanVideo and Wan 2.1. Our formulation expresses all four observed patterns (see fig. 3 (a)), achieving an average cosine similarity of 0.9831 with full attention across all heads. To generate attention masks, we devise the generation algorithm (algorithm 1). Given an input sequence of length L=T×H×WL=T× H× W, the mask generation algorithm permits an attention connection between query qiq_i and key kjk_j only when both the inter-frame and intra-frame conditions of ℳM are satisfied. Algorithm 1 PSA Mask Definition 1: Input: Q index i, K index j, Parameters ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) 2: Output: Boolean indicating if qiq_i attends to kjk_j 3: function IsAttending(i,j,ℳi,j,M) 4: (tq,sq)←(⌊i/(H⋅W)⌋,i%(H⋅W))(t_q,s_q)←( i/(H· W) ,i\%(H· W)) 5: (tk,sk)←(⌊j/(H⋅W)⌋,j%(H⋅W))(t_k,s_k)←( j/(H· W) ,j\%(H· W)) 6: Δt←tk−tq _t← t_k-t_q ⊳ inter-frame distance 7: inter_frame_cond←((|Δt−To|)modTs)≤Tw−1inter\_frame\_cond←((| _t-T_o|) T_s)≤ T_w-1 8: if not inter_frame_condinter\_frame\_cond then 9: return false 10: end if 11: Δs←sk−sq _s← s_k-s_q ⊳ intra-frame distance 12: intra_frame_cond←((|Δs−So|)modSs)≤Sw−1intra\_frame\_cond←((| _s-S_o|) S_s)≤ S_w-1 13: return intra_frame_condintra\_frame\_cond 14: end function Hardware-Aware Design. Our kernel design targets maximum computational efficiency on modern GPUs by ensuring full utilization of every scheduled thread block, eliminating idle compute units [35]. To this end, we introduce a fundamental computational abstraction called PSABlock, which represents the minimum data granularity necessary to fully saturate the resources of a single streaming multiprocessor (SM). Notably, PSABlock also aligns with the block-wise characteristics observed in the attention heatmaps (fig. 1), enabling the mask to operate naturally at this granularity. Since PSABlock is the minimum computable unit, PSA applies masking at PSABlock granularity rather than individual tokens. For video tokens organized in 3D structure, a PSABlock comprises constituent blocks along temporal, height, and width dimensions: PSABlock=PSABlockt×PSABlockh×PSABlockwPSABlock=PSABlock_t×PSABlock_h×PSABlock_w. High-Performance Kernel. We develop a high-performance CUDA kernel, PSA-Kernel, specifically optimized for the PSA framework on the NVIDIA Hopper GPU architecture. Built upon the ThunderKittens [21] library, PSA-Kernel draws inspiration from its highly efficient FlashAttention-3 pipeline design, as illustrated in fig. 3 (b). The key challenge lies in maintaining high computational efficiency across diverse sparse patterns. We address this through two synergistic mechanisms. First, we reorganize the Q,K,VQ,K,V data layout from the standard (T,H,W)(T,H,W) format to a PSABlock-centric arrangement: (T′,H′,W′CLOSE,(T ,H ,W , PSABlocktPSABlock_t, PSABlockhPSABlock_h, OPENPSABlockw)PSABlock_w), where (T′,H′,W′)(T ,H ,W ) denotes the grid of PSABlocks. This transformation ensures contiguous memory placement within each PSABlock, enabling coalesced memory accesses—a prerequisite for the efficient kernel design that follows. Second, building on this layout, our kernel achieves FA3-level efficiency while supporting flexible masks as shown in fig. 3(b). A dedicated producer warpgroup guided by the mask logic (algorithm 1) identifies required K,VK,V PSABlocks and asynchronously prefetches them into on-chip SRAM via TMA-driven ping-pong buffering. Concurrently, consumer warpgroups perform dense attention computations at PSABlock granularity using resident Q and prefetched K,VK,V data. This decoupling of data movement and computation enables hardware-optimized dense operations, maximizing GPU utilization even under sparse attention patterns. 3D-Padding for Resolution Flexibility. The block-aligned computation described above requires that the token dimensions (T,H,W)(T,H,W) be divisible by the corresponding PSABlock tile sizes (PSABlockt,PSABlockh,PSABlockw)(PSABlock_t, PSABlock_h, PSABlock_w). To support arbitrary video resolutions, we employ 3D-padding: each dimension is minimally padded to the smallest multiple of its corresponding PSABlock size, i.e., T′=⌈T/PSABlockt⌉⋅PSABlocktT = T/PSABlock_t ·PSABlock_t, ensuring alignment without restricting input geometry. This design enables support for arbitrary video resolutions with minimal computational overhead. For instance, at 720p resolution, padding the height dimension (H) from 45 to 48 tokens during the last 80% of generation steps incurs only 5% overhead (Overhead=psteps×rpaddingOverhead=p_steps× r_padding, where pstepsp_steps denotes the proportion of steps requiring padding and rpaddingr_padding represents the relative padding ratio). 3.3 Automatic and Efficient Sparsity Discovery Figure 4: Evaluation of the similarity of attention heatmaps across all 11 evaluation dimensions of VBench, where prompts 1–99 are compared against prompt 0 at the same head positions within a search space of 100(timesteps)×40(layers)×40(heads)100\ (timesteps)× 40\ (layers)× 40\ (heads). A central challenge in applying the PSA representation is assigning an optimal mask configuration ℳM to each attention head. We address this with PSA-Search, an automatic search algorithm grounded in a key empirical observation: attention patterns in DiT models vary significantly across timesteps, layers, and heads, yet remain remarkably stable with respect to input prompts [3] (see fig. 4 and Appendix G). This prompt-invariance enables an offline search strategy—optimal sparse configurations can be identified once using a representative dataset and subsequently applied to all future inferences, introducing no runtime overhead. The PSA-Search algorithm operates on a per-head basis, as illustrated in fig. 3 (c). For each attention head, the algorithm iterates through a predefined search space of candidate sparse configurations. Given a user-defined L2L_2 loss threshold α, the search first identifies all candidate masks satisfying this quality constraint (i.e., L2≤αL_2≤α), where the L2L_2 loss is computed between the outputs of the sparse attention and the full attention. Among these qualified candidates, PSA-Search selects the mask achieving maximum sparsity. Furthermore, PSA-Search supports a dual-interface: users can either specify an error tolerance α to maximize sparsity, or specify a target sparsity level and obtain the corresponding mask with minimal error. To enhance search efficiency, we leverage distributed parallelization across multiple GPUs via Ulysses [7]. Searching over 100 candidate masks completes in 2.5 hours on 8 H100 GPUs, assigning each head its maximum-sparsity mask under the error constraint without the prohibitive costs of training-based methods. 4 Experiments and Results Table 1: Comparative evaluation of PSA against SVG, SVG2 and STA baselines across different models. Spar denotes the average sparsity across all attention heads. “-” indicates metrics that are self-referential for the baseline. Models Methods Spar (%) E2E Speed (s)↓ OC↑ TS↑ MSE↓ PSNR↑ SSIM↑ LPIPS↓ HunyuanVideo (720×1280) FA2 0 1793.06 26.95 24.52 - - - - FA3 0 1172.62 (1x) 26.95 24.52 - - - - SVG 56.92 906.12 (1.29x) 22.32 21.11 0.0039 24.2716 0.8686 0.3400 SVG2 61.92 866.10 (1.35x) 25.96 23.28 0.0016 29.6303 0.8910 0.1625 Ours 59.74 777.28 (1.51x) 26.00 23.63 0.0012 30.3463 0.9062 0.1523 HunyuanVideo (768×1280) FA2 0 1684.60 26.97 24.61 - - - - FA3 0 1106.36 (1x) 26.97 24.61 - - - - STA 58.37 746.73 (1.48x) 26.19 23.39 0.0596 12.5239 0.6036 0.3117 SVG 56.98 797.57 (1.39x) 23.21 22.43 0.0053 22.9984 0.8654 0.1907 SVG2 63.23 773.47 (1.43x) 25.31 22.74 0.0027 25.7979 0.7855 0.2184 Ours 60.33 703.91 (1.57x) 26.33 23.64 0.0019 27.1983 0.8287 0.1568 Wan 2.1 14B (720×1280) FA2 0 1873.32 25.91 23.19 - - - - FA3 0 1214.69 (1x) 25.91 23.19 - - - - SVG 58.51 944.81 (1.29x) 25.29 20.32 0.0157 18.5862 0.7056 0.2881 SVG2 63.40 926.78 (1.31x) 24.69 23.33 0.0051 23.4689 0.8391 0.1703 Ours 57.57 886.64 (1.37x) 25.29 23.59 0.0048 23.4813 0.8372 0.1601 4.1 Setup Models. We evaluate two state-of-the-art text-to-video models: Wan 2.1 (14B parameters) with 81-frame sequences at 720×1280, and HunyuanVideo in two configurations—129 frames, 720×1280 and 117 frames, 768×1280. Metrics. For quantitative evaluation of frame-level fidelity and perceptual quality, we report Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) [36]. In addition to these per-frame metrics, we conduct a more holistic assessment using the VBench benchmark [34]. Specifically, we report scores for overall consistency (OC) and temporal style adherence (TS). Baselines. We compare against three sparse attention methods: SVG and SVG2, evaluated across all models to assess generalizability; and STA, evaluated only on HunyuanVideo at 768×1280 resolution due to its tile-based architectural constraints [35]. These methods represent the two paradigms discussed in section 1: SVG and STA employ predefined masks with limited flexibility, while SVG2 determines masks dynamically at runtime, sacrificing hardware efficiency for flexibility. Parameters. All methods target approximately 60% sparsity. For PSA, we construct a discrete candidate space for ℳM based on attention heatmap analysis, uniformly sampling about 100 configurations per head. The threshold α is set to 0.03, 0.018, and 0.007 for HunyuanVideo (720×1280), HunyuanVideo (768×1280), and Wan 2.1, respectively. Search uses 1–3 prompts (inter-prompt similarity is high) and evaluation uses all VBench prompts. The initial 20% of diffusion timesteps are exempted from sparsification, as early steps are critical for coarse layout and quality [28, 31, 29, 1, 2, 37]. 4.2 PSA Performance and Video Quality Table 2: Kernel performance of PSA across different models with the same mask pattern. Spar denotes the kernel sparsity of each individual attention head. Models Methods Spar (%) Kernel Speed (ms)↓ TFLOPs MFU (%)↑ HunyuanVideo (720×1280) FA3 0 251.30 (1.00x) 173.43 69.78 SVG 65.41 173.40 (1.45x) 59.99 34.98 SVG2 65.67 156.76 (1.60x) 59.54 38.40 Ours 66.00 121.58 (2.07x) 58.96 49.04 HunyuanVideo (768×1280) FA3 0 236.36 (1.00x) 163.07 69.76 STA 58.33 111.73 (2.12x) 67.95 61.50 SVG 58.43 178.59 (1.32x) 67.79 38.38 SVG2 57.68 182.79 (1.29x) 69.01 38.18 Ours 58.33 112.25 (2.11x) 67.95 61.21 Wan 2.1 14B (720×1280) FA3 0 185.55 (1.00x) 117.05 63.78 SVG 75.41 77.61 (2.39x) 28.78 37.50 SVG2 74.84 69.86 (2.66x) 29.45 42.62 Ours 75.00 55.12 (3.37x) 29.26 53.68 We evaluate PSA-Kernel at both kernel and end-to-end (E2E) levels on a single NVIDIA H100 GPU, benchmarking against FlashAttention-2 (FA2) [4] and FlashAttention-3 (FA3) [19]. E2E inference performance. As presented in table 1, PSA consistently outperforms existing sparse attention baselines across different model architectures. Compared to SVG, SVG2 and STA, it achieves relative speedups of 1.13×, 1.10×, and 1.06× on HunyuanVideo (768×1280768× 1280), respectively. Kernel performance. To isolate the computational efficiency of our core attention mechanism, we conduct detailed micro-benchmarks of PSA-Kernel against the kernels from SVG, SVG2 and STA under the same mask pattern, as shown in table 2. Implementation differences across methods may cause slight sparsity variations even under the same mask pattern. Notably, compared to the SVG and SVG2 kernels, our kernel achieves significant speedups of approximately 1.59× and 1.63× on HunyuanVideo (768×1280), respectively. Moreover, it achieves kernel performance competitive with STA while delivering superior E2E speed—the E2E advantage stems from our FA3-based implementation for unsparsified timesteps (the initial 20% of diffusion steps), whereas STA relies on a slower custom full-attention kernel during these steps. This kernel-level efficiency serves as the foundational driver of the end-to-end latency reductions, confirming that our method’s performance gains stem from a highly optimized computational core. Video Quality. Beyond computational performance, a critical question is whether PSA preserves video generation quality. As shown in table 1, PSA achieves the best VBench scores (OC and TS) among all sparse methods across all configurations, while also delivering the best frame-level fidelity metrics (MSE, PSNR, SSIM, LPIPS). Notably, on HunyuanVideo (720×1280), PSA achieves a PSNR of 30.35 at 59.74% sparsity, confirming that the parameterized mask ℳM faithfully captures each head’s attention structure. 4.3 Sensitivity Tests Quality sensitivity tests. As shown in table 3(a), we evaluate the generation quality metrics under varying levels of sparsity. The results clearly indicate that as sparsity increases from 30% to 60%, all evaluation metrics exhibit a smooth and controlled degradation. Notably, even at a high sparsity of 60%, the model maintains an acceptable generation quality (e.g., PSNR remains above 30). This demonstrates that the negative impact on generation quality is within a manageable range, even while unlocking significant computational savings. Kernel sensitivity tests. Table 3(b) presents kernel performance across varying sparsity levels. Our kernel consistently outperforms SVG across all sparsity settings. While STA exhibits a marginal latency advantage at the kernel level, our kernel supports arbitrary video resolutions via 3D-padding with negligible overhead, whereas STA is constrained to a single fixed resolution. This generality, combined with competitive kernel throughput, gives our method practical deployment value. Table 3: Sensitivity tests on quality and kernel performance. (a) Quality (HunyuanVideo 720×1280720× 1280) Sparsity MSE↓ PSNR↑ SSIM↑ LPIPS↓ 30% 0.0005 34.9341 0.9519 0.0834 40% 0.0007 33.7610 0.9431 0.0875 50% 0.0010 31.6886 0.9171 0.1250 60% 0.0012 30.3463 0.9062 0.1523 (b) Performance (ms, HunyuanVideo 768×1280768× 1280) Sparsity FA3 STA SVG Ours 60% 236.366 111.508 165.028 113.494 70% 236.366 79.702 125.161 80.438 80% 236.366 53.624 85.027 54.093 90% 236.366 25.882 45.643 26.292 4.4 Bracketing Comparison For all baselines, sparsity is an outcome rather than an input: STA’s is fixed by its predefined window masks, SVG’s is quantized to the nearest feasible window combination, and SVG2’s emerges from data-dependent runtime clustering, so an exactly matched sparsity budget cannot be enforced on the baseline side. We therefore evaluate two operating points on HunyuanVideo (768×1280768× 1280) whose sparsity strictly brackets all baselines: Ours-min below every baseline and Ours-max above every baseline. As shown in table 4, PSA dominates every baseline in both speed and quality at both endpoints: Ours-min is the fastest despite computing the most (1.51× vs. at most 1.48×), and Ours-max achieves the best PSNR despite computing the least (26.76 vs. 25.80), confirming that the conclusions of table 1 hold regardless of any residual sparsity mismatch. Table 4: Bracketing comparison on HunyuanVideo (768×1280768× 1280). Ours-min and Ours-max are two operating points whose sparsity strictly brackets that of all baselines. Method Spar (%) Speedup↑ PSNR↑ SVG 56.98 1.39× 23.00 STA 58.37 1.48× 12.52 SVG2 63.23 1.43× 25.80 Ours-min 56.47 1.51× 28.73 Ours-max 63.84 1.65× 26.76 4.5 Generality of the Stripe Structure To examine how broadly the structural premise underlying PSA holds, we measure the fraction of dominant-channel heads (eq. 3) on nine model settings, following the protocol of Appendix E. As shown in table 5, the dominant-channel fraction exceeds 78.9% on all nine settings (median ≈ 90%), spanning resolutions (480p/720p/768p), model sizes (14B/1.3B), architectures (M-DiT, text–visual concatenated, and factorized spatio-temporal attention), modalities (video/image), tokenizations (2D/3D patchify), and denoising schedules (50 vs. 40 steps). The attention heatmaps of all these models consistently exhibit the same periodic diagonal stripes as fig. 1. Notably, the RoPE-free spatial attention of Open-Sora v1.2 [38] still exhibits locality-induced stripes with stride W, indicating that spatio-temporal locality (eq. 1) holds independently of RoPE and tokenization. Table 5: Fraction of dominant-channel heads across nine model settings. Model Type Dominant-channel heads (%) Wan 2.1 14B (50-step) video DiT, 3D full attention 87.55 Wan 2.1 14B (40-step) video DiT, 3D full attention 89.30 Wan 2.1 1.3B video DiT, 3D full attention, 832×480 90.81 HunyuanVideo (720×1280720× 1280) video M-DiT 96.61 HunyuanVideo (768×1280768× 1280) video M-DiT 96.33 FLUX.1-schnell [9] image M-DiT, 2D RoPE 89.16 CogVideoX-5b [32] video DiT, concatenated attention 90.39 Open-Sora v1.2 video DiT, factorized attention 91.95 Hunyuan-DiT v1.2 [10] image DiT, 2D patchify 78.93 4.6 Fidelity of Searched Masks Our masks achieve an average cosine similarity of 0.9831 with full attention. At 60% sparsity, over 93.6% of heads maintain L2L_2 loss below 10−210^-2, confirming that PSA-Search produces high-fidelity sparse approximations. 4.7 Ablation Studies To further validate that ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) effectively and efficiently represents the sparse patterns of attention heads, we conduct a comprehensive ablation study. For a fair comparison, all experiments are conducted at a fixed sparsity level, and we evaluate the quality of the generated videos. The following five scenarios were tested: 1. SVG-like (ℳ(Tw,Sw)M(T_w,S_w)): This configuration simulates the approach of SVG, where only the width of the temporal and spatial attention bands can be controlled. 2. STA-like (ℳ(Tw,Sw,Ss)M(T_w,S_w,S_s)): This mask simulates the behavior of STA. STA’s method of applying sparsity along the tile’s t, w, and h axes corresponds in the attention heatmap to defining the inter-frame width (TwT_w), intra-frame width (SwS_w), and the stride of intra-frame diagonals (SsS_s). 3. Inter-frame-Only (ℳ(Tw,To,Ts)M(T_w,T_o,T_s)). 4. Intra-frame-Only (ℳ(Sw,So,Ss)M(S_w,S_o,S_s)). 5. Ours (ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s)). Table 6: Ablation study on mask at a fixed 60% sparsity (HunyuanVideo, 720×1280720× 1280). Config MSE↓ PSNR↑ SSIM↑ LPIPS↓ SVG-like 0.0027 26.0230 0.8364 0.2073 STA-like 0.0026 26.1466 0.8133 0.2343 Inter-frame 0.0019 27.8065 0.8505 0.1757 Intra-frame 0.0042 24.0145 0.7838 0.2730 Ours 0.0012 30.3463 0.9062 0.1523 As shown in table 6, the full six-parameter model outperforms all ablated versions, achieving a PSNR of 30.35—2.5 dB higher than the next best (Inter-frame-only). This confirms that offset and stride parameters are essential for capturing multi-diagonal and hybrid patterns. Inter-frame-only (PSNR 27.81) significantly outperforms Intra-frame-only (24.01), showing that inter-frame relationships are more critical, yet only the full model captures both. Notably, our SVG-like configuration even outperforms the actual SVG in table 1, as PSA-Search assigns more accurate per-head patterns than SVG’s fixed two-pattern strategy. 5 Conclusion We present PSA, a parameterized stripe attention framework for efficient video generation. Our central insight is that video DiT attention is not arbitrary but exhibits periodic diagonal stripe structures along both temporal and spatial dimensions. This structural understanding enables a unified parametric formulation ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) that resolves the flexibility–efficiency tension inherent in prior methods: it covers all observed attention patterns while permitting a single hardware-efficient CUDA kernel achieving up to 61% MFU. A training-free offline search algorithm then assigns each head its maximum-sparsity mask under an error constraint, introducing no runtime overhead. Experiments on HunyuanVideo and Wan 2.1 demonstrate end-to-end speedups of 1.57× and 1.37× over FlashAttention-3 with minimal quality degradation. Acknowledgments and Disclosure of Funding This work was supported by Alibaba Group through Alibaba Research Intern Program. Like most work on large-scale video generation, developing and evaluating our method required substantial computational resources, which carries environmental costs and is currently feasible only for research groups with access to large GPU clusters. While our method is training-free—built on top of existing pretrained video generation models without requiring additional fine-tuning—and reduces per-inference compute and energy, the overall computational demands of video generation remain significant. References [1] M. Adnan, N. Kurella, A. Arunkumar, and P. J. Nair (2025) Foresight: adaptive layer reuse for accelerated and high-quality text-to-video generation. External Links: 2506.00329, Link Cited by: §4.1. [2] A. Chen, B. Dong, J. Li, J. Lin, K. Tian, Y. Yao, and G. Wang (2025) RainFusion: adaptive video generation acceleration via multi-dimensional visual redundancy. External Links: 2505.21036, Link Cited by: §4.1. [3] P. Chen, X. Zeng, M. Zhao, P. Ye, M. Shen, W. Cheng, G. Yu, and T. Chen (2025) Sparse-vdit: unleashing the power of sparse attention to accelerate video diffusion transformers. arXiv preprint arXiv:2506.03065. Cited by: §2, §3.3. [4] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), Cited by: §2, §4.2. [5] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §1. [6] A. Hassani, S. Walton, J. Li, S. Li, and H. Shi (2023) Neighborhood attention transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6185–6194. Cited by: §2. [7] S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y. He (2023) DeepSpeed ulysses: system optimizations for enabling training of extreme long sequence transformer models. External Links: 2309.14509, Link Cited by: §3.3. [8] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) Hunyuanvideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: §1, §3.1. [9] B. F. Labs (2024) FLUX. Note: https://github.com/black-forest-labs/flux Cited by: Table 5. [10] Z. Li, J. Zhang, Q. Lin, J. Xiong, Y. Long, X. Deng, Y. Zhang, X. Liu, M. Huang, Z. Xiao, D. Chen, J. He, J. Li, W. Li, C. Zhang, R. Quan, J. Lu, J. Huang, X. Yuan, X. Zheng, Y. Li, J. Zhang, C. Zhang, M. Chen, J. Liu, Z. Fang, W. Wang, J. Xue, Y. Tao, J. Zhu, K. Liu, S. Lin, Y. Sun, Y. Li, D. Wang, M. Chen, Z. Hu, X. Xiao, Y. Chen, Y. Liu, W. Liu, D. Wang, Y. Yang, J. Jiang, Q. Lu, et al. (2024) Hunyuan-dit: a powerful multi-resolution diffusion transformer with fine-grained chinese understanding. External Links: 2405.08748 Cited by: Table 5. [11] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §1. [12] Y. Liu, Y. Hu, Z. Zhang, K. Jiang, and K. Yuan (2026) Mixture of distributions matters: dynamic sparse attention for efficient video diffusion transformers. External Links: 2601.11641, Link Cited by: §2. [13] Y. Liu, Z. Zhang, Y. Cai, R. Deng, Y. He, and K. Yuan (2026) RoPeSLR: 3d rope-driven sparse-lowrank attention for efficient diffusion transformers. External Links: 2605.20659, Link Cited by: §2. [14] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2022) Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems 35, p. 5775–5787. Cited by: §1. [15] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2025) Dpm-solver++: fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, p. 1–22. Cited by: §1. [16] E. Lu, Z. Jiang, J. Liu, Y. Du, T. Jiang, C. Hong, S. Liu, W. He, E. Yuan, Y. Wang, et al. (2025) Moba: mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189. Cited by: §2. [17] B. Pan, R. Panda, C. Fosco, C. Lin, A. Andonian, Y. Meng, K. Saenko, A. Oliva, and R. Feris (2021) VA-red2^2: video adaptive redundancy reduction. External Links: 2102.07887, Link Cited by: §3.1. [18] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4195–4205. Cited by: §1. [19] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024) Flashattention-3: fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems 37, p. 68658–68685. Cited by: §2, §4.2. [20] J. Song, C. Meng, and S. Ermon (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Cited by: §1. [21] B. F. Spector, S. Arora, A. Singhal, D. Y. Fu, and C. Ré (2024) ThunderKittens: simple, fast, and adorable ai kernels. External Links: 2410.20399, Link Cited by: §3.2. [22] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu (2024) Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §3.1. [23] W. Sun, R. Tu, Y. Ding, Z. Jin, J. Liao, S. Liu, and D. Tao (2025) VORTA: efficient video diffusion via routing sparse attention. arXiv preprint arXiv:2505.18809. Cited by: §2. [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §1, §2. [25] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: §1, §3.1. [26] X. Wang, Q. Si, J. Wu, S. Zhu, L. Cao, and L. Nie (2025) ReTaKe: reducing temporal and knowledge redundancy for long video understanding. External Links: 2412.20504, Link Cited by: §3.1. [27] J. Wu, L. Hou, H. Yang, X. Tao, Y. Tian, P. Wan, D. Zhang, and Y. Tong (2025) VMoBA: mixture-of-block attention for video diffusion models. arXiv preprint arXiv:2506.23858. Cited by: §2. [28] H. Xi, S. Yang, Y. Zhao, C. Xu, M. Li, X. Li, Y. Lin, H. Cai, J. Zhang, D. Li, et al. (2025) Sparse videogen: accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776. Cited by: §1, §2, §4.1. [29] Y. Xia, S. Ling, F. Fu, Y. Wang, H. Li, X. Xiao, and B. Cui (2025) Training-free and adaptive sparse attention for efficient long video generation. External Links: 2502.21079, Link Cited by: §4.1. [30] Q. Yang, J. Wang, X. Li, Y. Bai, X. Tong, H. Zhen, J. Hao, M. Yuan, and B. Li (2026) Why attention patterns exist: a unifying temporal perspective analysis. External Links: 2601.21709, Link Cited by: §3.1. [31] S. Yang, H. Xi, Y. Zhao, M. Li, J. Zhang, H. Cai, Y. Lin, X. Li, C. Xu, K. Peng, et al. (2025) Sparse videogen2: accelerate video generation with sparse attention via semantic-aware permutation. arXiv preprint arXiv:2505.18875. Cited by: §1, §2, §4.1. [32] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2024) Cogvideox: text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. Cited by: §1, Table 5. [33] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze (2025) FlashInfer: efficient and customizable attention engine for llm inference serving. External Links: 2501.01005, Link Cited by: §1. [34] F. Zhang, S. Tian, Z. Huang, Y. Qiao, and Z. Liu (2024) Evaluation agent: efficient and promptable evaluation framework for visual generative models. arXiv preprint arXiv:2412.09645. Cited by: §4.1. [35] P. Zhang, Y. Chen, R. Su, H. Ding, I. Stoica, Z. Liu, and H. Zhang (2025) Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507. Cited by: §1, §2, §3.2, §4.1. [36] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, Cited by: §4.1. [37] Y. Zhang, J. Xing, B. Xia, S. Liu, B. Peng, X. Tao, P. Wan, E. Lo, and J. Jia (2025) Training-free efficient video generation via dynamic token carving. External Links: 2505.16864, Link Cited by: §4.1. [38] Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You (2024) Open-sora: democratizing efficient video production for all. arXiv preprint arXiv:2412.20404. Cited by: §4.5. This appendix provides supplementary materials to support the main paper. We organize the content as follows: Appendix A discusses the limitations of the current work and outlines potential directions for future research. Appendix B presents the pseudocode implementation of PSA-Kernel, offering detailed algorithmic specifications for reproducibility. Appendix C introduces the PSA-Search process, including how to initialize mask candidates and the search results. Appendix D provides the formal proposition and derivation showing how video spatio-temporal locality alone produces periodic diagonal stripes in the attention heatmap. Appendix E provides extensive empirical evidence, through large-scale visualization experiments on Wan 2.1 and HunyuanVideo, to validate the dominance assumption regarding mλ∗m^*_λ proposed in section 3.1. Appendix F demonstrates that CogVideoX-v1.5 exhibits attention heatmap characteristics consistent with our observations, confirming that our method can be effectively applied to accelerate computation in this model as well. Appendix G visualizes attention heatmaps across various prompts, revealing that identical heads exhibit highly similar heatmap patterns regardless of input prompt variations. Appendix H presents the loss threshold sensitivity analysis. Appendix I presents a comprehensive evaluation of end-to-end performance and generation quality across varying sparsity levels. Appendix J provides qualitative comparisons with the FlashAttention-3 baseline, showcasing visual quality assessments and demonstrating that our method maintains comparable generation quality while achieving significant speedups. Appendix A Limitations While PSA demonstrates substantial acceleration for DiTs, two limitations warrant discussion. First, the current work is limited to video generation models, and it remains uncertain whether the same approach transfers effectively to other model families or domains, such as language models. Second, the current PSA-Search requires 2.5 hours on 8 H100 GPUs to evaluate all candidate masks for every attention head, which can be costly. In future work, we plan to accelerate this process via a two-stage search strategy: a coarse-grained stage that rapidly filters out unpromising candidates, followed by a fine-grained stage that performs detailed evaluation only on the surviving mask candidates. Appendix B Pseudocode for PSA-Kernel Algorithm 2 outlines the execution flow of PSA-Kernel, which employs a producer-consumer model to maximize hardware utilization. The workload is partitioned between two concurrently operating warpgroups within each GPU thread block. The producer warpgroup interprets the sparse mask configuration ℳM to identify required KV blocks and asynchronously loads them from HBM to shared memory. The consumer warpgroups perform block-wise attention computation on available KV blocks, including safe softmax and result accumulation. Double-buffering with kv_ready and compute_done flags coordinates the pipeline, effectively hiding memory latency behind computation. Algorithm 2 Pseudocode for PSA-Kernel 1: Input: Sparsity parameters ℳM, Tensors Q,K,VQ,K,V 2: Output: Result tensor Out 3: // Executed in parallel by dedicated warp groups within a thread block 4: 5: procedure Producer-Warpgroup 6: // Determine which KV blocks are needed for this Q block 7: KVlist←GetKVBlocks(Qblock,ℳ)KV_list ($Q_block$,M) ⊳ Uses logic from algorithm 1 8: for each KVblockKV_block in KVlistKV_list do 9: wait until compute_done flag is set 10: Reset compute_done flag 11: Load KVblockKV_block from HBM into SRAM 12: Set kv_ready flag 13: end for 14: end procedure 15: 16: procedure Consumer-Warpgroup 17: for k←1k← 1 to |KVlist||KV_list| do ⊳ Loop terminates after processing all blocks 18: wait until kv_ready flag is set 19: Reset kv_ready flag 20: // Perform attention on data in SRAM 21: S←Qblock×KSRAMTS← Q_block× K_SRAM^T ⊳ Compute attention scores 22: Spadded←3D-Padding(S,−∞)S_padded← 3D-Padding(S,-∞) ⊳ Pad invalid positions with −∞-∞ 23: P←SafeSoftmax(Spadded)P← SafeSoftmax(S_padded) ⊳ Apply safe softmax 24: Opartial←P×VSRAMO_partial← P× V_SRAM ⊳ Weighted sum of values 25: Accumulate OpartialO_partial into local output OlocalO_local 26: Set compute_done flag 27: end for 28: Write final accumulated OlocalO_local to global memory Out 29: end procedure Appendix C Process of PSA-Search C.1 Pseudocode for PSA-Search We present the detailed pseudocode of PSA-Search in algorithm 3. Given a list of candidate masks ℳC_M and a user-specified loss threshold α, the algorithm independently searches for the optimal mask configuration for each attention head. For a given head h, PSA-Search first evaluates all candidate masks by computing the L2L_2 loss between the sparse attention output and the full attention output, and sorts the results in ascending order of loss. Starting from the lowest-loss mask as the initial best candidate, the algorithm then scans through all qualified masks whose loss does not exceed α and greedily replaces the current best with any mask that achieves higher sparsity. This design ensures that the selected mask maximizes sparsity while remaining within the acceptable quality budget. After processing all heads, the algorithm returns a complete mask strategy map ℳS_M that assigns the sparsified yet faithful mask to each head. Notably, since PSA-Search records the per-head loss for every candidate mask, adjusting α only requires re-executing lines 10–21 without repeating the costly L2L_2 loss evaluation, making the threshold tuning process highly efficient. Algorithm 3 PSA-Search 1: Input: List of candidate masks ℳC_M, DiT model D, Loss threshold α 2: Output: An optimized mask strategy ℳS_M 3: function PSA-Search(,ℳ,αD,C_M,α) 4: ℳ←∅S_M← ⊳ Initialize an empty mask strategy map 5: for each head h in model D do ⊳ Iterate over all timesteps, layers, and heads 6: HeadResults←[]HeadResults←[] 7: for each mask ℳ∈ℳM _M do 8: Loss←L2Loss(PSA(h,ℳ),FullAttn(h))Loss 2Loss(PSA(h,M),FullAttn(h)) 9: Add (Loss,ℳ)(Loss,M) to HeadResultsHeadResults 10: end for 11: Sort HeadResultsHeadResults by loss in ascending order. 12: Bestℳ←HeadResults[0].maskBestM← HeadResults[0].mask 13: for each (Li,ℳi)(L_i,M_i) in HeadResultsHeadResults do 14: if Li≤αL_i≤α then 15: if Sparsity(ℳi)>Sparsity(Bestℳ)Sparsity(M_i)>Sparsity(BestM) then 16: Bestℳ←ℳiBestM _i 17: end if 18: end if 19: end for 20: ℳ[h]←BestℳS_M[h]← BestM 21: end for 22: return ℳS_M 23: end function C.2 Initialization of Mask Candidates. Before conducting PSA-Search, it is necessary to select appropriate masks for initialization. We choose the corresponding masks based on visualized heatmap results (fig. 1). In general, for a DiT with shape (T,H,W)(T,H,W) and PSABlock with shape (PSABlockt,PSABlockh,PSABlockw)( PSABlock_t, PSABlock_h, PSABlock_w), we configure ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) such that 0<Tw<T/PSABlockt0<T_w<T/PSABlock_t, To∈0,±1,±2T_o∈\0,± 1,± 2\, Ts∈1,2,3,∞T_s∈\1,2,3,∞\, 0<Sw<W/PSABlockw0<S_w<W/PSABlock_w, So∈0,±1S_o∈\0,± 1\, and Ss∈1,3,5,6,∞S_s∈\1,3,5,6,∞\. For example, in Wan 2.1, the DiT shape is 21×45×8021× 45× 80. After applying 3D-Padding to obtain 21×48×8021× 48× 80, we set the PSABlock size to 3×8×163× 8× 16. Then, the ranges for ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) are configured as follows: Tw<7T_w<7, To∈0,±1,±2T_o∈\0,± 1,± 2\, Ts∈1,2,3,∞T_s∈\1,2,3,∞\, Sw<5S_w<5, So∈0,±1S_o∈\0,± 1\, Ss∈1,3,5,6,∞S_s∈\1,3,5,6,∞\. Similarly, for HunyuanVideo at 720×1280720× 1280 resolution, the DiT shape is 33×45×8033× 45× 80. After 3D-Padding to 33×48×8033× 48× 80, we design the PSABlock as 3×8×163× 8× 16. The ranges for ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) are set as: Tw<11T_w<11, To∈0,±1,±2T_o∈\0,± 1,± 2\, Ts∈1,2,3,∞T_s∈\1,2,3,∞\, Sw<5S_w<5, So∈0,±1S_o∈\0,± 1\, Ss∈1,3,5,6,∞S_s∈\1,3,5,6,∞\. For HunyuanVideo at 768×1280768× 1280 resolution, the DiT shape is 30×48×8030× 48× 80 without 3D-Padding. We design the PSABlock as 6×8×86× 8× 8, and the ranges for ℳ(Tw,To,Ts,Sw,So,Ss)M(T_w,T_o,T_s,S_w,S_o,S_s) are configured as: Tw<5T_w<5, To∈0,±1,±2T_o∈\0,± 1,± 2\, Ts∈1,2,3,∞T_s∈\1,2,3,∞\, Sw<10S_w<10, So∈0,±1S_o∈\0,± 1\, Ss∈1,3,5,6,∞S_s∈\1,3,5,6,∞\. It is worth noting that setting Ts=∞T_s=∞ or Ss=∞S_s=∞ indicates that there is only one diagonal in the inter-frame or intra-frame attention, respectively. In practice, it suffices to set TsT_s no less than the number of frames T and SsS_s no less than H×WH× W. C.3 Wan 2.1 14B To demonstrate the practical effectiveness of our framework, we analyzed the mask distribution on the Wan 2.1 14B model. Using a loss tolerance of α=0.007α=0.007, our search achieved a significant 57.57% overall sparsity, as shown in table 7. Table 7: Top 10 most frequent PSA configurations discovered by the search algorithm in Wan 2.1 14B using a search threshold of α=0.007α=0.007. Mask Configuration Head Count (Tw,To,Ts)(T_w,T_o,T_s) (Sw,So,Ss)(S_w,S_o,S_s) # Heads % Total full window 32000 20.00 (7, 0, ∞) (3, 0, 3) 25257 15.79 (1, 0, ∞) (5, 0, 6) 11778 7.36 (7, 0, ∞) (1, 0, 3) 11069 6.92 (7, 0, ∞) (3, 0, 1) 9014 5.63 (7, 0, ∞) (3, 0, 5) 6810 4.26 (7, 0, ∞) (1, 0, 5) 5655 3.53 (7, 0, ∞) (5, 0, 3) 5470 3.42 (1, 0, 3) (5, 0, 6) 5088 3.18 (3, 0, ∞) (5, 0, 6) 4622 2.89 All Other Configurations 43237 27.02 Total 160000 100.00 C.4 HunyuanVideo 768×1280 On the HunyuanVideo model at a 768×1280 resolution, we set the L2L_2 loss threshold to α=0.018α=0.018 for PSA-Search. This resulted in a heterogeneous mask combination that achieved an overall sparsity of 60.33%. The distribution of the ten most prevalent mask configurations is presented in table 8. Table 8: Top 10 most frequent PSA configurations discovered for the HunyuanVideo model (768×1280 resolution) using a search threshold of α=0.018α=0.018. Mask Configuration Head Count (Tw,To,Ts)(T_w,T_o,T_s) (Sw,So,Ss)(S_w,S_o,S_s) # Heads % Total full window 14404 20.00 (1, 0, ∞) (10, 0, 6) 9213 12.80 (5, 0, ∞) (1, 0, 1) 5494 7.63 (5, 0, ∞) (1, 0, 3) 4889 6.79 (5, 0, ∞) (3, 0, 3) 4651 6.46 (5, 0, ∞) (3, 0, 1) 3828 5.32 (5, 0, ∞) (1, 0, 5) 1789 2.48 (5, 0, ∞) (1, 0, 6) 1701 2.36 (5, 0, ∞) (5, 0, 1) 1567 2.18 (5, 0, ∞) (7, 0, 6) 1522 2.11 All Other Configurations 22942 31.86 Total 72000 100.00 C.5 HunyuanVideo 720×1280 For the HunyuanVideo model at a 720×1280 resolution, we applied PSA-Search using a loss threshold of α=0.03α=0.03. This process yielded a mask combination achieving 59.74% overall model sparsity. table 9 summarizes the ten most frequently chosen mask configurations discovered during this search. Table 9: Top 10 most frequent PSA configurations for the HunyuanVideo model (720×1280 resolution) using a search threshold of α=0.03α=0.03. Mask Configuration Head Count (Tw,To,Ts)(T_w,T_o,T_s) (Sw,So,Ss)(S_w,S_o,S_s) # Heads % Total full window 14400 20.00 (11, 0, ∞) (1, 0, 1) 6223 8.64 (1, 0, ∞) (5, 0, 6) 4937 6.86 (3, 2, ∞) (5, 0, 6) 3995 5.55 (11, 0, ∞) (3, 0, 1) 3964 5.51 (11, 0, ∞) (1, 0, 3) 3511 4.88 (11, 0, ∞) (3, 0, 3) 2898 4.03 (1, 2, 2) (5, 0, 6) 2385 3.31 (3, 1, ∞) (5, 0, 6) 2348 3.26 (3, 0, ∞) (5, 0, 6) 2090 2.90 All Other Configurations 25249 35.06 Total 72000 100.00 Appendix D Locality-Induced Periodic Diagonal Stripes In section 3.1, we stated that video spatio-temporal locality alone can produce periodic diagonal stripes in the 2D attention heatmap. We now formalize this claim. Proposition 1 (Locality-induced diagonal stripes). Under the row-major token ordering i=t⋅HW+h⋅W+wi=t· HW+h· W+w, the locality conditions in eq. 1 produce periodic diagonal bands in the 2D attention heatmap: 1. Temporal locality (|Δt|<εt| t|< _t): For tokens at the same spatial position (h,w)(h,w), the 1D index difference is Δi=Δt⋅HW i= t· HW, forming diagonal bands with stride HWHW and half-width εt⋅HW _t· HW. 2. Column-wise locality (|Δh|<εh| h|< _h): For tokens in the same column within a frame, Δi=Δh⋅W i= h· W, forming diagonal bands with stride W and half-width εh⋅W _h· W. 3. Row-wise locality (|Δw|<εw| w|< _w): For tokens in the same row within a frame, Δi=Δw i= w, forming a continuous band of width 2εw2 _w along the main diagonal. The composition of these conditions yields periodic diagonal stripes with stride determined by the token layout (HW,W)(HW,W) and width determined by the locality radii (εt,εh,εw)( _t, _h, _w). Discussion. This proposition shows that locality alone suffices to produce periodic diagonal stripes without invoking RoPE periodicity: the stride is determined by the token layout (HWHW for inter-frame, W for intra-frame), and the width by the locality radii. For the 9.70% of heads without a dominant RoPE channel (see appendix E), stripes arise purely from this mechanism. For heads with a dominant channel, RoPE periodicity further modulates the stride to λT_λ (eq. 5) and introduces nonzero offset via ϕ(mλ∗)φ^(m^*_λ), as described in the Joint effect paragraph of section 3.1. Appendix E Empirical Support for mλ∗m^*_λ in RoPE-3D To empirically validate the existence of a predominant channel mλ∗∈Mλm^*_λ∈ M_λ that satisfies eq. 3 for structured attention patterns, we visualize the RoPE-channel contribution distributions of attention across a large number of attention heads in Wan 2.1 and HunyuanVideo, as shown in fig. 5. For each attention head, the RoPE-channel weights are computed by performing a dot product between the query and key vectors along each row, normalizing the resulting weights, and subsequently averaging over all rows at each corresponding position. The observations are summarized as follows: 1. Figures (a)–(b) illustrate the RoPE-channel weight distributions associated with the intra-frame pattern. In this case, the dominant channel m∗m^* is consistently localized within MwM_w. Moreover, as the channel index within the MwM_w dimension increases, the spacing between adjacent intra-frame diagonal stripes increases accordingly, which corroborates the periodicity formulation in eq. 5. 2. Figures (c)–(d) illustrate the RoPE-channel weight distributions associated with the inter-frame pattern. Here, the dominant channel m∗m^* is consistently localized within MtM_t. Analogously, as the channel index within the MtM_t dimension increases, the spacing between adjacent inter-frame diagonal stripes widens accordingly, consistent with the behavior described in eq. 5. 3. Figure (e) corresponds to the hybrid pattern. Under this regime, pronounced dominant channels m∗m^* simultaneously emerge across multiple sub-groups MtM_t, MhM_h, and MwM_w, reflecting a superposition of multiple distinct attention patterns. 4. Figure (f) corresponds to the uniform pattern. Under this regime, the weight distributions across all channels exhibit a disordered and non-structured configuration, with no discernible dominant channel. Quantitatively, we measure the dominance ratio as maxmwm/medianmwm _mw_m/median_mw_m for each head, where wmw_m denotes the normalized RoPE-channel weight. On Wan 2.1, 87.55% of all attention heads exhibit a dominance ratio exceeding 3×3×, confirming that the single-channel dominance assumption underlying eq. 3 holds for the vast majority of heads. Through experimental observation, among the remaining 12.45% of heads, 2.74% correspond to the uniform pattern, while the remaining 9.70% still exhibit diagonal stripe structures. These stripes arise from the spatio-temporal locality (εt _t, εh _h, εw _w) described in eq. 1, and can likewise be efficiently sparsified by PSA. As formalized in appendix D, locality alone produces periodic diagonal stripes with stride determined by the token layout (HWHW, W), confirming that these heads are not exceptions but a natural sub-case of the unified stripe structure. To verify that PSA effectively handles these locality-driven heads, we separately evaluate the 9.70% of heads without a dominant RoPE channel. At 60% sparsity, these heads achieve an average cosine similarity of 0.9872 with full attention, confirming that the locality-driven stripes are equally well captured by ℳM. Although the uniform pattern lacks a dominant m∗m^*, PSA-Search effectively handles these heads: the discovered masks fall into two categories—wide-stripe configurations (large TwT_w or SwS_w) and checkerboard patterns (periodic small-stride masks)—both of which achieve faithful approximations of the uniform distribution (cosine similarity >> 0.98) while maintaining meaningful sparsity. This confirms that the PSA mask space is expressive enough to cover all four observed pattern types. Figure 5: Empirical support for mλ∗m^*_λ in RoPE-3D Appendix F Heatmaps of Different Heads in CogVideoX Figure 6: CogVideoX Heatmap Patterns. As shown in fig. 6, we analyze various patterns in the attention heatmaps of CogVideoX, which exhibit behaviors consistent with HunyuanVideo and Wan 2.1. Four distinct pattern types are identified: • Figures (b), (e)-(h) demonstrate intra-frame attention patterns. These patterns are characterized by diagonal structures with varying widths, strides, and offsets within each frame, with the pattern repeating identically across all frames. • Figures (a), (d) showcase inter-frame attention patterns. In these patterns, attention weights within each frame are relatively uniform, while diagonal band structures with varying widths, strides, and offsets emerge between frames. • Figure (i) presents a hybrid pattern, which can be interpreted as a combination of intra-frame and inter-frame attention patterns. • Figure (c) displays uniform pattern, where attention is uniformly distributed across all spatio-temporal positions. Through further examination of the CogVideoX heatmap, we observe that it demonstrates sparse patterns analogous to those exhibited by Wan 2.1 and HunyuanVideo. The underlying reason is that these models all utilize DiT architectures for video generation, maintain structural similarities, and optimize toward identical objectives. Consequently, their weight distributions should also converge to comparable configurations, suggesting that the periodic locality characteristic of attention maps is universally applicable across DiT-based video generation scenarios. This demonstrates the general adaptability of PSA. Appendix G Heatmap Visualization Across Different Prompts As shown in fig. 7, it can be intuitively observed that the heatmaps of the same head across different prompts exhibit extremely high similarity (cosine similarity >> 0.99). Figure 7: Heatmap Comparison Between Prompts. Appendix H Loss Threshold Sensitivity Analysis As illustrated in fig. 8, we perform loss threshold sensitivity analysis on various α values for HunyuanVideo and Wan 2.1. The experimental results indicate that larger α values correspond to higher tolerable loss thresholds, which given the design principle of PSA-Search, result in increased sparsity. To maintain sparsity at approximately 60%, we configured α as 0.03, 0.018, and 0.007 for HunyuanVideo (720×1280)(720× 1280), HunyuanVideo (768×1280)(768× 1280), and Wan 2.1, respectively. This observation reveals that Wan 2.1 exhibits higher sensitivity to loss, thereby necessitating a smaller α value to attain equivalent sparsity levels. (a) HunyuanVideo (720×1280720× 1280) (b) HunyuanVideo (768×1280768× 1280) (c) Wan 2.1 14B Figure 8: Sparsity variation with different α values across different models. Furthermore, we obtain α values at various sparsity levels through our adaptive search algorithm for reference. As shown in table 10, setting the corresponding alpha value for a given sparsity level ensures optimal acceleration while minimizing the loss. Note that changing α does not require a time-consuming re-search, as PSA-Search saves the HeadResults for each head; consequently, after each adjustment of α, only the re-execution of algorithm 3 (lines 10–21) is needed, taking approximately 2 minutes. Table 10: Alpha-sparsity correspondence for different models. HunyuanVideo (720×1280720× 1280) HunyuanVideo (768×1280768× 1280) Wan 2.1 14B Sparsity (%) α Sparsity (%) α Sparsity (%) α 18.72 0.0003 20.59 0.0003 19.65 0.0003 28.65 0.0010 30.04 0.0010 31.63 0.0010 40.85 0.0030 42.16 0.0030 42.05 0.0021 49.73 0.0070 51.66 0.0070 51.51 0.0041 62.80 0.0600 60.33 0.0180 61.22 0.0110 67.22 0.6000 69.09 0.0600 65.46 0.3000 Appendix I End-to-End Performance and Quality Across Varying Sparsity Levels Beyond our initial analysis at 60% sparsity to match STA’s fixed sparsity (58.37%) for a direct comparison, we evaluate 10–70% sparsity (table 11). Our method consistently outperforms SVG2 in speed/quality, and achieves a better speed–quality trade-off than SVG. SVG only achieves marginal speedup at 70% sparsity while suffering severe quality degradation. Table 11: End-to-end performance and quality comparison across varying sparsity on HunyuanVideo (768 × 1280). Method Sparsity E2E(s) PSNR↑ SSIM↑ LPIPS↓ SVG 10.48% 1393.25 30.34 0.9144 0.1247 30.39% 1116.87 27.68 0.8862 0.1552 50.32% 865.78 25.67 0.8494 0.1914 70.16% 622.37 21.21 0.7771 0.2651 SVG2 10.49% 1333.93 31.87 0.9076 0.1206 30.72% 1129.91 26.75 0.7753 0.2040 50.67% 945.26 26.18 0.7314 0.2352 70.93% 694.07 24.54 0.6610 0.2985 Ours 11.28% 1057.19 38.83 0.9647 0.0526 30.04% 941.44 33.99 0.9318 0.0812 51.66% 776.87 30.21 0.8952 0.1244 69.09% 632.86 24.58 0.8220 0.2340 Appendix J Qualitative Comparison with Baselines As shown in figs. 9, 10 and 11, we execute multiple prompts across several models. By comparing the extracted video frames with the baseline using FlashAttention-3, we can observe that the quality is nearly lossless. Figure 9: HunyuanVideo(720×1280) FA3 Baseline vs. Ours. Figure 10: HunyuanVideo(768×1280) FA3 Baseline vs. Ours. Figure 11: Wan 2.1(720×1280) FA3 Baseline vs. Ours. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state three contributions of PSA (parameterized representation, efficient kernel, automatic search), which are consistent with the theoretical and experimental results in Sections 3–4. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discusses limitations in Appendix A. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The RoPE-3D attention logit decomposition (Eq. 1) and the dominant channel assumption (Eqs. 2–4) are presented with clear assumptions and derivations in Section 3.1. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The paper provides complete algorithm pseudocode (Algorithms 1–3), model configurations (Section 4.1), search parameter ranges (Appendix B), and kernel implementation details (Appendix A), sufficient to reproduce the main experiments. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code is publicly available at https://github.com/jxyjason/PSA, with sufficient instructions to faithfully reproduce the main experimental results. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Section 4.1 provides detailed descriptions of model configurations, evaluation metrics, baseline methods, search parameters, and experimental settings. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: The randomness in video generation evaluation primarily stems from initial noise, while our comparison is between deterministic sparse attention and full attention outputs, so error bars are not required. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: The paper states that all experiments are conducted on a single NVIDIA H100 GPU, and mentions in Section 3.3 that the search completes in 2.5 hours on 8 H100 GPUs. 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: This research is an efficiency optimization method for video generation and does not involve human subjects, sensitive data, or ethical risks. 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [N/A] Justification: This research is a foundational efficiency optimization method that accelerates inference of existing models and is not directly tied to specific application deployments, with no direct path to negative societal impacts. 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: This paper does not release new pre-trained models or datasets; it only provides an attention efficiency optimization method. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All existing assets used in this paper are properly credited, and their licenses and terms of use are explicitly stated and fully respected. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code is released at https://github.com/jxyjason/PSA with documentation provided in the repository; no other new assets are involved. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: This paper does not involve crowdsourcing or research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: This paper does not involve research with human subjects. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: LLMs are not a component of the core method in this research; the core method is sparse attention optimization based on parameterized stripe attention.