Paper deep dive
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
Jianlin Yu, Jing Lin, Linghui Kong, Aiyue Chen, Weiyi Sun, Chenyu Zeng, Wangli Lan, Jinxi Li, Zhuo Zheng, Ziyang Yue, Danning Ke, Fei Yi, Tianchi Hu, Yuan Ding, Yiwu Yao, Junsong Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The quadratic cost of attention is a major bottleneck in diffusion-based video generation models. MXFP4 attention provides a promising path toward efficient inference, but direct MXFP4 quantization often degrades generation quality due to two numerical issues: the clipping-underflow trade-off from power-of-two scaling and the row-wise normalization error introduced in the softmax loop. We propose MXAttention, a data-free post-training quantization framework for MXFP4 attention. MXAttention introduces two components: Universal Optimal Scaling (UOS), which exploits the periodic structure of power-of-two microscaling to derive a distribution-independent optimal scaling boundary Qmax=7.25 without calibration or search, and Pre-Normalization Quantization (PNQ), which quantizes unnormalized softmax exponentials before row-wise summation to preserve normalization by construction. Experiments on Wan2.2 and HunyuanVideo show that MXAttention closes at least 95% of the VBench Imaging Quality gap between OCP MXFP4 and FP16, substantially improves frame-level similarity, and preserves FP16-level generation quality with less than 0.01 absolute degradation on all reported VBench metrics. MXAttention also achieves performance competitive with strong NVFP4-based baselines with negligible overhead when fused into the attention pipeline. The implementation is publicly available in MindIE-SD.
Tags
Links
- Source: https://arxiv.org/abs/2607.24377v1
- Canonical: https://arxiv.org/abs/2607.24377v1
Trouble viewing inline? Open PDF directly →
Full Text
85,373 characters extracted from source content.
Expand or collapse full text
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention Jianlin Yu, Jing Lin, Linghui Kong, Aiyue Chen, Weiyi Sun, Chenyu Zeng, Wangli Lan, Jinxi Li Zhuo Zheng, Ziyang Yue, Danning Ke, Fei Yi, Tianchi Hu, Yuan Ding, Yiwu Yao, Junsong Wang Huawei Technologies Co., Ltd. Corresponding author. Abstract The quadratic cost of attention is a major bottleneck in diffusion-based video generation models. MXFP4 attention offers a promising path toward lower-cost inference, but directly applying standard MXFP4 quantization to attention often degrades generation quality. We attribute this degradation primarily to two numerical failure modes: power-of-two shared scaling creates a clipping–underflow trade-off within MXFP4 blocks, while direct MXFP4 quantization in the softmax loop breaks row-wise normalization, so the induced attention weights no longer sum to one after quantization. To address these issues, we propose MXAttention, a data-free post-training quantization framework for MXFP4 attention that achieves near-lossless generation quality through two key components. First, Universal Optimal Scaling (UOS) leverages the periodic structure induced by power-of-two microscaling to minimize a global MXFP4 quantization-error objective, deriving the closed-form, distribution-independent scaling boundary Qmax=7.25Q_ =7.25 without calibration or per-layer search. Second, Pre-Normalization Quantization (PNQ) quantizes unnormalized softmax exponentials before the row-wise summation, guaranteeing that the induced attention probabilities sum to one exactly and preventing row-sum errors from accumulating in the attention output. As a fully data-free method, MXAttention requires neither calibration nor QAT. Empirically, across Wan2.2 and HunyuanVideo, MXAttention closes at least 95% of the VBench Imaging Quality gap between vanilla OCP MXFP4 and FP16, while substantially improving frame-level similarity. It preserves FP16-level generation quality with less than 0.01 absolute degradation on all reported VBench metrics and achieves performance competitive with strong NVFP4-based baselines with negligible overhead when fused into the attention pipeline. The implementation of MXAttention has been integrated into the main branch of MindIE-SD and is publicly available at https://gitcode.com/Ascend/MindIE-SD/tree/master/mindiesd. 1 Introduction Video diffusion Transformers process spatiotemporal token sequences whose length grows with spatial resolution and video duration [12, 26, 28]. At every denoising step, the model performs a full forward pass through its Transformer blocks. Although fast samplers, distillation, and caching reduce the number of model evaluations or reuse intermediate computation [17, 27, 15, 34], the cost of each executed forward pass remains substantial. Attention computation is therefore a key target for low-precision acceleration: its two dominant operations, QK⊤QK and PVPV, are GEMMs that can directly exploit native FP4 throughput when Q, K, V, and the softmax-path values can be quantized with sufficiently low error [30, 33]. Native FP4 GEMM support and standardized microscaling formats have made 4-bit attention increasingly practical. MXFP4, defined by the OCP MX specification, is an open standard format supported across multiple accelerator families, including NVIDIA Blackwell GPUs, AMD Instinct MI350 series, and Ascend 950 series [22, 20, 1, 11]. It represents E2M1 values in 32-element blocks using a shared E8M0 power-of-two scale. Alongside this open standard, NVIDIA introduced NVFP4, which adopts a finer 16-value block granularity with E4M3 block scales and an additional tensor-level scale [19, 21]. MXFP4 reduces scale metadata overhead and enables simpler scale handling in low-precision GEMMs, but its coarser block granularity and power-of-two scaling restriction make accurate attention quantization challenging. Directly applying standard MXFP4 quantization to attention can therefore cause substantial degradation in generated video quality. Recent work has explored low-bit attention from several directions. Rotation-based PTQ methods such as QuaRot and SpinQuant use orthogonal transformations, including Hadamard transforms, to suppress activation outliers and have become widely used in low-bit quantization pipelines [2, 16]. These techniques improve the numerical distributions of Q and K, but do not directly address the format-specific scaling behavior of MXFP4, which can limit their effectiveness for MXFP4 attention. Attention-specific kernels such as the SageAttention series demonstrate the practicality of low-bit attention through smoothing, scaling, and kernel-level optimization [31, 29, 32, 30]. Quantization-aware training (QAT) further improves FP4 attention by adapting the model during training [33]. Recent MXFP4 quantization methods improve block quantization through overflow-aware scaling, adaptive scale selection, and metadata augmentation [5, 8, 9]. These methods improve quantization accuracy through block-dependent decisions, empirical optimization, or additional format flexibility, whereas MXAttention derives a fixed, data-free MXFP4 scaling boundary. Video-diffusion quantization methods also exploit spatiotemporal structure to quantize model weights and activations, but generally do not address the numerical constraints inside the fused attention softmax loop [14]. MXAttention instead targets a fully data-free PTQ method for MXFP4 attention, without calibration, QAT, or per-layer search. We characterize two numerical failure modes in directly quantized MXFP4 attention. First, power-of-two shared scaling creates a clipping–underflow trade-off within each MXFP4 block. A smaller shared scale preserves the resolution of smaller values but exposes larger values to saturation, whereas a larger shared scale reduces clipping at the cost of increased rounding and underflow. Second, directly inserting MXFP4 quantization into the FlashAttention online-softmax loop breaks row-wise normalization [6, 7, 24]. In this setting, the unnormalized softmax exponential tile is quantized for the output-accumulator update, while the row-wise sum is accumulated from its unquantized counterpart. Because the two updates use different representations, the induced attention weights no longer necessarily sum to one. To address these issues, we propose MXAttention, a data-free post-training quantization framework for MXFP4 attention. First, Universal Optimal Scaling (UOS) identifies a periodic structure induced by power-of-two microscaling and shows that it makes the optimal scaling boundary independent of the block-maximum distribution. Based on this observation, UOS derives the closed-form boundary Qmax=7.25Q_ =7.25 without calibration or per-layer search. Second, Pre-Normalization Quantization (PNQ) quantizes the unnormalized softmax exponentials before both the row-wise sum and output-accumulator updates. By using the same quantized values in both updates, PNQ preserves row-wise normalization by construction and avoids the additional scaling error caused by inconsistent quantization paths. Figure 1 provides an overview of MXAttention, which integrates Universal Optimal Scaling (UOS) and Pre-Normalization Quantization (PNQ) into the FlashAttention pipeline. Figure 1: Overview of MXAttention. MXAttention integrates Universal Optimal Scaling (UOS) and Pre-Normalization Quantization (PNQ) into the FlashAttention pipeline. A fixed Hadamard rotation is applied to Q/KQ/K for outlier suppression. UOS applies the distribution-independent MXFP4 scaling boundary Qmax=7.25Q_ =7.25 to all MXFP4-quantized attention tensors. During the online-softmax loop, PNQ quantizes the unnormalized softmax tile and reuses the same quantized tile for both row-wise sum and output-accumulator updates, preserving row-wise normalization by construction. Our main contributions are: • We characterize two numerical failure modes that arise when MXFP4 quantization is directly applied to attention: the clipping–underflow trade-off induced by power-of-two block scaling and the row-wise normalization error caused by inconsistent quantization in the online-softmax loop. • We identify a periodic structure induced by power-of-two microscaling and show that it makes the optimal MXFP4 scaling boundary independent of the block-maximum distribution. Based on this observation, Universal Optimal Scaling (UOS) derives a single fixed boundary Qmax=7.25Q_ =7.25 for selecting the shared power-of-two scale of each MXFP4 block and proves that it globally minimizes the proposed quantization-error objective without calibration or per-layer search. • We introduce Pre-Normalization Quantization (PNQ), which uses the same quantized softmax exponential tiles for both the row-wise sum and output-accumulator updates, preserving row-wise normalization by construction and avoiding the additional scaling error introduced by mixing quantized and unquantized paths. • We evaluate MXAttention on Wan2.2 and HunyuanVideo. Across both models, MXAttention closes at least 95% of the VBench Imaging Quality gap between vanilla OCP MXFP4 and FP16, substantially improves frame-level similarity, and preserves FP16-level generation quality, remaining within 0.01 of or exceeding FP16 on every reported VBench metric. It also achieves performance competitive with strong NVFP4-based baselines with negligible algorithmic overhead and a fusion-friendly design. 2 Preliminaries Attention and FlashAttention. Given query, key, and value matrices Q, K, and V, scaled dot-product attention computes S=QK⊤d,P=Softmax(S),O=PV,S= QK d, P=Softmax(S), O=PV, (1) where every row of P satisfies ∑jPij=1 _jP_ij=1. A conventional implementation materializes the score matrix S and probability matrix P, whose sizes grow quadratically with sequence length. FlashAttention avoids writing these matrices to HBM by partitioning Q, K, and V into blocks and evaluating attention through an online-softmax recurrence in on-chip memory [6, 7, 24]. For the recurrence below, we use the FlashAttention-2 form, which maintains an unnormalized output accumulator and applies the final normalization after all key/value blocks have been processed. We reserve P for the final normalized attention matrix, denote the unnormalized softmax exponential tile by P~ij P_ij, and denote the in-loop unnormalized output accumulator by O~i(j) O_i^(j). For query block QiQ_i and key/value blocks Kj,VjK_j,V_j, define Sij=QiKj⊤d.S_ij= Q_iK_j d. (2) FlashAttention maintains a row-wise running maximum mi(j)m_i^(j), a row-wise sum of exponentials ℓi(j) _i^(j), and an unnormalized output accumulator O~i(j) O_i^(j). Starting from mi(0)=−∞m_i^(0)=-∞, ℓi(0)=0 _i^(0)=0, and O~i(0)=0 O_i^(0)=0, it updates mi(j) m_i^(j) =max(mi(j−1),rowmax(Sij)), = \! (m_i^(j-1),rowmax(S_ij) ), (3) αi(j) _i^(j) =exp(mi(j−1)−mi(j)), = \! (m_i^(j-1)-m_i^(j) ), (4) P~ij P_ij =exp(Sij−mi(j)Bc⊤), = \! (S_ij-m_i^(j)1_B_c ), (5) ℓi(j) _i^(j) =αi(j)⊙ℓi(j−1)+P~ijBc, = _i^(j) _i^(j-1)+ P_ij1_B_c, (6) O~i(j) O_i^(j) =Diag(αi(j))O~i(j−1)+P~ijVj. =Diag\! ( _i^(j) ) O_i^(j-1)+ P_ijV_j. (7) Here, P~ij P_ij contains the unnormalized softmax exponentials and enters the second attention GEMM; it is not the final normalized probability tile. After all TcT_c key/value blocks have been processed, Oi=Diag(ℓi(Tc))−1O~i(Tc).O_i=Diag\! ( _i^(T_c) )^-1 O_i^(T_c). (8) FlashAttention therefore computes the normalized output from the row-wise sum ℓi(Tc) _i^(T_c) and the unnormalized output accumulator O~i(Tc) O_i^(T_c), without materializing the full probability matrix P. Materializing P for quantization would reintroduce quadratic intermediate storage and memory traffic. A FlashAttention-compatible low-bit implementation must instead integrate quantization into the tiled online-softmax loop. MXFP4 Block Quantization. MXFP4 is a microscaling format defined by the OCP MX specification [22, 23]. Each block contains B=32B=32 E2M1 elements that share an E8M0 power-of-two scale σ=2eσ=2^e. The E2M1 grid is E2M1=0,±0.5,±1,±1.5,±2,±3,±4,±6.G_E2M1=\0,± 0.5,± 1,± 1.5,± 2,± 3,± 4,± 6\. (9) For an element v, we write MXFP4 quantization as σ(v)=σΠE2M1(vσ),Q_σ(v)=σ\, _G_E2M1\! ( vσ ), (10) where ΠE2M1 _G_E2M1 denotes round-to-nearest projection onto the finite E2M1 grid, with out-of-range values saturated at ±6± 6. Let M=maxv∈ℬ|v|M= _v |v| be the maximum magnitude of a nonzero block ℬB. The standard MX conversion rule computes eOCP=⌊log2M⌋−emaxelem,σOCP=2eOCP,e_OCP= _2M -e_ ^elem, _OCP=2^e_OCP, (11) where emaxeleme_ ^elem is the exponent of the largest power-of-two value in the element format [23]. For E2M1, emaxelem=2e_ ^elem=2, corresponding to the grid value 44. We express different scale-selection rules through a common boundary QmaxQ_ : efloor(Qmax) e_floor(Q_ ) =⌊log2(MQmax)⌋+1, = _2 ( MQ_ ) +1, σfloor(Qmax) _floor(Q_ ) =2efloor(Qmax), =2^e_floor(Q_ ), (12) eceil(Qmax) e_ceil(Q_ ) =⌈log2(MQmax)⌉, = _2 ( MQ_ ) , σceil(Qmax) _ceil(Q_ ) =2eceil(Qmax). =2^e_ceil(Q_ ). (13) The exact OCP rule used in our experiments is efloor(8)e_floor(8), which maps the normalized block maximum into [4,8)[4,8). Because 88 is not representable in E2M1, normalized block maxima above 77 fall into an overflow-rounding region and are saturated to 66. TetraJet’s Truncation-Free Scaling (TFS) corresponds to eceil(6)e_ceil(6), giving 3<Mσceil(6)≤6,3< M _ceil(6)≤ 6, (14) which avoids overflow of the block maximum but can increase rounding and underflow errors for smaller values [4]. The floor- and ceiling-based forms agree except when M/QmaxM/Q_ is an exact power of two; this measure-zero endpoint distinction does not affect the continuous analysis in Section 4.1. Our UOS method adopts the ceiling-based scaling rule and derives a fixed boundary Qmax=7.25Q_ =7.25 by minimizing a global MXFP4 quantization-error objective that captures the trade-off between clipping large values and preserving smaller values. It requires neither calibration nor per-layer search. For an all-zero block, we set σ=1σ=1 and quantize every element to zero. 3 Failure Modes of Standard MXFP4 Attention Directly applying standard MXFP4 quantization to the attention pipeline can substantially degrade generated video quality. We trace this degradation to two distinct numerical failure modes: the clipping–underflow trade-off induced by block scaling and the normalization mismatch introduced in the online-softmax loop. Clipping–Underflow Trade-off. Under the standard OCP conversion rule, the normalized block maximum xmax=M/σOCPx_ =M/ _OCP lies in the interval [4,8)[4,8) [22, 23]. However, the largest finite E2M1 value is 66. Under round-to-nearest conversion followed by finite-range saturation, normalized maxima in (7,8)(7,8) fall into the overflow-rounding region: their unconstrained rounded value is 88, which is not representable in E2M1, and they are therefore saturated to 66. Under the uniform wrapped-phase approximation discussed in Appendix D, xmaxx_ has density p(x)=1xln2p(x)= 1x 2 over [4,8)[4,8). The probability that the block maximum enters this region is therefore Pr(xmax≥7)=∫781xln2x=log287=1−log274≈19.27%. (x_ ≥ 7)= _7^8 1x 2\,dx= _2 87=1- _2 74≈ 19.27\%. (15) Truncation-Free Scaling (TFS) avoids this overflow by using Qmax=6Q_ =6 [4]. When the OCP-normalized maximum exceeds 66, TFS doubles the shared scale and consequently halves the normalized magnitude of every value in the block. Although this prevents saturation of the largest values, it reduces the effective resolution available to smaller values and increases their probability of rounding to zero. OCP scaling and TFS therefore represent opposite ends of the same trade-off: the former better preserves small values but permits overflow saturation, whereas the latter removes saturation at the cost of increased rounding and underflow. Normalization Mismatch in Online Softmax. FlashAttention computes attention through a tiled online-softmax recurrence without materializing the full probability matrix [6, 7, 24]. In the direct MXFP4 baseline considered in this work, the unnormalized exponential tile is quantized for the output-accumulator update, while its unquantized counterpart is used for the row-wise sum update. Let P^ij=⋆(P~ij) P_ij=Q_ ( P_ij) denote the quantized exponential tile. The two update paths become ℓi(j) _i^(j) =αi(j)⊙ℓi(j−1)+P~ijBc, = _i^(j) _i^(j-1)+ P_ij1_B_c, (16) O~^i(j) O_i^(j) =Diag(αi(j))O~^i(j−1)+P^ijV^j. =Diag\! ( _i^(j) ) O_i^(j-1)+ P_ij V_j. (17) The row-wise sum and output accumulator are therefore updated from different representations of the same softmax exponentials. Let P~ieff P_i^eff and P^ieff P_i^eff denote the corresponding effective exponential rows after accounting for online rescaling factors. The induced attention weights satisfy P^idirect=Diag(P~ieff)−1P^ieff,P^idirect=(P^ieff)⊘(P~ieff)≠1. P_i^direct=Diag\! ( P_i^eff1 )^-1 P_i^eff, P_i^direct1= ( P_i^eff1 ) ( P_i^eff1 )≠ 1. (18) This mismatch is not an unavoidable consequence of element-wise quantization; it arises because the row-wise sum and output-accumulator updates use inconsistent representations of the same softmax exponentials. Zeroing, saturation, and rounding all perturb the quantized exponential mass. As quantified in Section 5.5.2, the induced row sums of the direct online MXFP4 baseline on Wan2.2 have an overall mean of 0.93360.9336 across five denoising steps, two prompts, and 40 attention layers. All ten step–prompt groups have mean row sums below one, although individual rows range from 0.71810.7181 to 1.11271.1127. The resulting row-dependent scaling error perturbs the magnitude of the attention output and propagates through subsequent Transformer blocks and denoising steps. 4 MXAttention This section presents MXAttention, which addresses MXFP4 quantization errors at two complementary stages of the attention pipeline. First, Universal Optimal Scaling (UOS) derives the closed-form, data-free optimal scaling boundary Qmax=7.25Q_ =7.25 by minimizing a global MXFP4 quantization-error objective. Second, Pre-Normalization Quantization (PNQ) quantizes the unnormalized softmax exponentials before both normalizer accumulation and output-accumulator updates, ensuring that the induced attention probabilities remain row-normalized by construction. Together, UOS and PNQ reduce block-level quantization error and eliminate the normalization mismatch introduced by direct MXFP4 quantization in online softmax. 4.1 Universal Optimal Scaling Under the ceiling-based scaling rule introduced in the Preliminaries, let q=Qmaxq=Q_ and let M be the maximum magnitude of a nonzero block. The shared-scale exponent and normalized block maximum are eq(M)=⌈log2(Mq)⌉,Xq=M2eq(M)∈(q2,q].e_q(M)= _2 ( Mq ) , X_q= M2^e_q(M)∈ ( q2,q ]. (19) We analyze nonzero blocks and assume that eq(M)e_q(M) lies within the E8M0 exponent range. The cases M=0M=0, for which log2M _2M is undefined, and exponents outside this range are handled by the MXFP4 conversion routine and do not affect the derivation of q. For a fixed block maximum M, increasing q can select a smaller shared scale. This increases all normalized magnitudes, reducing rounding and underflow for small values while moving the block maximum closer to the upper end of the E2M1 range. Decreasing q has the opposite effect. UOS formulates this trade-off as an optimization over q. Data-Free Quantization-Error Objective. Because the E2M1 grid is symmetric, it is sufficient to analyze nonnegative magnitudes. Let Π+ _G_+ denote round-to-nearest projection onto the nonnegative finite E2M1 grid +=0,0.5,1,1.5,2,3,4,6,G_+=\0,0.5,1,1.5,2,3,4,6\, (20) with saturation at 66 [22, 23]. We define the cumulative projection error and its normalized form as E(x)=∫0x(v−Π+(v))2v,D(x)=E(x)x3.E(x)= _0^x (v- _G_+(v) )^2\,dv, D(x)= E(x)x^3. (21) Equivalently, D(x)=1x2[1x∫0x(v−Π+(v))2v].D(x)= 1x^2 [ 1x _0^x (v- _G_+(v) )^2\,dv ]. (22) Thus, D(x)D(x) is the mean squared projection error over [0,x][0,x], divided by x2x^2. The factor 1/x21/x^2 removes the overall magnitude scale, making D(x)D(x) a dimensionless relative-error measure. It is a data-free analytical measure determined by the E2M1 grid rather than the empirical MSE of a particular block. Let gq(x)g_q(x) denote the density of XqX_q. We define the global MXFP4 quantization-error objective as (q)=[D(Xq)]=∫q/2qD(x)gq(x)x.J(q)=E[D(X_q)]= _q/2^qD(x)g_q(x)\,dx. (23) Log-Periodic Boundary Relation. A direct optimization of Eq. (23) appears to require knowledge of the block-maximum distribution, which can vary across tensors, layers, and models. We next show that power-of-two scaling imposes an exact log-periodic relation that removes this distribution dependence from the derivative sign. Recall that M is the block maximum, and let U=log2MU= _2M have an arbitrary absolutely continuous density fUf_U. Appendix A shows that, within the support of XqX_q, gq(x)=g(x)=H(log2x)xln2,H(t)=∑k∈ℤfU(t+k),g_q(x)=g(x)= H( _2x)x 2, H(t)= _k f_U(t+k), (24) Here, H is the periodized density of U, obtained by summing the integer shifts of fUf_U [18, 3]. It has period one, meaning H(t+1)=H(t)H(t+1)=H(t). The parameter q selects the interval (q/2,q](q/2,q], but does not otherwise change the density within that interval. The periodized density has unit mean over one period. Under standard Fourier-convergence conditions [25], it can be written as H(t)=1+∑n≠0cnei2πnt,H(t)=1+ _n≠ 0c_ne^i2π nt, (25) where the constant term corresponds to a uniform wrapped phase, or a log-uniform density in the linear domain, and the nonzero modes describe distribution-specific harmonics. These harmonics are not assumed to be small. The endpoints q/2q/2 and q differ by exactly one power-of-two scale interval. After taking log2 _2, they are one unit apart, exactly the period of H: log2(q/2)=log2q−1. _2(q/2)= _2q-1. (26) Consequently, every harmonic takes the same value at the two endpoints: ei2πnlog2(q/2)=ei2πn(log2q−1)=ei2πnlog2q,n∈ℤ.e^i2π n _2(q/2)=e^i2π n( _2q-1)=e^i2π n _2q, n . (27) Lemma 1 (Boundary Density Ratio). For almost every q>0q>0, the density of the normalized block maximum satisfies g(q2)=2g(q).g ( q2 )=2g(q). (28) Proof. For almost every q, periodicity gives H(log2q2)=H(log2q−1)=H(log2q).H ( _2 q2 )=H( _2q-1)=H( _2q). (29) Substituting this identity into Eq. (24) gives g(q2)=H(log2q)(q/2)ln2=2g(q).g ( q2 )= H( _2q)(q/2) 2=2g(q). (30) ∎ Since D(x)g(x)D(x)g(x) is locally integrable, J is absolutely continuous; Appendix C.1 justifies the differentiation below. Differentiating Eq. (23) with respect to q using the Leibniz integral rule and applying Lemma 1 yields, for almost every q, ′(q) (q) =D(q)g(q)−12D(q2)g(q2) =D(q)g(q)- 12D ( q2 )g ( q2 ) (31) =g(q)[D(q)−D(q2)]. =g(q) [D(q)-D ( q2 ) ]. (32) All dependence on the block-maximum distribution is contained in the nonnegative factor g(q)g(q). This factor can change the magnitude of the derivative and the shape of the objective, but it cannot reverse the sign determined by D(q)−D(q/2)D(q)-D(q/2). The location of the optimum is therefore determined by the E2M1 grid rather than by the block-maximum distribution. Wherever g(q)>0g(q)>0, the condition ′(q)=0J (q)=0 requires D(q)=D(q2),D(q)=D ( q2 ), (33) or equivalently, E(q)=8E(q2).E(q)=8E ( q2 ). (34) Appendix D provides the complete Fourier interpretation and additional wrapped-distribution analysis [18, 3]. Neither a Gaussian nor a log-uniform prior is required for the UOS result. Closed-Form E2M1 Solution. Equation (32) reduces the optimization to the grid-dependent difference D(q)−D(q/2)D(q)-D(q/2). We now evaluate this difference exactly for E2M1. We first evaluate it over q∈[6,8]q∈[6,8]. The endpoints recover TFS at q=6q=6 [4] and the ceiling-based counterpart of OCP at q=8q=8, which explains the choice of interval. Define Δ(q)=E(q)−8E(q2). (q)=E(q)-8E ( q2 ). (35) Since D(q)−D(q2)=Δ(q)q3,D(q)-D ( q2 )= (q)q^3, (36) the sign of Δ(q) (q) determines the direction of (q)J(q). Exact integration over the E2M1 decision intervals gives Δ(q)=−18,6≤q≤7,(4q−29)(4q−27)8,7≤q≤8. (q)= cases- 18,&6≤ q≤ 7,\\[6.0pt] (4q-29)(4q-27)8,&7≤ q≤ 8. cases (37) The first branch is negative. In the second branch, the factor 4q−274q-27 vanishes at q=27/4=6.75q=27/4=6.75, outside its valid interval [7,8][7,8]. The only admissible sign change is therefore q⋆=294=7.25.q = 294=7.25. (38) Within [6,8][6,8], the objective is nonincreasing before q⋆q and nondecreasing afterward; the inequalities are strict wherever g(q)>0g(q)>0. Appendix C extends this sign analysis to the complete domain q>0q>0. Theorem 1 (Global Optimality of UOS). Consider nonzero blocks for which the required E8M0 shared-scale exponent is representable. Under the ceiling-based scale family in Eq. (19), for any absolutely continuous law of U=log2MU= _2M, the objective in Eq. (23) has a global minimizer q⋆=294,Qmax⋆=q⋆=7.25.q = 294, Q_ =q =7.25. (39) If the periodized density H is positive almost everywhere over one period, the minimizer is unique. Figure 2 plots the UOS objective under a log-uniform density and several nonuniform wrapped densities. The curves have different shapes but share the same minimum at q=7.25q=7.25, illustrating that distribution-specific harmonics affect the objective without shifting its grid-determined optimum. (a) Objective under a log-uniform density (b) Objectives under wrapped densities Figure 2: UOS objective under different wrapped distributions. (a) Percentage increase in the objective relative to its value at q=7.25q=7.25 under a log-uniform density. (b) The same quantity under representative nonuniform wrapped densities. Distribution-specific harmonics change the curve shapes, but all cases share the minimum q=7.25q=7.25. UOS Quantization. Algorithm 1 summarizes the resulting MXFP4 quantization procedure. UOS modifies only the shared-exponent selection rule, while block formation and element-wise E2M1 quantization remain unchanged. Algorithm 1 UOS-Based MXFP4 Block Quantization 1:Input: A block ℬ=vii=1BB=\v_i\_i=1^B, B=32B=32; UOS boundary q⋆=7.25q =7.25 2:Output: E8M0 shared scale σ; E2M1 elements pii=1B\p_i\_i=1^B 3:M←max1≤i≤B|vi|M← _1≤ i≤ B|v_i| 4:if ℬ is an all-zero blockB is an all-zero block then 5: σ←1σ← 1 6: pi←0,i=1,…,Bp_i← 0, i=1,…,B 7: return σ,pii=1Bσ,\p_i\_i=1^B 8:end if 9:e←⌈log2(M/q⋆)⌉e← _2\! (M/q ) 10:σ←2eσ← 2^e 11:for i=1i=1 to B do 12: pi←ΠE2M1(vi/σ)p_i← _G_E2M1\! (v_i/σ ) 13:end for 14:return σ,pii=1Bσ,\p_i\_i=1^B For nonzero blocks, the normalized maximum after UOS scaling satisfies q⋆2<Mσ≤q⋆, q 2< Mσ≤ q , which yields the range (3.625,7.25](3.625,7.25] for q⋆=7.25q =7.25. Comparison with Existing Scaling Rules. Table 1 compares OCP, TFS, and UOS at the standard 32-element MXFP4 block granularity. The exact OCP baseline uses a floor-based conversion rule, whereas TFS and UOS belong to the ceiling-based family. The relation to OAS is given separately in Appendix G. Table 1: Comparison of MXFP4 scaling rules at B=32B=32. All normalized maxima above 66 map to the largest finite E2M1 value. The reported overflow-rounding interval is the stricter subset in which the nearest value on an unbounded E2M1 ladder would be 88 before finite-range saturation. The endpoint at 77 depends on the tie-breaking convention. Method Shared-scale exponent Normalized maximum range Overflow-rounding interval OCP ⌊log2M⌋−2 _2M -2 [4,8)[4,8) (7,8)(7,8) TFS ⌈log2(M/6)⌉ _2(M/6) (3,6](3,6] None UOS ⌈log2(M/7.25)⌉ _2(M/7.25) (3.625,7.25](3.625,7.25] (7,7.25](7,7.25] Relative to TFS, UOS selects a smaller shared scale for a subset of blocks, improving the resolution available to smaller values while admitting only the narrow overflow-rounding interval (7,7.25](7,7.25]. Relative to OCP, UOS substantially narrows this interval. The optimum is not obtained by separately equating a clipping term and an underflow term. Instead, Eq. (33) balances the total normalized projection errors at the two endpoints associated with adjacent power-of-two scale choices. Figure 3: Normalized block-maximum ranges under representative MXFP4 scaling rules. The exact OCP floor rule maps block maxima to [4,8)[4,8), with normalized maxima above 77 entering the overflow-rounding region. TFS uses (3,6](3,6] to avoid overflow of the block maximum, while UOS selects the intermediate range (3.625,7.25](3.625,7.25]. 4.2 Pre-Normalization Quantization Softmax normalizes each attention row: ∑jPij=1. _jP_ij=1. (40) Because MXFP4 quantizes microscaling blocks independently, it does not by itself preserve this row-wise sum. Consequently, where quantization is inserted in the online-softmax loop determines whether the induced attention weights remain normalized. Throughout this section, ⋆(⋅)Q_ (·) denotes block-wise MXFP4 quantization using the UOS boundary q⋆=7.25q =7.25. To isolate the effect of quantization placement, both the direct formulation and PNQ use the same quantizer ⋆Q_ . Figure 4: Pre-Normalization Quantization (PNQ). (A) Standard attention forms the normalized probability matrix P before computing O=PVO=PV. (B) FlashAttention avoids materializing P and updates both the row-wise sum of exponentials ℓi _i and the unnormalized output accumulator O~i O_i from the same exponential tile P~ij P_ij. (C) A direct placement uses the unquantized tile P~ij P_ij in the ℓi _i update but the quantized tile P^ij P_ij in the O~i O_i update. PNQ uses the same P^ij P_ij in both updates. Running-max rescaling and previous key/value tiles are omitted for clarity. Using the notation introduced in Section 2, consider a single tile before accounting for previous accumulators and running-max rescaling. The tile adds δℓi=P~ijBc,δO~i=P~ijVjδ _i= P_ij1_B_c, δ O_i= P_ijV_j (41) to the row-wise sum of exponentials and the unnormalized output accumulator. Both updates use the same exponential tile, as illustrated in Figure 4(B). For clarity, we omit the quantization mark on VjV_j; its treatment is identical in the direct and PNQ formulations and does not affect the row-normalization analysis. Normalization Mismatch under Direct Softmax-Path Quantization. A direct placement uses the unquantized exponential tile to update ℓi _i, but quantizes the same tile for the output-accumulator update: P^ij=⋆(P~ij),δℓidirect=P~ijBc,δO~idirect=P^ijVj. P_ij=Q_ ( P_ij), δ _i^direct= P_ij1_B_c, δ O_i^direct= P_ijV_j. (42) Here, P^ij P_ij denotes a quantized unnormalized exponential tile, not a normalized probability tile. The mismatch is already visible at the tile level because P^ijBc≠P~ijBc P_ij1_B_c≠ P_ij1_B_c (43) in general. The final row-wise sum is therefore computed from different weights than those used in the output update. Appendix I gives the corresponding full-row expression after all online rescaling factors are included. Pre-Normalization Quantization. PNQ quantizes the exponential tile before either update: P^ij=⋆(P~ij), P_ij=Q_ ( P_ij), (44) δℓiPNQ=P^ijBc,δO~iPNQ=P^ijVj.δ _i^PNQ= P_ij1_B_c, δ O_i^PNQ= P_ijV_j. (45) The row-wise sum and output accumulator are therefore updated from the same quantized exponential values, as shown in Figure 4(C). In the complete online-softmax recurrence, let αi(j)=exp(mi(j−1)−mi(j)) _i^(j)= \! (m_i^(j-1)-m_i^(j) ) (46) be the row-wise factor applied when the running maximum changes. PNQ updates ℓi(j) _i^(j) =αi(j)⊙ℓi(j−1)+P^ijBc, = _i^(j) _i^(j-1)+ P_ij1_B_c, (47) O~i(j) O_i^(j) =Diag(αi(j))O~i(j−1)+P^ijVj. =Diag\! ( _i^(j) ) O_i^(j-1)+ P_ijV_j. (48) Both FlashAttention states use the same running-max rescaling and the same quantized exponential tile. Appendix H shows that, after all key/value tiles have been processed, the two states can be written as ℓi=P^ieff,O~i=P^ieffV,Oi=Diag(ℓi)−1O~i, _i= P_i^eff1, O_i= P_i^effV, O_i=Diag( _i)^-1 O_i, (49) where P^ieff P_i^eff is a conceptual full-row matrix obtained by rescaling each quantized tile to the final row maximum and concatenating the resulting tiles. It is used only for analysis and is not materialized by the kernel. The normalized attention weights induced by PNQ are PiPNQ=Diag(P^ieff)−1P^ieff.P_i^PNQ=Diag\! ( P_i^eff1 )^-1 P_i^eff. (50) Proposition 1 (Row-Wise Normalization under PNQ). For every attention row considered in this work, the weights induced by PNQ satisfy, in exact arithmetic, PiPNQ=.P_i^PNQ1=1. (51) Proof. Substituting Eq. (50) gives PiPNQ=Diag(P^ieff)−1P^ieff=.P_i^PNQ1=Diag\! ( P_i^eff1 )^-1 P_i^eff1=1. (52) ∎ PNQ does not change the element-wise MXFP4 quantizer. Its role is to make the resulting quantization error enter the ℓi _i and O~i O_i updates consistently, preventing it from becoming an additional row-dependent scale error. For the attention rows considered in this work, the key set is nonempty. Appendix J shows that the tile containing the final row maximum contributes an exponential value of 11, which is represented exactly under q⋆=7.25q =7.25; hence, ℓi>0 _i>0. Ordinary floating-point accumulation and division may still introduce their usual rounding error; Proposition 1 concerns the normalization identity of the induced weights. PNQ requires no additional pass over the attention matrix. The quantized exponential tile already used by the low-precision PVPV GEMM is reused in the ℓi _i update. The mechanism-validation experiments evaluate the row-sum behavior of the direct placement and PNQ. 4.3 Overall MXAttention Pipeline MXAttention integrates UOS and PNQ into tiled attention computation. At runtime, a fixed orthogonal Hadamard rotation RHR_H is applied to Q and K for outlier suppression. Let ⋆(⋅)Q_ (·) denote block-wise MXFP4 quantization using the fixed UOS boundary q⋆=7.25q =7.25. The MXFP4 quantization of Q, K, V, and the unnormalized softmax exponential tiles P~ij P_ij all uses this boundary. PNQ reuses each quantized tile P^ij P_ij in both the row-wise sum and output-accumulator updates. Algorithm 2 MXAttention Forward Pass 1:Input: Query, key, and value tensors Q,K,VQ,K,V; fixed Hadamard rotation RHR_H; UOS boundary q⋆=7.25q =7.25 2:Output: Attention output O 3:Qrot←QRHQ^rot← QR_H 4:Krot←KRHK^rot← KR_H 5:Q^←⋆(Qrot) Q _ (Q^rot) 6:K^←⋆(Krot) K _ (K^rot) 7:V^←⋆(V) V _ (V) 8:for each query tile i do 9: mi←−∞,ℓi←0,O~i←0m_i←-∞, _i← 0, O_i← 0 10: for each key/value tile j do 11: Sij←Q^iK^j⊤/dS_ij← Q_i K_j / d 12: minew←max(mi,rowmax(Sij))m_i^new← \! (m_i,rowmax(S_ij) ) 13: αi←exp(mi−minew) _i← \! (m_i-m_i^new ) 14: P~ij←exp(Sij−minewBc⊤) P_ij← \! (S_ij-m_i^new1_B_c ) 15: P^ij←⋆(P~ij) P_ij _ ( P_ij) 16: ℓi←αi⊙ℓi+P^ijBc _i← _i _i+ P_ij1_B_c 17: O~i←Diag(αi)O~i+P^ijV^j O_i ( _i) O_i+ P_ij V_j 18: mi←minewm_i← m_i^new 19: end for 20: Oi←Diag(ℓi)−1O~iO_i ( _i)^-1 O_i 21:end for 22:return O 5 Experiments We evaluate MXAttention on two large-scale video diffusion models through end-to-end generation quality, fully 4-bit ablations, and mechanism-level analyses of UOS and PNQ. 5.1 Experimental Setup Models. We evaluate two large-scale open-source text-to-video models with distinct attention architectures: Wan2.2-14B and HunyuanVideo-13B. Wan2.2 employs separate self-attention and cross-attention modules, whereas HunyuanVideo adopts dual-stream and single-stream Transformer blocks with full attention. Datasets & Generation. We use the same fixed prompt subset from the Open-Sora prompt suite for all compared methods [35]. For each model and prompt, all methods use the same generation seed, enabling paired comparisons with the FP16 output. Evaluation Metrics. We evaluate generated-video quality with VBench [10], reporting Subject Consistency, Imaging Quality, and Aesthetic Quality, averaged over all generated videos. To measure frame-level similarity to the FP16 baseline, we report frame-level cosine similarity, Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio (PSNR), averaged over frames. Implementation Details. We follow the recommended inference settings of both models. Wan2.2 generates 81-frame videos at 720p resolution using 40 denoising steps, while HunyuanVideo generates 129-frame videos at 720p resolution using 50 denoising steps. Only the Wan2.2 main comparison in Table 2 uses a fixed hybrid-precision policy: Block 0 and the final two denoising steps (38 and 39) remain in high precision, while all remaining attention computations use 4-bit attention. All other experiments use fully 4-bit attention, including the HunyuanVideo main comparison, the ablation studies on both models, and all mechanism-validation experiments. Here, fully 4-bit attention means that every attention block at every denoising step uses the 4-bit attention kernel; non-attention model components retain their original precision. 5.2 Baselines and Configurations We compare MXAttention with FP16 and representative 4-bit attention configurations: • FP16. The unmodified full-precision attention implementation. • MXFP4 (OCP). A direct MXFP4 attention baseline using the exact floor-based OCP conversion rule [22, 23]. The unnormalized softmax tile P~ P is quantized into P^=⋆(P~) P=Q_ ( P) for the output-accumulator update, while the row-wise sum ℓ is updated from the unquantized tile. • NVFP4. A direct NVFP4 implementation in which Q, K, V, and P~ P are quantized for the two attention GEMMs. No additional smoothing or softmax-path scaling is applied [19, 21]. • NVFP4 + SageAttention. The NVFP4 baseline augmented with channel-wise Q/KQ/K smoothing and two-level scaling for the softmax-path tile, following the SageAttention design [30]. • MXAttention. Our method applies a fixed Hadamard rotation to Q/KQ/K for outlier suppression [2, 16], uses the UOS boundary Qmax=7.25Q_ =7.25 for all MXFP4 quantization steps, and applies PNQ so that the same quantized softmax tile updates both ℓ and the output-accumulator state. 5.3 Main Results Table 2: End-to-end evaluation of 4-bit attention methods. All 4-bit Wan2.2 methods use the same hybrid-precision policy, retaining high precision for Block 0 and denoising steps 38–39. HunyuanVideo uses fully 4-bit attention. Bold values denote the best result among quantized methods. Model Method VBench Metrics (↑ ) Similarity to FP16 (↑ ) Subject Imaging Aesthetic Cosine SSIM PSNR Wan2.2 FP16 0.9562 0.7085 0.6042 – – – MXFP4 (OCP) 0.9517 0.6414 0.6202 0.9290 0.5076 15.58 NVFP4 0.9583 0.6973 0.6158 0.9278 0.5275 15.52 NVFP4 + SageAttention 0.9558 0.6980 0.6127 0.9510 0.6177 17.72 MXAttention 0.9544 0.7054 0.6229 0.9536 0.6319 17.92 HunyuanVideo FP16 0.9665 0.6185 0.6242 – – – MXFP4 (OCP) 0.9781 0.4459 0.5928 0.9489 0.5954 16.25 NVFP4 0.9666 0.6221 0.6162 0.9635 0.6359 17.80 NVFP4 + SageAttention 0.9649 0.6063 0.6288 0.9800 0.7424 20.57 MXAttention 0.9690 0.6380 0.6344 0.9745 0.7061 19.23 Table 2 shows that direct OCP MXFP4 attention causes a substantial loss in generation quality. Imaging Quality decreases from 0.7085 to 0.6414 on Wan2.2 and from 0.6185 to 0.4459 on HunyuanVideo. MXAttention raises the scores to 0.7054 and 0.6380, respectively: it recovers 95.4% of the FP16–MXFP4 gap on Wan2.2 and fully closes the gap on HunyuanVideo, where it exceeds the FP16 score. Among the quantized methods, MXAttention achieves the highest Imaging and Aesthetic scores on both models. On Wan2.2, it also obtains the highest cosine similarity, SSIM, and PSNR, leading five of the six reported metrics. Its Imaging Quality is only 0.0031 below FP16, while its Aesthetic Quality exceeds the FP16 baseline. The HunyuanVideo comparison uses fully 4-bit attention across all blocks and denoising steps. Relative to direct OCP MXFP4, MXAttention improves Imaging Quality by 0.1921 and Aesthetic Quality by 0.0416, while increasing SSIM from 0.5954 to 0.7061 and PSNR from 16.25 to 19.23 dB. It also exceeds FP16 in both Imaging and Aesthetic Quality. NVFP4 + SageAttention obtains the highest paired frame-level similarity on this model, while MXAttention leads Imaging and Aesthetic Quality. Overall, MXAttention recovers the generation-quality loss of direct MXFP4 and substantially improves similarity to FP16. Figure 5 provides qualitative comparisons under identical prompts and random seeds. Direct OCP MXFP4 produces noticeable changes in object identity and appearance compared with FP16. NVFP4-based methods reduce these artifacts but still exhibit minor appearance variations in some cases. MXAttention better preserves the semantic content and visual appearance of the FP16 generation, consistent with the improvements in Imaging Quality and frame-level similarity. These results demonstrate that MXFP4 attention with proper quantization techniques can retain FP16-level generation quality and remain competitive with strong NVFP4-based pipelines. Figure 5: Qualitative comparison of 4-bit attention methods. Each row shows frames generated with the same prompt and random seed. The left three columns correspond to a Wan2.2 video and the right three columns correspond to a HunyuanVideo video, with multiple frames sampled from the same generated video. Direct OCP MXFP4 produces noticeable changes in object identity and appearance, while MXAttention better preserves the FP16 generation. 5.4 Ablation Studies We evaluate the contribution of the fixed Hadamard rotation, PNQ, and UOS under fully 4-bit attention. Unlike the main Wan2.2 comparison, no high-precision block or denoising-step fallback is used in this study. The rows form a sequential ablation: UOS is first removed from the full method, followed by PNQ and then the fixed Hadamard rotation. Table 3: Sequential ablation under fully 4-bit attention. All attention blocks and denoising steps are quantized without high-precision fallback. Bold values denote the best result within each model. Model Configuration Hadamard PNQ UOS VBench Metrics (↑ ) Similarity to FP16 (↑ ) Subject Imaging Aesthetic Cosine SSIM PSNR Wan2.2 Full ✓ ✓ ✓ 0.9424 0.6842 0.5994 0.9329 0.5519 15.96 w/o UOS ✓ ✓ – 0.9402 0.6822 0.6176 0.9143 0.5084 14.90 w/o PNQ and UOS ✓ – – 0.9442 0.6352 0.6081 0.9063 0.4727 14.77 OCP MXFP4 – – – 0.9461 0.5452 0.5998 0.9127 0.4636 14.78 HunyuanVideo Full ✓ ✓ ✓ 0.9689 0.6380 0.6344 0.9745 0.7061 19.23 w/o UOS ✓ ✓ – 0.9713 0.6423 0.6229 0.9676 0.6740 18.41 w/o PNQ and UOS ✓ – – 0.9759 0.5321 0.6046 0.9643 0.6520 17.87 OCP MXFP4 – – – 0.9781 0.4459 0.5928 0.9489 0.5954 16.25 Table 3 shows that UOS, PNQ, and the fixed Hadamard rotation provide complementary gains under fully 4-bit attention. UOS consistently improves cosine similarity, SSIM, and PSNR on both models. Compared with the configuration without UOS, the full method improves these metrics from 0.9143/0.5084/14.90 to 0.9329/0.5519/15.96 on Wan2.2 and from 0.9676/0.6740/18.41 to 0.9745/0.7061/19.23 on HunyuanVideo. The consistent gains in paired similarity support the analytical boundary Qmax=7.25Q_ =7.25 as a more accurate MXFP4 scale choice. PNQ produces the largest gain in Imaging Quality. Adding PNQ increases the score from 0.6352 to 0.6822 on Wan2.2 and from 0.5321 to 0.6423 on HunyuanVideo, while also improving all three paired similarity metrics. These gains agree with the analysis in Section 4.2: using the same quantized exponential tile for the row-wise sum and output-accumulator updates avoids the additional row-dependent scale error of direct softmax-path quantization. The fixed Hadamard rotation further reduces the effect of Q/KQ/K outliers. Adding it to direct OCP MXFP4 raises Imaging Quality from 0.5452 to 0.6352 on Wan2.2 and from 0.4459 to 0.5321 on HunyuanVideo. On HunyuanVideo, it also improves cosine similarity, SSIM, and PSNR. The full configuration combines the highest cosine similarity, SSIM, and PSNR on both models with the highest Imaging Quality on Wan2.2 and the highest Aesthetic Quality on HunyuanVideo. These results show that UOS, PNQ, and the fixed Hadamard rotation address complementary sources of MXFP4 error and together provide the strongest overall balance. Figure 6 provides qualitative ablation results under fully 4-bit attention. Removing UOS gradually introduces more visual artifacts, while further removing PNQ leads to additional degradation in generated details. Direct OCP MXFP4 exhibits the most severe quality degradation, with noticeable blurring and artifact patterns. The qualitative results are consistent with the quantitative ablation study, showing that UOS and PNQ provide complementary improvements. Figure 6: Qualitative ablation under fully 4-bit attention. From top to bottom: full MXAttention, without UOS, without UOS and PNQ, and direct OCP MXFP4. Removing UOS and PNQ progressively increases visible artifacts and reduces generation quality. 5.5 Mechanism Validation 5.5.1 Validation of Universal Optimal Scaling We compare UOS with the OCP boundary Qmax=8Q_ =8, TFS with Qmax=6Q_ =6, and a per-block empirical oracle obtained by sweeping QmaxQ_ from 6.00 to 8.00 in increments of 0.05. At the initial denoising step, the oracle independently minimizes MSE for Q, K, and V in each of the 40 evaluated attention blocks. Table 4: Per-block empirical QmaxQ_ optima. Values are percentages over 40 attention blocks. Model Tensor ≤7.15 ≤ 7.15 7.20 7.20 7.25 7.25 ≥7.30 ≥ 7.30 Wan2.2 Q 0.0 5.0 95.0 0.0 K 0.0 10.0 90.0 0.0 V 0.0 10.0 90.0 0.0 HunyuanVideo Q 45.0 0.0 52.5 2.5 K 0.0 62.5 32.5 5.0 V 0.0 50.0 50.0 0.0 For Wan2.2, Qmax=7.25Q_ =7.25 is the empirical optimum for 95.0%, 90.0%, and 90.0% of the evaluated Q, K, and V blocks, respectively. On HunyuanVideo, 7.257.25 is the most frequent optimum for Q and is tied with 7.207.20 for V, while the K optima concentrate at 7.207.20 (62.5%) and 7.257.25 (32.5%). Overall, 7.257.25 is the mode or co-mode in five of the six tensor–model combinations and the second-most frequent choice in the remaining case. The sweep minimizes finite-sample tensor MSE, whereas UOS minimizes the data-free analytical objective in Eq. (23). Their agreement supports using Qmax=7.25Q_ =7.25 without layer-wise calibration. The softmax path is validated separately below. 5.5.2 Validation of Pre-Normalization Quantization We use materialized attention probabilities as a diagnostic for the row-wise mass error introduced by block-wise MXFP4 quantization. FlashAttention does not materialize P during inference, and PNQ quantizes the unnormalized exponential tile P~ P. This diagnostic isolates how local MXFP4 rounding changes row-wise probability mass and complements the online-softmax analysis in Section 4.2. Figure 7: Row-sum diagnostics for MXFP4 probability quantization. (A) Across 80 Wan2.2 attention tensors at denoising step 9, covering two prompts and 40 layers, the mean row sum of each quantized tensor is below one. (B) The row-sum error is decomposed into zeroing, saturation, downward rounding, and upward rounding. (C) Pre-quant global scaling can restore the average row sum of a representative tensor but does not enforce normalization for individual rows. (D) High-entropy rows are predominantly under-normalized, whereas low-entropy rows may be under- or over-normalized. Across the 80 tensors in Figure 7(A), the average of the tensor-level mean row sums is 0.9266. This does not imply that every row has a sum below one; individual rows can be under- or over-normalized. It instead shows a consistent negative bias after aggregation across layers and prompts. Figure 7(B) separates the error by quantization outcome. Averaged across the tensors, zeroing contributes −0.0541-0.0541, saturation contributes −0.0190-0.0190, downward rounding contributes −0.0486-0.0486, and upward rounding contributes +0.0483+0.0483. The row-sum error is therefore not equivalent to a single global scaling factor. Figure 7(C) evaluates a pre-quant global scaling baseline, which applies a single scaling factor to the entire softmax tile before MXFP4 quantization. Although global scaling can restore the average row sum of a representative tensor, it substantially increases the number of rows with over-normalized attention weights. Correcting the mean therefore does not restore row-wise normalization. Finally, Figure 7(D) shows that row-sum behavior depends on attention entropy. Rows with entropy above 5 account for approximately 75% of the representative tensor and are almost always under-normalized, while lower-entropy rows exhibit errors in both directions. PNQ avoids this row-dependent normalization error by using the same quantized exponential tile in the row-wise sum and output-accumulator updates. 5.6 Algorithmic Overhead and Kernel Integration MXAttention preserves the tiled dataflow of low-precision attention. UOS changes only the boundary used in shared-scale selection; it retains the same block-maximum reduction, quantization granularity, and tensor traversal as direct MXFP4 quantization. PNQ reuses the quantized exponential tile P P, already required by the low-precision output-accumulator update, to update the row-wise sum ℓ . It therefore requires neither an additional quantization operation nor another traversal of the attention tiles. Moreover, rotation and quantization can be executed in parallel by leveraging orthogonal compute units (GEMM vs. non-GEMM units), eliminating resource contention. The fixed Hadamard transform is applied online only to Q and K; the corresponding transformation outside this path is folded into the model weights. Because the rotation and block quantization operate locally on Q/KQ/K tiles, they can be fused into the Q/KQ/K preprocessing stage. These modifications add no attention-matrix materialization or extra pass over the key/value sequence. They therefore introduce negligible algorithmic overhead and are compatible with a fused low-precision attention implementation. 6 Conclusion This paper presents MXAttention, a data-free post-training quantization framework for MXFP4 attention in video diffusion models. We identify two numerical failure modes in direct MXFP4 attention: the clipping–underflow trade-off introduced by power-of-two block scaling and the mismatch between the row-wise sum and output-accumulator update caused by inconsistent quantization in the online-softmax loop. To address them, Universal Optimal Scaling (UOS) derives the closed-form boundary Qmax=7.25Q_ =7.25 as the distribution-independent minimizer of a global MXFP4 quantization-error objective, while Pre-Normalization Quantization (PNQ) uses the same quantized softmax exponential tiles for both the row-wise sum and output-accumulator updates, preserving row-wise normalization by construction. Experiments on Wan2.2 and HunyuanVideo show that MXAttention recovers at least 95% of the VBench Imaging Quality gap between direct OCP MXFP4 and FP16, substantially improves frame-level similarity, and maintains FP16-level generation quality with less than 0.01 absolute degradation on all reported VBench metrics. MXAttention also achieves results competitive with strong NVFP4-based baselines. UOS changes only the shared-scale selection, and PNQ reuses the quantized exponential tiles already required by the low-precision output path; neither requires an additional pass over the attention matrix. These results demonstrate that accurate MXFP4 attention can be achieved without calibration, quantization-aware training, or layer-wise parameter search. References [1] Advanced Micro Devices, Inc. (2025) AMD Instinct MI350 Series GPUs. Note: https://w.amd.com/en/products/accelerators/instinct/mi350.htmlNative MXFP4 and MXFP6 datatype support Cited by: §1. [2] S. Ashkboos, A. L. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman (2024) QuaRot: outlier-free 4-bit inference in rotated LLMs. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §1, 5th item. [3] W. Bell and S. Nadarajah (2024) A review of wrapped distributions for circular data. Mathematics 12 (16), p. 2440. External Links: Document Cited by: Appendix D, §4.1, §4.1. [4] Y. Chen, H. Xi, J. Zhu, and J. Chen (2025) Oscillation-reduced MXFP4 training for vision transformers. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267. Cited by: §2, §3, §4.1. [5] J. Chhugani, G. Jeong, B. Su, Y. Pan, H. Yang, A. Ankit, J. Yu, S. Deng, Y. Chen, N. Satish, and C. Kim (2026) Unveiling the potential of quantization with MXFP4: strategies for quantization error reduction. In International Conference on Machine Learning, Cited by: Appendix G, §1. [6] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré (2022) FlashAttention: fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, Vol. 35, p. 10448–10459. Cited by: §1, §2, §3. [7] T. Dao (2024) FlashAttention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, Cited by: §1, §2, §3. [8] T. Gupta, H. Prairie, X. Wu, R. Abhyankar, Q. Wu, A. Silveria, P. Ponnusamy, J. Wang, B. Athiwaratkun, L. Song, T. Dao, D. Y. Fu, and C. De Sa (2026) Search your block floating point scales!. arXiv preprint arXiv:2605.12464. Cited by: §1. [9] W. Hu, C. Zhang, Z. Zhang, C. Guo, et al. (2026) M2xfp: a metadata-augmented microscaling data format for efficient low-bit quantization. arXiv preprint arXiv:2601.19213. Cited by: §1. [10] Z. Huang, Y. He, J. Yu, F. Zhang, Y. Zhang, T. Li, W. Zheng, Y. Qiao, and Z. Liu (2024) VBench: comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §5.1. [11] Huawei (2025) Leading a new paradigm for ai infrastructure. Note: https://w.huawei.com/en/news/2025/9/hc-xu-keynote-speechAnnounced MXFP4 support for the Ascend 950 series Cited by: §1. [12] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) HunyuanVideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: §1. [13] L. Kuipers and H. Niederreiter (1974) Uniform distribution of sequences. John Wiley & Sons, New York. Cited by: Appendix E. [14] X. Li, S. Tesfai, Z. Zhang, H. Xi, S. Yang, L. Zhang, Y. Sun, K. Peng, M. Agrawala, I. Stoica, K. Keutzer, J. Zhu, S. Han, Y. Lin, and M. Li (2026) DeltaQuant: 4-bit video diffusion models with spatiotemporal delta smoothing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 43578–43588. Cited by: §1. [15] F. Liu, S. Zhang, X. Wang, Y. Wei, H. Qiu, Y. Zhao, Y. Zhang, Q. Ye, and F. Wan (2025) Timestep embedding tells: it’s time to cache for video diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7353–7363. Cited by: §1. [16] Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort (2025) SpinQuant: LLM quantization with learned rotations. In International Conference on Learning Representations, Cited by: §1, 5th item. [17] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao (2024) Latent consistency models: synthesizing high-resolution images with few-step inference. In International Conference on Learning Representations, Cited by: §1. [18] K. V. Mardia and P. E. Jupp (2000) Directional statistics. Wiley Series in Probability and Statistics, John Wiley & Sons, Chichester. Cited by: Appendix D, §4.1, §4.1. [19] NVIDIA (2025) Introducing NVFP4 for efficient and accurate low-precision inference. Note: https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ Cited by: §1, 3rd item. [20] NVIDIA (2025) OpenAI triton on nvidia blackwell boosts ai performance and programmability. Note: https://developer.nvidia.com/blog/openai-triton-on-nvidia-blackwell-boosts-ai-performance-and-programmability/Demonstrates hardware-accelerated MXFP4 GEMM support on NVIDIA Blackwell Cited by: §1. [21] NVIDIA (2025) Using FP8 and FP4 with transformer engine. Note: https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html Cited by: §1, 3rd item. [22] Open Compute Project (2023) OCP Microscaling Formats (MX) Specification Version 1.0. Note: https://w.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf Cited by: §1, §2, §3, §4.1, 2nd item. [23] B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, et al. (2023) Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537. Cited by: §2, §2, §3, §4.1, 2nd item. [24] J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao (2024) FlashAttention-3: fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08608. Cited by: §1, §2, §3. [25] E. M. Stein and R. Shakarchi (2003) Fourier analysis: an introduction. Princeton Lectures in Analysis, Vol. 1, Princeton University Press, Princeton, NJ. Cited by: §D.1, Appendix E, §4.1. [26] Wan Team, A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314. Cited by: §1. [27] F. Wang, Z. Huang, A. W. Bergman, D. Shen, P. Gao, M. Lingelbach, K. Sun, W. Bian, G. Song, Y. Liu, X. Wang, and H. Li (2024) Phased consistency models. In Advances in Neural Information Processing Systems, Cited by: §1. [28] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2024) CogVideoX: text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072. Cited by: §1. [29] J. Zhang, H. Huang, P. Zhang, J. Wei, J. Zhu, and J. Chen (2025) SageAttention2: efficient attention with thorough outlier smoothing and per-thread INT4 quantization. In International Conference on Machine Learning, Cited by: §1. [30] J. Zhang, J. Wei, P. Zhang, X. Xu, H. Huang, H. Wang, K. Jiang, J. Zhu, and J. Chen (2025) SageAttention3: microscaling FP4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505.11594. Cited by: §1, §1, 4th item. [31] J. Zhang, J. Wei, P. Zhang, J. Zhu, and J. Chen (2025) SageAttention: accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations, Cited by: §1. [32] J. Zhang, X. Xu, J. Wei, H. Huang, P. Zhang, C. Xiang, J. Zhu, and J. Chen (2025) SageAttention2++: a more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21136. Cited by: §1. [33] P. Zhang, M. Noto, W. Tan, C. Jiang, W. Lin, W. Zhou, and H. Zhang (2026) Attn-qat: 4-bit attention with quantization-aware training. arXiv preprint arXiv:2603.00040. Cited by: §1, §1. [34] X. Zhao, X. Jin, K. Wang, and Y. You (2024) Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408.12588. Cited by: §1. [35] Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You (2024) Open-sora: democratizing efficient video production for all. arXiv preprint arXiv:2412.20404. External Links: 2412.20404 Cited by: §5.1. Appendix A Exact Density under Power-of-Two Scaling We analyze nonzero blocks for which the required shared-scale exponent lies within the E8M0 exponent range. The cases M=0M=0 and exponents outside this range are handled by the MXFP4 conversion routine and are omitted because they do not affect the derivation of q. Let M>0M>0 be the unscaled block maximum, let U=log2MU= _2M have density fUf_U, and define eq(M)=⌈log2(Mq)⌉,Xq=M2eq(M).e_q(M)= _2 ( Mq ) , X_q= M2^e_q(M). (53) For x∈(q/2,q)x∈(q/2,q), every preimage of x has the form M=2kx,U=log2x+k,k∈ℤ.M=2^kx, U= _2x+k, k . (54) Indeed, eq(2kx)=⌈k+log2(xq)⌉=k,e_q(2^kx)= k+ _2 ( xq ) =k, (55) because log2(x/q)∈(−1,0) _2(x/q)∈(-1,0). The Jacobian of U=log2x+kU= _2x+k is |dUdx|=1xln2. | dUdx |= 1x 2. (56) Summing over all preimages gives gq(x)=1xln2∑k∈ℤfU(log2x+k),x∈(q/2,q).g_q(x)= 1x 2 _k f_U( _2x+k), x∈(q/2,q). (57) Define the periodized log-domain density H(t)=∑k∈ℤfU(t+k).H(t)= _k f_U(t+k). (58) The function H has period one: H(t+1)=H(t)H(t+1)=H(t). Equation (57) can therefore be written as gq(x)=g(x)=H(log2x)xln2.g_q(x)=g(x)= H( _2x)x 2. (59) Thus, q selects a factor-of-two interval of a fixed periodized density rather than changing the density inside that interval. The density integrates to one over every such interval. Let a=log2qa= _2q. Then ∫q/2qg(x)x _q/2^qg(x)\,dx =∫a−1aH(t)t = _a-1^aH(t)\,dt (60) =∫01H(t)t = _0^1H(t)\,dt (61) =∑k∈ℤ∫k+1fU(u)u = _k _k^k+1f_U(u)\,du (62) =1. =1. (63) For almost every q>0q>0, periodicity gives g(q2) g ( q2 ) =H(log2q−1)(q/2)ln2 = H( _2q-1)(q/2) 2 (64) =2H(log2q)qln2 =2 H( _2q)q 2 (65) =2g(q). =2g(q). (66) This proves Lemma 1. The result requires only the exact periodicity of H, not a log-uniform, Gaussian, or small-perturbation assumption. Appendix B Integrated Quantization Error of the E2M1 Grid For nonnegative magnitudes, the finite E2M1 grid and its round-to-nearest decision boundaries are E2M1 _E2M1 =0,0.5,1,1.5,2,3,4,6, =\0,0.5,1,1.5,2,3,4,6\, (67) E2M1 _E2M1 =0.25,0.75,1.25,1.75,2.5,3.5,5.0. =\0.25,0.75,1.25,1.75,2.5,3.5,5.0\. (68) The value 66 is the largest finite E2M1 value, and 55 is the decision boundary between 44 and 66. Hence, every normalized value above 55 projects to 66. The threshold 77 has a different role: it is the midpoint between 66 and the next value 88 on the unbounded E2M1 ladder. Values above 77 would round upward to 88 before finite-range saturation forces them to 66. We refer to this subset as the overflow-rounding region. For an interval [a,b][a,b] mapped to a quantization value r, ∫ab(v−r)2v=(b−r)3−(a−r)33. _a^b(v-r)^2\,dv= (b-r)^3-(a-r)^33. (69) Applying Eq. (69) successively gives the cumulative errors in Table 5. Table 5: Exact cumulative quantization error E(x)E(x) at E2M1 decision boundaries and selected grid points. E(0)E(0) E(0.25)E(0.25) E(0.75)E(0.75) E(1.25)E(1.25) E(1.75)E(1.75) E(2.5)E(2.5) E(3.5)E(3.5) E(5)E(5) E(6)E(6) 0 1192 1192 164 164 5192 5192 7192 7192 112 112 16 16 1324 1324 78 78 For q≥6q≥ 6, every value above 55 maps to 66, giving E(q)=E(6)+∫6q(v−6)2v=78+13(q−6)3.E(q)=E(6)+ _6^q(v-6)^2\,dv= 78+ 13(q-6)^3. (70) For 6≤q≤76≤ q≤ 7, q/2∈[3,3.5]q/2∈[3,3.5], where values map to 33. Hence, E(q2)=18+13(q2−3)3.E ( q2 )= 18+ 13 ( q2-3 )^3. (71) For 7≤q≤87≤ q≤ 8, q/2∈[3.5,4]q/2∈[3.5,4], where values map to 44. Hence, E(q2)=524+13(q2−4)3.E ( q2 )= 524+ 13 ( q2-4 )^3. (72) Substitution into Δ(q)=E(q)−8E(q/2) (q)=E(q)-8E(q/2) gives Δ(q)=−18,6≤q≤7,(4q−29)(4q−27)8,7≤q≤8. (q)= cases- 18,&6≤ q≤ 7,\\[6.0pt] (4q-29)(4q-27)8,&7≤ q≤ 8. cases (73) Appendix C Global Optimality of the UOS Boundary Since D(x)=E(x)/x3D(x)=E(x)/x^3, D(q)−D(q2)=E(q)−8E(q/2)q3=Δ(q)q3.D(q)-D ( q2 )= E(q)-8E(q/2)q^3= (q)q^3. (74) C.1 Justification of Differentiation Let F(x)=D(x)g(x)F(x)=D(x)g(x). Since D is continuous on (0,∞)(0,∞) and g is locally integrable, F∈Lloc1(0,∞)F∈ L^1_loc(0,∞). Define G(t)=∫1tF(x)x.G(t)= _1^tF(x)\,dx. (75) Then G is absolutely continuous on every compact subinterval of (0,∞)(0,∞) and G′(t)=F(t)G (t)=F(t) for almost every t. Since (q)=G(q)−G(q/2),J(q)=G(q)-G(q/2), (76) J is absolutely continuous and, for almost every q, ′(q)=F(q)−12F(q/2).J (q)=F(q)- 12F(q/2). (77) Using Eq. (66), ′(q) (q) =D(q)g(q)−12D(q2)g(q2) =D(q)g(q)- 12D ( q2 )g ( q2 ) (78) =g(q)[D(q)−D(q2)] =g(q) [D(q)-D ( q2 ) ] (79) =g(q)q3Δ(q). = g(q)q^3 (q). (80) Because g(q)≥0g(q)≥ 0, the derivative sign is determined by Δ(q) (q). Table 6 gives the exact piecewise form of Δ(q) (q) over q>0q>0. Table 6: Exact piecewise form and sign of Δ(q)=E(q)−8E(q/2) (q)=E(q)-8E(q/2). Interval () (q) Sign 0<q≤140<q≤ 14 0 0 14<q≤12 14<q≤ 12 −(4q−1)232- (4q-1)^232 <0<0 12<q≤34 12<q≤ 34 16q2−24q+732 16q^2-24q+732 <0<0 34<q≤54 34<q≤ 54 −116- 116 <0<0 54<q≤32 54<q≤ 32 −16q2−40q+2732- 16q^2-40q+2732 <0<0 32<q≤74 32<q≤ 74 (4q−9)(4q−5)32 (4q-9)(4q-5)32 <0<0 74<q≤7 74<q≤ 7 −18- 18 <0<0 7<q≤107<q≤ 10 (4q−29)(4q−27)8 (4q-29)(4q-27)8 <0<0 before 29/429/4, =0=0 at 29/429/4, >0>0 afterward q>10q>10 48q2−864q+39838 48q^2-864q+39838 >0>0 The quadratic in the final row has discriminant −18240<0-18240<0 and a positive leading coefficient, so it is strictly positive. Therefore, Δ(q)=0,0<q≤14,<0,14<q<294,=0,q=294,>0,q>294. (q) cases=0,&0<q≤ 14,\\ <0,& 14<q< 294,\\ =0,&q= 294,\\ >0,&q> 294. cases (81) Combining Eqs. (80) and (81), (q)J(q) is nonincreasing before q=29/4q=29/4 and nondecreasing afterward. Hence q⋆=29/4q =29/4, equivalently Qmax⋆=7.25Q_ =7.25, is a global minimizer for every absolutely continuous law of log2M _2M within the stated scale family. If H is positive almost everywhere over one period, (q)J(q) is constant on 0<q≤1/40<q≤ 1/4, strictly decreasing on (1/4,29/4)(1/4,29/4), and strictly increasing on (29/4,∞)(29/4,∞). Therefore, Qmax⋆=29/4Q_ =29/4 is the unique global minimizer. Appendix D Wrapped-Distribution and Harmonic Interpretation Reducing a real-valued random variable modulo one yields a periodic density, commonly called a wrapped distribution [18, 3]. It can be represented either as a sum of translated densities or through Fourier coefficients. These classical results provide the mathematical tools used below; the contribution of UOS is the boundary-density consequence for power-of-two microscaling. Let Yq=(U−log2q)mod1∈[0,1).Y_q=(U- _2q) 1∈[0,1). (82) For noninteger U−log2qU- _2q, which occurs almost surely under an absolutely continuous law, Xq=q 2Yq−1.X_q=q\,2^Y_q-1. (83) At integer endpoints, the ceiling convention maps the value to q; this measure-zero distinction does not affect the density or the quantization-error integrals. If YqY_q is uniformly distributed, the Jacobian gives fXq(x)=1xln2,x∈(q/2,q].f_X_q(x)= 1x 2, x∈(q/2,q]. (84) For the ceiling-based q=8q=8 interval (4,8](4,8], which differs from the exact OCP interval [4,8)[4,8) only at measure-zero endpoints, this approximation gives Pr(X8>7)=∫781xln2x=log287≈19.27%. (X_8>7)= _7^8 1x 2\,dx= _2 87≈ 19.27\%. (85) This is a diagnostic estimate under the uniform wrapped-phase approximation, not an assumption required by Theorem 1. D.1 Fourier Representation The Fourier coefficients of the periodized density H are cn c_n =∫01H(t)e−i2πntt = _0^1H(t)e^-i2π nt\,dt (86) =∫−∞fU(u)e−i2πnuu = _-∞^∞f_U(u)e^-i2π nu\,du (87) =ϕU(−2πn), = _U(-2π n), (88) where ϕU(ω)=[eiωU] _U(ω)=E[e^iω U]. The zeroth coefficient is c0=1c_0=1. Under standard Fourier-convergence conditions [25], H(t)=1+∑n≠0ϕU(−2πn)ei2πnt.H(t)=1+ _n≠ 0 _U(-2π n)e^i2π nt. (89) For a general integrable density, the coefficient identity remains valid even when pointwise convergence of the Fourier series is not assumed. Equation (89) gives the harmonic interpretation of Lemma 1. For every integer n, ei2πn(log2q−1)=ei2πnlog2q.e^i2π n( _2q-1)=e^i2π n _2q. (90) Thus, the uniform term and every nonzero Fourier mode take the same value at the two scale boundaries. The harmonic components can change the objective value and the magnitude of its derivative, but they cannot reverse the derivative sign or shift the UOS minimizer. If U∼(μ,σ2)U (μ,σ^2), the Fourier series becomes H(t)=1+2∑n=1∞e−2π2n2σ2cos(2πn(t−μ)).H(t)=1+2 _n=1^∞e^-2π^2n^2σ^2 (2π n(t-μ) ). (91) The first-harmonic amplitude is a1(σ)=2e−2π2σ2.a_1(σ)=2e^-2π^2σ^2. (92) Table 7: First-harmonic amplitude under a Gaussian log-domain prior. The values describe the leading Fourier component, not the complete pointwise deviation of the wrapped density. σ σ^2 () a_1(σ) 0.10 0.01 1.6417 0.20 0.04 0.9081 0.30 0.09 0.3384 0.40 0.16 0.0850 0.50 0.25 0.0144 0.60 0.36 0.0016 1.00 1.00 5.35×10−95.35× 10^-9 Higher harmonics may remain non-negligible when σ2σ^2 is small, so a1(σ)a_1(σ) should not be interpreted as a bound on the total deviation from uniformity. Appendix E Wrapped-Phase Convergence for Location–Scale Families Let Uσ=μ+σZU_σ=μ+σ Z, where Z has an absolutely continuous density. For every nonzero integer n, |ϕUσ(−2πn)|=|ϕZ(−2πnσ)|. | _U_σ(-2π n) |= | _Z(-2π nσ) |. (93) Since Z has an integrable density, the Riemann–Lebesgue lemma gives lim|t|→∞ϕZ(t)=0 _|t|→∞ _Z(t)=0 [25]. Therefore, ϕUσ(−2πn)→0 _U_σ(-2π n)→ 0 for every n≠0n≠ 0. By the Fourier characterization of the uniform distribution modulo one [13], Uσmod1→(0,1).U_σ 1 dU(0,1). (94) Thus, broad absolutely continuous location–scale families approach a uniform wrapped phase. Without additional regularity assumptions, this weak convergence does not imply uniform convergence of the corresponding densities. Appendix F Empirical Wrapped-Phase Diagnostics The following figures visualize block-maximum distributions before and after logarithmic wrapping. Figure 8 shows empirical distributions from Wan2.2 attention tensors, including the wrapped log-domain phases used in our analysis. Real attention tensors can exhibit clear nonuniform harmonic structure; UOS does not require these harmonics to be small. Figure 8: Empirical block-maximum distributions in Wan2.2. The rows show the original block maxima, their logarithms, the wrapped log-domain phases, and the corresponding normalized linear-domain values for Q, K, V, and the normalized attention probability matrix P. The visible nonuniformity corresponds to distribution-specific harmonic components but does not change the UOS boundary condition. The P column is included only as a distributional diagnostic; PNQ quantizes the unnormalized softmax exponential tile. Figure 9 further examines how Hadamard rotation changes the empirical block-maximum distributions. Although the rotation modifies the log-domain harmonic structure, the periodic boundary relation used by UOS remains unchanged. Figure 9: Effect of Hadamard rotation on block-maximum distributions. Rotation changes the log-domain distribution and its harmonic content; the boundary density ratio remains unchanged. Appendix G Relation to OAS and the OCP Endpoint Convention At the common 32-element block granularity, the OAS scale-selection rule is equivalent to the ceiling-based boundary q=7q=7. Let e6=⌈log2(M6)⌉,x6=M2e6∈(3,6].e_6= _2 ( M6 ) , x_6= M2^e_6∈(3,6]. (95) Then e7=e6+⌈log2(x67)⌉=e6−1,3<x6≤3.5,e6,3.5<x6≤6.e_7=e_6+ _2 ( x_67 ) = casese_6-1,&3<x_6≤ 3.5,\\ e_6,&3.5<x_6≤ 6. cases (96) Thus, q=7q=7 halves the TFS scale when x6∈(3,3.5]x_6∈(3,3.5], mapping this interval to (6,7](6,7], and otherwise leaves the scale unchanged. This is the OAS scale-selection rule at the same block granularity [5]. The comparison does not include the smaller block size used in the full OAS configuration. The experimental OCP baseline uses the exact floor-based conversion rule eOCP=⌊log2M⌋−2=⌊log2(M8)⌋+1.e_OCP= _2M -2= _2 ( M8 ) +1. (97) The ceiling-based q=8q=8 rule used in the continuous boundary analysis is e8(M)=⌈log2(M8)⌉.e_8(M)= _2 ( M8 ) . (98) The two expressions agree whenever log2(M/8)∉ℤ _2(M/8) , since ⌈z⌉=⌊z⌋+1 z = z +1 for noninteger z. They differ when M/8M/8 is an exact power of two. Under an absolutely continuous law, these endpoints have measure zero and do not affect the quantization-error integrals or Theorem 1. All reported OCP experiments use the exact floor-based rule. Appendix H Full Online-Softmax Form of PNQ Let TcT_c be the number of key/value tiles processed for query block i. For each tile j, define the row-wise rescaling factor from its current running maximum to the final running maximum as βi(j)=exp(mi(j)−mi(Tc)). _i^(j)= \! (m_i^(j)-m_i^(T_c) ). (99) Because αi(r)=exp(mi(r−1)−mi(r)), _i^(r)= \! (m_i^(r-1)-m_i^(r) ), (100) the element-wise product of all subsequent rescaling factors telescopes: ∏r=j+1Tcαi(r)=exp(mi(j)−mi(Tc))=βi(j), _r=j+1^T_c _i^(r)= \! (m_i^(j)-m_i^(T_c) )= _i^(j), (101) where the empty product for j=Tcj=T_c is the all-ones vector. Lemma 2 (Effective Full-Row Form). After all key/value tiles have been processed, PNQ satisfies ℓi(Tc) _i^(T_c) =∑j=1Tcβi(j)⊙(P^ijBc), = _j=1^T_c _i^(j) ( P_ij1_B_c ), (102) O~i(Tc) O_i^(T_c) =∑j=1TcDiag(βi(j))P^ijV^j. = _j=1^T_cDiag\! ( _i^(j) ) P_ij V_j. Proof. Unrolling Eq. (48), the contribution of tile j is multiplied by every subsequent rescaling factor αi(r) _i^(r), for r=j+1,…,Tcr=j+1,…,T_c. Equation (101) reduces their element-wise product to βi(j) _i^(j), yielding Eq. (102). ∎ Define the effective quantized tile P^ijeff=Diag(βi(j))P^ij, P_ij^eff=Diag\! ( _i^(j) ) P_ij, (103) and the conceptual full-row matrix P^ieff=[P^i1eff,…,P^iTceff]. P_i^eff= [ P_i1^eff,…, P_iT_c^eff ]. (104) Let V V denote the corresponding vertical concatenation of V^1,…,V^Tc V_1,…, V_T_c. Lemma 2 then gives ℓi(Tc)=P^ieff,O~i(Tc)=P^ieffV^. _i^(T_c)= P_i^eff1, O_i^(T_c)= P_i^eff V. (105) The matrix P^ieff P_i^eff is an analytical representation of the online recurrence and is never materialized by the kernel. Appendix I Full-Row Mismatch under Direct Placement For the direct placement, define P~ijeff=Diag(βi(j))P~ij,P^ijeff=Diag(βi(j))P^ij, P_ij^eff=Diag\! ( _i^(j) ) P_ij, P_ij^eff=Diag\! ( _i^(j) ) P_ij, (106) and concatenate the tiles across the key sequence: P~ieff=[P~i1eff,…,P~iTceff],P^ieff=[P^i1eff,…,P^iTceff]. P_i^eff= [ P_i1^eff,…, P_iT_c^eff ], P_i^eff= [ P_i1^eff,…, P_iT_c^eff ]. (107) The final direct-placement states are ℓidirect=P~ieff,O~idirect=P^ieffV^. _i^direct= P_i^eff1, O_i^direct= P_i^eff V. (108) Hence, the final weights induced by the direct placement are P^idirect=Diag(P~ieff)−1P^ieff, P_i^direct=Diag\! ( P_i^eff1 )^-1 P_i^eff, (109) with row sums P^idirect=(P^ieff)⊘(P~ieff). P_i^direct1= ( P_i^eff1 ) ( P_i^eff1 ). (110) Here, ⊘ denotes element-wise division between the two row-wise sum vectors. The running-max rescaling is the same in both states, but the exponential tiles are not; the final row sum therefore need not equal one. Appendix J Positive PNQ Normalization Factor Fix any row r of query block i. All attention rows considered in this work have a nonempty key set. Let tile j⋆j contain an entry c⋆c attaining the final maximum of row r: Sij⋆[r,c⋆]=mi(Tc)[r].S_ij [r,c ]=m_i^(T_c)[r]. (111) When tile j⋆j is processed, the running maximum for this row reaches its final value: mi(j⋆)[r]=mi(Tc)[r].m_i^(j )[r]=m_i^(T_c)[r]. (112) Therefore, P~ij⋆[r,c⋆]=exp(Sij⋆[r,c⋆]−mi(j⋆)[r])=1, P_ij [r,c ]= \! (S_ij [r,c ]-m_i^(j )[r] )=1, (113) and βi(j⋆)[r]=1. _i^(j )[r]=1. (114) Every entry of P~ij⋆ P_ij lies in (0,1](0,1], and the MXFP4 block containing P~ij⋆[r,c⋆] P_ij [r,c ] therefore has maximum magnitude one. Under the UOS boundary q⋆=7.25q =7.25, eq⋆(1)=⌈log2(17.25)⌉=−2.e_q (1)= _2 ( 17.25 ) =-2. (115) The shared scale is 2−2=1/42^-2=1/4, so the normalized value is 12−2=4, 12^-2=4, (116) which is exactly representable in E2M1. Hence, ⋆(1)=1.Q_ (1)=1. (117) Since this entry also has βi(j⋆)[r]=1 _i^(j )[r]=1, it contributes exactly one to the final row-wise sum. Therefore, ℓi(Tc)[r]≥1>0. _i^(T_c)[r]≥ 1>0. (118) The normalization guarantee concerns the weights induced by the PNQ recurrence in exact arithmetic. It does not assert that the quantized weights equal the full-precision softmax weights. In an implementation, finite-precision accumulation and the final division may introduce ordinary floating-point rounding. PNQ removes the additional structural mismatch caused by updating ℓi _i with P~ij P_ij while updating O~i O_i with P^ij P_ij.