Paper deep dive
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Ziyang Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 10:36:33 AM
Summary
The paper investigates W4A4 (4-bit weight/activation) quantization errors in SwiGLU-based language models. It introduces a 'Reader/Generator' decomposition, identifying that while 'Readers' (qkv, w1, w3) are bounded by the residual stream, 'Generators' (o_proj, w2) create heavy-tailed distributions due to bilinear or complex interactions. The authors propose 'DR+sink' (Depth Registers with a register-magnitude hinge loss) as a training-time intervention that significantly reduces validation perplexity (from 1727 to 119) by controlling the residual axis. The study also evaluates post-hoc methods like SmoothQuant and QuaRot, finding that orthogonal rotations (Full QuaRot) fail to close the gap caused by the bilinear SwiGLU tail (w2).
Entities (10)
Relation Signals (7)
w1 → isa → Reader
confidence 100% · residual-axis readers (qkv, w1, w3)
o_proj → isa → Generator
confidence 100% · block-internal generators (o_proj, w2)
w2 → isa → Generator
confidence 100% · block-internal generators (o_proj, w2)
w3 → isa → Reader
confidence 100% · residual-axis readers (qkv, w1, w3)
qkv → isa → Reader
confidence 100% · residual-axis readers (qkv, w1, w3)
w2 → dominateserrorin → W4A4
confidence 90% · Delta-remove(w2) dominating
DR+sink → reducesperplexityof → W4A4
confidence 90% · DR+sink ... reduces this to 119 (about 14x)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 to 1727. A simple residual-axis training-time intervention -- Depth Registers with a register-magnitude hinge loss (DR+sink) -- reduces this to 119 (about 14x) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 PPL. The residual ~2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers (qkv, w1, w3) and block-internal generators (o_proj, w2). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2's bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ~0.28 nats across three matched checkpoints with Delta-remove(w2) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot -- adding online per-head value Hadamard plus online w2-input rotation -- does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge.
Tags
Links
- Source: https://arxiv.org/abs/2604.18128v1
- Canonical: https://arxiv.org/abs/2604.18128v1
Trouble viewing inline? Open PDF directly →
Full Text
57,789 characters extracted from source content.
Expand or collapse full text
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition Ziyang Liu Independent Researcher ziyang.liu.r@outlook.com Abstract We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.623.6 to 17271727. A simple residual-axis training-time intervention — Depth Registers with a register-magnitude hinge loss (DR+sink) — reduces this to 119119 (∼14× 14×) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.939.9 PPL. The residual ≈2≈ 2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers (qkv, w1, w3) and block-internal generators (o_proj, w2). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2’s bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ≈0.28≈ 0.28 nats across three matched checkpoints with Δremove(w2) ( w2) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot — adding online per-head value Hadamard plus online w2-input rotation — does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge. 1 Introduction W4A4 post-training quantization (4-bit weights, 4-bit activations) is the leading route to 2–3× inference throughput and memory savings on commodity GPUs. SwiGLU-based language models — LLaMA (Touvron et al., 2023), Mistral (Jiang et al., 2023), Qwen (Bai et al., 2023) — are reported to lose one to three orders of magnitude in validation perplexity under naive round-to-nearest W4A4; we reproduce this on a 300M-parameter SwiGLU decoder-only model trained from scratch on 5B FineWeb-Edu tokens (PPL 23.6→172723.6→ 1727). A family of calibration-based remedies — SmoothQuant (Xiao et al., 2023), QuaRot (Ashkboos et al., 2024), AWQ (Lin et al., 2024), FlatQuant (Sun et al., 2025) — recovers most of this gap by rescaling, rotating, or protecting specific linear layers. Each choice encodes an implicit hypothesis about which linears drive the damage and why. Prior work catalogs the phenomenology but does not separate the damage by input-activation site. Observational studies document a few hidden-state channels carrying very large magnitudes (Dettmers et al., 2022; Sun et al., 2024), an “attention sink” token emitting disproportionate values (Xiao et al., 2024), and GLU-family FFNs whose elementwise product amplifies channel-wise spikes (Yang et al., 2025). These say where outliers appear; they do not say which linears’ input activations are load-bearing for W4A4 damage, nor whether those inputs are reachable from the residual axis. We offer a clean operator-level account. In a SwiGLU block, five input-activation sites can be grouped by the source of their input: • Readers receive the residual stream after RMSNorm: qkv, w1, w3. Their input magnitude is directly bounded by any intervention that shapes the residual. • Generators receive a function of the block’s internal computation: o_proj’s input is the attention-weighted mixture ∑jαjV(x~j) _j _jV( x_j) — linear in the value projections conditional on the softmax weights αj _j, which themselves depend on x~ x; w2’s input is the SwiGLU product silu(w1(x~))⊙w3(x~)silu( w1( x)) w3( x), which is bilinear in the residual. Figure 1: The five input-activation sites of a SwiGLU block. Readers (qkv, w1, w3; teal) ingest RMSNormRMSNorm of the residual stream; generators (o_proj, w2; amber) ingest block-internal quantities — the softmax⋅V· V aggregate and the SwiGLU bilinear product. The algebra (Section˜3) predicts that a norm-constrained residual-axis intervention bounds reader inputs tightly, bounds o_proj’s input only up to ‖WV‖op \|W_V \|_op and the softmax weights the attention assigns to outlier positions, and does not bound the SwiGLU product’s heavy tail beyond a trivial product of bounds. We propose a residual-axis training method that exploits this decomposition directly: Depth Registers with a register-magnitude hinge loss (DR+sink). DR+sink partitions the residual stream into register and non-register dimensions and applies a hinge penalty pulling the maximum register magnitude below a fixed target (Section˜3.3). On a 300M × 5B FineWeb-Edu checkpoint, DR+sink reduces naive-RTN W4A4 perplexity from 1727 to 119 (∼× 14×) at matched FP16 PPL (23.623.6) and matched zero-shot capacity (three 1k-example probes, ±1.5± 1.5 p), and composes with SmoothQuant to reach 39.939.9 PPL (from 57.657.6 without the probe). Operator-level evidence aligns with the decomposition: DR+sink drives reader kurtosis to near-Gaussian (qkv: 35.9→2.8535.9→ 2.85) and leaves generator kurtosis essentially unchanged (w2: 1921→18601921→ 1860) across three matched checkpoints. The same separation shows up directly in W4A4 damage. We report skip-ablation observables in excess NLL (conditional-on-rest-quantized, not additive): ΔNLL(naive) (naive), ΔNLL(skip-readers) (skip-readers) — the reader-rescued residue — and single-site skip-removals (Section˜4.2). The reader-rescued residue is flat across all three matched checkpoints at ≈0.28≈ 0.28 nats (≈31≈ 31 PPL). A post-hoc alternative, Per-Linear QuaRot (independent random Hadamard rotations per linear input), substitutes for DR+sink on the reader axis almost exactly (25.525.5/25.425.4/25.425.4 W4A4 PPL, spread ≤0.1≤ 0.1). We therefore do not claim DR+sink is the uniquely best practical recipe for W4A4. Our claim is narrower: DR+sink is a training-time residual-axis probe that makes the reader/generator split experimentally visible, composes with residual-axis post-hoc scaling (SmoothQuant: 57.6→39.957.6→ 39.9 PPL, a reduction Per-Linear QuaRot does not produce), and produces an interpretable named partition rather than a structureless rotation. Full QuaRot (adding online per-head Hadamard on attention values and online orthogonal rotation on w2’s input) lands at 26.026.0/26.126.1/26.026.0 PPL (ΔNLL≈0.09 ≈ 0.09 nats), marginally worse than per-Linear QuaRot despite the online FFN-internal w2 rotation — the direct post-hoc test of Observation˜3.4: in our setting, orthogonal rotation does not close the bilinear SwiGLU residue. Contributions. 1. An operator-level decomposition of the five input-activation sites in a SwiGLU block into residual-axis readers and block-internal generators, with elementary norm arguments bounding reader inputs by the residual, bounding o_proj’s input up to a softmax- and WVW_V-dependent constant, and leaving w2’s bilinear product bounded only by the trivial product of factor bounds (Section˜3). 2. A residual-axis training method DR+sink (Depth Registers + register-magnitude hinge) that reduces naive-RTN W4A4 PPL from 17271727 to 119119 (∼14× 14×) at matched FP16 PPL (23.623.6) and matched zero-shot capacity (three 1k-example probes, ±1.5± 1.5 p) on a 300M × 5B checkpoint, and composes with SmoothQuant (57.6→39.957.6→ 39.9 PPL). Operator-level: reader kurtosis collapses (qkv: 35.9→2.8535.9→ 2.85) while generator kurtosis is essentially unchanged (w2: 1921→18601921→ 1860) across three matched checkpoints (Section˜4.1). 3. A W4A4 skip-ablation budget in excess NLL (Δremove(S) (S) defined as ΔNLL(naive)−ΔNLL(skip-S) (naive)- (skip-S), conditional-on-rest-quantized, not additive). The reader-rescued residue is flat at ≈0.28≈ 0.28 nats across all three matched checkpoints; Δremove(w2)=0.88/0.69 ( w2)=0.88/0.69 nats dominates (dirty/clean residual), Δremove(o_proj)=0.06 ( o\_proj)=0.06 nats at k=0k=0 and numerically small once readers are clean (Section˜4.2). 4. A post-hoc interaction study. Per-Linear QuaRot matches DR+sink on readers (25.5/25.4/25.425.5/25.4/25.4 PPL, ≤0.1≤ 0.1 spread) and reaches part of o_proj; full QuaRot (Ashkboos et al., 2024) adds the online w2 Hadamard the decomposition flags as the natural next intervention but does not close the gap (ΔNLL≈0.09 ≈ 0.09 nats) — consistent with Observation˜3.4’s reachability argument that orthogonal rotation need not suppress the bilinear product’s tail (Section˜4.3). Headline. DR+sink: 1727→1191727→ 119 PPL (∼14× 14×) at matched FP16 PPL (23.623.6) and matched downstream accuracy; composes with SmoothQuant to 39.939.9 PPL. The residual ≈2≈ 2 PPL gap is w2-dominant and residual-axis-irreducible: all three checkpoints hit the same reader-rescued residue (ΔNLL≈0.28 ≈ 0.28 nats); Δremove(w2)=0.88/0.69 ( w2)=0.88/0.69 nats dominates, Δremove(o_proj) ( o\_proj) is within single-seed noise of zero once readers are clean. Figure˜2 shows the operator-level mechanism and the residue flatness; Table˜1 collects the headline numbers across all three checkpoints and all post-hoc methods evaluated. (a) Per-linear input kurtosis, three models. (b) W4A4 excess-NLL decomposition. Figure 2: Readers and generators respond differently to residual-axis fixes. (2(a)) Per-layer channel-max excess kurtosis, layer mean. DR+sink collapses reader kurtosis; generator kurtosis essentially unchanged. (2(b)) W4A4 excess-NLL decomposition. Reader-rescued residue flat at ≈0.28≈ 0.28 nats; naive ΔNLL shrinks ≈2.7×≈ 2.7×. Conditional-on-rest-quantized. Table 1: Headline W4A4 PPL, 300M×5B FineWeb-Edu, single seed. Three matched checkpoints differing only in the DR+sink setting. FP16 PPL matched within 0.10.1. DR+sink reduces naive RTN W4A4 by ∼14× 14×; it composes with SmoothQuant to 39.939.9; per-Linear QuaRot substitutes for DR+sink on readers. Full method-by-method breakdown in Table˜4; tail and ΔNLL metrics in Sections˜4.1 and 4.2. Validation PPL k=0k=0 k=64k=64 (DR) k=64k=64+sink FP16 reference 23.6 23.7 23.7 Naive RTN W4A4 1727 671 119 SmoothQuant W4A4 57.6 58.1 39.9 QuaRot (per-Linear) W4A4 25.5 25.4 25.4 QuaRot (full) W4A4 26.0 26.1 26.0 2 Related Work Prior work on quantization outliers in transformers spans three lines: observational studies of where outliers appear, post-training methods that exploit those observations to scale or rotate activations, and architectural or training-time interventions that try to prevent outliers at the source. We organize by where along the network the intervention operates and highlight the one site that remains largely unaddressed from the residual axis: the SwiGLU down-projection input. Observational studies. Dettmers et al. (2022) identified a small set of “emergent” hidden-state channels whose magnitudes grow with scale and dominate INT8 quantization error. Sun et al. (2024) documented massive activations as an architectural phenomenon in modern LLMs. Xiao et al. (2024) observed an attention sink: the first token absorbs a disproportionate fraction of attention mass and emits unusually large hidden-state values, stabilizing attention at the cost of outliers. Yang et al. (2025) specifically examined GLU-family feed-forward networks and noted that the elementwise product in SwiGLU amplifies channel-wise activation spikes at the down-projection input — the observation our mechanism argument builds on. These papers catalog the phenomenology but do not decompose W4A4 damage by input-activation site. Post-training quantization methods. We organize PTQ methods by the axis they act on. Weight-only: GPTQ (Frantar et al., 2022) compensates weight-quantization error via second-order information and does not touch activations. Activation rescaling along the residual axis: SmoothQuant (Xiao et al., 2023) migrates magnitude between activations and weights per input channel; AWQ (Lin et al., 2024) selects salient weight channels by activation magnitude and protects them. Residual-axis rotation: QuaRot (Ashkboos et al., 2024) applies orthogonal (Hadamard) rotations to decorrelate outlier dimensions, with two flavors — a per-Linear Appendix variant that rotates each linear’s input independently, and a full variant that additionally fuses a shared residual-axis rotation into surrounding weights and applies an online Hadamard to w2’s input. Under our taxonomy the online w2 Hadamard is an FFN-internal intervention, not a residual-axis one. SpinQuant (Liu et al., 2025) learns rotations; FlatQuant (Sun et al., 2025) extends to per-Linear affine transforms; OmniQuant (Shao et al., 2024) learns per-layer weight-clip and channel-smooth parameters; RPTQ (Yuan et al., 2023) reorders activation channels into blocks before quantizing. Our experiments cover per-Linear QuaRot, SmoothQuant, AWQ, and full QuaRot (offline residual-axis rotation with RMSNorm-fusion plus online per-head attention-value Hadamard plus online orthogonal w2 rotation), the last of which is the direct post-hoc test of whether the FFN-internal rotation closes the residue our decomposition locates in w2. Attention-internal and training-time interventions. Bondarenko et al. (2023) trace outliers to “no-op” attention heads whose softmax puts full mass on a single token and propose clipped softmax and gated attention to prevent the problem at training time. Darcet et al. (2024) insert auxiliary register tokens into vision transformers to absorb artifactual high-norm activations, inspiring the channel-axis port we use as a probe. Our Depth Registers variant is a thin architectural intervention: we partition the residual channel axis into a semantic region and a register region and penalize register-region magnitude via a hinge loss. Compared to register tokens (which act along the token axis) and to rotation-based post-hoc methods, our probe targets the residual-axis component of the intervention toolkit at matched FP16 PPL and matched zero-shot capacity (±1.5± 1.5 p on three 1k-example probes), without mixing in per-Linear scaling or Hadamard rotations. Positioning. We do not claim novelty on “GLU internals matter” (Yang et al., 2025), “Hadamards help” (Ashkboos et al., 2024), or “registers/sinks help” (Darcet et al., 2024; Xiao et al., 2024). Our contribution is narrower and threefold. (i) Site-wise damage observables: three excess-NLL quantities — ΔNLL(naive) (naive), the reader-rescued residue ΔNLL(skip-readers) (skip-readers), and single-site skip-removals Δremove(S) (S) — that are conditional on the rest being W4A4, not an additive reader/generator/interaction decomposition. (i) Residual-axis reachability: which inputs a residual-axis magnitude bound reaches (readers tightly; o_proj up to a softmax- and WVW_V-dependent constant; w2 only via a trivial product bound) — a one-sided observation, not an impossibility result against all post-hoc methods. (i) Residual-axis-targeted training probe: DR+sink empirically isolates most of the reader-conditional damage at matched FP16 PPL and matched zero-shot capacity, independently of any post-hoc calibration or rotation. 3 Setup and Mechanism 3.1 Preliminaries A single SwiGLU transformer block (Shazeer, 2020) operates on a residual stream x∈ℝT×dx ^T× d, where T is the sequence length and d the model dimension: y y =x+Attn(RMSNorm1(x)), =x+Attn(RMSNorm_1(x)), (1) z z =y+MLP(RMSNorm2(y)), =y+MLP(RMSNorm_2(y)), (2) with Attn(x~)=o_proj(softmax(Q(x~)K(x~)⊤/dh)V(x~))Attn( x)= o\_proj(softmax(Q( x)K( x) / d_h)\,V( x)) and MLP(x~)=w2(silu(w1(x~))⊙w3(x~))MLP( x)= w2(silu( w1( x)) w3( x)). Here ⊙ denotes elementwise product. The five trainable input-activation sites are the linears qkv,w1,w3\ qkv, w1, w3\, each of which reads RMSNorm(⋅)RMSNorm(·), and o_proj,w2\ o\_proj, w2\, each of which reads a function of another linear’s output. Quantization setting. We study weight/activation quantization with 4-bit weights (RTN with group size 128128) and per-token dynamic 4-bit activations (W4A4). “Naive W4A4” applies this uniformly to every linear in the transformer blocks. The “skip” configurations in Section˜4 leave a named subset of linears at FP16 while quantizing the remainder. Tail metrics. We measure outlier severity of each linear’s input activations via the centered excess kurtosis kurtc(x)=[(x−μ)4]/Var(x)2−3kurt_c(x)=E[(x-μ)^4]/Var(x)^2-3, aggregated by taking the channel-wise maximum within each layer and reporting the mean across layers. A Gaussian has kurtc=0kurt_c=0; empirically, well-behaved activations have kurtc≲10kurt_c 10, while outlier-dominated channels reach kurtc>1000kurt_c>1000. An appendix table reports severity (max-channel/median-channel ratio) as a second, monotonic tail metric. 3.2 Operator-level taxonomy: readers vs. generators Figure˜1 (introduction) draws the five input-activation sites on a single SwiGLU block. Three of them — qkv, w1, w3 — are fed RMSNormRMSNorm of the residual stream; the remaining two — o_proj, w2 — are fed quantities produced inside the block (the attention-weighted V aggregate, and the SwiGLU bilinear product, respectively). We promote this split to a definition. Definition 3.1 (Readers and generators). The input-activation site of a trainable linear in a SwiGLU block is a reader if its input is RMSNorm(⋅)RMSNorm(·) of the residual stream, and a generator otherwise. Under this definition, qkv,w1,w3\ qkv, w1, w3\ are readers and o_proj,w2\ o\_proj, w2\ are generators. Table˜2 summarizes the classification and what each method in Section˜2 can reach. The taxonomy is mechanism-motivated bookkeeping, not a theorem contribution: it labels each input-activation site by the axis on which its magnitude is controllable, and predicts which ablations should expose which component of W4A4 damage. Three elementary observations follow. Table 2: Outlier taxonomy and method compatibility. Each trainable linear in a SwiGLU transformer block is classified by whether its input is the residual stream (post-RMSNorm reader) or a block-internal product (attention-side softmax V for o_proj; SwiGLU gate product for w2). A method can only reduce outliers on the axis it acts on — readers cannot be fixed by a generator intervention and vice-versa. Linear Input source Class qkv residual → RMSNorm reader (linear in residual) w1 residual → RMSNorm reader (linear in residual) w3 residual → RMSNorm reader (linear in residual) o_proj softmax(QK⊤)V(QK )V generator (convex-mix of V) w2 silu(w1x)⊙w3xsilu( w1x) w3x generator (quadratic product) Method Fixes readers? Fixes generators? DR + sink (this paper) ✓ ✗ SmoothQuant (Xiao et al., 2023) partial ✗ AWQ (Lin et al., 2024) partial ✗ QuaRot, per-Linear (Ashkboos et al., 2024) ✓ partial† QuaRot, full (online w2 Hadamard)‡ ✓ ✗ † Per-Linear QuaRot rotates each linear’s input independently and reaches part of the o_proj-side residue but leaves a w2-dominated gap to FP16 in our experiments (Table˜4). ‡ Full QuaRot additionally applies an online Hadamard rotation between the SwiGLU product and w2 — an FFN-internal intervention on the generator axis. Tested: ΔNLL≈0.09 ≈ 0.09 nats, marginally worse than per-Linear QuaRot on all three checkpoints (Table˜4), consistent with Observation˜3.4’s prediction that orthogonal rotation cannot bound the bilinear tail. Observation 3.2 (Readers are bounded by the residual). Let RMSNormγ(x)=γ⊙x/d−1∑ixi2+εRMSNorm_γ(x)=γ x/ d^-1 _ix_i^2+ denote RMSNorm with learned per-channel gain γ∈ℝdγ ^d and numerical stabilizer ε>0 >0. Any intervention that bounds the residual magnitude ‖x‖∞ \|x \|_∞ bounds every reader input, up to the learned gain: if ‖x‖∞≤M \|x \|_∞≤ M and d−1∑ixi2≥r2>0d^-1 _ix_i^2≥ r^2>0, then ‖RMSNormγ(x)‖∞≤‖γ‖∞⋅‖x‖∞r2+ε≤‖γ‖∞⋅Mr, \|RMSNorm_γ(x) \|_∞\;≤\; \|γ \|_∞· \|x \|_∞ r^2+ \;≤\; \|γ \|_∞· Mr, so reader inputs live in a controlled-tail regime whenever the residual does, modulo ‖γ‖∞ \|γ \|_∞ — an FP16 weight, quantization-invariant in our setting. RMSNorm is applied at every layer before each reader, so the bound holds layer-wise. Observation 3.3 (The attention generator is linear in the value projections). The input to o_proj at token t is ∑jαt,jWVx~j _j _t,jW_V x_j, with softmax weights αt,j≥0 _t,j≥ 0, ∑jαt,j=1 _j _t,j=1 themselves depending on x~ x through QK⊤/dhQK / d_h. Conditional on αt,j\ _t,j\, the map is linear in x~ x; using the induced ℓ∞→ℓ∞ _∞→ _∞ norm ‖WV‖∞→∞=maxi∑j|(WV)ij| \|W_V \|_∞→∞= _i _j |(W_V)_ij |, ‖o_proj input‖∞≤‖WV‖∞→∞⋅maxj‖x~j‖∞ \| o\_proj input \|_∞≤ \|W_V \|_∞→∞· _j \| x_j \|_∞. So a residual-axis bound admits only loose control of o_proj’s input: it is tight only when attention places mass on outlier positions and the corresponding WVW_V row sums coherently. Observation 3.4 (The MLP generator is bilinear in the residual). The input to w2 is the elementwise product u⊙vu v where u=silu(w1(x~))u=silu( w1( x)) and v=w3(x~)v= w3( x). Each factor is reader-like: its magnitude is bounded by a residual-axis intervention via Observation˜3.2. A bound on factor norms gives only a trivial product bound on u⊙vu v; in particular a rare large entry in one factor multiplied by a typical entry in the other produces an outlier not present in either factor, and residual-axis magnitude control does not suppress this. The exact fourth-moment identity is [(uivi)4]=[ui4vi4]=[ui4][vi4]+Cov(ui4,vi4),E[(u_iv_i)^4]=E[u_i^4v_i^4]=E[u_i^4]E[v_i^4]+Cov(u_i^4,v_i^4), so even when the marginal fourth moments of uiu_i and viv_i are individually controlled, positive dependence in their rare large events can preserve a heavy-tailed product. This is the SwiGLU multiplicative amplification noted empirically for GLU activations by Yang et al. (2025); it is the source of the stubborn w2 input tail we measure in Section˜4.1. We emphasize this is a reachability argument, not a no-go theorem: orthogonal rotations preserve second-order structure but could in principle suppress fourth-moment growth in the product — whether they do is an empirical question we test against full QuaRot in Section˜4.3. The practical reading of Observations˜3.2, 3.3 and 3.4 is scoped to what residual-axis magnitude control can reach: such an intervention bounds reader inputs tightly, admits only loose control of o_proj’s input, and does not bound w2’s input tail beyond a trivial product bound. A measurable per-operator damage decomposition is therefore predicted: residual-axis cleanup should collapse reader outliers and reader-side W4A4 damage while leaving an o_proj- and w2-shaped residue. Observations˜3.3 and 3.4 are elementary one-sided observations, not impossibility results: they describe the inherent looseness of residual-axis magnitude bounds on generator inputs, and say nothing about FFN-internal or attention-internal interventions — e.g., an online rotation fused into w2’s input — which act on axes our argument does not cover. 3.3 A residual-axis probe: Depth Registers with a sink loss We need an intervention that targets residual allocation during training while avoiding the interventions we want to dissociate from: no post-hoc calibration, no rotation, no per-Linear scaling, no FFN-internal or attention-internal surgery. Depth Registers (DR) with a register-magnitude hinge loss are that intervention. DR+sink is not a pure residual-axis projection: it changes parameterization, optimization dynamics, and the representations the rest of the network learns. What we claim is that its mechanism of action is a residual-axis magnitude hinge, and that the reader/generator separation we report does not mix in post-hoc calibration or rotation tricks. The capacity controls that make this comparison meaningful are (i) FP16 validation PPL matched across the three checkpoints to within 0.10.1 (23.623.6–23.723.7), and (i) matched zero-shot capacity on three 1k-example probes within ±1.5± 1.5 p (Table˜6, appendix). We partition the residual channel axis into a semantic region of d−kd-k channels and a register region of k channels and apply independent affine parameters within each region via a partitioned RMSNorm: RMSNormpart(x)i=γisem⋅xi/1d−k∑j∈semxj2,i∈sem,γireg⋅xi/1k∑j∈regxj2,i∈reg.RMSNorm_part(x)_i= casesγ^sem_i· x_i/ 1d-k _j x_j^2,&i ,\\ γ^reg_i· x_i/ 1k _j x_j^2,&i . cases (3) The sink loss is a hinge penalty on the register-region magnitude, ℒsink=λ⋅tmax(0,‖xtreg‖∞−τ)L_sink=λ·E_t (0, \|x_t^reg \|_∞-τ), with λ=0.01λ=0.01 and target τ=3.0τ=3.0. Its role is the same as the attention-sink token in Xiao et al. (2024): a dedicated, bounded outlet for magnitude that the rest of the network would otherwise deposit on semantic channels. The register channels absorb outlier mass, the semantic channels stay calibrated, and neither post-hoc calibration nor any rotation is applied. We treat DR+sink as one training-time probe of residual-axis magnitude control, not a deployment proposal and not a claim that Depth Registers are uniquely necessary. A full separation between the register partition and the hinge loss would require additional controls (a hinge applied to a standard-RMSNorm residual, or DR without the hinge); our partition-plus-hinge coupling is deliberate and reflects the mechanism both components jointly implement (named-channel sink with bounded magnitude). The k=64k=64 without sink loss configuration (reported in Section˜4.1) isolates the extra-capacity effect and shows it increases rather than decreases reader kurtosis, which we read as evidence that the hinge — not the partition — is load-bearing. Every experiment in Section˜4 uses k=64k=64 out of d=1024d=1024 or the k=0k=0 baseline. 4 Experiments Setup. Three 300M-parameter decoder-only transformers (20 layers, d=1024d=1024, 16 heads) are trained from scratch on 5B tokens of FineWeb-Edu (Penedo et al., 2024) at a matched FP16 validation perplexity of 23.623.6–23.723.7. The three configurations share optimizer, data, schedule, and weight initialization; they differ only in the DR+sink setting: k=0k=0 (baseline), k=64k=64 with no sink loss, and k=64k=64 with sink loss (λ=0.01λ=0.01, τ=3.0τ=3.0). All tail statistics are computed in FP32 on 1.5M held-out tokens. Every quantization result is per-token dynamic INT4 activations with RTN group-128128 INT4 weights; perplexity is reported on the same 1.5M-token held-out split. We flag one limitation up front: results are single-seed at 300M parameters and 5B tokens. The reader-vs.-generator separations we report are larger than any plausible seed noise (qkv kurtosis 35.9→2.8535.9→ 2.85, naive W4A4 PPL 1727→1191727→ 119), but a second-seed replication is the first follow-up we recommend (Section˜5). 4.1 Per-linear input-activation tails Figure˜2(a) plots the mean-over-layers of per-layer channel-max kurtckurt_c for the five input-activation sites, across the three checkpoints. Three observations. First, readers collapse with DR+sink consistent with Observation˜3.2: qkv reaches kurtc≈2.85kurt_c≈ 2.85, indistinguishable from a Gaussian, and w1, w3 drop by ≈2.8×≈ 2.8×. Second, neither generator moves materially: o_proj sits near kurtc≈100kurt_c≈ 100 across all three checkpoints, and w2 near kurtc≈1900kurt_c≈ 1900. This is consistent with Observation˜3.4’s reachability argument that a residual-axis magnitude intervention does not bound the bilinear product tail. Third, as a sanity check that the hinge is doing the work and not the extra channel capacity, k=64k=64 without the sink loss slightly increases reader kurtosis (qkv: 35.9→44.335.9→ 44.3); the extra partition capacity is used for semantic work unless the hinge penalty is applied. The per-layer heatmap (Figure˜4, appendix) shows the same pattern is consistent across depth: reader rows darken uniformly under DR+sink, generator rows do not. We checked that the reader/generator split is not an artifact of the kurtosis statistic: the severity ratio (max-channel / median-channel L∞L_∞, reported in Figure˜4) and a max-abs tail metric (not shown) separate the two groups in the same direction across all three checkpoints, differing only in how compressed the reader scale becomes after DR+sink. 4.2 Per-linear-type W4A4 damage budget We now quantify how each class of linears contributes to naive W4A4 damage by skip-ablation: for each named subset S, we leave S at FP16 and quantize the remaining linears to W4A4. Figure˜3 visualizes the sweep; Table˜3 gives the numbers. Figure 3: W4A4 skip-linear ablation. Log-scale PPL by skip target. Skipping all readers drops every checkpoint to the ∼31 31 PPL floor; generator-only skips leave near-naive damage; w2 dominates o_proj within each model. Table 3: W4A4 skip-linear ablation. Validation PPL when the named linears are held at FP16 and the rest quantized W4A4. Skipping all readers drops every checkpoint to ∼ 31 PPL; the unquantizable damage lives in the generators regardless of DR/sink. FP16 PPL ≈23.7≈ 23.7. Kept at FP16 k=0k=0 k=64k=64 (DR) k=64k=64+sink all quantized (W4A4) 1727 671 119 skip qkv (reader) 323 159 65.9 skip w1,w3 (readers) 141 126 66.6 skip all readers (qkv,w1,w3) 31.3 31.3 31.0 skip o_proj (generator) 1623 630 123 skip w2 (generator) 719 248 59.9 skip both generators (o_proj,w2) 694 249 62.4 Excess-NLL observables. PPL is non-additive, so we work in excess NLL. Let ℒL be the set of all trainable linears in the transformer blocks, ℓ(⋅) (·) validation cross-entropy (nats/token), Q(ℒ)Q(L) the all-W4A4 configuration, and Q(ℒ∖S)Q(L S) the configuration with S⊆ℒS held at FP16. We define ΔNLL(naive):=ℓ(Q(ℒ))−ℓ(FP16) (naive):= (Q(L))- (FP16), ΔNLL(skip-S):=ℓ(Q(ℒ∖S))−ℓ(FP16) (skip-S):= (Q(L S))- (FP16), and Δremove(S):=ΔNLL(naive)−ΔNLL(skip-S) (S):= (naive)- (skip-S). The reader-rescued residue is ΔNLL(skip-readers):=ΔNLL(skip-qkv,w1,w3) (skip-readers):= (skip-\ qkv, w1, w3\). All quantities use a per-checkpoint FP16 denominator. Δremove(S) (S) is a counterfactual reduction conditional on the complement being quantized, not an additive component: ∑SΔremove(S) _S (S) generally differs from ΔNLL(naive) (naive). Figure˜2(b) visualizes the three configurations: the reader-rescued residue is flat at 0.28/0.28/0.270.28/0.28/0.27 nats (PPL 31.3/31.3/31.031.3/31.3/31.0) across the three matched checkpoints; ΔNLL(naive) (naive) shrinks ≈2.7×≈ 2.7× under DR+sink, pulling Δremove(readers) (readers) with it. Three reads. First, reader rescue removes most of the damage. At k=0k=0, ΔNLL(naive W4A4)=4.32 (naive W4A4)=4.32 nats and holding readers at FP16 drops ΔNLL to 0.280.28 nats — Δremove(readers)=4.04 (readers)=4.04 nats (≈94%≈ 94\% of the excess NLL above FP16), the damage readers contribute conditional on generators being W4A4 (including reader/generator interaction, not their additive share). The reader-rescued residue ΔNLL(skip-readers) (skip-readers) is essentially identical (0.28/0.28/0.270.28/0.28/0.27 nats) across checkpoints, measuring the remaining damage after readers are held at FP16; per-Linear QuaRot (Section˜4.3) undercuts this residue by reaching part of the o_proj-side damage, so it is not a lower bound on residual-axis post-hoc methods. Second, the probe removes most of the reader-conditional damage without post-hoc calibration. ΔNLL(naive W4A4) (naive W4A4) drops from 4.324.32 nats (k=0k=0) to 1.621.62 nats (k=64k=64 + sink), while the reader-rescued residue stays at 0.270.27 nats: the reduction is Δremove(readers) (readers)-side, not residue-side (Figure˜2(b)). Third, the two generators are not equivalent. At k=0k=0, Δremove(o_proj)=0.06 ( o\_proj)=0.06 nats vs. Δremove(w2)=0.88 ( w2)=0.88 nats. At k=64k=64+sink, Δremove(o_proj) ( o\_proj) is numerically small (skip-o_proj PPL is −4.0-4.0 below naive, ∼ grid step; with one seed we cannot distinguish this from a modest positive effect), while Δremove(w2)=0.69 ( w2)=0.69 nats — a ∼10× 10× asymmetry unlikely to be seed-driven. o_proj’s skip-removal tracks residual cleanliness; w2’s persists under a clean residual, consistent with its bilinear origin. Δremove(S) (S) is conditional on the rest being quantized: neither 0.880.88 nor 0.690.69 is an additive w2 share. The per-layer heatmap (Figure˜4) confirms this is not a few-layer effect. A noise-injection sweep (Figure˜5, appendix) corroborates: Gaussian noise at fixed relative std on w2’s input yields ≈2.7×≈ 2.7× larger Δloss than the same injection at o_proj’s input across all three checkpoints. 4.3 Post-hoc-method interaction and the direct test of Observation˜3.4 We test four post-hoc methods against the DR+sink probe: SmoothQuant, AWQ, per-Linear QuaRot, and full QuaRot (offline residual-axis rotation with RMSNorm-fusion, online per-head Hadamard on attention values, and online orthogonal rotation on w2’s input). Full QuaRot adds the FFN-internal intervention Observation˜3.4 flags — rotation inside the SwiGLU product — and is the direct post-hoc test of the prediction that orthogonal rotation cannot bound the bilinear tail. Table 4: W4A4 perplexity across post-hoc methods. Per-Linear QuaRot substitutes for DR+sink on readers (spread ≤0.1≤ 0.1 PPL); full QuaRot adds the online w2-input rotation but does not close the gap (ΔNLL≈0.09 ≈ 0.09 nats, marginally worse than per-Linear). FP16 PPL ≈23.7≈ 23.7. Method k=0k=0 k=64k=64 (DR) k=64k=64+sink FP16 (reference) 23.7 23.7 23.7 RTN 1727 671 119 SmoothQuant (Xiao et al., 2023) 57.6 58.1 39.9 AWQ (Lin et al., 2024) 1698 698 111 QuaRot (per-Linear) (Ashkboos et al., 2024) 25.5 25.4 25.4 QuaRot (full) (Ashkboos et al., 2024) 26.0 26.1 26.0 Four observations. (i) Weight-only INT4 is near-lossless; damage is in the activation path. (i) Per-Linear QuaRot substitutes for DR+sink on the reader axis (25.525.5/25.425.4/25.425.4, spread ≤0.1≤ 0.1). (i) SmoothQuant composes partially with DR+sink (57.6→39.957.6→ 39.9); AWQ’s 1698→1111698→ 111 inherits from DR+sink’s residual cleanup. (iv) Full QuaRot does not close the gap either: 26.026.0/26.126.1/26.026.0 (ΔNLL≈0.09 ≈ 0.09 nats), marginally worse than per-Linear QuaRot despite adding offline residual rotation and online FFN-internal w2 rotation (FP16 sanity |Δloss|<10−4| |<10^-4, Appendix˜A). The leftover ≈2≈ 2 PPL is consistent with a bilinear tail not reducible by orthogonal transforms, motivating mixed-precision w2 or QAT (Section˜5). 5 Discussion and Limitations Takeaway and scope. Observation˜3.4 predicts orthogonal rotation need not bound the bilinear SwiGLU tail; empirically the reader-rescued residue stays at ≈0.28≈ 0.28 nats and full QuaRot at ΔNLL≈0.09 ≈ 0.09 nats. Single-seed at 300M×5B; we do not claim the w2-residue extends unchanged to pretrained 7B–70B models. Follow-ups: second-seed, scale-up to 1.2B×20B, LLaMA-7B validation, non-rotation w2 fixes (mixed-precision, QAT, product clipping). References Ashkboos et al. (2024) Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html. Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309.16609, 2023. doi: 10.48550/ARXIV.2309.16609. URL https://doi.org/10.48550/arXiv.2309.16609. Bondarenko et al. (2023) Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.c/paper_files/paper/2023/hash/edbcb7583fd8921dad78adecfe06a99b-Abstract-Conference.html. Darcet et al. (2024) Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1. Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. CoRR, abs/2208.07339, 2022. doi: 10.48550/ARXIV.2208.07339. URL https://doi.org/10.48550/arXiv.2208.07339. Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post-training quantization for generative pre-trained transformers. CoRR, abs/2210.17323, 2022. doi: 10.48550/ARXIV.2210.17323. URL https://doi.org/10.48550/arXiv.2210.17323. Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825, 2023. doi: 10.48550/ARXIV.2310.06825. URL https://doi.org/10.48550/arXiv.2310.06825. Lin et al. (2024) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Phillip B. Gibbons, Gennady Pekhimenko, and Christopher De Sa, editors, Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13-16, 2024. mlsys.org, 2024. URL https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9d64e9ba4a95c1ef21-Abstract-Conference.html. Liu et al. (2025) Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quantization with learned rotations. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=ogO6DGE6FZ. Penedo et al. (2024) Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html. Shao et al. (2024) Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. International Conference on Learning Representations (ICLR), 2024. Shazeer (2020) Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202. Sun et al. (2024) Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. CoRR, abs/2402.17762, 2024. doi: 10.48550/ARXIV.2402.17762. URL https://doi.org/10.48550/arXiv.2402.17762. Sun et al. (2025) Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. Flatquant: Flatness matters for LLM quantization. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https://proceedings.mlr.press/v267/sun25l.html. Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. doi: 10.48550/ARXIV.2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971. Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Proceedings of Machine Learning Research, pages 38087–38099. PMLR, 2023. URL https://proceedings.mlr.press/v202/xiao23c.html. Xiao et al. (2024) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=NG7sS51zVF. Yang et al. (2025) Jaewoo Yang, Hayun Kim, Junyung Ji, and Younghoon Kim. Mitigating quantization errors due to activation spikes in gated linear unit-based large language models. Future Internet, 17(4):185, 2025. doi: 10.3390/FI17040185. URL https://doi.org/10.3390/fi17040185. Yuan et al. (2023) Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. RPTQ: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089, 2023. Appendix A Implementation details Architecture. All three models share: 20 decoder layers, d=1024d=1024 model dimension, 16 attention heads (head dimension 6464), SwiGLU MLP with inner dimension 27522752 (parameter budget matched across configurations), RoPE positional embeddings, tied token-embedding / output head, vocabulary size 50,30450,304. Total trainable parameters: 302302M. Training. Optimizer AdamW with β=(0.9,0.95)β=(0.9,0.95), weight decay 0.10.1, gradient clipping at 1.01.0. Peak learning rate 6×10−46× 10^-4 with 6060M-token linear warmup and cosine decay to 6×10−56× 10^-5. Batch size 11M tokens (micro-batch 88, sequence length 20482048, gradient accumulation 88 across 1616 GPUs) for 50005000 steps on 55B tokens of FineWeb-Edu [Penedo et al., 2024]. Training precision BF16 mixed. One seed per configuration. Depth Registers. For k>0k>0 we partition the residual channel axis into sem=0,…,d−k−1sem=\0,…,d-k-1\ and reg=d−k,…,d−1reg=\d-k,…,d-1\. The partitioned RMSNorm applies independent (γsem,γreg)(γ^sem,γ^reg) affine parameters and independent root-mean-square denominators within each partition. The partition is applied at every RMSNorm site in the network, including the final pre-head norm. Sink loss. Let xt∈ℝdx_t ^d denote the residual stream at token t immediately before the first block. The sink loss is ℒsink=λ⋅tmax(0,‖xtreg‖∞−τ)L_sink=λ·E_t (0, \|x^reg_t \|_∞-τ) with λ=0.01λ=0.01 and τ=3.0τ=3.0, added to the cross-entropy loss. Quantization (shared across all methods). Weights are quantized with round-to-nearest symmetric INT4 at group size 128128 along the input axis. Activations are quantized per-token dynamically to INT4 symmetric: per-token scale st=maxi|at,i|/7s_t= _i |a_t,i |/7, quantized value a~t,i=st⋅round(clip(at,i/st,−8,7)) a_t,i=s_t·round(clip(a_t,i/s_t,-8,7)). The qkv linear is fused (single weight of shape 3dheadnh×d3d_headn_h× d); its W4A4 input scale is shared across Q, K, V slices. All post-hoc baselines (SmoothQuant, AWQ, per-Linear QuaRot, full QuaRot) use identical weight and activation quantizers to this default. Calibration set. SmoothQuant, AWQ, per-Linear QuaRot, and full QuaRot’s offline-rotation step all use the same 256,000256,000-token calibration slice (disjoint from the 1.51.5M-token evaluation split, drawn from the same FineWeb-Edu validation shard). Activation statistics (channel-wise maxima for SmoothQuant, per-channel L2L_2 importance for AWQ) are computed once per checkpoint and cached. SmoothQuant. α∈0.3,0.5,0.7,0.8α∈\0.3,0.5,0.7,0.8\ swept per checkpoint; we report the setting with lowest calibration-set NLL. Selected: α=0.5α=0.5 for k=0k=0, α=0.7α=0.7 for k=64k=64, α=0.7α=0.7 for k=64k=64+sink. AWQ. Default group size 128128, salient-channel fraction swept over 0.01,0.02,0.05,0.1\0.01,0.02,0.05,0.1\; best per checkpoint reported. Unlike the original AWQ paper (which targets weight-only), we apply AWQ weight scales and then quantize activations per-token dynamically, so the comparison is W4A4 throughout. Per-Linear QuaRot. The Appendix variant of Ashkboos et al. [2024]: each linear’s input receives an independent random orthogonal Hadamard rotation (size matched to the linear’s input dimension) before quantization. No shared residual-axis rotation, no online w2 Hadamard. Rotation seeds are fixed across the three checkpoints. Full QuaRot. Main-body procedure of Ashkboos et al. [2024]: (i) fuse each RMSNorm’s γ into the next Linear’s weights; (i) generate a shared residual-axis orthogonal R (block-diagonal diag(Rsem,Rreg)diag(R_sem,R_reg) when k>0k>0, since rotation within each partition commutes with partitioned-RMSNorm); (i) fold R into E’s output, residual-reading Linears’ input, residual-writing Linears’ output, and the head’s input (untying the head from E if tied); (iv) apply an online per-head Hadamard of size dhead=64d_head=64 to attention values, fused into o_proj’s input side; (v) apply an online orthogonal rotation to w2’s input (27522752 is not a power of 22, so a random orthogonal; Kronecker-factored Hadamard alternatives run-timed within 1%1\%). At inference, steps (iv) and (v) are fused into o_proj’s and w2’s W4A4 kernels as a pre-rotation of the quantized activation; offline cost is a single matmul per linear call. Rotation seeds are fixed across checkpoints. Full QuaRot port sanity check. We verified the full QuaRot implementation with an FP16 sanity check: with online rotations disabled, the offline residual-axis rotation is a pure change of basis and post-rotation FP16 PPL must match baseline FP16 PPL. Across all three checkpoints and both rotation kinds (randomized Hadamard, random orthogonal), |Δloss|<10−4| |<10^-4 nats on the 1.51.5M-token validation split, confirming the offline pipeline. The ≈0.09≈ 0.09-nat residue reported in Table˜4 is therefore a property of the method under W4A4 quantization, not a port artifact. Evaluation. All perplexity numbers are reported on a 1.51.5M-token held-out FineWeb-Edu split, evaluated in FP32. Downstream accuracy in Table˜6 uses the first 10001000 examples of each dataset’s standard evaluation split; decoding is greedy. Appendix B Supplementary tail metrics Table 5: Full method matrix (W-only and W4A4). Validation PPL on FineWeb-Edu. Top block: weight-only INT4/INT8 quantization (all methods work). Bottom block: W4A4 activation quantization, where residual-axis rescaling (SmoothQuant, AWQ) fails and only QuaRot’s generator-axis Hadamard recovers the gap. Method k=0k=0 k=64k=64 k=64k=64+sink Weight-only FP16 23.65 23.66 23.68 RTN W8 23.66 23.66 23.68 RTN W4 (g128) 24.26 24.25 24.24 RTN W4 (per-channel) 24.64 24.64 24.61 GPTQ W4 (g128) 23.99 23.97 24.04 AWQ W4 24.16 24.15 24.18 SmoothQuant W4 26.28 25.90 25.76 QuaRot W4 24.26 24.22 24.23 W4A4 (weights + activations) RTN W4A4 1727 671 119 SmoothQuant W4A4 57.64 58.10 39.94 AWQ W4A4 1698 698 111 QuaRot W4A4 25.49 25.41 25.42 Table˜5 expands Table˜4 to the full set of weight-only and W4A4 configurations we evaluated. The weight-only block (top) confirms that INT4 weights with group size 128128 are essentially lossless on all three checkpoints. In the W4A4 block (bottom), per-Linear QuaRot is the only method that reaches within 22 PPL of FP16 across all three checkpoints. Figure 4: Per-layer input-activation kurtosis. kurtckurt_c per linear per layer, log color scale. Under DR+sink (k=64k=64 + sink, bottom row), reader rows (qkv, w1, w3) dim uniformly across all 2020 layers, while generator rows (o_proj, w2) are indistinguishable from the k=0k=0 baseline at every layer. Severity (max-channel / median-channel ratio, not shown) varies across layers and linears by 22–8×8× and is therefore a mild discriminator; the decisive reader/generator separation is in tail shape (kurtosis), not maximum magnitude. Appendix C Noise-injection sensitivity The main-text Δremove observables in Section˜4.2 compare W4A4 against FP16 and therefore conflate two things: how large an input tail is, and how sensitive the downstream loss is to perturbations at that site. To isolate the sensitivity factor we run a controlled noise-injection sweep: at inference, Gaussian noise with per-channel relative standard deviation σrel∈0,0.005,0.01,0.02,0.05,0.1,0.2 _rel∈\0,0.005,0.01,0.02,0.05,0.1,0.2\ is added to the input of either o_proj or w2 at every layer, and validation Δ is measured against the unperturbed FP16 run. The sweep is repeated for all three 300M×5B300M× 5B checkpoints. This is a complementary witness to the W4A4 damage budget: it does not by itself establish the linear-vs.-bilinear propagation argument — noise here is injected at the generator’s input, bypassing the upstream residual-axis magnitude story that drives the tails of those inputs — but it directly probes the functional sensitivity of each generator’s output to perturbation. Figure˜5 shows the result. At σrel=0.1 _rel=0.1, averaged across the three checkpoints, Δ at o_proj’s input is 1.8×10−31.8× 10^-3 nats and at w2’s input is 4.9×10−34.9× 10^-3 nats — a 2.7×2.7× asymmetry. At σrel=0.2 _rel=0.2 the same ratio is 2.7×2.7× (7.3×10−37.3× 10^-3 vs. 2.0×10−22.0× 10^-2). The superlinear growth at w2’s input (approximately quadratic in σrel _rel) is consistent with Observation˜3.4’s prediction that the SwiGLU bilinear product amplifies perturbations of its factors; the near-linear growth at o_proj’s input is consistent with Observation˜3.3’s prediction that the attention generator is linear in V(x~)V( x) conditional on the softmax weights. Crucially the curves for the three checkpoints lie essentially on top of each other at both generators: the sensitivity asymmetry is a property of the SwiGLU block’s architecture, not of the particular residual-axis cleanup performed on the readers. This rules out one specific alternative explanation for the main-text w2-dominant residue — that DR+sink happens to change w2’s input distribution in a way that breaks linearity of loss in tail magnitude — and reinforces that the residue reflects a structural property of the operator, not an accident of our particular checkpoints. Figure 5: Sensitivity of validation loss to Gaussian noise injected at generator inputs. For each of the three checkpoints, Gaussian noise with per-channel relative standard deviation σrel _rel is added to the input of either o_proj (left) or w2 (right) at every layer, and Δ is measured against FP16. Noise at o_proj’s input grows close to linearly in σrel _rel; noise at w2’s input grows superlinearly and is architecture-invariant across the three checkpoints. This is a complementary sensitivity witness: it does not by itself establish the linear-vs.-bilinear propagation argument — noise here is injected at the generator’s input, bypassing the upstream residual-axis magnitude story — but it is consistent with Observation˜3.4’s prediction that w2’s input tolerates less perturbation than o_proj’s. Appendix D Downstream accuracy Table 6: Downstream task accuracy at 300M × 5B. Evaluated on the first 1000 examples of each dataset. Adding Depth Registers and the attention-sink loss does not harm zero-shot downstream performance relative to the k=0k=0 baseline, confirming capacity is preserved. Model HellaSwag ↑ Winogrande ↑ LAMBADA acc ↑ LAMBADA PPL ↓ k=0k=0 (baseline) 35.7 48.8 20.3 65.8 k=64k=64 (DR only) 38.0 48.8 20.3 67.8 k=64k=64 + sink 36.5 48.9 20.3 64.7 Table˜6 reports FP16 downstream accuracy on three standard zero-shot benchmarks, using the first 10001000 examples of each dataset’s standard split with greedy decoding. Differences between the three checkpoints are within ±1.5± 1.5 percentage points on all three tasks. We treat this as a capacity sanity check, not a definitive downstream-equivalence result: 10001000 examples and three datasets cannot rule out modest task-specific regressions. A full quantization-by-downstream evaluation matrix with bootstrap CIs is future work.