Paper deep dive
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei, Milad Hosseini, Adrian Weller
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Most post-training quantization pipelines fit each weight matrix to its pretrained counterpart, one matrix at a time. Whether that proxy tracks what an attention block actually computes, or how errors in the Q, K and V projections compound inside the softmax, is rarely checked. We write the objective on the attention output instead, over all three projections at once, and reuse it throughout the pipeline. JAB defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights (GPTQ warm start, then STE with learnable scales), and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this works. At 3 bits JAB recovers 77-90% of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a point. It stops working once MLP layers enter the allocation. A role-aware offset rule needing no sensitivity estimate at all beats JAB on GPT-2's MLP and on the full Mistral-7B model: with a 3-bit floor it quantizes 96.4% of the weights to 4.5 bits per parameter at 6.933 perplexity, within 4.4% of full precision (6.643) at 3.56x compression, against 7.158 for JAB at the same budget. Which matrix a weight sits in matters more than any sensitivity estimate we computed. Two things came out sideways. Block-local reconstruction is an unreliable proxy for end-to-end perplexity: one run improved a block's own objective 4.6x while perplexity rose 32x, which is why every allocation here is validated end-to-end. And on attention-only quantization, fine-tuning moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights.
Tags
Links
- Source: https://arxiv.org/abs/2609.34866v1
- Canonical: https://arxiv.org/abs/2609.34866v1
Trouble viewing inline? Open PDF directly →
Full Text
111,366 characters extracted from source content.
Expand or collapse full text
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers Nafiseh HosseinpourFardi1,∗ Negar Alihadi1,∗ Mahmoudreza Babaei2,3,4 Milad Hosseini5 Adrian Weller6 1Sharif University of Technology, Tehran, Iran 2GISMA University of Applied Sciences, Potsdam, Germany 3BRAINS, Brandenburg Research Center for Applied Intelligent Systems, Potsdam, Germany 4Max Planck Institute for Security and Privacy, Bochum, Germany 5Computer Vision Center (CVC), Barcelona, Spain 6University of Cambridge, United Kingdom Abstract Most post-training quantization pipelines keep each quantized weight matrix close to its pretrained counterpart, one matrix at a time, and stop there. Whether that proxy tracks what the attention block actually computes, or how errors in the Q, K and V projections compound once they meet inside the softmax, is rarely checked. The obvious fix is to write the objective on the attention output itself, over all three projections at once, and to use that same objective wherever the pipeline needs a signal. That is what JAB does. It defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block’s real causally-masked attention output, and uses it both to fit the quantized weights (GPTQ warm start, then STE with learnable scales) and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this pays off: at 3 bits JAB recovers 7777–90%90\% of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a perplexity point. Once MLP layers enter the allocation, though, JAB is beaten by a role-aware offset rule that needs no sensitivity estimate at all, both on GPT-2 with QKV and MLP quantized together and on the full Mistral-7B model. With a 3-bit floor, this rule quantizes 96.4%96.4\% of Mistral-7B’s weights to 4.5 bits per parameter at 6.9336.933 perplexity — within 4.4%4.4\% of full precision (6.6436.643) at 3.56×3.56× compression, 14 GB to 3.93 GB — against 7.1587.158 for JAB at the same budget, and to 3.5 bits at 7.8757.875 (4.57×4.57×). Whichever matrix a weight belongs to inside its block matters more than any sensitivity estimate we managed to compute. Two things came out sideways. Block-local reconstruction turned out to be an unreliable proxy for end-to-end perplexity: in one run a 4.6×4.6× improvement in a block’s own objective came with a 32×32× increase in perplexity, which is why every allocation in this paper is validated end-to-end rather than on local objectives alone. And on attention-only quantization, fine-tuning consistently moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights. 1 Introduction Large language models (LLMs) face severe memory bottlenecks during inference, and quantization mitigates these costs by mapping real-valued weights to discrete integers (Rakka et al., 2026). Because quantization-aware training (QAT) remains prohibitively resource-intensive, post-training quantization (PTQ) has become the dominant paradigm for efficient compression (Cao and Aref, 2025). Early quantization methods relied on the inverse Hessian matrix of the Optimal Brain Surgeon (OBS) algorithm, which proved computationally intractable for modern deep networks (Hassibi and Stork, 1990; Liu et al., 2025). GPTQ resolved this by processing weights in blocks via Cholesky decomposition, which made it possible to compress 175-billion-parameter LLMs such as GPT-3, models that are prohibitively expensive to deploy in standard FP16 precision, to 3 or 4 bits efficiently. Building on OBS, GPTQ minimizes a layer-wise weight-reconstruction loss; this local weight-space objective can, however, fail to capture end-to-end Transformer fidelity (Frantar et al., 2023). Pushing below 4 bits exposes structural vulnerabilities to heavy-tailed activation outliers. Outlier-Aware Weight Quantization (OWQ) addresses this by preserving a subset of sensitive weights in higher precision (Lee et al., 2024). To automate variable bit-width allocation under strict budgets, HAWQ-V2 sets bit widths from the average Hessian trace of the task loss, which serves as a measure of parameter sensitivity (Dong et al., 2020), and the resulting assignment is optimized with Multiple-Choice Knapsack Problem (MCKP) solvers (Sinha and Zoltners, 1979; Lee et al., 2026). HAQ learns the allocation instead (Wang et al., 2019). Moving beyond independent layer-wise optimization, BRECQ introduced block-wise reconstruction to capture cross-layer dependencies (Li et al., 2021). Attention-Aware PTQ (APTQ) narrows the gap for attention layers by folding attention information — the gradients of the non-linear softmax — into its sensitivity estimate to guide optimization, but that criterion is still a weight-space proxy, not the reconstruction objective itself (Guan et al., 2024). In most PTQ pipelines, then, the objective that reconstructs the quantized weights is not the criterion that sets their bit widths (Appendix ). This disconnect between the sensitivity-estimation criterion and the actual reconstruction loss, which we term the proxy gap, motivates the Joint Attention-Based (JAB) framework: we set out to test whether one objective could do both jobs, produce the weights and set the budget. JAB (Joint Attention-Based allocation) uses a single loss for both steps. We define ℒJABL_JAB over the concatenated [WQ∣WK∣WV][W_Q W_K W_V] parameters of a block and evaluate it against that block’s own causally-masked attention output and attention map. We then use it twice. Minimizing it directly produces the weights: a GPTQ warm start, then straight-through fine-tuning with learnable scales. Differentiating it twice, through Hutchinson probes of the Hessian trace, scores the block for an MCKP allocation. We ask whether the same machinery transfers once allocation extends past attention to the MLP and to the whole model. Findings Attention-only and heterogeneous full-model quantization separate cleanly. On attention-only Mistral-7B, JAB closes most of the gap between uniform GPTQ and full precision at 3 bits, recovering 7777–90%90\% of it across all four calibration/evaluation corpus pairings. Its Hessian-trace criterion tracks the end-to-end oracle closely too, and at a small fraction of the model evaluations. The same advantage does not carry over to the MLP or the full model. On GPT-2’s MLP the oracle allocates bits mainly by matrix role rather than depth, and the Hessian criterion does not pick that structure up. On the full Mistral-7B model a seven-parameter role offset rule reaches lower perplexity than JAB at the same parameter-weighted budget, with no sensitivity estimation at all. With a 3-bit floor on every unit, the same rule reaches 6.9336.933 perplexity at 4.5 bits per parameter, within 4.4%4.4\% of full precision (6.6436.643) at 3.56×3.56× compression, and 7.8757.875 at 3.5 bits (4.57×4.57×). Two further results bear on block-local refinement. Improving a block’s local reconstruction objective does not guarantee better end-to-end quality: in one experiment a 4.6×4.6× reduction in the local objective came with a 32×32× increase in end-to-end perplexity. And on attention-only quantization, fine-tuning moves the weights farther from their floating-point values while making the attention outputs more accurate, which points to functional recovery mattering more than weight-space fidelity. Contributions • Joint objective for reconstruction and allocation. We introduce JAB: a single joint QKV attention loss used for both quantized-weight refinement and sensitivity-based bit allocation. We also separate the joint from the per-matrix treatment of QKV. The two agree in expectation, but the per-matrix form can underestimate the variability of the true joint loss (Section ). • Evidence for role-dependent allocation. MLP and full-model experiments show that matrix role can provide a stronger allocation signal than the curvature criterion JAB uses, which motivates a Hessian-free role-based prior (Sections and ). • Local-to-global failure mode. We isolate a case where improving a block-local reconstruction objective substantially worsens end-to-end perplexity. Local refinement objectives alone do not guarantee global quality (Section ). • Attention-space interpretation of refinement. We introduce the amplification factor ρ=εA/εWρ= ^A/ ^W to separate weight fidelity from functional fidelity. Attention-aware refinement can improve the latter while moving the weights farther from their floating-point values (Section ). A parallel line of PTQ work improves the quantizer itself rather than the allocation, through activation-aware scaling (Lin et al., 2024; Xiao et al., 2023), learned clipping (Shao et al., 2024), or incoherence processing (Tseng et al., 2024). These are orthogonal to bit allocation and compose with it; we hold the quantizer fixed and vary only how bits are assigned. 2 Method Figure 1: Overview of the proposed mixed-precision PTQ pipeline. The method separates the process into two stages. (a) Bit allocation: candidate sensitivity criteria determine how many bits each unit receives under a parameter-weighted budget. (b) Quantization and refinement: GPTQ provides the initial quantized weights, after which the joint attention loss ℒJABL_JAB is minimized through the quantizer using STE++LSQ. 2.1 Layer-wise PTQ Let X∈ℝB×T×dX ^B× T× d be a calibration input to one block and y=xWy=xW a sublayer with W∈ℝdin×doutW ^d_in× d_out; the QKV projection fuses W=[WQ∣WK∣WV]W=[W_Q W_K W_V], and attention is A(X)=softmax(QK⊤/dhead+M)VA(X)=softmax (QK / d_head+M )V with causal mask M. GPTQ minimizes the local surrogate ℒ(W^)=∑x∈X‖xW−xW^‖22,H=2X⊤X,L( W)= _x∈ X\|xW-x W\|_2^2, H=2X X, (1) placing each scalar on a symmetric grid with a per-(row, group) scale s, Q(w)=clip(round(w/s),−qmax,qmax)⋅s,qmax=2b−1−1,Q(w)=clip\! (round(w/s),-q_ ,q_ )· s, q_ =2^b-1\!-\!1, (2) and compensating not-yet-quantized columns with the OBS update (Appendix ). H is accumulated by a forward hook on the model in its current state, so block i’s Hessian already reflects blocks 0,…,i−10,…,i-1 being quantized. The solve runs in float64; in float32 the column loop’s repeated division by the inverse-Hessian diagonal makes perplexity non-monotone in the bit-width. 2.2 The joint attention loss Let A(X),P(X)A(X),P(X) be the attention output and post-softmax map of the unmodified block, and A^(X,w),Q(X,w) A(X;w),Q(X;w) those of a differentiable re-implementation evaluated at the quantized w=[vec(W^Q);vec(W^K);vec(W^V)]w=[vec( W_Q);vec( W_K);vec( W_V)]. The joint loss is ℒJAB(w)=∥A(X)−A^(X;w)∥F2+λKLDKL(P∥Q(w)).L_JAB(w)= \|A(X)- A(X;w) \|_F^2+ _KL\,D_KL (P\,\|\,Q(w) ). (3) Because the two terms are normalized differently, the effective ratio of KL to MSE is 10310^3–10410^4 at λKL=0.1 _KL=0.1: what is optimized is attention-map distillation with an output-MSE regularizer (Appendix ). 2.3 Why one joint objective rather than three Attention is bilinear in (WQ,WK)(W_Q,W_K): the scores depend on the pair only through WQWK⊤W_QW_K , whose perturbation contains a cross term δQδK⊤ _Q _K that no per-matrix model represents. Treating the three separately is the block-diagonal model HsepH_sep of Hℒ=∇w2ℒJABH_L=∇^2_wL_JAB, discarding cross-blocks Γ=Hℒ−Hsep =H_L-H_sep that are structurally non-zero: ℒJABL_JAB is invariant under WQ↦WQRW_Q W_QR, WK↦WKR−⊤W_K W_KR^- per head, so HℒH_L has a null space of dimension at least nheaddhead2n_headd_head^2 (49,15249,152 per GPT-2 block) on which the separable model reports damage where there is none. With a zero-mean rounding residual of variance σ2σ^2, the two agree in expectation but not in spread: [ΔℒJAB]=[ΔℒJABsep],Var[ΔℒJAB]−Var[ΔℒJABsep]=12σ4‖Γ‖F2> 0.E[ _JAB]=E[ _JAB^sep], [ _JAB]-Var[ _JAB^sep]= 12σ^4\| \|_F^2\;>\;0. (4) A per-matrix model is therefore unbiased on average but over-confident about the tail — the wrong profile for allocation, since perplexity is convex and effectively unbounded in the low-bit direction. The argument bears on reconstruction and fine-tuning; for scoring, a trace sees only diagonal blocks and the two coincide (Appendix ). 2.4 Sensitivity and allocation The criterion is the Hessian trace of the same loss w.r.t. the same vector, estimated with Hutchinson’s method (Hutchinson, 1990), Tr(Hℒ)≈1n∑ivi⊤HℒviTr(H_L)≈ 1n _iv_i H_Lv_i with Rademacher viv_i, each product computed by two chained backward passes (Pearlmutter, 1994). Because ℒJABL_JAB is zero at the float point w∗w , the Hessian there reduces to Hℒ(w∗)=2JA⊤JA+λKLJQ⊤ℐ(P)JQ,H_L(w )=2J_A J_A+ _KL\,J_Q I(P)\,J_Q, (5) a squared attention-output Jacobian plus an attention-map Fisher information, not a generic curvature probe. Following HAWQ-V2, the allocator scores each candidate width by (C1)Ωi(b)=Tr(Hi)‖Qb(Wi)−Wi‖F2,(C2)Ωiprop(b)=(L−i)Ωi(b),(C1)\;\; _i(b)=Tr(H_i)\, \|Q_b(W_i)-W_i \|_F^2, (C2)\;\; ^prop_i(b)=(L-i)\, _i(b), (6) where C2 re-weights by the number of layers an error propagates through. As a bound, the oracle (C3) measures each (i,b)(i,b) pair’s end-to-end logit KL by quantizing only layer i (≈73≈ 73 model loads). Each layer then chooses bi∈ℬ=2,3,4,8,16b_i =\2,3,4,8,16\ by min∑iΩi(bi)s.t.∑ic(bi)≤B, _i _i(b_i) .t. _ic(b_i)≤ B, (7) solved greedily or on the pooled Pareto frontier (Appendix ). 2.5 Joint fine-tuning through the quantizer GPTQ solves equation , not equation . The final stage takes GPTQ’s solution as a warm start and descends ℒJABL_JAB through the quantizer, using the straight-through estimator on weights (Bengio et al., 2013) and the LSQ gradient on the per-(row, group) scales (Esser et al., 2020). Because GPTQ’s output sits exactly on grid points, learning rates are set in units of the mean grid step s¯ s (ηw=αws¯ _w= _w s, ηs=αss¯ _s= _s s) over 200 Adam steps; teacher and student see the same partially-quantized input; a held-out batch gates every write-back; and we log the flip rate, the fraction of weights whose grid position changed, since a stage that changes nothing and one that changes everything can leave perplexity equally unmoved (Appendix ). 2.6 Metrics We report perplexity and next-token accuracy under a sliding window, the weight-space error εW=‖W−W^‖F/‖W‖F ^W=\|W- W\|_F/\|W\|_F, the attention-space error εA ^A, and the amplification factor ρ=εA¯/εW¯ρ= ^A/ ^W, which separates weight fidelity from functional fidelity (Appendix ). 3 Experimental Setup We study GPT-2 small (Radford et al., 2019) (124M parameters, 12 blocks) and Mistral-7B-v0.1 (Jiang et al., 2023) (32 layers, grouped-query attention, rotary embeddings). Both use group size 128, activation ordering, λKL=0.1 _KL=0.1 unless ablated (except the MLP study, where it is 0 by construction), and 10 Hutchinson probes per layer. The attention-only study quantizes the QKV projection: on GPT-2 it sweeps four criteria over seven budgets with and without fine-tuning, calibrating on WikiText-2 (Merity et al., 2017) or C4 (Raffel et al., 2020); on Mistral-7B it covers all four calibrate→ pairings. The MLP-only study quantizes GPT-2’s c_fc and mlp.c_proj. The full-model study quantizes all seven linear modules of Mistral-7B. We evaluate GPT-2 on the full WikiText-2 test split and Mistral-7B on 32 windows of 512 tokens, so Mistral perplexities are exact within a study but not comparable to published full-split figures. All results are single-seed. Complete configurations are in Appendix . 4 Attention-Only Quantization Table 1: Attention-only Mistral-7B at 3 bits, all four calibrate→ pairings (W==WikiText-2), with the share of the uniform-versus-fp16 gap closed. Full sweep: Appendix . Pairing fp16 Uniform Adaptive Gap closed Δ Acc. W → W 4.818 8.738 5.228 89.5%89.5\% +0.031+0.031 C4 → C4 7.492 13.506 8.898 76.6%76.6\% +0.014+0.014 W → C4 7.492 13.575 8.622 81.4%81.4\% +0.018+0.018 C4 → W 4.818 8.875 5.428 85.0%85.0\% +0.028+0.028 At 7B scale, allocation carries the result On Mistral-7B, adaptive allocation dominates uniform GPTQ at 3 bits in all four corpus pairings, recovering 7777–90%90\% of the uniform-versus-fp16 gap and 22–33 accuracy points (Table ). By 4 bits every method is within 11 -2%2\% of fp16, and refinement is second-order: at 3 bits allocation is worth 33–55 perplexity points and fine-tuning at most 0.170.17 in either direction. The allocation transfers across corpus mismatch, and the GPT-2 trace ranking is nearly calibration-invariant (Appendix ). A cheap criterion matches an expensive oracle Across the four GPT-2 budgets ≥3.5≥ 3.5 bits, the Hutchinson criterion and the ≈73≈ 73-load oracle differ by 0.170.17 perplexity on average (Table ). On Mistral, second-order scoring beats a first-order Fisher proxy at 3 bits (5.2685.268 vs. 5.3025.302), while greedy and Pareto solvers are indistinguishable: the score matters, the solver does not. Mixed precision is not uniformly safe, and the objective says why On GPT-2 at exactly 3.03.0 bits, adaptive allocation reaches 86.3686.36 against uniform’s 38.3538.35. This follows from the objective. At an average of exactly 3 bits, an assignment with no block below 3 must place every block at 3, which is the uniform solution; the adaptive assignment differs from uniform, so at least one block sits at 2 bits, and one is enough to dominate the total (uniform 2-bit scores 5137.65137.6). Minimizing a sum of Ωi(bi) _i(b_i), the allocator cannot see that perplexity is convex and unbounded below. Depth re-weighting C2 halves the damage (43.9343.93) at no cost. Mixed precision earns its keep at fractional budgets instead, beating a geometric interpolation between neighboring uniform solutions by 2.2×2.2× at 2.5 bits and 1.11×1.11× at 3.5 (Appendix ). Refinement recovers attention behavior, not weights On GPT-2, joint fine-tuning improves 18 of 21 configurations, and does so by moving the weights away from float: at 4 bits εW ^W rises 19%19\% (0.174→0.2080.174→ 0.208) while εA ^A falls 33%33\% (0.238→0.1600.238→ 0.160), and both perplexity and accuracy improve. Every GPTQ-only arm has ρ>1ρ>1 and every fine-tuned arm ρ<1ρ<1, with the ordering ρGPTQ>ρMSE-only>ρMSE+KL _GPTQ> _MSE-only> _MSE+KL holding at every budget on both corpora (Appendix ). The attention map is what must be preserved: the KL term supplies 88%88\% of the gain, and MSE-only refinement lowers εA ^A yet still degrades perplexity. 5 Beyond Attention: the MLP The JAB machinery is not intrinsically tied to the softmax. To test whether it generalizes beyond attention, we apply the same allocation-and-refinement pipeline to the GPT-2 MLP, quantizing mlp.c_fc and mlp.c_proj with attention kept in fp32; the MLP holds 66.6%66.6\% of the non-embedding weights against 25.0%25.0\% for QKV. Two structural differences follow. The two matrices are sequential rather than a jointly coupled projection, so their Hessians are collected in execution order; and there is no attention map to distill, so the KL component is absent (λKL=0 _KL=0) and ℒJABL_JAB reduces to output MSE. The experiment therefore tests which parts of JAB transfer once its attention-specific signal is removed. Table 2: GPT-2 MLP, WikiText-2 perplexity (fp32 control 24.35724.357) at the two budgets with a uniform baseline. Full sweep: Appendix , Table . B (bits) Uniform JAB (C1) Oracle (C3) 3.0 53.71 120.70 46.15 4.0 26.23 27.46 26.23 The Hessian criterion does not improve over uniform allocation, whereas the oracle does At the two budgets for which a uniform baseline is available, the Hessian criterion performs worse than uniform quantization: 120.70120.70 vs. 53.7153.71 perplexity at 3 bits and 27.4627.46 vs. 26.2326.23 at 4 bits (Table ). This failure differs qualitatively from the QKV collapse, since the 4-bit allocation contains no unit below 3 bits. The oracle, in contrast, improves on uniform at 3 bits, reaching 46.1546.15, a 14%14\% reduction. Thus, useful allocation structure exists in the MLP, but the Hessian criterion fails to identify it. The oracle allocates by role, not depth At every fractional budget the oracle returns the same flat allocation in every block, split only by matrix role: mlp.c_fc at b+1b+1 and mlp.c_proj at b−1b-1 (Appendix , Table ). The criterion cannot find that structure, by construction. One trace over the concatenated block vector is identical for both units of a block, so the only thing that can tell them apart is the perturbation factor of Eq. (). What the criterion sees instead is a depth gradient rising 4.5×4.5×, unlike the QKV trace on the same network, which peaks at block 3 with a 41×41× spread. Sensitivity profiles are a property of the sublayer, not of the architecture alone. 6 Full-Model Quantization We now put QKV and MLP under one shared budget, first on GPT-2 small, then on every linear module of Mistral-7B. 6.1 GPT-2: QKV and MLP Together The fused QKV projection and both MLP matrices of every GPT-2 block now share one budget: 36 units, 91.7%91.7\% of block weights, giving 2.21×2.21× end-to-end compression at 4 bits against 1.66×1.66× for the MLP alone. Four allocators are compared at five budgets, calibrated on C4, GPTQ-only, evaluated on WikiText-2 (fp32 control 24.35724.357) and a disjoint C4 slice with the same ranking (Appendix ). Sensitivity tables are normalized per block type first, since raw JAB scores are not commensurate across types. The shared block trace was also replaced by an unbiased per-unit trace in one of the sweeps, but no improvement was obtained. Table 3: GPT-2 with QKV and MLP quantized together (C4 calibration, GPTQ-only): WikiText-2 perplexity (fp32 control 24.35724.357) and accuracy at the two budgets that separate the allocators. Full table: Appendix ; in-domain C4, same ranking: Appendix . B=3.5B=3.5 B=4.5B=4.5 Allocator PPL Acc PPL Acc JAB (C1) 75.009 0.296 29.041 0.394 JAB++prop (C2) 67.337 0.307 28.694 0.397 Type-offset 64.896 0.310 28.211 0.395 Role beats curvature once both sublayers share the budget Type-offset needs no scoring pass at all and still leads at the fractional budgets: 64.89664.896 at 3.53.5 bits and 28.21128.211 at 4.54.5, against JAB++prop’s 67.33767.337 and 28.69428.694 (Table ). JAB’s attention-defined sensitivity over-invests in QKV and under-funds the MLP, which now dominates the parameter count; only near saturation, at 6 bits, does uniform win. Two observations carry over to Mistral (Appendix ): εA ^A stops tracking perplexity once the MLP shares the budget, and the joint STE stage raises perplexity at every budget, by 16.35×16.35× at 3 bits. 6.2 Mistral-7B: All Seven Modules Together We now quantize every linear module of all 32 Mistral-7B layers under one budget: 224 units, 6.986.98B parameters, 96.4%96.4\% of the model. A parameter-weighted budget The sublayer studies had units of equal size, so per-unit and per-parameter costs coincided. Here grouped-query attention makes k_proj and v_proj (4.24.2M) a quarter of q_proj and o_proj (16.816.8M), while the three SwiGLU matrices hold 58.758.7M each — a 14×14× span, with the MLP carrying 80.8%80.8\% of the weights. A per-unit cost would price those equally, so we define the budget per parameter, B=∑iPibi∑iPi,c(bi)=Pibi,B\;=\; _iP_i\,b_i _iP_i, c(b_i)=P_i\,b_i, (8) with PiP_i the parameter count of unit i. On the quantized weights this makes storage 16/B16/B times smaller than fp16: 13.9613.96 GB →3.93→ 3.93 GB (3.56×3.56×) at B=4.5B=4.5. Type-offset: a Hessian-free role prior Motivated by the MLP oracle’s role rule, Type-offset gives each unit kind κ a fixed integer offset Δκ _κ and assigns bi=nearestℬ(t⋆+Δκ(i)),t⋆=argmint|∑ic(bi)−B∑iPi|,b_i\;=\;nearest_B\! (t + _κ(i) ), t = _t |\, _ic(b_i)-B _iP_i |, (9) followed by single-unit moves meeting the budget exactly: seven integers, no scoring pass. We set Δ=0 =0 for q,k,v,oq,k,v,o, +1+1 for gate and up, −1-1 for down, transposed from the GPT-2 oracle rather than tuned, and also test JAB-norm, which divides each trace by diag(Hi)¯ diag(H_i). Table 4: Mistral-7B, all seven modules quantized: WikiText-2 perplexity (fp16 control 6.6436.643) and compression of the quantized weights. Budgets B are bits per parameter (Eq. ()). Left: criteria at fractional budgets; Type-offset needs no scoring pass. Right: uniform quantization, defined only at integer budgets, and the adaptive JAB arm. Best per budget in bold. B (bits/param) 2.5 3.5 4.5 Compression 6.40×6.40× 4.57×4.57× 3.56×3.56× Type-offset 6624.3 32.33 6.970 JAB (C1) 1799.6 8.570 7.250 JAB-norm 648.0 109.6 10.77 Arm B Compr. PPL Uniform 2.0 8.00×8.00× 64779.9 Uniform 3.0 5.33×5.33× 9.126 Uniform 4.0 4.00×4.00× 6.998 JAB adaptive 4.3 3.72×3.72× 7.373 6.3 Layer Role Outperforms Curvature The role prior wins at a matched budget At B=4.5B=4.5 bits per parameter, Type-offset reaches 6.9706.970 perplexity against 7.2507.250 for JAB and 10.77010.770 for JAB-norm (Table ). All three realize the same budget to three decimals, 4.5004.500 bits per parameter, but Type-offset computes no Hessian and runs no Hutchinson probes. The curvature information JAB pays for does not buy a better allocation than seven integers. The MLP experiment showed the same thing: there too the end-to-end oracle preferred a flat allocation set by matrix role rather than depth. That it recurs in Mistral-7B, with grouped-query attention, SwiGLU and 58×58× the parameters, is what makes us read it as a property of heterogeneous transformers rather than of one architecture. Why the JAB criterion misses the role structure JAB computes one Hessian trace for the concatenated parameter vector of each decoder block, so all seven linear modules in a block receive the same trace value. The criterion can separate them only through the perturbation term in Eq. (). What it does vary over is depth: the trace rises 381×381× from block 0 to block 31. The oracle’s allocation has no depth structure at all, only a role split, so JAB spends bits along the wrong axis. It distinguishes where a layer sits rather than lemphwhich matrix it is. A per-unit trace would remove this limitation (Appendix ). Section removes this limitation and finds the ranking unchanged. Type-offset is useful, but its offsets are not universally transferable Type-offset degrades sharply below 4.54.5 bits: at B=3.5B=3.5 the fixed Δdown=−1 _down=-1 offset assigns 2 bits to 3030 of the 3232 downprojdown_proj matrices, giving 32.3332.33 perplexity. The GPT-2 MLP experiment pointed to this direction, but Mistral’s downprojdown_proj receives heavier-tailed inputs, so the same offset need not apply; the floor of Section removes the failure entirely. Each candidate offset costs one evaluation, so we report the 4.54.5-bit result as a floor for the role-prior family, not a tuned choice. 6.4 Block-Local Objectives Can Diverge from End-to-End Quality Table 5: Joint refinement of the adaptive B=4.3B=4.3 allocation. Block 0 is the only block whose weights change, so refining blocks 0–3 reproduces the full-depth result exactly. Full table: Appendix . Refinement Blocks refined Blocks changed PPL None (GPTQ only) – – 7.373 Joint FT 0–31 1 (block 0) 239.52 Joint FT 0–3 1 (block 0) 239.52 A locally improved block can substantially degrade the complete model Joint refinement of the adaptive B=4.3B=4.3 allocation reduces the block-local objective of block 0 from 1.851.85 to 0.41×10−30.41× 10^-3, a 4.6×4.6× improvement. End-to-end perplexity increases from 7.3737.373 for GPTQ-only to 239.52239.52 after refinement. Refining only blocks 0–3 gives exactly the same perplexity and the same error metrics (Table ), and in both runs block 0 is the only block whose quantized weights move at all; the other 31 return to their GPTQ grid points. This is not a failure of the local optimization. The refinement reduces the objective it was given and passes the held-out checkpoint gate. One refined block still raises perplexity 32.5×32.5×. Why block 0 is especially vulnerable Two properties coincide in this block. The adaptive allocator assigns 2 bits to six of its seven linear modules, giving a weight-space error of εW=1.015 ^W=1.015: the quantized block is already farther from its floating-point weights than the zero matrix is, in relative Frobenius error. And the output of block 0 passes through all 31 downstream decoder blocks. Refinement is therefore working on a severely quantized representation at the point of maximum downstream leverage. We read the failure as follows. Minimizing a block-local objective on a small calibration set can move the weights toward calibration-specific behavior while losing what the downstream network needs. The held-out gate cannot catch this, because it evaluates the same local objective rather than the end-to-end metric. Granularity matters too. Under uniform 4-bit quantization the same refinement stage changes three blocks and raises perplexity by 1.6×1.6× rather than 32.5×32.5×, so in these two runs the mismatch is far more severe when low-bit assignments are concentrated. Implication for refinement The problem is not specific to ℒJABL_JAB. Any refinement procedure that optimizes a block-local target and validates against that same target gives no guarantee of better end-to-end quality. At minimum, refinement should track a global metric alongside a per-block flip-rate diagnostic (Appendix , Eq. ()), which is what identifies a locally accepted update with outsized downstream effect. The same block also generalizes the 2-bit pathology we prove for GPT-2 in Section . The additive objective equation cannot see it, and it appears whenever the allocator is free to concentrate low widths on a single block. 6.5 A 3-Bit Floor with Per-Unit Curvature Each diagnostic suggests a change: the additive objective cannot see the low-bit cliff, so we impose the floor bi≥3b_i≥ 3; and the shared block trace cannot rank units within a block, so we replace it with an unbiased per-unit trace at no extra cost, partitioning v⊤Hv \!Hv over each unit’s coordinate range (Appendix ). Table 6: Mistral-7B, all seven modules, floor bi≥3b_i≥ 3 and per-unit traces (GPTQ-only; fp16 6.6436.643; uniform 4-bit 6.9386.938; adaptive B=4.3B=4.3 7.2117.211). Compression is on the quantized weights. Not comparable to Table (64 vs 128 Hessian batches); see Appendix . B Compr. JAB (C1) JAB-norm Type-offset 3.0 5.33×5.33× 9.713 9.713 9.713 3.5 4.57×4.57× 7.514 8.215 7.875 4.5 3.56×3.56× 7.158 7.592 6.933 The floor removes the catastrophic regime No arm now exceeds 9.729.72, against 32.3332.33 and 109.6109.6 at B=3.5B=3.5 without it; at B=3.0B=3.0 the floor saturates the budget, so all four criteria return the uniform assignment and the identical 9.7139.713. The usable operating points are Type-offset at B=3.5B=3.5 (7.8757.875, 4.57×4.57×) and B=4.5B=4.5 (6.9336.933, 3.56×3.56×): 1414 GB reduced to 3.933.93 GB for 4.4%4.4\% higher perplexity than fp16. Per-unit curvature does not change the ranking The traces now discriminate strongly within a block, yet Type-offset still leads at matched B=4.5B=4.5, 6.9336.933 against 7.1587.158: q and k take 77%77\% of the trace mass while holding 19.2%19.2\% of the weights, since ℒJABL_JAB is defined on the attention output and map. The limitation is the objective the curvature comes from, not the shared trace this experiment removes (Appendix ). 7 Conclusion Using one loss for both reconstruction and allocation lets us analyze the two stages of mixed-precision PTQ in a single frame, and on attention it works: adaptive allocation recovers most of the uniform-to-fp16 gap at 7B. Beyond attention it does not transfer. Where units are heterogeneous, matrix role carries more signal than the curvature we computed, and per-unit resolution does not change that, since the curvature comes from an attention objective. Two lessons hold independently of JAB. An additive objective cannot see the low-bit cliff, and a floor bi≥3b_i≥ 3 repairs it at no cost. And a certified local improvement is not evidence of a global one (Appendix ). Both regimes matter, but not equally. Compression is the reason to quantize, and the ratio moves fastest at the low end: 16/316/3 is 5.3×5.3× against 3.56×3.56× at 4.54.5 bits. Large models are the ones forced to live there, and 3.53.5 bits is where the curvature criterion is ahead of the role prior (7.517.51 against 7.877.87). The regime JAB serves is therefore the one that grows in importance with model size, although we test only to 7B. AI use statement In this work, we used generative AI tools (a large language model assistant) for editing and condensing the manuscript text, drafting and checking LaTeX, writing experiment, analysis and plotting code, and cross-checking the numbers reported in the text against our result files. We have not used generative AI tools to produce any experimental result: every number in the paper is computed by our code from the models’ outputs, and every AI-drafted script was run and checked by the authors. We have reviewed all AI-assisted work and take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. Reproducibility Statement Code, configurations and seeds: anonymous.4open.science/r/…-6080. All runs use seed 42 and are single-seed; full hyperparameters are in Appendix . References Bengio et al. (2013) Y. Bengio, N. Léonard, and A. Courville Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. Cited by: §B.6, §2.5. Cao and Aref (2025) D. Cao and S. Aref Enhancing ultra-low-bit quantization of large language models through saliency-aware partial retraining. In Modeling Decisions for Artificial Intelligence: 22nd International Conference, p. 354–383. External Links: Document Cited by: §1. Dantzig (1957) G. B. Dantzig Discrete-variable extremum problems. Operations Research 5 (2), p. 266–288. Cited by: §B.5. Dong et al. (2020) Z. Dong, Z. Yao, D. Arfeen, A. Gholami, M. W. Mahoney, and K. Keutzer HAWQ-V2: hessian aware trace-weighted quantization of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Table 7, §1. Esser et al. (2020) S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha Learned step size quantization. In International Conference on Learning Representations (ICLR), Cited by: §B.6, §2.5. Frantar et al. (2023) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR), Cited by: Table 7, §1. Guan et al. (2024) Z. Guan, H. Huang, Y. Su, H. Huang, N. Wong, and H. Yu APTQ: attention-aware post-training mixed-precision quantization for large language models. In ACM/IEEE Design Automation Conference (DAC), Cited by: Table 7, §1. Hassibi and Stork (1990) B. Hassibi and D. G. Stork Second order derivatives for network pruning: optimal brain surgeon. In Advances in Neural Information Processing Systems (NeurIPS), p. 164–171. Cited by: §1. Hutchinson (1990) M. F. Hutchinson A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics – Simulation and Computation 19 (2), p. 433–450. Cited by: §2.4. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed Mistral 7b. arXiv preprint arXiv:2310.06825. Cited by: Appendix C, §3. Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: §B.6. Lee et al. (2024) C. Lee, J. Jin, T. Kim, H. Kim, and E. Park OWQ: outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence, External Links: Document Cited by: §1. Lee et al. (2026) D. Lee, S. Chu, and H. O. Song Q-strata: hierarchical bit allocation for mixed-precision quantization of mixture-of-experts llms. External Links: 2608.30564, Link Cited by: §1. Li et al. (2021) Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu BRECQ: pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), Cited by: §1. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of Machine Learning and Systems (MLSys), Cited by: §1. Liu et al. (2025) J. Liu, T. Koike-Akino, Y. Wang, H. Mansour, and M. Brand AWP: activation-aware weight pruning and quantization with projected gradient descent. External Links: 2506.10205, Link Cited by: §1. Merity et al. (2017) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), Cited by: Appendix C, §3. Michel et al. (2019) P. Michel, O. Levy, and G. Neubig Are sixteen heads really better than one?. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Appendix D. Pearlmutter (1994) B. A. Pearlmutter Fast exact multiplication by the Hessian. Neural Computation 6 (1), p. 147–160. Cited by: §2.4. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. Technical report OpenAI. Cited by: Appendix C, §3. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. Cited by: Appendix C, §3. Rakka et al. (2026) M. Rakka, M. Fournarakis, O. Krestinskaya, J. Bazzi, K. Salama, F. Kurdahi, A. Eltawil, and M. Fouda Mixed-precision quantization for language models: techniques and prospects. In ACM Comput. Surv., p. 0360–0300. Cited by: §1. Shao et al. (2024) W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo OmniQuant: omnidirectionally calibrated quantization for large language models. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §1. Sinha and Zoltners (1979) P. Sinha and A. A. Zoltners The multiple-choice knapsack problem. Operations Research 27 (3), p. 503–515. Cited by: §1. Tseng et al. (2024) A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa QuIP#: even better LLM quantization with hadamard incoherence and lattice codebooks. In Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235. External Links: Link Cited by: §1. Wang et al. (2019) K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han HAQ: hardware-aware automated quantization with mixed precision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8612–8620. Cited by: §1. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), Cited by: §1. Appendix A Positioning Against Prior Work Table contrasts the reconstruction objective each method minimizes with the importance criterion it allocates by. JAB is the only entry for which the two are the same functional; the oracle bounds what any criterion can achieve, and Type-offset (Section ) sidesteps the question by using no criterion at all. Table 7: Relationship between reconstruction objective and importance criterion. JAB closes the gap for attention; the type-offset baseline sidesteps it entirely. Method Recon. objective Importance criterion Same? GPTQ (Frantar et al., 2023) ‖XW−XW^‖2\|XW-X W\|^2/layer N/A (uniform bits) – HAWQ-V2 (Dong et al., 2020) ‖XW−XW^‖2\|XW-X W\|^2/layer Tr(H)Tr(H) of task loss No APTQ (Guan et al., 2024) Weight recon. ++ attn. grad. Hessian trace, weight-space Related JAB-Hessian (ours) Joint QKV attn. loss ℒJABL_JAB Tr(∇2ℒJAB)Tr(∇^2L_JAB) Identical Fisher-coupling (ours) Joint QKV attn. loss ℒJABL_JAB ‖∇ℒJAB‖2\| _JAB\|^2 at 2-bit probe First-order Oracle (ours) Joint QKV attn. loss ℒJABL_JAB End-to-end logit KL Bound Type-offset (ours) Layer-local GPTQ Role-based offset table Hessian-free Appendix B Method Details B.1 GPTQ core OBS update OBS gives the closed form for rounding weight wqw_q while optimally compensating the not-yet-quantized weights F: δF=−wq−Q(wq)[H−1]qqHF,q−1, _F=- w_q-Q(w_q)[H^-1]_q\,H^-1_F,q, (10) i.e. for column i, err=(wcol−qcol)/H−1[i,i]err=(w_col-q_col)/H^-1[i,i] and W[:,i+1:]-=outer(err,H−1[i,i+1:])W[:,i+1:] -=outer(err,H^-1[i,i+1:]). Both standard refinements are used: group-wise scaling and activation ordering (diag(H)diag(H)-descending column permutation). Numerical precision is load-bearing The column loop divides by an inverse-Hessian diagonal dind_in times per row, so fp32 rounding compounds and produces non-monotone perplexity across bit-widths. GPT-2 can afford a float64 solve; Mistral cannot (float64 is 32×32× slower on a T4 over a 40964096-step loop), so the streaming build accumulates H in float64 but solves in float32 — a caveat carried into Section . B.2 Effective weight of the KL term The nominal λKL _KL is not the effective weight ℒMSEL_MSE takes an element-wise mean over B⋅T⋅dB\!·\!T\!·\!d entries, whereas ℒKLL_KL uses batchmean, dividing only by B while summing over B⋅nhead⋅T2B\!·\!n_head\!·\!T^2. The measured ratio λKLℒKL/ℒMSE _KLL_KL/L_MSE is 10310^3–10410^4 at λKL=0.1 _KL=0.1, so what is optimized is attention-map distillation with an output-MSE regularizer. We keep the implemented convention and quantify it by ablation (Section ), since it carries most of the benefit. B.3 Why one joint objective rather than three: full argument Section states the result; this subsection gives the argument. It cannot be worse The grid is a product set =Q×K×VG=G_Q×G_K×G_V — each matrix rounds on its own scales, and no choice for one constrains the others — so any separately obtained triple is feasible for the joint problem and minw∈ℒJAB(w)≤ℒJAB(W^Qsep,W^Ksep,W^Vsep). _w L_JAB(w)\;≤\;L_JAB ( W^sep_Q, W^sep_K, W^sep_V ). (11) The content lies in when equation is strict — and in the fact that a separate pipeline does not minimize ℒJABL_JAB at all, but a sum of per-matrix surrogates, so two gaps stack. What separability discards Partition HℒH_L into 3×33× 3 blocks indexed by Q,K,VQ,K,V. Separate treatment is exactly the block-diagonal model Hsep=blockdiag(HQQ,HKK,HVV)H_sep=blockdiag(H_Q,H_K,H_V), and the error it commits at second order is exactly the cross-blocks, ΔℒJAB−ΔℒJABsep=δQ⊤HQKδK+δQ⊤HQVδV+δK⊤HKVδV. _JAB- _JAB^sep= _Q H_QK _K+ _Q H_QV _V+ _K H_KV _V. (12) These do not vanish, because attention is bilinear in (WQ,WK)(W_Q,W_K): the scores depend on that pair only through M=WQWK⊤M=W_QW_K , and W^QW^K⊤=M+δQWK⊤+WQδK⊤+δQδK⊤ W_Q W_K =M+ _QW_K +W_Q _K + _Q _K , whose last term no per-matrix model can represent. In the Gauss–Newton form equation , HQK=2JAQ⊤JAK+λKLJQQ⊤ℐ(P)JQKH_QK=2J_A^Q J_A^K+ _KLJ_Q^Q I(P)J_Q^K — a Gram cross-block that vanishes only if perturbing WQW_Q and WKW_K moved the attention output in orthogonal directions. They do not: both move it through M. The discarded term is unbounded in relative terms Take one head with dhead=1d_head=1, so WQ,WKW_Q,W_K reduce to vectors q,kq,k and the block depends on them only through qk⊤qk ; measure damage by ‖Δ(qk⊤)‖F2\| (qk )\|_F^2. With δq=εq _q= q and δk=−ε1+εk _k=- 1+ k the product is reproduced exactly: true damage 00, separable estimate Θ(ε2)‖qk⊤‖F2 ( ^2)\|qk \|_F^2 — an infinite relative error at any ε . With δq=εq _q= q, δk=εk _k= k the true damage is ≈4ε2‖qk⊤‖F2≈ 4 ^2\|qk \|_F^2 against a separable estimate of 2ε2‖qk⊤‖F22 ^2\|qk \|_F^2 — an underestimate by exactly 2×2×. Identical per-matrix perturbation norms; true damage anywhere from zero to twice the separable prediction, decided entirely by relative alignment, which is what a per-matrix model cannot see. This is not pathological: ℒJABL_JAB is exactly invariant under WQ↦WQRW_Q W_QR, WK↦WKR−⊤W_K W_KR^- per head, so HℒH_L is singular with a null space of dimension at least nheaddhead2=49,152n_headd_head^2=49,152 per GPT-2 block. Every such direction has nonzero Q and K components, so on it δ⊤Hℒδ=0δ H_Lδ=0 while δ⊤Hsepδ>0δ H_sepδ>0. Nor is the separable model conservative: if HQK≠0H_QK≠ 0, take u,vu,v with u⊤HQKv=c≠0u H_QKv=c≠ 0; then (u,v,0)(u,v,0) and (u,−v,0)(u,-v,0) have identical block-diagonal forms while their true forms differ by ±2c± 2c, so it over- and under-estimates in equal measure. Unbiased in the mean, over-confident in the tail Model the rounding residual as zero-mean and uncorrelated with variance σ2=s2/12σ^2=s^2/12. The cross-blocks contribute no diagonal, so Tr(Hsep)=Tr(Hℒ)Tr(H_sep)=Tr(H_L) and [ΔℒJAB]=σ22Tr(Hℒ)=[ΔℒJABsep]E[ _JAB]= σ^22Tr(H_L)=E[ _JAB^sep]: the two agree exactly in expectation. Their variances do not. With Γ=Hℒ−Hsep =H_L-H_sep, the shared diagonal gives ‖Hℒ‖F2=‖Hsep‖F2+‖Γ‖F2\|H_L\|_F^2=\|H_sep\|_F^2+\| \|_F^2, so Var[ΔℒJAB]−Var[ΔℒJABsep]=12σ4‖Γ‖F2> 0.Var[ _JAB]-Var[ _JAB^sep]= 12σ^4\| \|_F^2\;>\;0. A per-matrix model is thus unbiased on average and systematically over-confident about the tail, by exactly the squared norm of the cross-blocks — the worst profile here, since Section shows perplexity is convex and effectively unbounded in the low-bit direction, so what an allocator must avoid is the tail, not the mean. Scope of the argument Two limits matter. First, this bears on the reconstruction and fine-tuning half only: a trace sees only diagonal entries, so Tr(Hℒ)=Tr(HQQ)+Tr(HKK)+Tr(HVV)Tr(H_L)=Tr(H_Q)+Tr(H_K)+Tr(H_V) identically, and for the scoring half joint and separate coincide exactly — the criterion’s advantage over HAWQ-V2 lies in which loss the diagonal blocks come from, not in jointness. Second, the argument concerns ℒJABL_JAB, not perplexity, and ‖Γ‖F/‖Hℒ‖F\| \|_F/\|H_L\|_F was never measured (L5). The decisive test is cheap, and equation predicts its signature: equal means, heavier tail. B.4 Fisher-coupling Fisher-coupling (first-order alternative) Since ∇wℒJAB(w∗)=0 _wL_JAB(w )=0 identically, a gradient-based score must be evaluated off the float point: Fisher-coupling scores layer i by the squared gradient norm of ℒJABL_JAB at a coarse round-to-nearest 2-bit probe. It is cheaper than a double-backward HVP but is not a decomposition of HℒH_L — a first-order proxy taken near the low-bit operating point. Section compares the two. B.5 Allocators MCKP is NP-hard and exhaustive search (5125^12 here, 5325^32 on Mistral) is intractable, so two heuristics are compared, both consuming the same Ωi(b) _i(b) table. Greedy: start every layer at the highest width, repeatedly apply the downgrade minimizing ΔΩ/Δc / c until the budget is met, then spend leftovers on the best upgrade — the classical fractional-knapsack walk (Dantzig, 1957), optimal only for the continuous relaxation. MCKP/Pareto: restrict each layer’s (b,Ω)(b, ) points to their Pareto-efficient frontier first, then walk down the pooled frontier by best ratio, matching HAWQ-V2’s formulation more closely. B.6 Joint STE fine-tuning: the five design decisions GPTQ solves equation , not equation . The final stage takes GPTQ’s solution as a warm start and descends ℒJABL_JAB through the quantizer. Five decisions distinguish it from a naive STE loop. (1) STE on weights, LSQ on scales. Two quantities need two gradient rules: the fake-quantized weight w w of equation , and the per-group scale s that defines the grid. The weight uses the straight-through estimator (Bengio et al., 2013) — gradients pass through the rounding step unchanged — while the scale is a learnable parameter that can widen or narrow the grid, updated with the LSQ gradient (Esser et al., 2020). With v=w/sv=w/s and q=clip(round(v),±qmax)q=clip(round(v),± q_ ), ∂w^∂w=1,∂w^∂s=q−v,|v|≤qmax,sign(v)qmax,else, ∂ w∂ w=1, ∂ w∂ s= casesq-v,&|v|≤ q_ ,\\ sign(v)\,q_ ,&else, cases (13) the scale gradient further scaled by (nwqmax)−1/2(n_wq_ )^-1/2 as in LSQ. One scale is kept per (output row, group) rather than per weight: a per-weight scale would let qisiq_is_i take any real value, silently restoring full precision and defeating the bit-width constraint. (2) Grid-relative learning rates. GPTQ’s output sits exactly on grid points, so a latent weight must travel ≥s/2≥ s/2 before round(⋅)round(·) changes. Absolute rates are therefore meaningless; we set ηw=αws¯,ηs=αss¯, _w= _w s, _s= _s s, (14) with s¯ s the mean grid step, αw=0.05 _w=0.05, αs=0.02 _s=0.02, cosine-annealed over 200 Adam (Kingma and Ba, 2015) steps. (3) Teacher and student share the same input. Targets A(X),P(X)A(X),P(X) use the block’s float weights on the same X the student sees, taken from the partially-quantized model — GPTQ’s own target=WfloatXquanttarget=W_floatX_quant convention. A parallel all-float trajectory would fold upstream error, which block i cannot fix, into its residual. (4) Held-out checkpoint selection with a commit gate. One calibration batch is held out of the training pool; every 10 steps the current weights are scored on it, the best checkpoint retained, and written back only if it beats the GPTQ-only start. (5) Flip-rate diagnostic. Since a stage that changes nothing and one that changes everything can both leave perplexity unmoved, we log flipi=1d′|j:round(wjGPTQ/sj)≠round(wjFT/sj)|.flip_i= 1d |\j:round(w^GPTQ_j/s_j) (w^FT_j/s_j)\ |. (15) Why these decisions matter An earlier build used a fixed absolute rate and recorded a flip rate of 0.0000.000 on eleven of twelve blocks: each step moved weights less than a tenth of the way to the nearest grid boundary, so no quantized value could change regardless of gradient quality. The grid-relative rate of equation over 200 steps raises the flip rate to 0.120.12–0.180.18. When fine-tuning through a quantizer warm-started from a grid-exact solution, the learning rate must be expressed in units of the grid, and the flip rate must be logged. B.7 Evaluation metrics in full All four are computed in the same quantize-and-evaluate pass, after quantization and any fine-tuning. M1 — Perplexity. PPL=exp(1N∑iNLLi)PPL= ( 1N _iNLL_i ) under the sliding-window protocol, only the non-overlapping target span contributing, so no token is scored twice and every token carries ≥ 512 tokens of context. M2 — Next-token top-1 accuracy. Acc=1N∑i[argmaxvpθ(v|x<i)=xi]Acc= 1N _i1[ _vp_θ(v|x_<i)=x_i], on the identical windows, mask and forward pass as M1 — a decision-level metric, insensitive to how mass is spread over the rest of the vocabulary. M3 — Weight-space quantization error. εiW=‖Wi−W^i‖F/‖Wi‖F ^W_i=\|W_i- W_i\|_F/\|W_i\|_F, block-averaged: pure weight space, no data, and up to normalization exactly what GPTQ’s objective equation and the perturbation term of equation measure. M4 — Attention-space reconstruction error. εiA ^A_i, the relative Frobenius error of the merged pre-c_proj attention output, teacher and student evaluated on the same X from the live partially-quantized model, averaged over |ℰ|=8|E|=8 fixed 512-token spans. Derived — amplification factor. ρ=εA¯/εW¯ρ= ^A/ ^W. M3 and M4 are relative errors of the same form on either side of the attention operator, so ρ>1ρ>1 means causal softmax attention amplifies the relative weight perturbation it is handed and ρ<1ρ<1 that it attenuates it. Reporting both makes the report’s premise falsifiable: if attention space is the real objective and weight space only a proxy, then a pipeline optimizing the former should move away from the float weights (M3 up) while moving toward the float attention output (M4 down), and improve M1 and M2 while doing so. Appendix C Experimental Setup in Full Models and scope GPT-2 small (Radford et al., 2019) (124M, d=768d=768, 12 blocks) and Mistral-7B-v0.1 (Jiang et al., 2023) (56×56× larger, 32 layers, d=4096d=4096), which additionally uses grouped-query attention, rotary embeddings and a sliding-window mask. In both, only the QKV projection is quantized, so absolute perplexities are not comparable to end-to-end quantized systems. Runs Three runs are reported, all using group_size=128 group\_size=128, act_order, λKL=0.1 _KL=0.1 unless ablated, and 10 Hutchinson probes on 2 calibration batches per layer. Run A calibrates and evaluates GPT-2 on WikiText-2 and sweeps criterion × 7 budgets × GPTQ, ++FT. Run B calibrates GPT-2 on C4 and evaluates on both corpora, sweeping 5 modes × 5 budgets under all four metrics. Run C sweeps the same 5 modes × 5 budgets on Mistral-7B in all four WikiText-2/C4 calibrate→ pairings, plus a sensitivity-source × allocator cross. Mistral-7B (14.5 GB in fp16) does not fit on a free-tier T4 alongside the float teacher copy the fine-tuning stage requires, so Run C uses a layer-streaming re-implementation: shards are fetched tensor-by-tensor onto the GPU and each decoder layer is materialized on the meta device and discarded after use, at the cost of two passes — score, then quantize-and-evaluate — per configuration. This is why Run C is a re-implementation rather than a re-run. Evaluation corpora Runs A and B score GPT-2 on the full WikiText-2 (Merity et al., 2017) test split; Run B additionally scores the same weights in-domain on a disjoint C4 (Raffel et al., 2020) validation sample, which is what lets calibration effects be separated from evaluation effects in Sections and . Since C4 has no canonical evaluation subset, those absolute perplexities are specific to this draw; and Run C uses 32 windows (≈ 33k tokens) per configuration, so its perplexities are subset figures — exact within a pairing, not comparable to published Mistral-7B numbers — with εW,εA ^W, ^A on the streaming pipeline’s raw loss scale. Appendix D The Sensitivity Criterion Figure 2: Sensitivity criterion across both models. (a) Hessian trace per layer, normalized by each model’s maximum, against normalized depth; stars mark the most sensitive layer. The profile is strongly non-uniform in both models (41×41× spread on GPT-2, 135×135× on Mistral) but its shape inverts with architecture. (b) The sensitivity tables Ωi(b) _i(b) of equation are near-parallel in both models — a shared, steep sub-4-bit cliff — so bit-width dominates layer identity by orders of magnitude. Sensitivity is strongly non-uniform, but its depth profile is architecture-specific All traces are positive, as equation requires, and both models span one to two orders of magnitude across layers (41×41× on GPT-2, 135×135× on Mistral), so the flat profile a uniform allocation implicitly assumes is clearly wrong — the QKV analogue of the head-level heterogeneity reported by Michel et al. (2019). The shape, however, does not transfer (Fig. (a)): GPT-2 peaks at block 3 of 12 and decays, whereas Mistral rises with depth and peaks at its final layers. This falsifies C2 as a general rule, since the (L−i)(L-i) weighting of equation protects early layers — which rescues GPT-2 at 3 bits (Section ) but at Mistral scale would suppress exactly the layers the criterion ranks highest. Bit-width dominates layer identity The per-layer Ωi(b) _i(b) curves are near-parallel in both models (Fig. (b)): dropping from 4 to 3 bits multiplies Ω by roughly 5×5× and from 3 to 2 by a further 55–7×7×, almost independently of which layer is scored. This shared cliff is a property of the construction, not of one model, and it means a ratio-based allocator has little to differentiate layers on until the budget forces one below 4 bits — where the choice then matters a great deal. The ranking is a property of the model, not of the calibration text Traces computed under WikiText-2 (Run A) and C4 (Run B) calibration correlate at r=0.9995r=0.9995 and induce the same ordering up to one adjacent pair differing by under 1%1\% — not obvious, since both the Hessian and the perturbation term of equation are activation-dependent. The score matters; the solver does not A better sensitivity estimate pays off only where allocation is genuinely hard, and the choice of solver hardly matters at all. Crossing the two sensitivity sources with the two allocators on Mistral (Wiki→ , joint FT throughout), second-order Hutchinson scoring beats the first-order Fisher proxy at the aggressive 3-bit budget (5.2685.268 vs. 5.3025.302 perplexity, 0.60900.6090 vs. 0.60710.6071 accuracy), whereas greedy and MCKP/Pareto given identical scores are indistinguishable there, and all four combinations lie within 0.0070.007 perplexity at 6 bits. This mirrors the criterion-versus-oracle result of Section : the double-backward cost is worth paying, the Pareto pre-filter is not. Appendix E Bit Allocation Figure 3: Bit allocation on the QKV projection. (a) GPT-2: perplexity vs. average bits for four criteria, GPTQ-only (solid) and ++joint fine-tuning (dotted). (b) Mistral-7B: perplexity normalized by each pairing’s fp16 baseline, uniform GPTQ vs. Hutchinson-greedy adaptive. (c) Share of the uniform-GPTQ gap recovered by adaptive allocation at 3 bits: Mistral recovers 7777–90%90\% in every pairing, while on GPT-2 the same criterion loses 343%343\% (bar clipped), reduced to −40%-40\% by the depth re-weighting C2 and −50%-50\% by the oracle. Table 8: GPT-2 Run A: full WikiText-2 test perplexity across criteria, budgets, and modes. fp32 control 24.357. “–” marks combinations that do not exist. Best per row in bold. Uniform JAB (C1) JAB++prop (C2) Oracle (C3) bits GPTQ ++FT GPTQ ++FT GPTQ ++FT GPTQ ++FT 2.0 5137.6 516.8 – – – – – – 2.5 – – 201.7 126.5 201.7 126.5 218.0 120.1 3.0 38.35 37.81 86.36 48.31 43.93 41.99 45.36 42.45 3.5 – – 28.32 28.38† 28.32 28.38† 28.61 29.09 4.0 26.23 25.85 26.23 25.85 26.23 25.85 26.23 25.85 4.5 – – 26.20 25.62 26.20 25.62 25.87 25.68 6.0 – – 25.05 24.79 25.05 24.79 25.11 24.80 † the only budget at which fine-tuning hurts; see Section . Mixed precision helps above 4 bits, ties at 4, and can be catastrophic below On GPT-2 (Table , Fig. (a)) adaptive allocation improves on uniform 4-bit by −1.18-1.18 perplexity at 6 average bits by upgrading high-trace blocks to 8 bits. At exactly 4.0 bits it returns an all-4-bit assignment and reproduces the uniform baseline exactly — correct, since a budget the uniform solution already saturates leaves nothing to trade. At 3.0 bits it instead reaches 86.3686.36 against uniform 3-bit’s 38.3538.35: the allocator is destroying the model. That failure is a provable consequence of the additive objective Suppose the 3.0-bit assignment contained no block below 3 bits. Then ∑ibi≥36 _ib_i≥ 36, and the budget caps ∑ibi≤36 _ib_i≤ 36, so equality would force bi=3b_i=3 everywhere — the uniform assignment and its 38.345. It does not, so the allocation must contain at least one 2-bit block, which Run B confirms by inspection (blocks 0 and 1). One such block suffices: uniform 2-bit GPTQ scores 5137.6, over 200×200× the fp32 control. Minimizing a sum of Ωi(bi) _i(b_i), the allocator cannot see that perplexity is convex and effectively unbounded in the low-bit direction, so the downgrade looks ratio-favourable in equation and is not. Consistent with this, the depth re-weighting C2 — which costs nothing and leaves every other budget’s assignment unchanged — halves the damage (43.9343.93 vs. 86.3686.36) simply by making the sacrifice of an early block look expensive. Mixed precision earns its keep at budgets uniform cannot express Its real value is at fractional budgets, where no uniform alternative exists: at 2.5 and 3.5 average bits it beats a geometric interpolation between the neighbouring uniform solutions by 2.2×2.2× and 10.7%10.7\%. A cheap proxy is competitive with a 73-model-load oracle The oracle C3 costs ≈73×≈ 73× the model instantiations of C1 and buys little: across the four budgets ≥3.5≥ 3.5 bits the mean absolute difference is 0.170.17 perplexity, with the cheap criterion ahead in half the cases. C3 helps decisively only at 3.0 bits, where it fixes the same 2-bit pathology the free depth re-weighting largely fixes — and neither fixes it as well as declining to use mixed precision there. Above 3.5 bits the allocation is essentially solved, and the residual gap to fp32 lies in the quantizer. At 7B scale allocation carries the result, and the 3-bit collapse does not reproduce Adaptive allocation dominates uniform GPTQ at 3 bits in all four corpus pairings (Table ), recovering 7777–90%90\% of the uniform-versus-fp16 perplexity gap (Fig. (c)) and 22–33 accuracy points; by 4 bits every method is within 11–2%2\% of fp16 and by 6 bits within 0.5%0.5\%, the same diminishing-returns pattern as GPT-2. What does not appear is a 3-bit collapse, even though the budget arithmetic applies equally: Mistral’s allocator must also place some layer below 3 bits. The per-layer assignment is unavailable, so we report this as an open discrepancy. Two untested explanations are plausible: with 32 blocks rather than 12, one low-bit layer is a much smaller share of the budget; and GQA shrinks the K/V projections, changing both c(b)c(b) and the curvature landscape. Table 9: Mistral-7B, all four calibrate→ corpus pairings. Upper block: perplexity (lower better). Lower block: next-token top-1 accuracy (higher better). fp16 baselines: PPL 4.818 (WikiText-2 eval) / 7.492 (C4 eval); Acc 0.6257 / 0.5500. “–” marks a flat-bit mode at a fractional budget, left undefined rather than silently rounded. Best per column in bold; ties bolded. Wiki→ C4→ 4 Wiki→ 4 C4→ Method 3b 3.5b 4b 4.5b 6b 3b 3.5b 4b 4.5b 6b 3b 3.5b 4b 4.5b 6b 3b 3.5b 4b 4.5b 6b Perplexity Uniform (GPTQ) 8.738 – 4.855 – 4.822 13.506 – 7.571 – 7.495 13.575 – 7.580 – 7.496 8.875 – 4.870 – 4.820 Adaptive (Hutch., greedy) 5.228 5.129 4.855 4.850 4.834 8.898 8.670 7.571 7.556 7.518 8.622 8.340 7.580 7.577 7.530 5.428 5.302 4.870 4.861 4.839 Joint (uniform bits) 8.833 – 4.875 – 4.822 13.615 – 7.617 – 7.520 13.658 – 7.624 – 7.496 8.855 – 4.881 – 4.821 JAB++Joint (Hutch., greedy) 5.268 5.134 4.875 4.867 4.845 8.732 8.507 7.617 7.595 7.553 8.681 8.373 7.624 7.608 7.561 5.411 5.268 4.881 4.873 4.843 JAB++Joint, no KL 5.231 5.121 4.854 4.849 4.835 8.805 8.563 7.575 7.559 7.539 8.582 8.337 7.583 7.575 7.531 5.477 5.293 4.866 4.858 4.838 Accuracy Uniform (GPTQ) 0.5786 – 0.6236 – 0.6252 0.5094 – 0.5478 – 0.5504 0.5086 – 0.5486 – 0.5498 0.5752 – 0.6230 – 0.6256 Adaptive (Hutch., greedy) 0.6093 0.6146 0.6236 0.6248 0.6249 0.5229 0.5254 0.5478 0.5485 0.5498 0.5269 0.5309 0.5486 0.5483 0.5499 0.6034 0.6084 0.6230 0.6235 0.6253 Joint (uniform bits) 0.5762 – 0.6230 – 0.6252 0.5066 – 0.5470 – 0.5494 0.5067 – 0.5475 – 0.5498 0.5731 – 0.6232 – 0.6254 JAB++Joint (Hutch., greedy) 0.6090 0.6128 0.6230 0.6233 0.6238 0.5245 0.5286 0.5470 0.5484 0.5493 0.5269 0.5298 0.5475 0.5480 0.5489 0.6058 0.6100 0.6232 0.6239 0.6247 JAB++Joint, no KL 0.6112 0.6145 0.6240 0.6249 0.6245 0.5245 0.5286 0.5473 0.5483 0.5493 0.5272 0.5307 0.5488 0.5489 0.5499 0.6033 0.6079 0.6235 0.6226 0.6246 Appendix F Quantization Method: GPTQ vs. Joint Fine-Tuning Figure 4: Joint fine-tuning across models, corpora and budgets, on symmetric-log axes so that both the low-bit and near-lossless regimes are legible. (a) Relative perplexity change from adding the joint STE stage to the same allocation; negative is better, shaded band ±2%± 2\%. Gains are large exactly where quantization damage is large and shrink to a few percent in the usable regime, where the sign becomes model- and corpus-dependent. (b) The KL ablation, λKL:0.1→0 _KL:0.1→ 0: on GPT-2 removing the term is harmful at every budget on both corpora, catastrophically so below 4 bits; on Mistral, where the stage moves far less, the effect is within ±1%± 1\% and changes sign. Fine-tuning helps most where quantization damage is largest On GPT-2 the joint stage improves 18 of the 21 configurations of Table . Gains scale with the damage — −45%-45\% to −90%-90\% below 3 bits — and settle at 11–2%2\% in the usable regime, bringing the best arm (JAB at 6 average bits) to 24.78624.786 against an fp32 control of 24.35724.357. A flip rate of 0.120.12–0.180.18 throughout confirms this reflects the deployed weights, not latent shadow weights. Its one failure is a proxy–objective gap the commit gate cannot see All three exceptions sit at 3.5 average bits, where the gate of Section (4) certifies an improvement on the held-out attention loss while perplexity worsens slightly. There the allocation holds only 3- and 4-bit blocks, so every block carries substantial error and none is nearly exact; the optimizer has room for large local improvements, some of which trade against the global objective. Fine-tuning’s instabilities are corpus effects, and they separate cleanly Scoring the identical Run B weights on both corpora isolates cause from symptom. The 3.5-bit anomaly is an evaluation-corpus artifact: in-domain on C4 the sign reverses (−4.5%-4.5\% instead of +0.5%+0.5\%) although nothing about the models changed, so the agreement between Runs A and B here is better read as two measurements sharing an evaluation corpus than as independent confirmation — a caution for any single-corpus ablation. The 3-bit sign flip against Run A is instead a calibration-corpus effect: moving to the matched corpus makes it worse (+60.8%+60.8\% versus +16.1%+16.1\%), so cross-corpus evaluation cannot be the cause. Both are confined to fractional and aggressive budgets. At 7B scale, refinement is second-order to allocation Fig. (a) places every fine-tuning comparison on one axis. On Mistral the correction is marginally negative on WikiText-2 and marginally positive on C4; the magnitudes are what matter, since at 3 bits allocation is worth 33–55 perplexity points and the fine-tuning at most 0.170.17 either way. From an independent model and codebase this reinforces the central claim — how bits are spent matters more than how the resulting weights are locally refined — subject to two caveats: the LSQ and grid-relative refinements were not re-verified at this scale, and the streaming build’s smaller step budget makes its fine-tuning stage less comparable to GPT-2’s than its allocator is. The KL term is what makes fine-tuning work Holding bit-widths fixed on GPT-2, removing the attention-map term recovers only −0.052-0.052 of the −0.444-0.444 total gain, so the KL term supplies 88%88\% of it — consistent with the reduction imbalance of Section making the objective effectively attention-map distillation. Fig. (b) sweeps the ablation: dropping the term is worse than keeping it at every budget on both evaluation corpora, catastrophically so below 4 bits. The mechanism is visible in the metrics rather than assumed. MSE-only fine-tuning does reduce attention-output error relative to no fine-tuning (εA:0.238→0.188 ^A:0.238→ 0.188 at 4 bits) — it succeeds at the objective it was given — yet perplexity still worsens. Matching the attention output is therefore not sufficient: an unconstrained MSE descent can reach a lower output error through a distribution the downstream stack was never trained against, and the KL term forbids that. Flip rate measures movement, not benefit At a fixed assignment the MSE-only run moves more grid positions than the KL run (0.1770.177 vs. 0.1360.136) while ending 0.3920.392 perplexity worse, so activity and benefit must be logged separately. Benefit is monotone in fine-tuning depth — restricting the stage to the first four blocks lands between GPTQ-only and full-depth — which refutes a compounding-drift hypothesis: had each block accumulated error against a stale target, a prefix would have beaten full depth. Appendix G Where the Error Lives, and Whether Accuracy Agrees Table 10: GPT-2 Run B (C4 calibration): five pipeline modes, five budgets, four metrics, evaluated on both corpora. εW ^W is a property of the quantized weights alone and is therefore shared; the amplification factor ρ=εA¯/εW¯ρ= ^A/ ^W is plotted in Fig. (b). Controls: WikiText-2 24.357 / 0.4148; C4 30.778 / 0.3786. “–” marks a flat-bit mode at a fractional budget. Evaluated on WikiText-2 (cross-corpus) Evaluated on C4 (in-domain) Mode bits PPL Acc εA ^A PPL Acc εA ^A εW ^W uniform 3 39.338 0.3612 0.4711 54.862 0.3339 0.4621 0.3753 uniform 4 26.536 0.4038 0.2382 33.638 0.3692 0.2370 0.1739 uniform 6 24.488 0.4140 0.0695 30.955 0.3774 0.0702 0.0409 jab 3 80.140 0.2924 0.4463 99.206 0.2751 0.4287 0.4321 jab 3.5 29.425 0.3949 0.3579 36.958 0.3608 0.3504 0.2787 jab 4 26.536 0.4038 0.2382 33.631 0.3692 0.2370 0.1739 jab 4.5 26.315 0.4055 0.2206 33.348 0.3703 0.2193 0.1591 jab 6 25.042 0.4107 0.1178 31.528 0.3751 0.1145 0.0908 joint 3 45.653 0.3456 0.3208 88.226 0.3182 0.2912 0.4595 joint 4 25.867 0.4078 0.1602 33.347 0.3705 0.1472 0.2075 joint 6 24.465 0.4143 0.0440 30.953 0.3773 0.0414 0.0492 adaptive_joint 3 57.753 0.3188 0.3135 91.149 0.2914 0.2801 0.5192 adaptive_joint 3.5 29.585 0.3922 0.2475 35.301 0.3598 0.2246 0.3313 adaptive_joint 4 25.867 0.4078 0.1602 33.619 0.3705 0.1472 0.2075 adaptive_joint 4.5 25.478 0.4105 0.1474 33.022 0.3717 0.1348 0.1899 adaptive_joint 6 24.738 0.4138 0.0819 30.838 0.3757 0.0748 0.1064 kl0 (λKL=0 _KL=0) 3 199.247 0.2052 0.4461 200.900 0.1970 0.4157 0.5554 kl0 3.5 34.980 0.3703 0.3062 45.137 0.3417 0.2819 0.3599 kl0 4 26.865 0.4017 0.1883 33.819 0.3690 0.1712 0.2167 kl0 4.5 26.348 0.4050 0.1749 33.219 0.3707 0.1593 0.1978 kl0 6 25.353 0.4099 0.0996 31.136 0.3750 0.0908 0.1133 Figure 5: (a) Attention-space against weight-space error, one point per GPT-2 Run B arm; filled markers WikiText-2 eval, hollow markers C4 eval. Every GPTQ-only arm sits in the shaded amplifying region above the ρ=1ρ=1 diagonal, every fine-tuned arm below it. (b) The same as the amplification factor ρ by budget: the ordering GPTQ-only >> MSE-only >> MSE++KL holds at every budget on both corpora without a crossing. (c) Accuracy against perplexity, each normalized by its own full-precision baseline, pooling GPT-2 Run B with the Mistral-7B 3-bit arms. Fine-tuning trades weight fidelity for attention fidelity This measurement most directly tests the report’s premise, and the premise survives. On the flat 4-bit pair, joint fine-tuning moves the weights 19%19\% further from float (εW:0.174→0.208 ^W:0.174→ 0.208) while moving the attention output 33%33\% closer (εA:0.238→0.160 ^A:0.238→ 0.160), and both perplexity and accuracy improve; the same trade appears at 3 and 6 bits. A solution measurably worse by the objective GPTQ minimizes is therefore better on the metric that matters and better on the task, so the weight-space objective is not the binding constraint. The amplification factor separates the two regimes without exception Causal softmax attention amplifies the relative weight perturbation it is handed, and fine-tuning inverts that: every GPTQ-only arm has ρ>1ρ>1 and every fine-tuned arm ρ<1ρ<1, with the ordering ρGPTQ>ρMSE-only>ρMSE+KL _GPTQ> _MSE-only> _MSE+KL holding at every budget on both corpora without a crossing (Fig. (a,b)). The single departure — jab at 3 bits, where ρ moves from 1.031.03 to 0.990.99 between corpora — is smaller than single-draw Hutchinson estimates at one seed can resolve. On Mistral the same dissociation is produced by the allocator instead At 3 bits, adaptive allocation attains a lower attention-reconstruction error than uniform on both calibration corpora (1.131.13 vs. 1.53×10−31.53× 10^-3 under WikiText-2 calibration, 1.321.32 vs. 1.65×10−31.65× 10^-3 under C4) while incurring a higher weight error (0.4670.467 vs. 0.4330.433 and 0.4740.474 vs. 0.4390.439) — exactly what spending bits on high-curvature layers implies — and again it is that trade, not weight fidelity, that tracks perplexity and accuracy. The same signature arising through two mechanisms in two independent codebases is stronger evidence than either alone. (Both errors are on the streaming pipeline’s raw scale.) Accuracy confirms the perplexity ordering Next-token accuracy is almost perfectly rank-inverse to perplexity across all 21 GPT-2 arms (Spearman ρs=−0.996 _s=-0.996 on WikiText-2, −0.990-0.990 on C4), and Fig. (c) shows the relation holding across models, corpora and modes once each configuration is normalized by its own full-precision baseline — so the differences above are not artifacts of the likelihood tail. Only two arms reorder by more than one rank, both at budgets already flagged as unstable. Appendix H Cross-Corpus Robustness Quantization cost is a property of calibration and budget, not of the scoring corpus Uniform 4-bit GPTQ costs +8.9%+8.9\% perplexity against the WikiText-2 control and +9.3%+9.3\% against the C4 control despite baselines differing by 6.4 absolute points. Allocation transfers across corpus mismatch; refinement does not On Mistral, adaptive allocation recovers 81.4%81.4\% and 85.0%85.0\% of the uniform gap under corpus mismatch, against 89.5%89.5\% and 76.6%76.6\% matched — so the mismatch penalty is smaller than the spread between the two matched pairings (Fig. (c)). The sensitivity ranking is therefore a property of the model’s architecture and weight structure rather than an artifact overfit to the calibration text: the functional counterpart of the r=0.9995r=0.9995 trace correlation measured directly on GPT-2. Fine-tuning behaves oppositely — both of its instabilities are corpus-induced (Section ) — while no allocation result changes sign under either intervention. Appendix I Beyond QKV: the MLP Sublayer The pipeline of Section is not specific to attention: ℒJABL_JAB is a block-level reconstruction loss, and nothing in the Hutchinson estimator, the Ωi(b) _i(b) table or the allocator refers to a softmax. We therefore ported the whole system to the MLP sublayer, quantizing c_fc and mlp.c_proj and leaving all attention weights in fp32. Two changes are structural. First the MLP is two sequential matrices rather than one fused one, so each block contributes two allocation units and two Hessians, of dimension d and 4d4d; the second is collected only after the first is quantized, preserving within-block the sequential property Section obtains across blocks. Second, and decisively, the attention map has no MLP analogue: ℒKLL_KL cannot be formed, so λKL=0 _KL=0 by construction and the objective reduces to output MSE alone. The KL ablation of Section is correspondingly vacuous here, and we verified it reproduces the default arm bit-for-bit (Δ=0.0e+00 =0.0e+00). The scope is worth stating, because it inverts. The two MLP matrices hold 8d28d^2 parameters per block against QKV’s 3d23d^2: 56.656.6M versus 21.221.2M on GPT-2, or 66.6%66.6\% of non-embedding weights against 25.0%25.0\%. Every result below therefore covers 2.67×2.67× more of the model than Table , and no MLP budget is reachable by any QKV-only scheme — even 0-bit QKV leaves a larger model than 4-bit MLP. Compression at 4 bits is 1.66×1.66× end-to-end, the residue being the 39.439.4M embedding parameters (32%32\% of GPT-2 small) that no arm in this report touches. Table 11: GPT-2 MLP: full WikiText-2 test perplexity, same pipeline, criteria and budgets as Table , with c_fc and mlp.c_proj quantized and attention left in fp32. fp32 control 24.357; budget is average bits per matrix over 24 units. “–” marks combinations that do not exist. Best per row in bold. Contrast the Uniform column with Table : here the criterion loses to it at both comparable budgets. Uniform JAB (C1) JAB++prop (C2) Oracle (C3) bits GPTQ ++FT GPTQ ++FT GPTQ ++FT GPTQ ++FT 2.0 3698.3 8462.0‡ – – – – – – 2.5 – – 2458.3 1439.6 395.7 248.2 351.3 275.1 3.0 53.71 46.45 120.70 94.49 53.71 46.45 46.15 39.13 3.5 – – 40.12 37.55 29.90 28.80 28.54 27.99 4.0 26.23 25.96 27.46 26.64 26.23 25.96 26.23 25.96 4.5 – – 26.14 25.89 26.11 25.76 25.60 25.43 6.0 – – 25.68 25.20 24.92 24.98‡ 24.76 24.68 ‡ the two budgets at which fine-tuning hurts. The criterion loses to uniform, and the oracle does not Two budgets admit a uniform baseline, and C1 is beaten at both: 120.70120.70 against 53.7153.71 at 3 bits (2.25×2.25× worse) and 27.4627.46 against 26.2326.23 at 4 bits. This is not the 3.0-bit pathology of Section , whose cause was a 2-bit block entering the allocation; here the 4-bit assignment contains nothing below 3 bits and still loses. Adaptive allocation as implemented is simply harmful on this sublayer. That the oracle C3 beats uniform at 3 bits (46.1546.15 vs. 53.7153.71, a 14%14\% improvement) establishes that exploitable structure exists and that the Hutchinson criterion is failing to find it — a cleaner separation of criterion from ceiling than the QKV sweep affords, where C1 and C3 agree to 0.170.17 perplexity above 3.5 bits. The depth profile inverts again, and is far flatter Fig. (a) reported a QKV trace peaking at block 3 of 12 and decaying, with a 41×41× spread. The MLP trace on the same model rises monotonically from block 5 to the last layer and spans only 4.5×4.5× (365→1543365→ 1543). “Architecture-specific”, the reading offered in Section , is therefore too coarse: the profile is sublayer-specific, and inverts between two sublayers of one block. This weakens the (L−i)(L-i) re-weighting of equation further than Fig. (a) already does — C2’s advantage over C1 here (e.g. 29.9029.90 vs. 40.1240.12 at 3.5 bits) comes from opposing a depth gradient it was designed to create. The trace cannot separate the two matrices, by construction A single Hutchinson trace is taken over the concatenated [Wfc∣Wproj][W_fc W_proj] vector, so Tr(H)Tr(H) is identical for the two units of a block and all discrimination between them falls to the perturbation factor of equation . This is the exact MLP analogue of Section ’s observation that the trace sees only diagonal blocks: joint and separate coincide for scoring. Here that identity is not benign, because — unlike Q, K and V — the two MLP matrices have different input dimensions, different input distributions, and, as the oracle shows, genuinely different sensitivities. Table 12: GPT-2 MLP, bit assignments at the 3.5-bit budget by unit and depth. The oracle’s rule is constant in depth and splits by matrix role; the Hutchinson criterion follows a depth gradient instead and pays 11.611.6 perplexity for it. Criterion Unit Assignment (blocks 0–11) PPL Oracle (C3) c_fc 4 4 4 4 4 4 4 4 4 4 4 4 28.54 c_proj 3 3 3 3 3 3 3 3 3 3 3 3 JAB (C1) c_fc 4 3 3 3 3 3 4 4 4 4 4 4 40.12 c_proj 3 3 3 3 3 3 3 4 4 4 4 4 The oracle finds a role rule, not a depth rule Table makes the failure concrete, and the oracle’s assignments are the substantive finding of this section. At every fractional budget it returns a flat allocation split by matrix role: c_fc at b+1b+1 and mlp.c_proj at b−1b-1, with no depth variation whatever (3/2 at 2.5 bits, 4/3 at 3.5, 8/4 at 6). C1 instead spends its budget along depth — at the 4-bit budget it robs c_proj in blocks 0–4 down to 3 bits to fund an 8-bit c_proj in block 11, and loses 1.231.23 perplexity against leaving everything at 4. A plausible mechanism is that both factors of Ωi(b)=Tr(Hi)‖Qb(Wi)−Wi‖F2 _i(b)=Tr(H_i)\,\|Q_b(W_i)-W_i\|_F^2 scale with magnitude while GPTQ’s per-group scales already normalize for it, so the criterion reads activation scale — which grows with depth in a residual stream — as curvature. The measured block reconstruction losses track this directly, rising from 0.0330.033 at block 0 to 0.1130.113 at block 10. A scale-normalized trace Tr(Hi)/diag(Hi)¯Tr(H_i)/ diag(H_i) is the obvious test and was not run. Fine-tuning is inert in the deep layers Section reports a flip rate of 0.120.12–0.180.18 throughout and reads it as evidence the stage reaches deployed weights. The MLP run does not reproduce this. Pooling the 20 fine-tuned sweep arms, the flip rate is exactly zero in blocks 9, 10 and 11 in 1515, 1212 and 1717 arms respectively, against zero occurrences in blocks 0–3; the reconstruction loss is unchanged to six decimals in those blocks, so the commit gate of Section rejected every one of 200 steps. The signature is the one Section already attributes to a fixed absolute learning rate, and the deep blocks are precisely those carrying the largest reconstruction error, so the stage fails where it is most needed. The aggregate consequence is that fine-tuning is worth ≈1%≈ 1\% here against 11–2%2\% on QKV, it hurts at two budgets rather than one, and the depth-restricted control lands at 26.1226.12 between GPTQ-only 26.2126.21 and full-depth 25.9325.93 — a much narrower band than the QKV analogue. We report the Objective-3 results on this sublayer as unresolved rather than negative. What this says about the central claim Section establishes that the KL attention-map term supplies 88%88\% of the fine-tuning gain, and Fig. (b) that removing it is harmful at every budget. The MLP port is the limiting case of that ablation: the term is not down-weighted but structurally absent, and both halves of the pipeline degrade together — the criterion falls behind uniform, and refinement falls to a percent. The honest reading is that what transfers from this work is the attention-specific machinery, not a general-purpose mixed-precision recipe, and that a sublayer without a distributional target to distill against needs a different objective rather than the same one with a term deleted. What is not measured here A single seed was run for Table (λKL=0 _KL=0, seed 42, full WikiText-2 test split, 242242 minutes over 44 quantize-and-evaluate passes), so L1 and L5 apply unchanged. Integer budgets carry no information about allocation in this setting: at 3.0 and 4.0 bits every criterion returns a flat assignment and the four columns of Table coincide exactly, as the budget arithmetic of Section requires. A second, C4-calibrated run does instrument εW ^W, εA ^A and ρ A separate MLP sweep, calibrated on C4 rather than WikiText-2 and following Run B’s protocol (four modes, five budgets, scored on both corpora), defines εA ^A as the relative Frobenius error of the MLP block’s own output — the direct analogue of the attention-output εA ^A of Section , with mlp.c_proj’s input playing the role QKV’s output plays there — and reports ρ=εA/εWρ= ^A/ ^W alongside it (Table ). Table 13: GPT-2 MLP, C4 calibration: four pipeline modes, five budgets, evaluated on both corpora, with the amplification factor ρ=εA/εWρ= ^A/ ^W reported per corpus (εW ^W is shared). Controls: WikiText-2 24.357/0.4148; C4 32.770/0.3736. Not directly comparable to Table , which is calibrated on WikiText-2. WikiText-2 (cross-corpus) C4 (in-domain) Mode bits PPL Acc εA ^A ρ PPL Acc εA ^A ρ εW ^W uniform 3 54.152 0.3241 0.4749 1.332 61.630 0.3035 0.4632 1.299 0.3565 uniform 4 26.423 0.4053 0.2112 1.349 34.765 0.3652 0.2068 1.321 0.1566 uniform 6 24.420 0.4149 0.0483 1.338 32.648 0.3734 0.0472 1.307 0.0361 jab 3 600.798 0.1219 0.4650 1.192 574.850 0.1188 0.4457 1.143 0.3900 jab 3.5 46.236 0.3421 0.3292 1.273 54.889 0.3177 0.3237 1.251 0.2587 jab 4 26.423 0.4053 0.2112 1.349 34.765 0.3652 0.2068 1.321 0.1566 jab 4.5 26.237 0.4065 0.2009 1.387 34.621 0.3661 0.1959 1.353 0.1448 jab 6 25.499 0.4103 0.1113 1.305 33.779 0.3690 0.1094 1.283 0.0853 joint 3 48.121 0.3331 0.4199 1.038 58.676 0.3044 0.4021 0.994 0.4045 joint 4 26.055 0.4054 0.1951 1.123 34.411 0.3665 0.1873 1.078 0.1738 joint 6 24.434 0.4143 0.0469 1.206 32.731 0.3737 0.0452 1.162 0.0389 adaptive_joint 3 157.931 0.2067 0.4203 0.904 152.766 0.2195 0.4015 0.863 0.4651 adaptive_joint 3.5 40.772 0.3474 0.2864 0.965 51.422 0.3197 0.2742 0.924 0.2967 adaptive_joint 4 26.055 0.4054 0.1951 1.123 34.411 0.3665 0.1873 1.078 0.1738 adaptive_joint 4.5 25.920 0.4062 0.1847 1.140 34.283 0.3669 0.1764 1.089 0.1620 adaptive_joint 6 25.349 0.4099 0.0965 0.959 33.503 0.3689 0.0914 0.909 0.1006 ρ separates GPTQ-only from fine-tuned arms less cleanly than on QKV GPTQ-only allocation (uniform, jab) has ρ>1ρ>1 at every budget on both corpora, as on QKV. But the two fine-tuned modes do not fall cleanly below ρ=1ρ=1: joint stays above 11 except at the tightest budget, while adaptive_joint dips below 11 at 3, 3.5 and 6 bits but rises back above it at 4 and 4.5, tracking the flat-assignment coincidence with jab noted in Section rather than the fine-tuning stage itself. The clean, crossing-free separation of Fig. (a,b) is therefore a QKV-specific regularity, not a general property of ρ. The 3-bit allocator failure is far more severe under C4 calibration Table ’s WikiText-2-calibrated jab arm loses to uniform at 3 bits by 2.25×2.25× (120.70120.70 vs. 53.7153.71); under C4 calibration the same criterion at the same budget loses by 11.1×11.1× (600.798600.798 vs. 54.15254.152), with adaptive_joint similarly worse (157.931157.931 vs. Table ’s finetuned 94.4994.49). The failure mode is the same one-layer 2-bit collapse described in Section , but its severity is calibration-corpus dependent in a way the QKV sweep’s r=0.9995r=0.9995 trace correlation (Section ) did not predict. Appendix J Full-Model Quantization: Extended Analysis Sections – quantize the QKV projection and Section the MLP, each in isolation. We now compose them: every linear module of every Mistral-7B decoder layer is quantized under a single budget, covering 6.986.98B parameters (96.4%96.4\% of the model, 224 allocation units). This is the regime practitioners deploy, and it tests three claims that the sublayer studies could only suggest: whether a sensitivity criterion helps once units are heterogeneous, whether block-local refinement remains safe, and whether the 22-bit allocation pathology of Section is specific to GPT-2. J.1 Setting Implementation Four Hessians suffice per block: q,k,vq,k,v share the input RMSNorm1(h)RMSNorm_1(h) and gate, up share RMSNorm2(h)RMSNorm_2(h). They are collected in forward order qkv→o→gate,up→downqkv→ o→\ gate, up\→ down, each after the preceding units are quantized, extending within the block the sequential property of Section . Weight layout is an explicit per-architecture flag, since n.Linear stores (dout,din)(d_out,d_in), the transpose of GPT-2’s Conv1D, and a silent transpose would quantize along the wrong axis. Unlike the streaming build of Section , this run solves GPTQ in float64 throughout with per-block offloading on a single 96 GB GPU, so the float32-solve caveat of Section does not apply. A hand-written block forward reproducing RoPE, RMSNorm, GQA head repetition and SwiGLU is validated against MistralDecoderLayer before any arm runs. Evaluation uses 32 WikiText-2 windows of 512 tokens (fp16 control 6.6436.643), one seed, and 64 Hessian batches. Figure 6: Full-model quantization of Mistral-7B. (a) Perplexity against bits per parameter, GPTQ only. At B=4.5B=4.5 Type-offset is best; the adaptive JAB arm (⋆ ) spends 4.34.3 bits per parameter yet scores worse than uniform 44-bit. (b) Mean assigned bits per unit kind at B=4.5B=4.5. Type-offset is flat within a role and offset between roles; JAB over-buys k_proj, which grouped-query attention makes cheap; JAB-norm assigns 88 bits to every o_proj while starving q,k,vq,k,v to 22–33 bits. J.2 Layer Role Outperforms Curvature A seven-integer prior beats the Hessian criterion at matched budget At B=4.5B=4.5 (Table ), Type-offset reaches 6.9706.970 against JAB’s 7.2507.250 and JAB-norm’s 10.7710.77 — all three at exactly 4.5004.500 realized bits per parameter — while computing no Hessian, no Hutchinson probe and no scoring pass. This is the second architecture in which role-based allocation matches or beats the learned criterion: the GPT-2 MLP oracle found the same flat, role-split structure under GELU (Section ), and here it prevails under SwiGLU with grouped-query attention and 56×56× more parameters. The sensitivity-driven allocator also fails a weaker test. At B=4.3B=4.3 the adaptive JAB arm reaches 7.3737.373, worse than uniform 44-bit (6.9986.998) while spending 7.5%7.5\% more bits. Uniform 44-bit remains within 0.030.03 of the best mixed-precision arm at B=4.5B=4.5, so in this regime no allocator we tested improves on uniform quantization per bit spent; what distinguishes them is how much each loses. Why the criterion cannot find role structure The JAB trace is computed once per block over the concatenated parameter vector, and is therefore identical for all seven units of a layer (e.g. 1478.61478.6 for every unit of block 5). Between units of one block, the criterion can discriminate only through the perturbation factor of Eq. . Its dominant signal is instead a depth gradient spanning 381×381× (59.259.2 at block 0 to 22,57522,575 at block 31) — the one axis on which the winning allocation has no structure at all. Fig. (b) shows the consequence: JAB’s largest purchases are GQA-cheap k_proj units rather than any role. Per-unit traces would remove this limitation at four double-backward passes per block. Scale normalization trades one bias for another Section hypothesized that the criterion reads activation magnitude as curvature and proposed normalizing the trace by diag(Hi)¯ diag(H_i). We tested this (JAB-norm). It is worse than JAB at both usable budgets (10.7710.77 vs. 7.2507.250 at B=4.5B=4.5; 109.6109.6 vs. 8.5708.570 at B=3.5B=3.5). Dividing by the mean Hessian diagonal rewards units whose input spectrum is flat, and o_proj, fed by the merged attention output, is far flatter than the post-RMSNorm stream feeding q,k,vq,k,v; JAB-norm consequently assigns 8.08.0 mean bits to o_proj and 7.47.4 to down_proj while leaving q,k,vq,k,v at 2.52.5–3.03.0. The normalizer exchanges a magnitude bias for a conditioning bias. The offset sign matters, and is the main open parameter Type-offset degrades sharply at B=3.5B=3.5 (32.3332.33). With Δdown=−1 _ down=-1, the allocation places 3030 of 3232 down_proj matrices — 25.2%25.2\% of the quantized weights — at 22 bits, the width that destroys uniform quantization. At B=4.5B=4.5 the grid ℬB absorbs the offset (4−14-1 and 4+14+1 round back to 44) and the budget is instead spent promoting individual gate and up units to 88 bits. SwiGLU’s down_proj receives silu(gate(x))⊙up(x)silu( gate(x)) up(x), a heavier-tailed input than the post-GELU activation of its GPT-2 counterpart, so the transposed negative offset is plausibly the wrong sign. Because each candidate offset vector costs a single evaluation, fitting Δ is inexpensive; we report the untuned vector, so the B=4.5B=4.5 result is a lower bound on what the family achieves. J.3 Block-Local Objectives Can Diverge from End-to-End Quality Section identified a gap between the fine-tuning objective and perplexity that the commit gate cannot detect, of under 1%1\% on QKV. Under full coverage the same mechanism produces a 32×32× degradation, and a depth-restricted control localizes it to a single block (Table ). Table 14: Joint fine-tuning under full coverage. “Changed” counts blocks whose quantized weights differ from the GPTQ warm start (flip rate >0>0, Eq. ); “Block-0 loss” is that block’s own objective before and after refinement. Restricting refinement to blocks 0–3 reproduces the full-depth result exactly. Allocation, refinement Blocks refined Changed Block-0 loss PPL Adaptive B=4.3B=4.3, GPTQ only – – – 7.373 + joint FT 0–31 1 (0) 1.85→0.41×10−31.85→ 0.41× 10^-3 239.52 + joint FT 0–3 1 (0) 1.85→0.41×10−31.85→ 0.41× 10^-3 239.52 + joint FT, λKL=0 _KL=0 0–31 1 (0) MSE only 52.07 Uniform 44, GPTQ only – – – 6.998 + joint FT 0–31 3 (0,3,4) 1.0→0.9×10−51.0→ 0.9× 10^-5 11.39 The measurement Refining the adaptive allocation raises perplexity from 7.3737.373 to 239.52239.52. Refining only blocks 0–3 yields 239.52239.52 as well, identical to four decimals together with every weight- and activation-error metric. The flip-rate diagnostic explains the coincidence: in both runs block 0 is the only block whose quantized weights change (flip rate 0.0430.043); in the other 31 blocks every weight returns to its GPTQ grid point. Within block 0, the refinement succeeded on its own terms: its objective fell 4.6×4.6×, from 1.85×10−31.85× 10^-3 to 0.41×10−30.41× 10^-3, and the held-out commit gate accepted it. One refined block in 32 multiplied end-to-end perplexity by 32.532.5. Mechanism Two conditions coincide at block 0. The adaptive allocator assigns it 22 bits on six of its seven units, giving a mean relative weight error εW=1.015 ^W=1.015 (1.421.42 for q_proj): the quantized block lies farther from its float weights than the zero matrix does. And block 0’s output feeds all 31 blocks above it, so its residual is the least local quantity in the network. A 200200-step descent that reduces this block’s objective on eight calibration batches, from a starting point already beyond the zero-error reference, fits noise with maximal downstream leverage. Severity tracks grid coarseness as this account predicts: on the uniform 44-bit allocation the same stage changes three blocks and degrades perplexity by 1.6×1.6× rather than 32×32×. Implication beyond JAB Nothing in this mechanism is specific to ℒJABL_JAB. Any post-training pipeline that refines blocks against block-local targets and validates on held-out calibration data admits the same failure, and the local measurement alone places no bound on the global damage. A certified improvement in a block’s own reconstruction objective is therefore not evidence of end-to-end improvement. Reporting a global metric per refinement stage, together with a per-block flip rate, is the minimum needed to detect it. The KL term amplifies an ill-posed refinement On QKV the attention-map term supplies 88%88\% of the fine-tuning gain (Section ). Under full coverage, removing it reduces the damage from 239.52239.52 to 52.0752.07, with the block-0 flip rate falling from 0.0430.043 to 0.0340.034. Both remain far worse than no refinement, so this compares two degraded configurations; it shows that the KL term drives the stage harder, which helps when the stage is well posed and harms when it is not. Its effect is a property of the regime, not of the objective. J.4 The 2-Bit Pathology Is Not Specific to GPT-2 Section proved that GPT-2’s 3.03.0-bit assignment must contain a sub-33-bit block, and recorded as an open discrepancy that no comparable collapse appeared on Mistral. Full coverage resolves it. The adaptive allocation at B=4.3B=4.3 places its only sub-33-bit units in block 0 — six of seven at 22 bits, εW=1.015 ^W=1.015, the only block with εW>1 ^W>1 — and the arm loses to uniform while spending more. The earlier absence was a matter of scope rather than of depth or grouped-query attention: with only QKV quantized, a single 22-bit projection is 1/1281/128 of the quantized weights; with all seven modules under Eq. , the allocator can concentrate low widths on one block, and the additive objective of Eq. still cannot represent that perplexity is convex and unbounded in the low-bit direction. The floor bi≥3b_i≥ 3 proposed in Section removes every sub-33-bit unit from this assignment as it does from GPT-2’s; whether it recovers uniform-level quality at 77B is untested. Scope All results in this section come from one seed. The 0.280.28 separating Type-offset from JAB at B=4.5B=4.5 and the effects of Section are large relative to that limitation; the 0.030.03 separating Type-offset at B=4.5B=4.5 from uniform 44-bit is not, and the two differ in budget. Perplexities are exact within this run but computed on a 32-window subset. Beyond block 0, the fine-tuning stage changed no weights in 31 of 32 blocks, the mis-scaled step signature of Section ; refinement is therefore under-exercised here independently of the divergence above, and Type-offset was evaluated without it. J.5 GPT-2: QKV++MLP, Full WikiText-2 Table Role beats curvature once both sublayers share the budget. At the aggressive fractional budgets, Type-offset needs no scoring pass yet leads: 64.89664.896 at 3.53.5 bits and 28.21128.211 at 4.54.5, against JAB+prop’s 67.33767.337 and 28.69428.694 (Table ). JAB’s attention-defined sensitivity over-invests in QKV, a quarter of the quantized weights, and under-funds the MLP, which now dominates the parameter count; Type-offset’s flat role split needs no such trade-off. Only near saturation, at 6 bits, does uniform win (24.54624.546): perplexity is nearly flat there, so a mixed 4-/8-bit average cannot beat a flat assignment, and Type-offset still shifts bits toward the MLP while QKV lags. Attention error stops tracking perplexity. On QKV alone, attention-output error εA ^A tracked perplexity closely. Once the MLP shares the budget this relationship breaks: εA ^A sees only QKV and is blind to the MLP, which dominates the quantized weights. Type-offset has the worst εA ^A at 3.53.5 and 4.54.5 bits yet the best perplexity, and accuracy tracks perplexity rather than εA ^A throughout (Table ). Sublayer damage also compounds: QKV and MLP errors multiply once both are quantized, so at 3 bits the composed model is substantially farther from fp32 than either sublayer alone would suggest — consistent with each MLP being calibrated on an already-quantized attention block. Block-local fine-tuning wins locally, loses globally. Adding the joint STE stage to uniform assignments raises perplexity at every budget — 16.35×16.35× at 3 bits — even as block-reconstruction loss falls several-fold and the held-out gate commits nearly every block. The stage minimizes block-output MSE while perplexity is a causal, end-to-end quantity that no block-local loss can see; the held-out gate certifies exactly the updates responsible for the damage. No fine-tuned arm beats its GPTQ-only counterpart: GPTQ-only Type-offset at 4.54.5 bits (28.21128.211) beats fine-tuned uniform 4-bit (30.6530.65). Table 15: GPT-2 with QKV and MLP quantized together (C4 calibration, GPTQ-only unless noted): WikiText-2 perplexity (fp32 control 24.35724.357), accuracy, attention error εA ^A, weight error εW ^W, and ρ=εA/εWρ= ^A/ ^W. C4 (in-domain, control 32.77032.770) gives the same ranking; full C4 numbers are in Appendix (Table ). Best per budget in bold. Allocator B PPL Acc εA ^A εW ^W ρ Uniform 3.0 141.821 0.232 0.414 0.464 0.892 JAB 3.5 75.009 0.296 0.267 0.383 0.697 JAB+prop 3.5 67.337 0.307 0.320 0.379 0.845 Type-offset 3.5 64.896 0.310 0.421 0.303 1.390 All (flat) 4.0 29.308 0.393 0.238 0.210 1.137 JAB 4.5 29.041 0.394 0.218 0.188 1.159 JAB+prop 4.5 28.694 0.397 0.194 0.210 0.924 Type-offset 4.5 28.211 0.395 0.237 0.144 1.645 Uniform 6.0 24.546 0.413 0.070 0.048 1.449 JAB 6.0 26.868 0.404 0.071 0.152 0.470 JAB+prop 6.0 25.899 0.407 0.121 0.141 0.859 Type-offset 6.0 27.441 0.400 0.245 0.128 1.914 J.6 GPT-2: C4 Results for QKV+MLP Section reports the GPT-2 QKV+MLP results on WikiText-2 (Table ) and states that C4, the in-domain calibration corpus, gives the same ranking. Table gives those C4 numbers in full. Type-offset leads at 3.53.5 bits (78.23978.239) and 4.54.5 bits (37.04537.045), uniform wins at 6.06.0 bits (32.93532.935), and attention error εA ^A is again uninformative — Type-offset has the worst εA ^A at 3.53.5 and 4.54.5 bits while still leading on perplexity and accuracy — reproducing every qualitative claim of Section on a disjoint, in-domain corpus. Table 16: GPT-2 with QKV and MLP quantized together, evaluated on C4 (in-domain; fp32 control 32.77032.770), matching the WikiText-2 results of Table . Same allocators, budgets B (bits/parameter), and metrics. Best per budget in bold. Allocator B PPL Acc εA ^A εW ^W ρ Uniform 3.0 155.246 0.220 0.400 0.448 0.894 JAB 3.5 83.311 0.279 0.261 0.367 0.711 JAB+prop 3.5 81.408 0.280 0.316 0.371 0.852 Type-offset 3.5 78.239 0.287 0.421 0.303 1.390 All (flat) 4.0 39.053 0.352 0.237 0.205 1.160 JAB 4.5 38.626 0.355 0.219 0.183 1.195 JAB+prop 4.5 38.132 0.356 0.193 0.206 0.936 Type-offset 4.5 37.045 0.357 0.237 0.144 1.642 Uniform 6.0 32.935 0.373 0.070 0.047 1.496 JAB 6.0 35.725 0.364 0.068 0.149 0.458 JAB+prop 6.0 34.217 0.367 0.120 0.138 0.871 Type-offset 6.0 36.334 0.361 0.248 0.128 1.939 J.7 Per-Unit Traces and the 3-Bit Floor Estimator With a Rademacher probe v over the concatenated seven-unit vector and HvHv the existing Hessian-vector product, let SuS_u be unit u’s coordinate range and tu=∑j∈Suvj(Hv)jt_u= _j∈ S_uv_j(Hv)_j. Since [vjvk]=δjkE[v_jv_k]= _jk, [tu]=∑j∈SuHjj=Tr(Huu)E[t_u]= _j∈ S_uH_j=Tr(H_u): the partial sums are unbiased for the diagonal-block traces, and ∑utu _ut_u is exactly the pooled estimate used elsewhere, which we assert at run time. The cross-block terms cancel in the pooled sum but not per unit, so we raise the probe count from 10 to 30; the scoring pass runs once per run, not once per arm. JAB-norm Normalizing by diag(Hi)¯ diag(H_i) favors units with flatter input spectra: without the floor it allocates substantially more bits to oprojo_proj while assigning only 22–33 bits to q,k,vq,k,v, and it performs worse than JAB at both usable budgets, in that run and in the floored run of Table . The trace mass is concentrated on Q and K Median per-unit traces across the 32 blocks are 2081.62081.6 (q), 1245.61245.6 (k), 365.5365.5 (down), 211.0211.0 (v), 160.2160.2 (up), 146.0146.0 (o) and 93.393.3 (gate). Attention thus receives 85%85\% of the median trace mass while holding 19.2%19.2\% of the quantized weights. This is a property of the objective rather than of the estimator: ℒJABL_JAB is evaluated on the attention output and post-softmax map, so perturbing an MLP matrix affects it only through the block output term. Within-block discrimination Where the pooled trace gave one value for all seven units of a block, the per-unit traces in block 5 range from 878.9878.9 for qprojq_proj to 7.57.5 for gateprojgate_proj, a 117×117× spread within a single block. Allocations at B=4.5B=4.5 Mean assigned bits per unit kind, JAB versus Type-offset: q 7.477.47 vs. 3.973.97, k 7.387.38 vs. 4.004.00, v 4.064.06 vs. 4.004.00, o 3.883.88 vs. 3.973.97, gate 3.943.94 vs. 5.125.12, up 4.094.09 vs. 4.754.75, down 4.624.62 vs. 4.004.00. JAB spends its surplus lifting q and k to nearly 8 bits; Type-offset spends it on gate and up. Both realize 4.5004.500 bits per parameter exactly, and the second allocation scores 0.2250.225 perplexity lower. Protocol and comparability One seed, floor bi≥3b_i≥ 3 so the candidate set is 3,4,8,16\3,4,8,16\, 30 Hutchinson probes, 128 Hessian batches, 32 WikiText-2 windows of 512 tokens. Budgets below the floor are infeasible and were skipped, so this run has no B=2.5B=2.5 column. Because the Hessian batch count differs from the run of Table (128 against 64), absolute perplexities are not interchangeable between the two tables — uniform 4-bit moves from 6.9986.998 to 6.9386.938 — and only within-table comparisons are exact. Appendix K Limitations in Full L1. Single-draw trace estimates. Ten Hutchinson probes on 2 batches, no repeated-seed variance report; in particular the block-0/block-1 trace gap driving the 3.0-bit failure is not shown to exceed sampling noise, and neither is the 4.5×4.5× MLP depth gradient of Section . L2. Scope. Sections – quantize the QKV projection and Section the MLP; Section composes all seven Mistral-7B modules (96.4%96.4\% of parameters). The composition was run on Mistral only, so the sublayer interaction is unmeasured on GPT-2. Embeddings and the output head are never quantized, capping end-to-end compression at 1.66×1.66× on GPT-2 at 4 bits. L3. Greedy allocation is unverified at full scale. Its optimality gap was never measured, exhaustive enumeration being intractable, so how much of the 3.0-bit failure is the criterion and how much the solver is unknown; that greedy and MCKP/Pareto agree exactly on Mistral is suggestive, not a proof. L4. The joint-versus-separate gap is unmeasured. Section shows the cross-blocks Γ are structurally non-zero and predicts what a per-matrix baseline would look like, but no Q/K/V-separate arm was run and ‖Γ‖F/‖Hℒ‖F\| \|_F/\|H_L\|_F was never estimated, so the size of the effect on real blocks is unknown. L5. The MLP fine-tuning result is confounded. Section records a zero flip rate in the deepest blocks, the signature Section attributes to a mis-scaled learning rate. Until the grid step is confirmed to be computed per unit rather than pooled across units of differing input dimension, the weak Objective-3 result there cannot be separated from an optimizer defect. L6. Single seed; the Mistral half is convergent evidence, not a replicate. One seed per pairing, a re-implementation not checked against the full GPT-2 validation suite, a float32 rather than float64 solve (Section ), and a different εW,εA ^W, ^A normalization.