Paper deep dive

Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 47

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 8/24/2026, 5:41:55 AM

Summary

The paper introduces Quantization-Aware Healing (QAH), a method to recover the performance of large language models that have undergone both structural compression and 4-bit quantization. Unlike traditional Quantization-Aware Training (QAT) which re-fits models to hard labels, QAH distills the compressed, quantized student model directly from the original, uncompressed full-precision teacher model using KL divergence. This approach avoids the capacity ceiling imposed by intermediate recovered checkpoints and demonstrates faster convergence, greater stability, and superior or comparable benchmark performance compared to QAT baselines, while significantly reducing memory and compute requirements.

Entities (8)

Quantization-Aware Healing · method · 98%MXFP4 · quantization-format · 95%Quantization-Aware Training · method · 95%Hypernova-60B · model · 92%gpt-oss-120b · model · 90%KL-divergence · loss-function · 90%LiveCodeBench · benchmark · 85%Multiverse Computing · organization · 80%

Relation Signals (6)

Quantization-Aware Healing → uses → KL-divergence

confidence 95% · The QAH loss is a teacher–student KL divergence on output logits

Hypernova-60B → isgeneratedby → Quantization-Aware Healing

confidence 92% · the QAH student... is released open-weight as Hypernova-60B

Quantization-Aware Healing → distillsfrom → gpt-oss-120b

confidence 90% · QAH distills the 4-bit student directly from the original, uncompressed model... GPT-OSS 120B

Hypernova-60B → isquantizedinformat → MXFP4

confidence 90% · On a GPT-OSS 120B to 60B to MXFP4 pipeline... released open-weight as Hypernova-60B

Quantization-Aware Healing → outperforms → Quantization-Aware Training

confidence 90% · Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training

Quantization-Aware Healing → improves → LiveCodeBench

confidence 85% · reaches the original 120B model on LiveCodeBench

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B to 60B to MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher's parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

44,113 characters extracted from source content.

Expand or collapse full text

Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs Bakbergen Ryskulov Iker García-Ferrero David Montero David Jansen Ali Hashemi Affiliation: Jezabel R. Garcia, Antonio Tiene, Román Orús Abstract Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B→ 60B→ 4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4×4× less weight memory and half the teacher’s parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7×7× faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyperparameter search. Model (Hypernova-60B) — https://huggingface.co/MultiverseComputingCAI/Hypernova-60B-2605 1 Introduction COMPRESSION PIPELINEQAH MECHANISM Original Model uncompressed ⋅· full size capability multi-stage post-training compress &quantize Compressed Student ↓ significant capability loss compressed & quantized ⋅· 2×2× smaller Healed ↑ capability recovered compressed & quantized teacher logits frozen ⋅· no gradient ⋅· precomputed offline student logitsMXFP4 ⋅· fake quantizersKL DivergenceKL(pt∥ps)KL(p_t\,\|\,p_s)updates student weights Figure 1: QAH overview. Structural compression followed by 4-bit quantization sharply reduces capability. QAH heals the compressed, quantized student by distilling from the original uncompressed model (frozen teacher, dashed arrow), rather than re-fitting hard labels or distilling from the recovered full-precision checkpoint. The memory and compute cost of large language models (LLMs) has made low-precision, reduced-size deployment a practical necessity rather than an option. A now-common production pattern combines two compression steps in sequence: structural compression, which reduces a model to a fraction of its original parameter count by modifying its architecture, and 4-bit quantization, which further cuts memory and inference cost. Applied together, these steps deliver large efficiency gains, but at a measurable cost in reasoning, mathematical problem-solving, code generation, and long-context ability. For models that have already been through multi-stage post-training (supervised fine-tuning, RLHF/RLAIF, model merging, agentic-behavior tuning), this degradation compounds across stages and must be recovered before the model can be shipped. We refer to this recovery step as healing. The default healing recipe is quantization-aware training (QAT) (10; 17), which inserts fake-quantizers into the forward pass and continues training under a task (cross-entropy) loss. In our deployment setting we found QAT to be both expensive and operationally fragile: it converges slowly, and if training is allowed to continue past its peak it collapses, so shipping a QAT checkpoint safely requires careful early stopping against a held-out signal. Recent work on 4-bit inference reports the same fragility and proposes quantization-aware distillation (QAD), which replaces the task loss with a KL objective against a frozen full-precision copy of the model (23). QAD is a strong recipe when compression is quantization-only, because a full-precision counterpart of the quantized model exists by construction. Under structural compression that assumption breaks. The natural QAD teacher is the structurally compressed bfloat16 checkpoint that precedes quantization—but that checkpoint is not an independently trained model. It is a distillation-recovered approximation of the original, and it already carries the capacity loss from compression. Distilling the 4-bit student from it anchors the student to a degraded target and caps its accuracy at the recovered checkpoint’s ceiling. This left us with a concrete deployment question: how should we heal a model that has been both structurally compressed and 4-bit quantized? Our answer, and the recipe this paper documents, is Quantization-Aware Healing (QAH). QAH heals the compressed, quantized student by distilling from the original, uncompressed model—a strictly stronger teacher—rather than from the recovered checkpoint (Figure 1). The teacher and student no longer share an architecture; the student is supervised only through the teacher’s output distribution, which is architecture-agnostic. Concretely, on a two-stage pipeline that compresses a GPT-OSS 120B model to 60B and then re-quantizes it to MXFP4, the QAH student matches or exceeds its own bfloat16 source on 7 of 9 benchmarks, and reaches the original 120B model on LiveCodeBench—while using roughly 4×4× less weight memory and running at half the teacher’s parameter count. The practical reading is that the quantization stage is not a lossy postprocessing step to be minimized, but a second opportunity to apply teacher supervision that the bfloat16 checkpoint never received. The recipe described here was used to produce Hypernova-60B, an open-weight model released by Multiverse Computing under Apache 2.0; the public release incorporates further training beyond the pipeline evaluated in this paper, so the numbers we report are our own measurements of the pipeline rather than the released checkpoint’s published figures. This is an experience-and-recipe paper rather than a claim of a new algorithmic primitive: QAH combines well-understood ingredients (knowledge distillation and fake-quantized training) in a way that fits the realities of a compress-then-quantize production pipeline. Our contribution is in identifying where the standard recipe fails in that pipeline, in a recipe that works reliably without a hyperparameter search, and in the operational lessons that made it deployable. Contributions. • We identify why the standard QAD recipe is a poor fit once structural compression precedes quantization: the only available full-precision teacher is a recovered checkpoint that caps the student’s accuracy. We describe QAH, which distills from the original uncompressed model instead (§3). • On a GPT-OSS 120B→ 60B→ 4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4×4× lower weight memory, and reaches the 120B model on LiveCodeBench (§5). • We report the deployment lessons that made QAH usable in practice: it converges about 7×7× faster than a matched QAT baseline and does not require hand-tuned early stopping, and we document a large, reproducible quality gap between distributed-training backends that practitioners should be aware of (§6). 2 Background and Related Work Low-precision formats. Microscaling formats (MX, MXFP4, MXFP6, NVFP4) replace per-tensor exponent overhead with block-wise scaling factors, giving aggressive memory and compute reductions while preserving most of the bfloat16 forward pass (31; 20). MXFP4 quantizes weights only and is the format behind the public GPT-OSS release (24); NVFP4 quantizes weights and activations and is used by Nemotron 3 Nano (23). Post-training quantization (PTQ) methods—GPTQ (6), AWQ (16), SmoothQuant (42)—suffice at moderate compression ratios but leave a non-negligible 4-bit accuracy gap on reasoning and coding for models above roughly 10B parameters, which is the regime we operate in. Quantization-aware training. QAT inserts straight-through-estimator fake-quantizers into the forward pass so weights can adapt to the quantized representation (10; 15). For LLMs, LLM-QAT (17) closes much of the PTQ gap for moderate-scale dense models, and LR-QAT (5) reduces its memory overhead. All of these optimize a task loss, which forces the student to re-acquire behavior already present in the original model—a re-traversal that is doubly costly when the student has also been structurally modified. We use QAT as our baseline (§3.3). Knowledge distillation. Distillation (9) trains a student to match a teacher’s output distribution rather than rediscover it from labels; for LLMs, MiniLLM (8) and DistiLLM (14) show it outperforms supervised fine-tuning on reasoning and instruction following. Crucially, distillation does not require teacher and student to share an architecture—cross-architecture distillation is standard, from BERT compression (DistilBERT (33), TinyBERT (13)) to vision-language pruning (38). QAH relies on exactly this property, since its teacher (the uncompressed model) and student (the compressed, quantized model) differ in architecture. The idea of combining distillation with quantization itself is not new (25); our contribution is the choice of teacher in the compress-then-quantize setting. Structural compression. A complementary family of methods reduces parameter count by modifying architecture: structured pruning of heads, neurons, or layers (18; 41), outlier-aware sparsity (45; 19), embedding-dimension slicing (SliceGPT (1)), low-rank decomposition (SVD-LLM (39), ASVD (46)), and tensor-network methods (21; 43; 37). The common property is that the compressed model is never independently trained at full precision: its weights are derived from the original via a compression operator, and any full-precision version of it is itself a distillation-recovered approximation. This is precisely why a QAD teacher taken from that checkpoint is suboptimal, and why healing under structural compression calls for a different teacher—the original pre-compression model. Quantization-aware distillation. QAD, formalized for NVFP4 by 23, starts from the original bfloat16 model as a frozen teacher and trains a fake-quantized student to match its logits via KL divergence. The reported advantages over QAT are no re-traversal of post-training, robustness to data coverage, and single-stage recovery to near-full-precision accuracy. Standard QAD presupposes architectural identity between teacher and student, which holds when compression is quantization-only but not when structural compression comes first. QAH targets that compound-compression regime by distilling directly from the original model, making no assumption about shared layer structure. To our knowledge, healing for models that have undergone both structural compression and low-precision quantization has not been studied. 3 Method 3.1 The Compound-Compression Regime Let ℳHPM_HP be the original, uncompressed model, which has already completed multi-stage post-training (SFT, RLHF/RLAIF, and any downstream behavior tuning). We call the pipeline of structural compression followed by low-precision quantization the compound-compression regime. In it, ℳHPM_HP undergoes three modifications before deployment. Structural compression. A compression operator S reduces the parameter count of ℳHPM_HP by modifying its architecture—removing attention heads, feed-forward neurons, or transformer layers, or reducing embedding dimensionality. The output is a compressed intermediate ℳ=⁡(ℳHP)M_C=S(M_HP), 2×2× smaller in our experiments. ℳM_C is never trained at full precision: its weights are derived from ℳHPM_HP by the operator, and no full-precision checkpoint of the compressed architecture is trained from scratch. Full-precision recovery. Before quantization, ℳM_C is recovered in bfloat16 to a checkpoint ℳBF16M_BF16 by distilling from ℳHPM_HP via KL divergence. This adapts the compressed weights to recover capability lost to compression. It is not independent training: the recovered checkpoint is tethered throughout to ℳHPM_HP’s output distribution and inherits the capacity ceiling of the compressed architecture. Quantization. A quantization operator Q casts ℳBF16M_BF16 to MXFP4, yielding ℳFP4=⁡(ℳBF16)M_FP4=Q(M_BF16). The full pipeline ℳFP4=⁡(R⁡(⁡(ℳHP)))M_FP4=Q(R(S(M_HP))), with R the recovery operator, defines the regime. Quantization adds a further accuracy gap on top of the compression loss. Healing goal. Healing returns ℳFP4′=H⁡(ℳFP4)M_FP4 =H(M_FP4) such that (i) ℳFP4′M_FP4 approaches ℳHPM_HP on a target evaluation suite, (i) ℳFP4′M_FP4 remains a valid MXFP4 deployment artifact, and (i) healing does not regress a broader sanity suite. The defining feature of healing, as opposed to generic fine-tuning, is that ℳHPM_HP exists and is accessible: the target behavior is observable on a frozen teacher, so the student need not rediscover it from labels. 3.2 Quantization-Aware Healing QAH replaces the recovered checkpoint ℳBF16M_BF16 as teacher with the original model ℳHPM_HP. Teacher and student now have different architectures: ℳHPM_HP is full-size and uncompressed, while the student ℳFP4M_FP4 is structurally compressed. Notably, the advantage comes from the teacher being uncompressed, not from a difference in precision: ℳHPM_HP is the original GPT-OSS release, whose weights are already largely MXFP4 (§4), so QAH distills a compressed 4-bit student from an uncompressed teacher of comparable precision rather than from a higher-precision one. The student is supervised only through the teacher’s output distribution. The QAH loss is a teacher–student KL divergence on output logits: ℒQAH _QAH =x∼[KL(softmax(ℳHP(x)/τ) =E_x [KL (softmax\! (M_HP(x)\,/\,τ ) ∥softmax(ℳFP4(x;Qθ)/τ))], 25.00003pt \|\;softmax\! (M_FP4(x;\,Q_θ)\,/\,τ ) ) ], (1) where QθQ_θ are straight-through-estimator MXFP4 fake-quantizers inserted into the student’s forward pass, τ is the distillation temperature (τ=1τ=1 throughout), and D is a healing corpus. The KL uses the standard next-token shift and is averaged over masked positions. Following 23, teacher logits are precomputed offline from the released (MXFP4) Hugging Face checkpoint of ℳHPM_HP, once per example, and truncated to the top-k logits (k=100k=100) to bound storage. The student sees no hard labels—only the teacher’s distribution. 3.3 Quantization-Aware Training Baseline QAT inserts the same fake-quantizers QθQ_θ but optimizes a task loss: ℒQAT=(x,y)∼[CE⁡(y,ℳFP4(x,Qθ))],L_QAT=E_(x,y) [CE\! (y,\;M_FP4(x;\,Q_θ) ) ], (2) with (x,y)(x,y) input–label pairs and CECE next-token cross-entropy. Unlike QAH, QAT does not condition on ℳHPM_HP at training time; it must re-acquire, from labels, behavior that ℳHPM_HP already encodes. 3.4 Making QAH Fit the Memory Budget Healing at 16k–32k context within a fixed GPU-memory envelope is the main engineering obstacle. A direct implementation of Eq. 1 materializes the full [B,L,V][B,L,V] student log-softmax and its autograd graph, whose peak memory O⁡(BLV)O(BLV) becomes prohibitive at L=32L=32k with a vocabulary V of order 2×1052×10^5. We instead adopt the offline top-K logits and fused chunked-KL implementation of 32: the teacher’s top-k logits are cached once rather than recomputed each step, and the KL loss and its gradient are accumulated block-by-block along the sequence, so peak intermediate memory becomes linear in sequence length rather than proportional to vocabulary size while remaining bit-identical to the dense loss. This is what lets QAH train at 32k context on the same hardware as short-context QAT; we refer the reader to that work for the derivation and kernel details. We additionally apply a two-phase masking schedule—an attention-mask-only stability window, then assistant-focused masking for the bulk of training—which further reduces effective sequence length; loss-mask-only training consistently underperformed. 4 Experimental Setup Models and pipeline. We evaluate GPT-OSS 120B and 20B (24), both released with their MoE weights (the large majority of parameters) in MXFP4 and remaining tensors in BF16; we refer to these released checkpoints as MXFP4 throughout. Our pipeline has two stages. First we compress each model—120B to 60B and 20B to 9B—using a tensor-network compression operator (36; 4; 12), then recover each compressed checkpoint by continued bfloat16 training with KL distillation from the corresponding uncompressed GPT-OSS teacher. Second, we re-quantize the recovered 60B and 9B checkpoints to MXFP4, again distilling from the uncompressed teacher (QAH) or, for the baseline, training under cross-entropy (QAT). Data. Both stages train on a mixture of NVIDIA Nemotron (22; 44) and SmolTalk 2 (2) data, covering general-domain, science, coding, mathematics, safety, and reasoning tasks. Training. All runs use 8 nodes of NVIDIA H200 GPUs with FSDP2 for parameter sharding. Both stages optimize a pure KL objective against the uncompressed teacher (QAT excepted), with assistant-focused loss masking. For the MXFP4 QAH stage we use sequence length 32k, global batch size 64, learning rate 5×10−65×10^-6, and 400 steps. Quantization-sensitive submodules (embeddings, layer norms, selected attention components) are frozen to limit destructive drift. Evaluation. We evaluate on nine benchmarks: MMLU-Pro (40) (general knowledge), GPQA Diamond (30) (science), AIME 2025 (47) (mathematics), IFBench (26) (instruction following), SciCode (35) (scientific coding), LiveCodeBench (11) (coding), τ2τ^2-bench (3) (agentic tool use), Aider (7) (agentic coding), and A-LCR (34) (long-context reasoning). 5 Results Figure 2: Benchmark performance of three checkpoints in our pipeline: the original gpt-oss-120B (grey), the compressed-and-recovered gpt-oss-60B in bfloat16 (hatched blue), and the same 60B model re-quantized to MXFP4 under QAH (red). The 120B model is the frozen teacher for both the recovery and the quantization stages. The 4-bit QAH student matches or beats its own bfloat16 source on 7 of 9 benchmarks. 5.1 Quantization as a Second Training Opportunity Figure 2 compares three checkpoints across nine benchmarks: the original gpt-oss-120B teacher, the 60B bfloat16 student recovered from compression, and the same 60B student re-quantized to MXFP4 under QAH. Naively quantizing the recovered bfloat16 checkpoint to MXFP4 without healing incurs a substantial accuracy drop that QAH must recover; as the figure shows, it more than recovers it on most benchmarks. The central practical finding is that the 4-bit checkpoint is not a degraded copy of the 16-bit one—it is generally better. The MXFP4 60B student matches or beats its bfloat16 source on 7 of 9 benchmarks; the two exceptions, MMLU-Pro and SciCode, lose by only 0.20.2 and 1.41.4 points. Gains on the rest are substantial: +7.4+7.4 on A-LCR (42.742.7 vs. 35.335.3), +5.6+5.6 on AIME 2025 (76.376.3 vs. 70.770.7), +2.7+2.7 on Aider (40.940.9 vs. 38.238.2), and +2.3+2.3 on τ2τ^2-bench (61.761.7 vs. 59.459.4). This is a direct consequence of the QAH recipe rather than of quantization per se: the quantization stage performs a second pass of KL distillation against the original 120B distribution, so the quantized student receives teacher supervision that the bfloat16 checkpoint never saw. The deployment-relevant takeaway is that in a distillation-based healing pipeline, the quantization step is not a lossy postprocessing step to be minimized but a place to add capability. We are deliberate about what this does and does not show: it does not establish that 4-bit representations are inherently superior; it shows that, given the extra distillation pass QAH performs at quantization time, the deployable 4-bit artifact is at least as capable as its bfloat16 source at a fraction of the memory (§7). Despite being half the teacher’s parameter count and quantized to 4 bits, the QAH student retains most of the teacher’s capability. On LiveCodeBench it slightly exceeds the 120B model (66.566.5 vs. 66.066.0; a difference well within run-to-run noise, so we read it as matching the teacher), and on GPQA Diamond it closes to within 1.61.6 points (67.467.4 vs. 69.069.0). MMLU-Pro and IFBench show gaps of 4.24.2 and 3.43.4 points. The largest residual gap is on A-LCR (−7.3-7.3), an extreme long-context benchmark where capacity lost to compression is hardest to recover. Efficiency. These accuracy results come with real serving wins. At 4-bit precision the QAH model uses roughly 4×4× less weight memory than the bfloat16 student, and at half the teacher’s parameter count it roughly halves compute per token, enabling deployment on substantially smaller hardware. For model families released in bfloat16 rather than 4-bit (e.g., Qwen (27)), the combined parameter and precision reduction would amount to roughly 8×8× less compute per token. QAH therefore inverts the usual low-bit trade-off: instead of trading accuracy for efficiency, it delivers a model that is at once cheaper to serve, lighter in memory, and at least as strong as its bfloat16 counterpart. 5.2 QAH vs. QAT: Cost and Stability Figure 3: Average performance of QAH and QAT on MMLU-Pro, LiveCodeBench, and GPQA Diamond as training progresses, quantizing GPT-OSS 9B to MXFP4. QAH peaks at 54.954.9 in roughly 100100 steps and stays stable through 12001200 steps. QAT reaches a comparable peak (54.654.6) only at step 700700, then collapses, losing nearly 1919 points by step 12001200. For a practitioner, the choice between healing recipes turns less on peak accuracy than on how much it costs to reach that peak and how safe the resulting checkpoint is to ship. Figure 3 compares QAH and QAT when quantizing GPT-OSS 9B to MXFP4, plotting average performance on MMLU-Pro, LiveCodeBench, and GPQA Diamond against training steps. This 20B→ 9B configuration was produced and validated for an internal client deployment (details withheld), giving us a second end-to-end setting—alongside the released 60B model—in which the QAH recipe was applied in practice. The two recipes reach a comparable peak (54.954.9 for QAH vs. 54.654.6 for QAT), but differ sharply on the two axes that matter for deployment. First, cost: QAH peaks in about 100100 steps versus roughly 700700 for QAT—about 7×7× fewer steps; the same pattern holds at 60B, where QAH peaks in roughly 400400 steps. Second, stability: QAH stays within about two points of its peak through all 12001200 steps, whereas QAT collapses once past its peak, losing nearly 1919 points by step 12001200 (from 54.654.6 at step 700700 to roughly 3636). Similar QAT-style instability has been reported in concurrent work on NVFP4 distillation (23). We attribute the asymmetry to the loss. KL distillation against a frozen teacher pins the student to the teacher’s distribution and gives it no incentive to drift once matched; the cross-entropy objective keeps pushing the quantized student toward hard labels and eventually erodes unrelated capabilities inherited from the original model. Operationally this is the difference between a recipe that can be run to convergence and shipped, and one that requires hand-tuned early stopping against a held-out signal to avoid deploying a collapsed model. In a production setting the second is a standing operational liability, and it is a large part of why we adopted QAH. 6 Deployment Lessons Beyond the recipe itself, three lessons shaped whether healing worked in practice. We report them because they cost us time to learn and are, we believe, transferable. The distributed backend is a hyperparameter. FSDP2 (48) and DeepSpeed ZeRO-3 (28; 29) are usually treated as throughput-equivalent, interchangeable backends for sharded data parallelism, and in principle the choice should not affect optimization quality. Empirically we found a large and consistent gap. Table 1 summarizes the best QAT run under each backend on the 120B student, drawn from a sweep of eleven configurations spanning context lengths 2,4,8\2,4,8\k, batch sizes 128,256\128,256\, 4,8\4,8\ nodes, and learning rates from 10−710^-7 to 5×10−45×10^-4 (full sweep in Appendix A). No DeepSpeed configuration reaches the released 120B MXFP4 baseline on GPQA Diamond, plateauing at 65.1565.15 across the entire sweep, while FSDP2 reaches 73.7473.74. The gap is largest on reasoning-heavy GPQA Diamond (−8.6-8.6 points) and persists on MMLU-Pro and AIME 2025. We suspect an interaction between DeepSpeed’s mixed-precision communication path and the MXFP4 weight encoding and leave the precise diagnosis to future work; the immediate, actionable lesson is that for MXFP4 distillation workloads the backend must be treated as a tuned hyperparameter, and we default to FSDP2. Backend GPQA:D MMLU-P AIME25 LCB DeepSpeed 65.1565.15 69.4569.45 76.6776.67 66.7966.79 FSDP2 73.7473.74 70.9570.95 80.0080.00 66.4266.42 Target (120B MXFP4) 69.069.0 78.078.0 80.080.0 66.066.0 Table 1: Best QAT run under each distributed backend on the 120B student. The DeepSpeed row is the strongest of eight configurations (8k context, lr 5×10−65×10^-6, bs 256256, 8 nodes, 5000 steps); FSDP2 is the strongest of three (2k context, lr 10−510^-5, bs 128128, 4 nodes, 2200 steps). Across the full DeepSpeed sweep no configuration exceeds 65.1565.15 on GPQA Diamond. Bold marks the per-column best among QAT runs. Freeze quantization-sensitive submodules. In our sweep, unfreezing layer norms and embedding projections during the quantization stage—especially at higher learning rates—produced checkpoints worse than the unhealed MXFP4 model. Freezing embeddings, layer norms, and selected attention components was necessary for stable healing under both QAT and QAH. Long-context healing is memory-bound, but tractable. Healing at 16k–32k context is the binding constraint on the memory envelope. The chunked-KL implementation of §3.4 is what made 32k-context QAH fit the same hardware as short-context QAT: it reduces peak intermediate memory from O⁡(BLV)O(BLV) to O⁡(BcV)O(BcV) while remaining bit-identical to the dense loss, so the long-context capability that compression damages most can actually be healed rather than skipped. 7 Limitations and Open Questions We are explicit about scope, in the spirit of reporting what did not get tested alongside what did. The direct QAD baseline is missing. Our central argument is that distilling from the original model beats distilling from the recovered bfloat16 checkpoint (standard QAD). We motivate this from the quantization literature—a quantized-and-recovered checkpoint is by construction no stronger than the model it approximates—but we did not run the head-to-head experiment (QAH vs. QAD-from-ℳBF16M_BF16 at matched configuration). It is the single most valuable experiment to add, and until it is run our claim of a recovered-teacher ceiling should be read as well-motivated but not directly measured. Single run, small-n benchmarks. Every number is a single run with no seed variance or confidence intervals, and some benchmarks are small (AIME 2025 has 30 problems), so individual deltas should be read as indicative rather than significant. Where two systems are within a point or two—most notably the LiveCodeBench comparison to the teacher—we read the result as a match, not a win. One model family, one format, one data mixture. All experiments use GPT-OSS MoE transformers (120120B→ 6060B and 2020B→ 99B) quantized to MXFP4 with a single Nemotron ++ SmolTalk mixture, owing to the high compute cost. We do not evaluate other families (Llama, Qwen, Mistral) or formats (NVFP4, INT4, FP8). Proprietary compression operator. The operator producing the 60B and 9B students is proprietary. QAH is agnostic to the compression method—it needs only logit-level access to the uncompressed teacher—but we have not verified that the gains transfer to other structural-compression approaches such as layer pruning, SliceGPT, or low-rank decomposition. 8 Conclusion We described Quantization-Aware Healing, the recipe we adopted for recovering LLMs that have been both structurally compressed and 4-bit quantized. QAH heals by distilling the compressed, quantized student from the original uncompressed model rather than from a recovered full-precision checkpoint. On a GPT-OSS 120B→ 60B→ 4 pipeline the resulting 4-bit model matches or beats its own bfloat16 source on 7 of 9 benchmarks and reaches the 120B model on LiveCodeBench, at roughly 4×4× lower weight memory and half the parameter count. Against a matched QAT baseline it converges about 7×7× faster and, unlike QAT, does not collapse under continued training—so it can be shipped without hand-tuned early stopping. Alongside the recipe we reported the operational lessons that made it work, including a large and reproducible quality gap between distributed-training backends, and we were explicit about the comparison we did not run. For practitioners, the practical message is that in a distillation-based healing pipeline the quantization step is not a cost to be minimized but an additional opportunity for teacher supervision, yielding a model that is simultaneously cheaper to serve, lighter in memory, and at least as accurate as its full-precision counterpart. The recipe is not merely a research prototype: it produced the open-weight Hypernova-60B release and a separate model built for an internal client deployment, both healed with the pipeline described here. References Ashkboos et al. (2024) S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman SliceGPT: compress large language models by deleting rows and columns. In Proceedings of ICLR, External Links: Link Cited by: §2. Bakouch et al. (2025) E. Bakouch, L. Ben Allal, A. Lozhkov, N. Tazi, L. Tunstall, C. M. Patiño, E. Beeching, A. Roucher, A. J. Reedi, Q. Gallouédec, K. Rasul, N. Habib, C. Fourrier, H. Kydlicek, G. Penedo, H. Larcher, M. Morlon, V. Srivastav, J. Lochner, X. Nguyen, C. Raffel, L. von Werra, and T. Wolf SmolLM3: smol, multilingual, long-context reasoner. Note: https://huggingface.co/blog/smollm3 Cited by: §4. Barres et al. (2025) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan Tau2-bench: evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982. Cited by: §4. Bercovich et al. (2024) A. Bercovich, T. Ronen, T. Abramovich, N. Ailon, N. Assaf, M. Dabbah, I. Galil, A. Geifman, Y. Geifman, I. Golan, et al. Puzzle: distillation-based nas for inference-optimized llms. arXiv preprint arXiv:2411.19146. Cited by: §4. Bondarenko et al. (2024) Y. Bondarenko, R. D. Chiaro, and M. Nagel Low-rank quantization-aware training for LLMs. arXiv preprint arXiv:2406.06385. External Links: Link Cited by: §2. Frantar et al. (2023) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh GPTQ: accurate post-training quantization for generative pre-trained transformers. In Proceedings of ICLR, External Links: Link Cited by: §2. Gauthier (2024) P. Gauthier Aider LLM Leaderboards: Polyglot Coding Benchmark. Note: https://aider.chat/docs/leaderboards/\#polyglot-leaderboardAccessed: 2024 Cited by: §4. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In Proceedings of ICLR, External Links: Link Cited by: §2. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. External Links: Link Cited by: §2. Jacob et al. (2018) B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of CVPR, External Links: Link Cited by: §1, §2. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §4. Jansen et al. (2026) D. Jansen, R. Rausch, A. Hashemi, D. Montero, and R. Orús Block removal for large language models through constrained binary optimization. arXiv preprint arXiv:2602.00161. External Links: Link Cited by: §4. Jiao et al. (2020) X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu TinyBERT: distilling BERT for natural language understanding. In Findings of EMNLP, External Links: Link Cited by: §2. Ko et al. (2024) J. Ko, S. Kim, T. Chen, and S. Yun DistiLLM: towards streamlined distillation for large language models. In Proceedings of ICML, External Links: Link Cited by: §2. Krishnamoorthi (2018) R. Krishnamoorthi Quantizing deep convolutional networks for efficient inference: a whitepaper. arXiv preprint arXiv:1806.08342. External Links: Link Cited by: §2. Lin et al. (2024) J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han AWQ: activation-aware weight quantization for LLM compression and acceleration. In Proceedings of MLSys, External Links: Link Cited by: §2. Liu et al. (2023) Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra LLM-QAT: data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888. External Links: Link Cited by: §1, §2. Ma et al. (2023) X. Ma, G. Fang, and X. Wang LLM-Pruner: on the structural pruning of large language models. In Proceedings of NeurIPS, External Links: Link Cited by: §2. McGowan et al. (2024) J. McGowan, W. S. Lai, W. Chen, H. Aldridge, J. Clarke, J. Garcia, R. Xia, Y. Liang, G. Hennequin, and A. Bernacchia Efficient model compression techniques with FishLeg. arXiv preprint arXiv:2412.02328. External Links: Link Cited by: §2. Micikevicius et al. (2022) P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, and H. Wu FP8 formats for deep learning. arXiv preprint arXiv:2209.05433. External Links: Link Cited by: §2. Novikov et al. (2015) A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov Tensorizing neural networks. In Proceedings of NeurIPS, External Links: Link Cited by: §2. NVIDIA (2025) NVIDIA Nemotron 3 Nano: open, efficient mixture-of-experts hybrid Mamba-Transformer model for Agentic reasoning. Note: Technical report External Links: Link Cited by: §4. NVIDIA (2026) NVIDIA Quantization-aware distillation for NVFP4 inference accuracy recovery. arXiv preprint arXiv:2601.20088. External Links: Link Cited by: §1, §2, §2, §3.2, §5.2. OpenAI (2025) OpenAI GPT-OSS: open-weight models model card. arXiv preprint arXiv:2508.10925. External Links: Link Cited by: §2, §4. Polino et al. (2018) A. Polino, R. Pascanu, and D. Alistarh Model compression via distillation and quantization. In Proceedings of ICLR, External Links: Link Cited by: §2. Pyatkin et al. (2026) V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi Generalizing verifiable instruction following. Advances in Neural Information Processing Systems 38. Cited by: §4. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §5.1. Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He Zero: memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, p. 1–16. Cited by: §6. Rasley et al. (2020) J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of KDD, Cited by: §6. Rein et al. (2023) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: a graduate-level google-proof Q&A benchmark. arXiv preprint arXiv:2311.12022. External Links: Link Cited by: §4. Rouhani et al. (2023) B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, S. Dusan, V. Elango, M. Golub, A. Heinecke, P. James-Roxby, D. Jani, G. Kaul, C. Liu, L. Melnick, M. Mesmakhosroshahi, A. Rodriguez, M. Schulte, R. Shafipour, L. Shao, M. Siu, P. Dubey, P. Micikevicius, M. Naumov, C. Verrilli, R. Wittig, D. Burger, and E. Chung Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537. External Links: Link Cited by: §2. Ryskulov et al. (2026) B. Ryskulov, I. García-Ferrero, D. Montero, D. Jansen, A. Hashemi, J. R. Garcia, A. Tiene, and R. Orús Offline top-k logits and a fused chunked KL loss. External Links: 2608.03796, Link Cited by: §3.4. Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. External Links: Link Cited by: §2. Team (2025) A. A. Team Artificial analysis long context reasoning benchmark(lcr). Artificial Analysis, Inc.. Cited by: §4. Tian et al. (2024) M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, et al. Scicode: a research coding benchmark curated by scientists. Advances in Neural Information Processing Systems 37, p. 30624–30650. Cited by: §4. Tomut et al. (2024) A. Tomut, S. S. Jahromi, A. Sarkar, U. Kurt, S. Singh, F. Ishtiaq, C. Muñoz, P. S. Bajaj, A. Elborady, G. del Bimbo, et al. Compactifai: extreme compression of large language models using quantum-inspired tensor networks. arXiv preprint arXiv:2401.14109, p. 12. Cited by: §4. Tomut et al. (2025) A. Tomut, S. S. Jahromi, A. Sarkar, U. Kurt, S. Singh, F. Ishtiaq, C. Muñoz, P. S. Bajaj, A. Elborady, G. del Bimbo, M. Alizadeh, D. Montero, P. Martin-Ramiro, M. Ibrahim, O. T. Alaoui, J. Malcolm, S. Mugel, and R. Orus CompactifAI: extreme compression of large language models using quantum-inspired tensor networks. In Proceedings of the 33rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), External Links: Link Cited by: §2. Wang et al. (2023) T. Wang, W. Zhou, Y. Zeng, and X. Zhang EfficientVLM: fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning. In Findings of ACL, External Links: Link Cited by: §2. Wang et al. (2024a) X. Wang, Y. Zheng, Z. Wan, and M. Zhang SVD-LLM: truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378. Note: Verify author list against arXiv before camera-ready. External Links: Link Cited by: §2. Wang et al. (2024b) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Hu, A. Zhu, B. Li, Y. Liang, M. Ding, B. Chen, J. Yang, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. In Proceedings of NeurIPS, External Links: Link Cited by: §4. Xia et al. (2022) M. Xia, Z. Zhong, and D. Chen Structured pruning learns compact and accurate models. In Proceedings of ACL, External Links: Link Cited by: §2. Xiao et al. (2023) G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han SmoothQuant: accurate and efficient post-training quantization for large language models. In Proceedings of ICML, External Links: Link Cited by: §2. Xu et al. (2023) M. Xu, Y. L. Xu, and D. P. Mandic TensorGPT: efficient compression of the embedding layer in LLMs based on the tensor-train decomposition. arXiv preprint arXiv:2307.00526. External Links: Link Cited by: §2. Yang et al. (2026) Z. Yang, Z. Liu, Y. Chen, W. Dai, B. Wang, S. Lin, C. Lee, Y. Chen, D. Jiang, J. He, R. Pi, G. Lam, N. Lee, A. Bukharin, M. Shoeybi, B. Catanzaro, and W. Ping Nemotron-cascade 2: post-training llms with cascade rl and multi-domain on-policy distillation. Cited by: §4. Yin et al. (2023) L. Yin, Y. Wu, Z. Zhang, C. Hsieh, Y. Wang, Y. Jia, M. Pechenizkiy, Y. Liang, Z. Wang, and S. Liu Outlier weighed layerwise sparsity (OWL): a missing secret sauce for pruning LLMs to high sparsity. arXiv preprint arXiv:2310.05175. External Links: Link Cited by: §2. Yuan et al. (2023) Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun ASVD: activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. External Links: Link Cited by: §2. Zhang and Math-AI (2025) Y. Zhang and T. Math-AI American invitational mathematics examination (aime) 2025. Cited by: §4. Zhao et al. (2023) Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li PyTorch FSDP: experiences on scaling fully sharded data parallel. In Proceedings of VLDB, External Links: Link Cited by: §6. Appendix A QAT Training-Systems Sweep Table 2 reports the eleven-run QAT sweep on the 120B student that underlies the backend lesson in §6. Every run starts from the same recovered checkpoint with the same frozen-submodule set, Nemotron ++ SmolTalk corpus, gradient clipping at 1.01.0, and 100-step warmup. The variables probed are the distributed backend, effective sequence length, learning rate, batch size, cluster size, and layer-norm freezing strategy. Only the FSDP2 configuration (R11) reaches the released 120B MXFP4 baseline on GPQA Diamond; every DeepSpeed configuration plateaus 3.53.5–14.214.2 points below it across a four-order-of-magnitude learning-rate range. R11 is the QAT arm used in the matched comparison in the main text. Run Backend SL lr GPQA:D MMLU-P LCB R1 FSDP2 2k 10−510^-5 69.1969.19 70.8670.86 61.9461.94 R2 DeepSpeed 4k 10−510^-5 65.1565.15 69.1569.15 66.4266.42 R3 FSDP2 2k 10−510^-5 65.6665.66 70.2870.28 64.9364.93 R4 DeepSpeed 4k 10−510^-5 65.1565.15 69.3969.39 63.8163.81 R5 DeepSpeed 8k 5×10−65×10^-6 65.1565.15 69.4569.45 66.7966.79 R6 DeepSpeed 8k 5×10−65×10^-6 63.6463.64 68.8968.89 63.4363.43 R7 DeepSpeed 8k 5×10−45×10^-4 59.0959.09 66.8666.86 56.7256.72 R8 DeepSpeed 8kf 10−710^-7 65.1565.15 69.2269.22 67.5467.54 R9 DeepSpeed 8kf 5×10−75×10^-7 58.5958.59 69.1169.11 64.1864.18 R10 DeepSpeed 8kf 10−610^-6 64.1464.14 69.2569.25 65.3065.30 R11 FSDP2 2k −10^-5 73.7473.74 70.9570.95 66.4266.42 Target (120B MXFP4) 72.7372.73 79.4779.47 68.6668.66 Table 2: QAT sweep (R1–R11) over the 120B student; best checkpoint score per metric. R11 repeats R1 with ∼ 2×2× more steps. 8kf denotes 8k context with a position-id fix. R11 is the sole run reaching the MXFP4 baseline on GPQA Diamond and is the QAT reference arm in the main text.

Instant research discovery