Paper deep dive
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Ilya Koziev, Leonid Sinev, Ivan Oseledets
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.15504v1
- Canonical: https://arxiv.org/abs/2609.15504v1
Trouble viewing inline? Open PDF directly →
Full Text
23,781 characters extracted from source content.
Expand or collapse full text
Losslessness in Speculative Decoding: The Orthrus Case How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus Journal: Preprint. Under review Ilya Koziev Leonid Sinev Ivan Oseledets Abstract Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors’ checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance. 1 Introduction Autoregressive (AR) language models have become the dominant paradigm for text generation, but their decoding procedure remains inherently sequential. Given a generated prefix, the model must compute the next-token distribution before the next token can be appended to the context. Consequently, generating a sequence of N tokens generally requires N decoding steps and repeatedly accesses the growing key-value (KV) cache. This sequential dependency limits hardware utilization and makes inference increasingly expensive as model and context lengths grow. Diffusion language models and related parallel decoding methods address this bottleneck by predicting multiple future tokens simultaneously. However, relaxing the strict autoregressive dependency can introduce discrepancies from the original model distribution. Orthrus Nguyen et al. (2026) proposes a particularly attractive alternative: rather than replacing or substantially modifying the pretrained autoregressive model, it augments a frozen AR backbone with a lightweight diffusion view. During inference, the AR component constructs the context representation while the diffusion component predicts multiple future tokens in parallel. An intra-model consensus mechanism then uses the autoregressive view to validate the proposed tokens. The authors report substantial inference acceleration while claiming that this procedure is strictly lossless, i.e., that the accelerated system preserves the exact predictive behavior of the original autoregressive model. The present work investigates this losslessness claim experimentally. We independently implement the Orthrus architecture and training procedure and develop a configurable training framework that enables systematic investigation of training objectives, data distributions, and hyperparameters. This investigation leads to two observations. First, the distribution of the data used to train the diffusion view is important for parallel decoding efficiency. Because the diffusion component is trained under teacher forcing, training on generic human-written continuations does not necessarily reproduce the states that the frozen AR model encounters during its own generation process. We therefore construct a teacher-generated distillation corpus by prompting the frozen AR model and recording its greedy-decoded continuations. The resulting data follows the model’s own prediction trajectories. Our independently trained model achieves higher Tokens Per Forward (TPF) than the released checkpoint in most evaluation domains. Second, and more importantly, our experiments reveal that the practical observation of losslessness depends on numerical precision. The consensus mechanism of Orthrus can provide exact trajectory equivalence under an idealized arithmetic model, but an implementation using finite-precision arithmetic need not reproduce the reference computation bit-for-bit. Because generation is discrete, even small numerical differences can eventually change the selected token and lead to divergent trajectories. We demonstrate that this effect occurs in practical Orthrus inference. When generated sequences are compared directly against those of the corresponding frozen AR model, we observe non-zero rates of output divergence under BF16 inference. This finding is particularly relevant because Orthrus explicitly characterizes its generation procedure as strictly lossless. The observation does not imply that the consensus mechanism is ineffective: Orthrus can still retain extremely high fidelity while providing substantial acceleration. Rather, it shows that the term “lossless” requires a more precise operational definition when applied to neural inference systems implemented with finite-precision arithmetic and stateful KV cache. Interestingly, the numerical differences do not necessarily manifest as a degradation in standard task-level evaluation. In some experiments, Orthrus obtains benchmark scores that are slightly higher than those of the corresponding autoregressive model. This observation further illustrates why benchmark-level equality cannot establish exact inference equivalence: two systems may obtain identical or statistically indistinguishable task scores while producing different token sequences. Conversely, a small numerical perturbation may occasionally move a generation toward a benchmark-preferred answer. Our contributions are therefore threefold. First, we provide an independent implementation of Orthrus training and inference and evaluate it alongside the released checkpoint. Second, we show that an independently trained model using teacher-generated, on-policy distillation data achieves competitive or higher Tokens Per Forward across most of the evaluated domains. Third, we show that exact sequence-level equivalence is highly sensitive to numerical precision: substantial trajectory divergence occurs under BF16, whereas the same evaluation yields exact trajectory matching under FP32. Taken together, these results do not undermine the utility of Orthrus as an inference-acceleration technique. Instead, they clarify an important limitation of strong losslessness claims for neural decoding systems: an algorithm may preserve the intended autoregressive computation in principle while still producing different discrete outputs when implemented with finite-precision arithmetic. We therefore argue that evaluations of “lossless” language-model acceleration should specify both the operational criterion for equivalence and the numerical precision under which it is measured. 2 Training Orthrus The effects described in Section 3 and Section 4 are observed not only for the original chiennv/Orthrus-Qwen3-1.7B11 1 https://huggingface.co/chiennv/Orthrus-Qwen3-1.7B checkpoint, but also for a model of the same capacity trained independently by us using the procedure described below. The two models differ substantially in both training-data composition and training hyperparameters. These differences make the independently trained model useful for assessing whether the observed effects depend on the specific training setup of the released checkpoint. We therefore describe our training procedure below, focusing on the aspects relevant to the experiments rather than on implementation details. 2.1 Training Data The training data was obtained by distilling the autoregressive model Qwen/Qwen3-1.7B (Yang et al., 2025). It was constructed from a mixture of publicly available datasets on HuggingFace, using only their prompts. For each prompt, the response was generated by the autoregressive model using greedy decoding. Only prompts containing between 50 and 1,000 characters were retained for distillation. The resulting dataset contains 4,113,358 prompt–response samples. 2.2 Training Parameters Training was performed on eight NVIDIA H100 GPUs using CUDA 13.3.73, PyTorch 2.13.0, and Transformers 5.8.0. The training configuration was as follows: • number of epochs: 1; • initial learning rate: 2×10−42× 10^-4; • batch size: 10; • loss function: cross-entropy; • block size: 8; • number of blocks: 32; • maximum sequence length: 3,072 tokens. The training configuration differs substantially from that used in the original Orthrus experiments. 2.3 Evaluation All evaluations in this section were conducted using a specially curated set of prompts, grouped into 12 text domains with 100 prompts per domain, except for “gec-en”, which contains 90 prompts. The domains are described in Table 1. This breakdown makes it possible to assess variation in generation trajectories and their statistical properties across domains. Table 2compares the Tokens Per Forward (TPF) of our model variant with the authors’ released checkpoint across the evaluation domains described above. Our model achieves a slightly higher TPF in 10 of the 12 domains, while the authors’ checkpoint performs better in the remaining two. The differences in TPF may reflect differences in training data and training configuration between the two models. Table 1: Datasets and task domains used for trajectory evaluation. Domain Data Source Task Description code me-aas/python-code-dataset-500k Python code generation in English code-ru MERA-evaluation/MERA Python code generation in Russian creative-en IsDeeCee/StoryMaker English story generation gec-en jhu-clsp/jfleg English grammatical error correction gec-ru https://github.com/ReginaNasyrova/LORuGEC Human-annotated Russian grammatical error correction math HaimingW/math_train_decontaminated English-language mathematical problem solving math-ru evilfreelancer/MATH-500-Russian Russian-language mathematical problem solving poetry-en checkai/instruction-poems English poetry generation poetry-ru n/a (closed sources) Russian poetry generation qa smd20/social-engineering-qa-english General question answering in English qa-ru MERA-evaluation/MERA General question answering in Russian wmt ru-en wmt/wmt19 Russian-to-English machine translation Table 2: Tokens Per Forward across different evaluation domains. Higher values indicate more effective parallel token generation. Values are means with 95% confidence intervals. Domain chiennv/Orthrus-Qwen3-1.7B Orthrus-1.7B-final code 3.24±0.203.24± 0.20 3.44±0.133.44± 0.13 code-ru 3.59±0.223.59± 0.22 3.93±0.133.93± 0.13 creative-en 1.85±0.091.85± 0.09 2.02±0.102.02± 0.10 gec-en 5.00±0.305.00± 0.30 4.23±0.174.23± 0.17 gec-ru 2.48±0.182.48± 0.18 3.23±0.173.23± 0.17 math 8.08±0.658.08± 0.65 4.99±0.164.99± 0.16 math-ru 4.04±0.284.04± 0.28 4.35±0.144.35± 0.14 poetry-en 1.61±0.041.61± 0.04 1.86±0.041.86± 0.04 poetry-ru 1.87±0.191.87± 0.19 2.37±0.142.37± 0.14 qa 2.01±0.062.01± 0.06 2.24±0.062.24± 0.06 qa-ru 1.58±0.141.58± 0.14 2.00±0.202.00± 0.20 wmt ru-en 2.04±0.092.04± 0.09 2.46±0.122.46± 0.12 The independently trained model exhibits qualitatively similar behavior to the released checkpoint while achieving competitive or higher TPF in most domains. We next test the stronger claim of exact trajectory equivalence. 3 When Lossless Decoding Is Not Lossless This section analyzes differences between the token trajectories generated by Orthrus and the original autoregressive model. Experimental setup We use Python 3.10.12, PyTorch 2.8.0+cu128, CUDA 12.8, and Transformers 5.8.1. All experiments are performed on an NVIDIA GeForce RTX 3090 GPU with 23 GB of memory (compute capability 8.6). Models are evaluated using BF16 precision and the eager attention implementation (attn_implementation="eager"). For generation, we use the following arguments of the Transformers generate() method: • max_new_tokens=128, • do_sample=False, • temperature=0.0. Thus, all models use greedy decoding, with no sampling applied during generation. All subsequent results are presented for two variants of Orthrus: the original chiennv/Orthrus-Qwen3-1.7B and our Orthrus-1.7B-final, the training procedure for which is described in Section 2. Table 3shows the proportions of Orthrus trajectories that fully match Qwen trajectories and those that contain at least one divergence, without a breakdown by text domain. The aforementioned proportions, broken down by text domain, are shown in Table 4. Table 3: Orthrus–Qwen3 trajectory matching statistics under BF16 inference. Values are proportions with 95% confidence intervals. Model No. Trajectories Sequence Match Rate Diverging Trajectory Rate Orthrus-Qwen3-1.7B 1,190 0.45±0.030.45± 0.03 0.55±0.030.55± 0.03 Orthrus-1.7B-final 1,190 0.43±0.030.43± 0.03 0.57±0.030.57± 0.03 Table 4: Orthrus–Qwen3 trajectory matching rates by domain under BF16 inference. Values are proportions of exactly matching trajectories with 95% confidence intervals. Domain Authors Ours code 0.36±0.100.36± 0.10 0.35±0.100.35± 0.10 code-ru 0.58±0.100.58± 0.10 0.52±0.100.52± 0.10 creative-en 0.12±0.060.12± 0.06 0.11±0.060.11± 0.06 gec-en 0.88±0.070.88± 0.07 0.88±0.070.88± 0.07 gec-ru 0.52±0.100.52± 0.10 0.49±0.100.49± 0.10 math 0.54±0.100.54± 0.10 0.54±0.100.54± 0.10 math-ru 0.59±0.100.59± 0.10 0.62±0.100.62± 0.10 poetry-en 0.11±0.060.11± 0.06 0.08±0.050.08± 0.05 poetry-ru 0.14±0.070.14± 0.07 0.11±0.060.11± 0.06 qa 0.25±0.090.25± 0.09 0.17±0.070.17± 0.07 qa-ru 0.73±0.090.73± 0.09 0.70±0.090.70± 0.09 wmt ru-en 0.64±0.100.64± 0.10 0.63±0.100.63± 0.10 The conditional perplexity values, denoted as PPL (response|prompt), shown in the tables were calculated by the Qwen3-1.7B model. For a prompt x and generated response y=(y1,…,y|y|)y=(y_1,…,y_|y|), the response-conditional perplexity is: PPLQwen(y∣x)=exp(−1|y|∑t=1|y|logpQwen(yt∣x,y<t)).PPL_Qwen(y x)= (- 1|y| _t=1^|y| p_Qwen (y_t x,y_<t ) ). Table 5: Mean response-conditional PPL for matching and diverging Orthrus trajectories. Values are means with 95% confidence intervals. Model Matching trajectories Diverging trajectories Orthrus-Qwen3-1.7B 1.11±0.011.11± 0.01 1.28±0.011.28± 0.01 Orthrus-1.7B-final 1.10±0.011.10± 0.01 1.28±0.011.28± 0.01 Diverging trajectories exhibit higher response-conditional perplexity under the reference model, as shown by Table 5. To test whether this association persists after accounting for response length and domain, we fit the logistic regression with exact matching as the binary response and logPPL(y∣x) (y x), response length, and domain as predictors: logitP(Y=1)=β0+β1logPPL(y∣x)+β2L+γD.logitP(Y=1)= _0+ _1 (y x)+ _2L+ _D. Here, Y=1Y=1 denotes an exact trajectory match and γD _D represents domain effects. A negative β1 _1 therefore indicates that higher response-conditional perplexity is associated with a lower probability of exact matching. Logistic regression results performed using the statsmodels (Seabold and Perktold, 2010) are shown in Table 6. The results indicate a strong negative association between response-conditional perplexity and the probability of exact trajectory matching. For the chiennv/Orthrus-Qwen3-1.7B model, the estimated coefficient for logPPL(y∣x) (y x) is β1=−8.10 _1=-8.10 (95% CI [−10.50,−5.71][-10.50,-5.71], p=3×10−11p=3× 10^-11). A similar, and somewhat stronger, association is observed for Orthrus-1.7B-final, with β1=−10.92 _1=-10.92 (95% CI [−13.60,−8.24][-13.60,-8.24], p=1.3×10−15p=1.3× 10^-15). Thus, for both implementations, responses that are assigned higher conditional perplexity by the reference Qwen3-1.7B model are substantially less likely to be reproduced exactly by Orthrus. The confidence intervals exclude zero by a wide margin, indicating that this association remains statistically significant after controlling for response length and prompt domain. These results indicate that trajectory divergence is systematically associated with higher response-conditional perplexity under the reference model, rather than occurring uniformly across inputs. Table 6: Logistic regression results for the association between response-conditional perplexity and exact trajectory matching. Model β _1 95% CI p-value chiennv/Orthrus-Qwen3-1.7B −8.10-8.10 [−10.50,−5.71][-10.50,\,-5.71] 3×10−113× 10^-11 Orthrus-1.7B-final −10.92-10.92 [−13.60,−8.24][-13.60,\,-8.24] 1.3×10−151.3× 10^-15 Despite the observed trajectory divergence, these deviations do not translate into systematic degradation in downstream task performance. 4 When Losses Become Gains We evaluated Qwen3 and both Orthrus models on GSM8K, HumanEval, and IFEval using lm-eval-harness (Gao et al., 2023). Model and generation parameters are the same as those described in Section 3. As shown in Table 7, our Orthrus model has higher point estimates than the autoregressive Qwen3 baseline on all three benchmarks. These differences should not be interpreted as statistically significant improvements given the reported uncertainty. If the original autoregressive trajectory is assumed to provide the reference behavior, deviations from this trajectory reported in Section 3 might be expected to reduce downstream performance. Instead, the observed deviations do not consistently have a negative effect and can coincide with higher task scores. Thus, trajectory divergence does not imply systematic degradation in downstream task performance. Table 7: Comparison of Qwen3-1.7B, the authors’ Orthrus-Qwen3-1.7B, and our Orthrus-1.7B-final on lm-eval-harness benchmarks. Task Metric Qwen3-1.7B Orthrus-Qwen3-1.7B Orthrus-1.7B-final GSM8K exact match (flexible) 0.4003 ± 0.0135 0.4238 ± 0.0136 0.4147 ± 0.0136 HumanEval pass@1 0.4024 ± 0.0384 0.3659 ± 0.0377 0.4146 ± 0.0386 IFEval prompt-level loose accuracy 0.2015 ± 0.0173 0.2052 ± 0.0174 0.2181 ± 0.0178 IFEval prompt-level strict accuracy 0.1682 ± 0.0161 0.1756 ± 0.0164 0.1830 ± 0.0166 5 The Effect of Numerical Precision The BF16 experiments demonstrate that neither Orthrus variant consistently reproduces the autoregressive trajectory exactly. We next ask whether this discrepancy persists under higher numerical precision. We therefore repeat the evaluation using FP32 precision for both Orthrus variants and their corresponding reference computations while keeping the model parameters, decoding procedure, and evaluation prompts unchanged. Under FP32 inference, Orthrus produces exactly the same token trajectories as the autoregressive reference on all 1,190 prompts in the trajectory evaluation (Table 8), whereas the corresponding BF16 configuration exhibits substantial trajectory divergence (Table 3). Table 8: Orthrus–Qwen3 trajectory matching statistics under FP32 inference. Model No. of Trajectories Sequence Match Rate Diverging Trajectory Rate Orthrus-Qwen3-1.7B 1,190 1.00±0.001.00± 0.00 0.00±0.000.00± 0.00 Orthrus-1.7B-final 1,190 1.00±0.001.00± 0.00 0.00±0.000.00± 0.00 These observations demonstrate that the practical behavior of Orthrus is sensitive to numerical precision. The apparent violations of exact trajectory equivalence observed under BF16 disappear when the computations are performed in FP32. We therefore attribute the observed trajectory divergence to finite-precision numerical effects, without attributing them to any particular layer or computational operation. Identifying the specific computational stages responsible for these precision-dependent deviations remains an interesting direction for future work. The sensitivity of LLM inference to numerical precision has also been observed in studies of inference reproducibility, where changes in floating- point precision and hardware configuration can alter outputs even under greedy decoding (Yuan et al., 2025). Our setting differs in that we study numerical precision specifically in the context of lossless speculative decoding: the question is not merely whether an LLM is reproducible across inference configurations, but whether an accelerated model reproduces the exact trajectory of its autoregressive reference. 6 Conclusion We investigated the losslessness claim of Orthrus by directly comparing its generated trajectories with those of the corresponding autoregressive model. Under BF16 inference, exact trajectory equivalence was not preserved: 45% of trajectories matched the released checkpoint and 43% matched our independently trained model. Trajectory matching was strongly associated with the response-conditional perplexity of the reference model, while the observed divergences did not result in systematic degradation on the evaluated downstream benchmarks. Crucially, repeating the same trajectory evaluation with FP32 yielded exact matching on all 1,190 evaluated prompts. These results indicate that the practical losslessness of Orthrus depends on numerical precision and that claims of lossless language-model acceleration should specify the numerical precision and operational criterion under which equivalence is assessed. Acknowledgements. We thank Sergei Markov for supporting this research with computational resources and Valery Ternovsky for a methodologically sound question regarding the benchmarking of our Orthrus variant. References Gao et al. (2023) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou A framework for few-shot language model evaluation. Zenodo. External Links: Document Cited by: §4. Nguyen et al. (2026) C. V. Nguyen, C. Hegde, V. C. Pham, R. A. Rossi, F. Dernoncourt, and T. H. Nguyen Orthrus: memory-efficient parallel token generation via dual-view diffusion. preprint arXiv:2605.12825. External Links: 2605.12825, Link Cited by: §1. Seabold and Perktold (2010) S. Seabold and J. Perktold Statsmodels: econometric and statistical modeling with Python. In Proceedings of the 9th Python in Science Conference, p. 92–96. External Links: Document, Link Cited by: §3. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. preprint arXiv:2505.09388. External Links: 2505.09388, Link Cited by: §2.1. Yuan et al. (2025) J. Yuan, H. Li, X. Ding, W. Xie, Y. Li, W. Zhao, K. Wan, J. Shi, X. Hu, and Z. Liu Understanding and mitigating numerical sources of nondeterminism in LLM inference. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, Main Conference, p. 169819–169851. External Links: Document Cited by: §5.