Paper deep dive
Hierarchical Latent Prediction for Language Models
Chang Shi, Tim Pearce, Manan Tomar, Siddhartha Sen, John Langford
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/8/2026, 4:05:36 AM
Summary
The paper introduces Hierarchical Latent Prediction (HiLP), a training method for language models that incorporates multi-scale self-predictive learning into transformer pretraining. HiLP uses a sliding-window attention module to create higher-level abstract latents from lower-level token latents, enabling the model to predict further into the future in the latent space. This approach mitigates compounding errors associated with standard Next-Token Prediction (NTP) and single-step latent prediction methods like NextLat. Experiments demonstrate that HiLP improves performance on coding and multi-step reasoning benchmarks and enhances speculative decoding efficiency, while the hierarchical components are removed at inference time to maintain zero architectural overhead.
Entities (10)
Relation Signals (7)
Chang Shi → affiliatedwith → University of Texas at Austin
confidence 98% · Chang Shi1... 1University of Texas at Austin
HiLP → comparedwith → NextLat
confidence 95% · We evaluate the resulting models on downstream benchmark accuracy and speculative decoding efficiency, comparing HiLP with MTP and NextLat.
HiLP → usescomponent → Sliding Window Attention
confidence 95% · A sliding-window attention (SWA) module with window k produces higher-level abstract latents
HiLP → mitigates → compounding error
confidence 92% · We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts.
Chang Shi → affiliatedwith → Microsoft Research
confidence 90% · Work done during internship at Microsoft Research.
HiLP → improves → HumanEval
confidence 90% · Results in Tab. 1 and Fig. 2 show that HiLP is improve performance from the multi-scale latent prediction... HiLP 11.33
NextLat → suffersfrom → limited horizon
confidence 85% · However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2608.05806v1
- Canonical: https://arxiv.org/abs/2608.05806v1
Trouble viewing inline? Open PDF directly →
Full Text
29,473 characters extracted from source content.
Expand or collapse full text
Hierarchical Latent Prediction for Language Models Chang Shi1, Tim Pearce2, Manan Tomar2, Siddhartha Sen2, John Langford2 1University of Texas at Austin, 2Microsoft Research Work done during internship at Microsoft Research. Correspondence to: Chang Shi <changshi@utexas.edu> Abstract While standard Next-Token Prediction (NTP) lays the foundation of language model pretraining, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency. Hierarchical Latent Prediction for Language Models Chang Shi1†thanks: Work done during internship at Microsoft Research. Correspondence to: Chang Shi <changshi@utexas.edu>, Tim Pearce2, Manan Tomar2, Siddhartha Sen2, John Langford2 1University of Texas at Austin, 2Microsoft Research 1 Introduction Next-token prediction (NTP) with teacher forcing has yielded transformers of remarkable capability, yet it also introduces a training-inference mismatch known as exposure bias: during inference time, the auto-regressive token generation relies on its own previous outputs, leading to compounding errors which could degrade long-range generation quality. Recent work NextLat (Teoh et al., 2025) partially mitigates the issue through enforcing a latent space self-prediction loss. However, its latent dynamics model predicts one step ahead. This means that the self-predictive learning signal, while effective in inducing local transition consistency, provides only indirect gradient pressure to form representations that capture structure at longer temporal scales. In practice, long-horizon dependencies still remain attenuated by sequential dynamics unrolling. Multi-scale temporal abstraction methods have a long history in sequence modeling and reinforcement learning (Sutton et al., 1999; Bacon et al., 2017). Inherently, language incorporates hierarchical structures at multiple scales, from characters, phrases to sentences, paragraphs and discourses. Therefore, it is reasonable to extend NextLat with hierarchical learning signals at different granularities, for the sake of richer latent transition dynamics. This observation motivates a natural question: can we introduce temporal hierarchy directly into the latent-space self-prediction objective, shaping the representation of a language model transformer to encode multi-scale predictive structure during pretraining? In this paper, we answer affirmatively by introducing Hierarchical Latent Prediction (HiLP). Our contributions are as follows: 1. We propose HiLP, a hierarchical representation training method that introduces multi-scale self-predictive learning into transformer pretraining via sliding-window attention over latents, a higher-level dynamics model, and a combined NTP head. 2. We evaluate the resulting models on downstream benchmark accuracy and speculative decoding efficiency, comparing HiLP with MTP and NextLat. 3. We demonstrate that the entire hierarchical apparatus can be removed at inference time with no architectural overhead, preserving NextLat’s option of being a pure training-time intervention. 2 Methodology We use X1:TX_1:T to denote the token sequence prefix. A transformer GθG_θ produces lower-level latents t=Gθ(X1:t)h_t=G_θ(X_1:t). A sliding-window attention (SWA) module with window k produces higher-level abstract latents t=SWA(t−k+1:t)u_t=SWA(h_t-k+1:t), a deterministic function of the last k lower states. A higher-level dynamics model is trained to predict k steps ahead in this abstract space. This explicit coarser-scale prediction objective encourages lookahead planning beyond single-step latent transitions. A combined NTP head then conditions the next-token prediction on both level latents, taking advantage of abstract lookahead information to improve subsequent predictions. Four parameterised maps are jointly learned: output head pθp_θ, lower dynamics pψp_ψ, upper dynamics pϕp_φ, and combined head pρp_ρ. An illustration of the architecture is shown in Fig. 1. Training objectives. The total training objective is then a weighted sum of five terms: the standard NTP loss, the NextLat (lower-level) transition consistency loss, the KL term, the higher-level transition consistency loss, and the combined NTP loss. At inference time, only the standard NTP head is used, the entire hierarchical apparatus serves purely as an auxiliary training signal. ℒntp _ntp =t<T[−logpθ(Xt+1∣t)] =E_t<T\! [- p_θ(X_t+1 _t) ] (A) ℒh _h =t[1d∑i=1dSL1(sg[t+i],^t+i)] =E_t\! [ 1d _i=1^dSL1(sg[h_t+i], h_t+i) ] (B) ℒKL _KL =t[1d∑i=1dDKL(pθsg(⋅∣sg[t+i])∥ =E_t\! [ 1d _i=1^dD_KL\! (p_θ^sg(· [h_t+i])\, \| . . pθsg(⋅∣^t+i))] . .p_θ^sg(· h_t+i) ) ] (C) ℒu _u =t[SmoothL1(sg[t+k],^t+k)] =E_t\! [SmoothL1(sg[u_t+k],\, u_t+k) ] (D) ℒcntp _cntp =t<T[ =E_t<T\! [ −logpρ(Xt+1∣t,~t)] - p_ρ(X_t+1 _t, u_t) ] (E) Where SL1(⋅,⋅)SL1(·,·) denotes the SmoothL1 loss, sg[⋅]sg[·] the stop-gradient operator, and ~t:=SWA(sg[t−k+1:t]) u_t\!:=\!SWA(sg[h_t-k+1:t]) the higher-level latent with sgsg applied inside SWA so gradients do not flow back into h. We use this stop-gradient in the combined NTP path to keep the token-level representation th_t governed by the standard NTP and lower-level transition consistency losses, while allowing the combined head to train the SWA module and its higher-level representation. Without this separation, the combined-head cross-entropy would also update the lower latents through the SWA window, double-counting token-level supervision and creating a competing optimization signal for th_t. Consistency conditions. At optimality, the losses enforce: pθ(Xt+1∣t) p_θ(X_t+1 _t) =ℙ(Xt+1∣X1:t) =P(X_t+1 X_1:t) (I) pψ(t+1∣t,Xt+1) p_ψ(h_t+1 _t,X_t+1) =ℙ(t+1∣X1:t+1) =P(h_t+1 X_1:t+1) (I) pϕ(t+k∣t) p_φ(u_t+k _t) =ℙ(t+k∣X1:t) =P(u_t+k X_1:t) (I) pρ(Xt+1∣t,t) p_ρ(X_t+1 _t,u_t) =ℙ(Xt+1∣X1:t) =P(X_t+1 X_1:t) (IV) where (I) and (IV) enforce next-token consistency, (I) and (I) enforce transition consistency. Figure 1: Overview of HiLP with sliding-window size W=3W=3. The higher-level latent is computed by sliding-window attention over backbone latents, the higher next-latent predictor maps utu_t to u^t+W u_t+W without token input, and both the standard NTP head and combined NTP head are trained, but the combined NTP head is dropped during inference to ensure no slow down in inference speed. 3 Experiments 1B-parameter models are trained on 100B tokens using 8 × NVIDIA B200 GPUs. 3.1 Coding and Multi-step Reasoning Benchmarks After pretraining, we use LM Evaluation Harness (Gao et al., 2024) to evaluate the zero-shot performance of the models on HumanEval coding benchmark Chen et al. (2021), and DataComp for LLMs (Li et al., 2025) to evaluate the models on a set of symbolic and multi-step benchmarks. Results in Tab. 1 and Fig. 2 show that HiLP is improve performance from the multi-scale latent prediction. Model HumanEval pass@1 NTP 8.77 MTP 9.21 NextLat 10.58 HiLP 11.33 Table 1: Code-generation benchmark accuracy on HumanEval. HumanEval results are computed from 1000 samples per task. Figure 2: DCLM symbolic/multi-step eval results Figure 3: Latent cross-entropy curves for NextLat and HiLP on web and code splits. HiLP has comparable near-term latent prediction loss and lower long-horizon future and excess cross-entropy, indicating more stable multi-step latent rollout. 3.2 Speculative Decoding We evaluate speculative decoding on held-out validation splits of code and Nemotron-Climbmix data(Diao et al., 2026). For each model, we report the average number of accepted tokens per drafting step, and Avg match, the per-position draft-verifier argmax agreement averaged over K=1…4K=1…4. Note that K=0K=0 NTP position is always accepted and is omitted from the per-K columns; Model Avg Acc. Tok.↑ Avg match↑ K=1K=1 K=2K=2 K=3K=3 K=4K=4 K=5K=5 K=6K=6 K=7K=7 K=8K=8 MTP 2.57 0.545 0.707 0.633 0.531 0.310 – – – – NextLat 3.26 0.580 0.918 0.652 0.422 0.328 0.309 0.285 0.234 0.234 HiLP (NTP head) 3.41 0.592 0.938 0.645 0.414 0.367 0.324 0.324 0.293 0.340 Table 2: Speculative decoding results evaluated on a held-out validation split of code data. Model Avg Acc. Tok.↑ Avg match↑ K=1K=1 K=2K=2 K=3K=3 K=4K=4 K=5K=5 K=6K=6 K=7K=7 K=8K=8 MTP 2.11 0.162 0.301 0.141 0.109 0.098 – – – – NextLat 2.57 0.483 0.641 0.504 0.410 0.375 0.324 0.312 0.246 0.254 HiLP (NTP head) 2.59 0.491 0.699 0.492 0.379 0.395 0.328 0.406 0.289 0.289 Table 3: Speculative decoding results evaluated on a held-out validation split of Nemotron-ClimbMix data. 4 Discussion Longer horizon prediction. The latent prediction cross-entropy losses across rollout horizons up to 8 steps ahead are shown in 3. As the curves show, HiLP preserves the near-term cross-entropy of NextLat while producing lower future prediction error at longer horizons, suggesting that the hierarchical latent mitigates multi-step error accumulation. 5 Related Work Our work sits at the intersection of several active research threads: pretraining objectives beyond next-token prediction, latent-space language modeling, and hierarchical models. 5.1 Pretraining Objectives Beyond Next-Token Prediction The next-token prediction objective has largely limited the capacity of language models in downstream tasks that require longer-horizon reasoning and planning(Bachmann and Nagarajan, 2024; Nagarajan et al., 2025). Recent works have started to introduce auxiliary learning signals to mitigate this myopic gap through further future prediction (Gloeckle et al., ; Ahn et al., 2025; Teoh et al., 2025; Mahajan et al., 2025). However, these approaches usually operate on a single granularity scale and have limited horizon capability due to compounding error. HiLP differs in two ways: it introduces two latent prediction pathways at different temporal scales, so the coarser pathway supervises long-horizon structure directly rather than through repeated single-step unrolling; and all auxiliary heads are dropped at inference, so the longer-horizon signal costs nothing at deployment time. 5.2 Latent-Space Language Modeling Another line of work moves language generation itself into a continuous latent space. Large Concept Models (Barrault et al., 2024) perform autoregressive prediction over sentence-level embeddings, treating each sentence as a “concept” in a shared representation space. CALM (Shao et al., 2025) replaces next-token prediction with next-vector prediction, compressing a chunk of K tokens into a single continuous vector to raise the semantic bandwidth of each generative step. Coconut Shao et al. (2025) lets the model reason in latent space by feeding its own hidden state back as the next input embedding instead of decoding to tokens. All of these approaches change the inference-time generation process to operate on latents, requiring bespoke decoding procedures or latent-to-text decoders. HiLP is the opposite design point: latent prediction is used purely as an auxiliary training signal to shape representations, while inference remains standard token-level autoregressive decoding with zero added latency. 5.3 Hierarchical Models Hierarchical model design has been researched in both language modeling area and other domains. MegaByte (Yu et al., 2023) stacks a global patch-level transformer over a local byte-level one, the Byte Latent Transformer (Pagnoni et al., 2025) dynamically segments bytes into entropy-based patches that serve as the units of computation, and H-Net Hwang et al. (2025) learns content-dependent chunking end-to-end within a hierarchical U-Net-like network. In model-based reinforcement learning, Hierarchical Planning with Latent World Models (Zhang et al., 2026) learns world models at multiple temporal scales within a shared latent space, using long-horizon latent predictions as subgoals for short-horizon control. In representation learning theory, Learning Discrete Concepts in Latent Hierarchical Models (Kong et al., 2024) formalizes concepts as discrete latent variables organized in a hierarchical causal model and derives identifiability conditions for recovering such hierarchical concept structure from high-dimensional unsupervised data. In time series analysis. HiTime Tao et al. (2024) employs a hierarchical feature encoder together with a hybrid prompting strategy to align time series and text modalities, improving multivariate time series classification with large language models. These works demonstrate the benefit of hierarchical latent structure across planning, representation identifiability, and time series domains. HiLP differs in a way that the hierarchy complexity is not consumed at inference time: to our knowledge it is the first to introduce hierarchical latent prediction as a pretraining objective for autoregressive language models, where the hierarchy shapes the representation during training and is then removed entirely. 6 Conclusion HiLP introduces temporal hierarchy into the latent-space self-prediction objective, shaping the representation of language model transformers to encode multi-scale predictive structure during pretraining. Experiments show that it improves both speculative decoding and language models on tasks that need longer-horizon reasoning and planning. 7 Future Work In current work, the abstract latent prediction horizon is a manually set hyperparameter, which limits the flexibility of lookahead planning. Further designs including dynamically choosing the lookahead horizon could be beneficial. Also, the NTP head is used during inference in the current implementation, but combined NTP can also be used if we trade speed for accuracy. References K. Ahn, A. Lamb, and J. Langford (2025) Efficient joint prediction of multiple future tokens. arXiv preprint arXiv:2503.21801. Cited by: §5.1. G. Bachmann and V. Nagarajan (2024) The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963. Cited by: §5.1. P. Bacon, J. Harb, and D. Precup (2017) The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, Vol. 31. Cited by: §1. L. Barrault, P. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale, et al. (2024) Large concept models: language modeling in a sentence representation space. arXiv preprint arXiv:2412.08821. Cited by: §5.2. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374 Cited by: §3.1. S. Diao, Y. Yang, Y. Fu, X. Dong, D. Su, M. Kliegl, Z. Chen, P. Belcak, Y. Suhara, H. Yin, et al. (2026) Nemotron-climb: clustering-based iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems 38. Cited by: §3.2. L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2024) The language model evaluation harness. Zenodo. External Links: Document, Link Cited by: §3.1. [8] F. Gloeckle, B. Y. Idrissi, B. Rozière, D. Lopez-Paz, and G. Synnaeve Better & faster large language models via multi-token prediction, 2024. URL https://arxiv. org/abs/2404.19737. Cited by: §5.1. S. Hwang, B. Wang, and A. Gu (2025) Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955. Cited by: §5.3. L. Kong, G. Chen, B. Huang, E. Xing, Y. Chi, and K. Zhang (2024) Learning discrete concepts in latent hierarchical models. Advances in Neural Information Processing Systems 37, p. 36938–36975. Cited by: §5.3. J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Gadre, H. Bansal, E. Guha, S. Keh, K. Arora, S. Garg, R. Xin, N. Muennighoff, R. Heckel, J. Mercat, M. Chen, S. Gururangan, M. Wortsman, A. Albalak, Y. Bitton, M. Nezhurina, A. Abbas, C. Hsieh, D. Ghosh, J. Gardner, M. Kilian, H. Zhang, R. Shao, S. Pratt, S. Sanyal, G. Ilharco, G. Daras, K. Marathe, A. Gokaslan, J. Zhang, K. Chandu, T. Nguyen, I. Vasiljevic, S. Kakade, S. Song, S. Sanghavi, F. Faghri, S. Oh, L. Zettlemoyer, K. Lo, A. El-Nouby, H. Pouransari, A. Toshev, S. Wang, D. Groeneveld, L. Soldaini, P. W. Koh, J. Jitsev, T. Kollar, A. G. Dimakis, Y. Carmon, A. Dave, L. Schmidt, and V. Shankar (2025) DataComp-lm: in search of the next generation of training sets for language models. External Links: 2406.11794, Link Cited by: §3.1. D. Mahajan, S. Goyal, B. Y. Idrissi, M. Pezeshki, I. Mitliagkas, D. Lopez-Paz, and K. Ahuja (2025) Beyond multi-token prediction: pretraining llms with future summaries. arXiv preprint arXiv:2510.14751. Cited by: §5.1. V. Nagarajan, C. H. Wu, C. Ding, and A. Raghunathan (2025) Roll the dice & look before you leap: going beyond the creative limits of next-token prediction. External Links: 2504.15266, Link Cited by: §5.1. A. Pagnoni, R. Pasunuru, P. Rodriguez, J. Nguyen, B. Muller, M. Li, C. Zhou, L. Yu, J. E. Weston, L. Zettlemoyer, et al. (2025) Byte latent transformer: patches scale better than tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9238–9258. Cited by: §5.3. C. Shao, D. Li, F. Meng, and J. Zhou (2025) Continuous autoregressive language models. arXiv preprint arXiv:2510.27688. Cited by: §5.2. R. S. Sutton, D. Precup, and S. Singh (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial intelligence 112 (1-2), p. 181–211. Cited by: §1. X. Tao, T. Pan, M. Cheng, Y. Luo, Q. Liu, and E. Chen (2024) Hierarchical multimodal llms with semantic space alignment for enhanced time series classification. ACM Transactions on Intelligent Systems and Technology. Cited by: §5.3. J. Teoh, M. Tomar, K. Ahn, E. S. Hu, T. Pearce, P. Sharma, A. Krishnamurthy, R. Islam, A. Lamb, and J. Langford (2025) Next-latent prediction transformers learn compact world models. arXiv preprint arXiv:2511.05963. Cited by: §1, §5.1. L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis (2023) Megabyte: predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems 36, p. 78808–78823. Cited by: §5.3. W. Zhang, B. Terver, A. Zholus, S. Chitnis, H. Sutaria, M. Assran, R. Balestriero, A. Bar, A. Bardes, Y. LeCun, et al. (2026) Hierarchical planning with latent world models. arXiv preprint arXiv:2604.03208. Cited by: §5.3. Appendix A HiLP Training Procedure Algorithm 1 summarizes one HiLP training step, combining the five objectives into a single backward pass. The SWA module is a causal sliding-window self-attention over the lower latents, so every position t carries its own higher-level latent tu_t summarizing the last k lower latents. All latent-space prediction targets are stop-gradiented, and both distributions in the KL term are decoded through a frozen copy of the NTP head (pθsgp_θ^sg), so each auxiliary loss shapes the representation only through its designated pathway: the rollout losses reach h through pψp_ψ, the higher-level consistency loss trains pϕp_φ and the SWA module, and the combined cross-entropy trains pρp_ρ and the SWA module alone via the stop-gradient inside ~t u_t. At inference, only GθG_θ and pθp_θ are kept: the SWA module, both dynamics models, and the combined head are discarded. Algorithm 1 One HiLP training step. E(⋅)E(·) denotes the backbone, k the SWA window size and lookahead offset, and d the rollout depth. Expectations over t average over all valid positions in the batch. 1:batch X1:TX_1:T; backbone GθG_θ with NTP head pθp_θ; lower dynamics pψp_ψ; SWA module; higher dynamics pϕp_φ; combined head pρp_ρ; loss weights λntp,λh,λKL,λu,λcntp _ntp, _h, _KL, _u, _cntp 2:1:T←Gθ(X1:T)h_1:T← G_θ(X_1:T) ⊳ one backbone pass 3:ℒntp←t[−logpθ(Xt+1∣t)]L_ntp _t [- p_θ(X_t+1 _t) ] ⊳ Eq. (A) 4:Lower-level rollout (teacher-forced tokens): 5:^t←t h_t _t for all t; ℒh,ℒKL←0L_h,L_KL← 0 6:for i=1,…,di=1,…,d do 7: ^t+i←pψ(^t+i−1,E(Xt+i)) h_t+i← p_ψ ( h_t+i-1,\,E(X_t+i) ) 8: ℒh+=1dt[SL1(sg[t+i],^t+i)]L_h += 1d\,E_t [SL1 (sg[h_t+i],\, h_t+i ) ] ⊳ Eq. (B) 9: ℒKL+=1dt[DKL(pθsg(⋅∣sg[t+i])∥pθsg(⋅∣^t+i))]L_KL += 1d\,E_t [D_KL (p_θ^sg(· [h_t+i])\, \|\,p_θ^sg(· h_t+i) ) ] ⊳ Eq. (C) 10:end for 11:Higher-level channel: 12:t←SWA(t−k+1:t)u_t (h_t-k+1:t) ⊳ abstract latent 13:^t+k←pϕ(t) u_t+k← p_φ(u_t) ⊳ k-step lookahead, no token input 14:ℒu←t[SmoothL1(sg[t+k],^t+k)]L_u _t [SmoothL1 (sg[u_t+k],\, u_t+k ) ] ⊳ Eq. (D) 15:Combined head (sg keeps h governed by NTP and rollout): 16:~t←SWA(sg[t−k+1:t]) u_t (sg[h_t-k+1:t] ) 17:ℒcntp←t[−logpρ(Xt+1∣t,~t)]L_cntp _t [- p_ρ(X_t+1 _t, u_t) ] ⊳ Eq. (E) 18:ℒ←λntpℒntp+λhℒh+λKLℒKL+λuℒu+λcntpℒcntpL← _ntpL_ntp+ _hL_h+ _KLL_KL+ _uL_u+ _cntpL_cntp 19:One optimizer step on ∇ℒ w.r.t. θ, ψ, ϕφ, ρ, and the SWA module Appendix B Experiment hyperparameters Group Hyperparameter Value Backbone model Vocabulary size 100,352 Training sequence length 8,192 Hidden layers 24 Attention heads 16 Latent input combination GLU-cross Architecture Hidden multiplier 2 Latent window size 4 Latent lookahead steps 4 NTP loss weight 1.0 MSE loss weight 10.0 Loss weights KL loss weight 0.1 Higher-level MSE loss weight 1.0 Combined NTP loss weight 0.5 Input combination GLU-cross Combined head Downstream LM head NTP Draft LM head NTP Optimization Latent learning rate 1.0×10−31.0× 10^-3 Latent evaluation Evaluation horizon 8 Table 4: HiLP hyperparameter configuration. Appendix C Latent overhead and efficiency. Table 5 compares the four 100B-token code-data runs along three axes: total parameter count, the subset of parameters actually used during greedy NTP decoding, and training throughput. All four models share the same 1.06 B-parameter trunk and tied-free LM head, so the verifier path used at inference is identical and the additional parameters in MTP, NextLat, and HiLP are auxiliary draft components that are not required to produce the next-token distribution. NextLat adds a small (≈ 19 M) next-latent predictor on top of the trunk, while HiLP and MTP each add a ≈ 190–200 M draft module. Training throughput scales inversely with the size of the auxiliary loss graph: the NTP baseline reaches ≈ 126 K tokens/s/GPU, NextLat retains ≈ 83% of that throughput, HiLP retains ≈ 65%, and MTP drops to ≈ 49% because its four future-token heads must each be evaluated against the LM logits at every step. Model # Params Inf. Params Training Step Breakdown (ms) Training Throughput Step Fwd Bwd Optim Tok/s/GPU↑ TF/GPU↑ MFU↑ Samp/s↑ NTP 1.06 B 1.06 B 449 119 293 37 126,278 898 0.399 123.3 MTP 1.25 B 1.06 B 916 631 243 43 61,789 509 0.226 60.3 NextLat 1.08 B 1.06 B 542 205 295 42 104,505 884 0.393 102.1 HiLP 1.27 B 1.06 B 690 329 315 46 81,653 780 0.347 79.7 Table 5: Parameter counts and training efficiency for the 100B-token runs on 8×8\!×\!B200 GPUs. # Params: total trained parameters. Inf. Params: verifier path (trunk + LM head) used during greedy NTP decoding, identical across all models. Step breakdown: timing/step_time_sec and its components (forward, backward, optimizer), in milliseconds. Throughput: tokens/s/GPU, TFLOP/s/GPU, model FLOPs utilization, and samples/s (all GPUs). Appendix D Input-Combination Modes We include an implementation-level ablation comparing the historical concat input-combination mode against the current glu_cross mode used by latent prediction modules. Let D denote the hidden size, m the latent-head hidden multiplier, and V the vocabulary size. In the lower next-latent predictor, concat forms [t;et+1]∈ℝ2D[h_t;e_t+1] ^2D and feeds it directly to the SwiGLU predictor. This gives an intermediate width of 2mD2mD and costs 10mD210mD^2 parameters and multiply-adds per token for the gate, up, and down projections, ignoring biases. By contrast, glu_cross computes Wht⊙σ(Weet+1)∈ℝD,W_hh_t σ(W_ee_t+1) ^D, adding two D×D× D projections but reducing the following SwiGLU width to mDmD. Its corresponding cost is therefore (3m+2)D2(3m+2)D^2. With the default m=4m=4, this is 14D214D^2 versus 40D240D^2, or about 35%35\% of the concat predictor cost. The same distinction appears in the combined NTP head. For concat, the combined head maps [t;t]∈ℝ2D[h_t;u_t] ^2D to logits, costing 2DV2DV. For glu_cross, the gated fusion costs 2D22D^2 and the logits are produced from a D-dimensional vector, costing DVDV, for a total of DV+2D2DV+2D^2. Thus glu_cross is especially attractive when V≫DV D: it approximately halves the combined-head logit computation while also reducing the latent predictor from a 2D2D-wide to a D-wide SwiGLU input. Mode NTP loss↓ Step (ms)↓ concat 4.113 919.5 glu_cross 3.942 818.9 Table 6: Input-combination ablation comparing next-token prediction loss and training step time. Figure 4: concat input-combination mode used by the latent and combined NTP heads. Figure 5: glu_cross input-combination mode, which gates a projected lower latent with the conditioning latent while preserving hidden width D.