Paper deep dive
Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference
Leonid Sinev, Ilya Koziev, Vladislav Leshchuk
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/18/2026, 4:50:47 AM
Summary
The paper introduces Zarya, a hybrid language model architecture that combines autoregressive (AR) and masked diffusion (MDM) objectives. Zarya uses a slotted training approach with a curriculum that gradually increases slot granularity, allowing the model to learn both sequential prediction and parallel reconstruction. It supports dual-mode inference: MDM sampling with first-hitting denoising and slotted speculative decoding, both utilizing full KV cache reuse. The authors release Zarya models in 0.6B, 1.7B, and 4B sizes, demonstrating competitive performance on benchmarks like GSM8K, HellaSwag, IFEval, and MBPP.
Entities (15)
Relation Signals (12)
Zarya → combines → Autoregressive Language Models
confidence 95% · We introduce Zarya, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective...
Zarya → combines → Masked Diffusion Models
confidence 95% · We introduce Zarya, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective...
Zarya → evaluatedon → GSM8K
confidence 95% · We evaluate on standard benchmarks including GSM8K...
Zarya → evaluatedon → MBPP
confidence 95% · We evaluate on standard benchmarks including ... MBPP...
Zarya → evaluatedon → IFEval
confidence 95% · We evaluate on standard benchmarks including ... IFEval...
Zarya → evaluatedon → HellaSwag
confidence 95% · We evaluate on standard benchmarks including ... HellaSwag...
Zarya → isbasedon → Qwen3
confidence 90% · Zarya wraps a Qwen3 backbone Yang et al., 2025 with a custom Zarya class...
Zarya → supportsinferencemode → Slotted Speculative Decoding
confidence 90% · At inference, Zarya provides two distinct decoding paradigms... (ii) slotted speculative decoding...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autoregressive language models (ARMs) are constrained by sequential, left-to-right generation, while masked diffusion models (MDMs) enable parallel decoding but suffer from high computational overhead due to the inability to reuse Key-Value (KV) cache and from incoherent generation arising from learning dependencies over an intractable space of token combinations. We introduce Zarya, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture. Zarya structures training data into variable-size slots and employs a curriculum that gradually increases slot granularity, enabling a smooth transition from fine-grained AR learning to coarse-grained diffusion learning. At inference, Zarya provides two distinct decoding paradigms through a unified interface: (i) MDM sampling with first-hitting denoising, and (ii) slotted speculative decoding that interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, achieving full KV cache reuse. The training and inference regimes are fully decoupled, allowing a model trained with any configuration to be deployed in either mode. Extensive configurability --- including grouped noise patterns (Prefix Completion, Fill-In-the-Prefix, Fill-In-the-Middle), ordered sampling schedules, and noise-level permutation strategies --- enables flexible research exploration. We release Zarya models publicly in sizes 0.6B, 1.7B, and 4B, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms.
Tags
Links
- Source: https://arxiv.org/abs/2609.19868v1
- Canonical: https://arxiv.org/abs/2609.19868v1
Trouble viewing inline? Open PDF directly →
Full Text
24,247 characters extracted from source content.
Expand or collapse full text
Zarya: A Hybrid Autoregressive–Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference Leonid Sinev Ilya Koziev Vladislav Leshchuk Abstract Autoregressive language models (ARMs) are constrained by sequential, left-to-right generation, while masked diffusion models (MDMs) enable parallel decoding but suffer from high computational overhead due to the inability to reuse Key-Value (KV) cache and from incoherent generation arising from learning dependencies over an intractable space of token combinations. We introduce Zarya, a family of hybrid language models that jointly optimizes an autoregressive (AR) objective and a masked-diffusion objective within a single architecture. Zarya structures training data into variable-size slots and employs a curriculum that gradually increases slot granularity, enabling a smooth transition from fine-grained AR learning to coarse-grained diffusion learning. At inference, Zarya provides two distinct decoding paradigms through a unified interface: (i) MDM sampling with first-hitting denoising, and (i) slotted speculative decoding that interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, achieving full KV cache reuse. The training and inference regimes are fully decoupled, allowing a model trained with any configuration to be deployed in either mode. Extensive configurability — including grouped noise patterns (Prefix Completion, Fill-In-the-Prefix, Fill-In-the-Middle), ordered sampling schedules, and noise-level permutation strategies — enables flexible research exploration. We release Zarya models publicly in sizes 0.6B, 1.7B, and 4B, demonstrating performance on standard benchmarks while offering a principled integration of autoregressive and diffusion paradigms. https://github.com/ai-forever/zarya 1 Introduction Autoregressive models (ARMs) have achieved remarkable success in a wide range of natural language tasks Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023. However, their sequential, left-to-right decoding fundamentally limits inference throughput, preventing parallelization Chen et al., 2023; Cai et al., 2024. Masked diffusion models (MDMs) offer a compelling alternative by enabling parallel generation through an iterative denoising process without a fixed generation order Li et al., 2026b. However, most MDMs suffer from two critical drawbacks: (i) they exclude Key-Value (KV) caching, incurring high computational overhead during inference; and (i) they learn dependencies over an intractable space of token combinations, leading to incoherent generation Li et al., 2026a. Recent hybrid approaches have sought to bridge these paradigms. Block Diffusion (BD3-LMs) Arriola et al., 2025 interpolates between AR and MDM by grouping tokens into fixed-size blocks and unmasking them from left to right, but the block size is fixed and lacks flexibility. Esoteric Language Models (Eso-LMs) Sahoo et al., 2026 fuse AR and MDM paradigms at the loss level, enabling KV caching for MDMs and achieving faster inference than contemporary MDMs. ReFusion Li et al., 2026a elevates parallel decoding from the token level to a higher slot level, interleaving inter-slot diffusion-based selection with intra-slot autoregressive infilling, outperforming Qwen3-8B Yang et al., 2025 on GSM8K Cobbe et al., 2021 and MBPP Austin et al., 2021 while being 2.33× faster on average. We introduce Zarya, a novel hybrid architecture with the following key contributions: 1. Training with gradually increasing slot length: We partition the predicted sequence into fixed-length, consecutive sub-sequences, referred to as slots. The size of the slots gradually increased during training. For each sequence, we randomly mask several slots, reorder the input so that clean slots precede masked ones, and also permute the original order of both masked and clean slots. The model simultaneously learns AR next-token prediction on visible slots for sequential generation, and a denoising loss on the masked slots for context-aware parallel reconstruction. 2. Training–inference decoupling: Unlike prior work where the training configuration dictates the inference mode, Zarya allows any trained model to be deployed in either MDM sampling or slotted speculative decoding mode via a single inference flag, offering higher flexibility. Both modes are fully using KV cache with causal attention masks 2 Related Work Block Diffusion (BD3-LMs). BD3-LMs interpolate between AR and MDM by grouping tokens into blocks and unmasking them left-to-right Arriola et al., 2025. However, the block size is fixed, limiting flexibility. Esoteric Language Models (Eso-LMs). Eso-LMs fuse AR and MDM paradigms using causal attention, enabling exact likelihood computation and KV caching for MDMs Sahoo et al., 2026. They achieve low perplexities among diffusion models on One Billion Words LM1B; Chelba et al., 2014 and OpenWebText OWT; Gokaslan et al., 2019 datasets. ReFusion. ReFusion introduces slot-level parallel decoding, elevating generation from tokens to fixed-length slots Li et al., 2026a. It interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, reordering newly generated slots ahead of remaining masks after each iteration. This design unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot-level permutation space. However, ReFusion inference is tied only for prefix completion tasks. 3 Zarya Architecture 3.1 Model Backbone and Configuration Zarya wraps a Qwen3 backbone Yang et al., 2025 with a custom Zarya class registered with Hugging Face’s AutoModel and AutoConfig systems. We initialize Zarya from the Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B checkpoints, respectively, and fine-tune it for 1 epoch on a diverse 32M-sample instructional SFT dataset (approximately 37.7B tokens) covering mathematics, coding, and general instruction-following tasks. We release Zarya in three sizes (see Table 1). Because the Qwen3 backbone retains causal attention, masked positions cannot attend to future masked positions. Thus, the diffusion objective used by Zarya is a causal masked-reconstruction objective rather than the fully bidirectional masked-token objective commonly used in masked diffusion language models. The reordering of visible and masked positions ensures that all masked positions can attend to the visible prefix while preserving the causal attention pattern and KV-cache compatibility. Table 1: Model architecture of Zarya models Model Layers Heads (Q / KV) Tie Embedding Hidden Size Intermediate Size Context Length Zarya-0.6B 28 16 / 8 Yes 1024 3072 2K Zarya-1.7B 28 16 / 8 Yes 2048 6144 2K Zarya-4B 36 32 / 8 Yes 2560 9728 2K The architecture is parameterized with diffusion-specific hyperparameters, which extends common AR model parameters: • α0 _0 (default 0.25) and ϵε (default 0.001): parameters of the linear noise schedule αt=α0(1−t) _t= _0(1-t); • λ=diffusion_loss_proportionλ= diffusion\_loss\_proportion (default 0.5): weighting between MDM and AR losses; • sequential_shuffle / diffusion_shuffle: control slot/token shuffling for each phase; • ordered_sampling: monotonically increases pmaskp_mask left-to-right; • grouped_noise and max_span_length: enable Prefix Completion, Fill-in-the-Prefix, Fill-in-the-Middle, and random-span masking patterns; • noise_sorting: reorders tokens by mask/unmask state before the forward pass. 4 Training 4.1 Slotted Training When slotted_training=True, the forward_process() transforms each batch as follows (this regime is heavily inspired by Li et al. (2026a)): 1. Slot partitioning: Each answer is split into slots of size slot_size (from slot_size_set, e.g., for released model checkpoints [2,4,8,16,32,64][2,4,8,16,32,64]). 2. Mask sampling: For each example in batch, sample mask probability (pmaskp_mask) uniformly. This determines what fraction of slots will be masked (treated as the diffusion task). 3. Slot assignment: • AR slots: tokens remain visible; the model predicts the next token within each slot, yielding ℒseqL_seq. • MDM slots: all tokens are replaced with the special ⟨mdm_mask⟩ mdm\_mask token; the model reconstructs the original tokens, yielding ℒdifL_dif. Each token inside slot assigned pmaskp_mask value to be used in per-token normalization as 1pmask 1p_mask weighting. The final loss is a linear combination: ℒ=λ⋅ℒdif+(1−λ)⋅ℒseq,L=λ·L_dif+(1-λ)·L_seq, (1) where λ is diffusion loss proportion. This design enables the model to simultaneously learn next-token prediction (AR) and masked-token reconstruction (MDM) on the same input, with explicit slot boundaries providing a structured inductive bias. Optional flags include: • ordered_sampling: inside each slot modify per-token pmaskp_mask so that it increases left-to-right, as ℒdifL_dif is scaled with the value of 1pmask 1p_mask, this makes tne model learn that correct prediction of the slot beginning tokens is more important; 4.1.1 Slot-Size Curriculum Slot sizes evolve during training via slot_step_borders (epoch or step thresholds). For example, with slot_size_set =[2,4,8,16,32,64], =[2,4,8,16,32,64], slot_step_borders =[0.06,0.2,0.4,0.6,0.8,1.0], =[0.06,0.2,0.4,0.6,0.8,1.0], the slot size gradually increases from 2 to 64 over the course of training. This curriculum eases the model from fine-grained AR learning (small slots, many predictions) to coarse-grained diffusion learning (large slots, holistic reconstruction), providing a smooth transition between paradigms. Worth noting that increase in slot size also increases calculated loss (see Figure 1). Figure 1: Loss curve during training of Zarya-4B model. Step-by-step increasing of slot size causes increase of loss too 4.2 Non-Slotted Training When slotted_training=False, the model runs objective over raw sequences without slot partitioning (this regime is heavily inspired by Sahoo et al. (2026)): 1. For each example in batch, sample noise level (t) uniformly. 2. Construct noisy input. Noise schedule is αt=α0(1−t) _t= _0(1-t) and the per-token mask probability is pmask=1−αtp_mask=1- _t. 3. Run two separate forward passes: • sequential phase: clean sequence x0x_0, predict only tokens at masked positions (others set to ignore_index). • diffusion phase: noisy sequence xtx_t, reconstruct masked tokens. 4. Final loss is linear combination of losses at sequential and diffusion phases, calculated the same way as in Section 4.1. Optional flags include: • noise_sorting: reorders tokens by mask/unmask state before both forward passes, with logits permuted back afterward; • grouped_noise: masks contiguous spans (Prefix Completion, Fill-in-the-Prefix, Fill-in-the-Middle, random spans) up to max_span_length; • ordered_sampling: offsets t per token position so that pmaskp_mask increases left-to-right. 5 Inference: Dual-Mode Unified Interface Zarya provides two distinct decoding paradigms through a single model.generate() call, routed via generation_config.slotted_generation. Critically, the training and inference regimes are fully decoupled: a model trained with slotted_training=False can still be deployed with slotted_generation=True, and vice versa. 5.1 Mode A: MDM Sampling When slotted_generation=False, inference code executes the first-hitting denoising process Zheng et al., 2025; Sahoo et al., 2026. Starting from the prompt padded with ⟨mdm_mask⟩ mdm\_mask tokens up to max_length, the model iteratively reveals tokens: 1. Mask budget planning: _tokens_unmasked_per_step() determines how many masks to reveal per step. α0 _0 is the expected fraction of masked tokens generated with diffusion process. • If number of discretization steps is set to T>0T>0: exactly T diffusion steps are used. Binomial distribution is used to calculate number of masked tokens to denoise through diffusion process and tokens thats left after that to denoise sequentially (this mode is heavily inspired by Sahoo et al. (2026, Appendix B.5)). • If number of discretization steps is set to T=0T=0 (ignoring noise calculations): T steps are auto-calculated as 14 14 of masked tokens. 2. Reordering: Input sequence is reordered so that masked tokens are always after unmasked. 3. Per-step sampling from categorical distribution: At each step, the model receives the progressively filled sequence (with KV cache reuse) and yields logits for masked positions. Gumbel noise (Gumbel, 1935; Zheng et al., 2025, Appendix F)is added for categorical sampling, and standard sampling parameters (temperature, top_p, repetition_penalty) are honored. 4. Restoration: After all steps, the sequence is restored to the original token order. The KV cache can be reused because the sequence is reordered so that tokens whose values are fixed at a given denoising step precede the remaining masked positions. Under causal attention, the cached prefix states therefore remain unchanged when masked positions are progressively filled. This property would not hold for a bidirectional masked-diffusion attention pattern, where changing any previously masked token could affect the representations of other masked positions. 5.2 Mode B: Slotted Speculative Decoding When slotted_generation=True, inference code achieves parallelization by elevating decoding units from tokens to slots, fully reusing KV cache to avoid recomputation. This mode follows the ReFusion paradigm Li et al., 2026a: 1. Reordering: Before the first forward pass, input sequence is reordered so that masked tokens are always after unmasked. 2. Block construction: max_new_tokens count of masked tokens is divided into serial_num_blocks blocks of length block_size=⌊max_new_tokens/serial_num_blocks⌋ block\_size= max\_new\_tokens/ serial\_num\_blocks . Within each block, tokens are grouped into slots of size slot_size. If max_new_tokens is small, serial_num_blocks is forced to 1 to prevent zero-length blocks. 3. Draft phase: An MDM forward pass drafts tokens for all slots in the current block in parallel. The confidence of each slot is estimated as the probability of its first token. 4. Sampling from categorical distribution: If temperature is positive, apply Gumbel noise to logits for sampling from categorical distributions. 5. Slot selection: Slots with confidence exceeding slot_threshold are accepted immediately. If no slots are confident enough, select the most confident one, so we always have at least one slot to process further. 6. Verification phase: Selected slots undergo an AR verification forward pass. Tokens with probability exceeding token_threshold are accepted; those below are iteratively refined in a speculative loop. 7. KV cache update: The cache is updated incrementally with accepted tokens, avoiding recomputation for verified prefixes. 8. Restoration: After all steps, the sequence is restored to the original token order. 5.3 Parameter Decoupling Summary Table 2summarizes the scope of key configuration parameters. Table 2: Scope of configuration parameters across training and inference modes. Parameter Training Inference slotted_training ✓ — slot_size_set / slot_step_borders ✓ — diffusion_loss_proportion (λ) ✓ — noise_sorting ✓ — ordered_sampling ✓ — grouped_noise / max_span_length ✓ — add_loss_path / scale_by_batch ✓ — slotted_generation — ✓(routes Mode A/B) T (diffusion steps) — ✓(Mode A) sequential_shuffle / diffusion_shuffle ✓ ✓(Mode A only) slot_size / serial_num_blocks — ✓(Mode B) slot_threshold / token_threshold — ✓(Mode B) temperature / top_p / repetition_penalty — ✓(both modes) 6 Evaluation We evaluate on standard benchmarks including GSM8K (mathematical reasoning) Cobbe et al., 2021, HellaSwag (commonsense reasoning) Zellers et al., 2019, IFEval (instruction following) Zhou et al., 2023, and MBPP (code generation) Austin et al., 2021. Preliminary results for Zarya-0.6B are shown in Table 3. Hardware info: GPU A100; GPU driver CUDA version 13.2; GPU driver version 595.71.05; Docker info: Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002; lm-eval 0.4.12; Inference info: BF16, apply chat template, slotted_generation=true, slot_size=16, serial_num_blocks=4, slot_threshold=0.9, token_threshold=0.4 Table 3: Preliminary evaluation results for Zarya-0.6B Tasks Version Filter n-shot Metric Value Stderr gsm8k 3 flexible-extract 5 exact_match ↑ 0.2646 ± 0.0122 strict-match 5 exact_match ↑ 0.2646 ± 0.0122 hellaswag 1 none 0 acc ↑ 0.3526 ± 0.0048 none 0 acc_norm ↑ 0.4270 ± 0.0049 ifeval 4 none 0 inst_level_loose_acc ↑ 0.5372 ± N/A none 0 inst_level_strict_acc ↑ 0.5108 ± N/A none 0 prompt_level_loose_acc ↑ 0.4177 ± 0.0212 none 0 prompt_level_strict_acc ↑ 0.3993 ± 0.0211 mbpp 1 none 3 pass_at_1 ↑ 0.1500 ± 0.0160 mbpp_plus 1 none 3 pass_at_1 ↑ 0.2249 ± 0.0215 7 Conclusion We introduced Zarya, a hybrid AR–MDM language model that combines slotted training, dual-mode inference, and extensive configurability. By structuring data into slots and gradually increasing slot size during training, Zarya achieves a principled integration of autoregressive and diffusion paradigms. The unified inference interface supports both MDM sampling and slotted speculative decoding mode with full KV cache reuse. Zarya represents a step toward flexible, efficient language models that leverage the strengths of both generation paradigms. We release models at three sizes and provide a comprehensive training framework for further research. Limitations While Zarya demonstrates strong potential, several limitations remain: 1. Threshold sensitivity: The performance of slotted speculative decoding depends critically on slot selection threshold and token selection threshold. Poorly tuned thresholds can lead to low acceptance rates or quality degradation. An adaptive thresholding mechanism is needed. 2. Mode-specific ignorance: Researchers must remember that ordered_sampling and noise_sorting configuration options do nothing during inference. If these are tuned heavily during training, users may see no effect at inference time. Acknowledgments We thank the open-source community for providing the foundation upon which this work is built. References Arriola et al. (2025) M. Arriola, S. S. Sahoo, A. Gokaslan, Z. Yang, Z. Qi, J. Han, J. T. Chiu, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. External Links: 2108.07732, Link Cited by: §1, §6. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33, p. 1877–1901. External Links: Link Cited by: §1. Cai et al. (2024) T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao Medusa: simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, External Links: Link Cited by: §1. Chelba et al. (2014) C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson One billion word benchmark for measuring progress in statistical language modeling. External Links: 1312.3005, Link Cited by: §2. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. External Links: 2302.01318, Link Cited by: §1. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. External Links: 2110.14168, Link Cited by: §1, §6. Gokaslan et al. (2019) A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex OpenWebText corpus. External Links: Link Cited by: §2. Gumbel (1935) E. J. Gumbel Les valeurs extrêmes des distributions statistiques. 5 (2), p. 115–158. External Links: Link Cited by: item 3. Li et al. (2026a) J. Li, J. Guan, W. Wu, and C. Li ReFusion: a diffusion large language model with parallel autoregressive decoding. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 53846–53869. External Links: Link Cited by: §1, §1, §2, §4.1, §5.2. Li et al. (2026b) T. Li, M. Chen, B. Guo, and Z. Shen A survey on diffusion language models. External Links: 2508.10875, Link Cited by: §1. OpenAI (2023) OpenAI GPT-4 technical report. External Links: 2303.08774, Link Cited by: §1. Sahoo et al. (2026) S. S. Sahoo, Z. Yang, Y. Akhauri, J. Liu, D. Singh, Z. Cheng, Z. Liu, E. P. Xing, J. Thickstun, and A. Vahdat Esoteric language models: bridging autoregressive and masked diffusion llms. In ICLR 2026 Workshop on Multimodal Intelligence, External Links: Link Cited by: §1, §2, §4.2, 1st item, §5.1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Link Cited by: §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1, §3.1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 4791–4800. External Links: Document, Link Cited by: §6. Zheng et al. (2025) K. Zheng, Y. Chen, H. Mao, M. Liu, J. Zhu, and Q. Zhang Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 63186–63227. External Links: Link Cited by: item 3, §5.1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. External Links: 2311.07911, Link Cited by: §6.