Paper deep dive
DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
Ziyu Zhang, Tianlun Zuo, Hanzhao Li, Haoyu Zhang, Lei Xie
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/20/2026, 4:06:11 AM
Summary
The paper introduces DiTAR+, a dual-optimization framework for continuous-latent Autoregressive Diffusion Transformer (AR-DiT) speech synthesis. It addresses instability in long utterances and complex linguistic structures by proposing Dilated Context Sampling (DCS) to expand the historical receptive field without violating temporal continuity, and Hierarchical Acoustic Masking (HAM) to decouple semantic alignment from acoustic detail reconstruction. Experiments on the Emilia dataset show DiTAR+ outperforms baselines like DiTAR, CosyVoice 2, and MaskGCT in Word Error Rate (WER) and Speaker Similarity (SIM), particularly on challenging sets like ZH-Hard and ZH-Long.
Entities (14)
Relation Signals (14)
Dilated Context Sampling → partof → DiTAR
confidence 95% · The proposed framework contains two complementary components: Dilated Context Sampling (DCS) and Hierarchical Acoustic Masking (HAM).
Hierarchical Acoustic Masking → partof → DiTAR
confidence 95% · The proposed framework contains two complementary components: Dilated Context Sampling (DCS) and Hierarchical Acoustic Masking (HAM).
DiTAR → trainedon → Emilia
confidence 95% · All models are trained from scratch on the large-scale multilingual speech dataset, Emilia
DiTAR → uses → Dilated Context Sampling
confidence 95% · we introduce Dilated Context Sampling to expand the macro-level historical receptive field
DiTAR → uses → Hierarchical Acoustic Masking
confidence 95% · we propose Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context
DiTAR → evaluatedon → ZH-Hard
confidence 90% · On the linguistically challenging ZH-Hard set, DiTAR+ reduces the word error rate
DiTAR → evaluatedon → ZH-Long
confidence 90% · on extended utterances of 25 to 35 seconds it improves speaker similarity
DiTAR → outperforms → MaskGCT
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continuous-latent Autoregressive Diffusion Transformer (AR-DiT) models have demonstrated immense potential in zero-shot speech generation. However, they still suffer from limited decoding stability when synthesizing long utterances or complex linguistic structures. This instability primarily stems from a restricted historical receptive field and an acoustic inertia dependency within the diffusion decoder, which causes the model to ignore semantic conditions. To address these challenges, we propose DiTAR+, a dual-optimization framework. First, we introduce Dilated Context Sampling to expand the macro-level historical receptive field without violating physical temporal continuity, thereby preventing cumulative error propagation. Second, we propose Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reconstruction. Extensive experiments show that our framework effectively mitigates pronunciation errors and semantic hallucinations, enhances generation robustness on challenging sentences, and maintains exceptionally high speaker similarity throughout the entirety of long-form utterances. On the linguistically challenging ZH-Hard set, DiTAR+ reduces the word error rate from 12.478% to 9.893%, and on extended utterances of 25 to 35 seconds it improves speaker similarity from 0.741 to 0.759 while simultaneously lowering the word error rate from 2.778% to 2.173%, outperforming both discrete-token and pure flow-matching baselines.
Tags
Links
- Source: https://arxiv.org/abs/2609.13909v1
- Canonical: https://arxiv.org/abs/2609.13909v1
Trouble viewing inline? Open PDF directly →
Full Text
39,722 characters extracted from source content.
Expand or collapse full text
DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis Ziyu Zhang Tianlun Zuo Hanzhao Li Haoyu Zhang Lei Xie(✉) Affiliation: ASLP@NPU, Northwestern Polytechnical University, Xi’an, China E-mail ziyu_zhang@mail.nwpu.edu.cn, lxie@nwpu.edu.cn Abstract Continuous-latent Autoregressive Diffusion Transformer (AR-DiT) models have demonstrated immense potential in zero-shot speech generation. However, they still suffer from limited decoding stability when synthesizing long utterances or complex linguistic structures. This instability primarily stems from a restricted historical receptive field and an acoustic inertia dependency within the diffusion decoder, which causes the model to ignore semantic conditions. To address these challenges, we propose DiTAR+, a dual-optimization framework. First, we introduce Dilated Context Sampling to expand the macro-level historical receptive field without violating physical temporal continuity, thereby preventing cumulative error propagation. Second, we propose Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reconstruction. Extensive experiments show that our framework effectively mitigates pronunciation errors and semantic hallucinations, enhances generation robustness on challenging sentences, and maintains exceptionally high speaker similarity throughout the entirety of long-form utterances. On the linguistically challenging ZH-Hard set, DiTAR+ reduces the word error rate from 12.478% to 9.893%, and on extended utterances of 25 to 35 seconds it improves speaker similarity from 0.741 to 0.759 while simultaneously lowering the word error rate from 2.778% to 2.173%, outperforming both discrete-token and pure flow-matching baselines. Keywords: AR-DiT Continuous-Latent Modeling Zero-Shot Speech Synthesis. 1 Introduction Recent advances in speech generation have shown a clear trend from discrete acoustic token modeling toward continuous representation modeling [9, 16, 17, 20]. Compared with discrete codec tokens, continuous acoustic latents preserve richer prosodic, timbral, and fine-grained spectral details, thereby offering a higher upper bound for natural and expressive speech synthesis [24, 12, 14]. Built upon this direction, autoregressive diffusion Transformer architectures, which combine the sequence modeling ability of causal language models with the high-fidelity generation capability of diffusion decoders, have demonstrated strong potential in zero-shot and long-form speech generation [19, 11, 24, 18]. However, despite their impressive quality, continuous-latent AR-DiT models still suffer from limited generation stability, especially when synthesizing long utterances or linguistically difficult sentences. This instability is mainly reflected in speaker similarity degradation, word repetition, pronunciation disorder, insertion, deletion, and semantic misalignment. Existing speech generation methods based on autoregressive modeling can be broadly categorized into two paradigms: discrete-token-based autoregressive models and continuous-latent autoregressive diffusion models. Discrete-token-based methods first convert speech into codec indices and then employ a language model to predict acoustic tokens [21, 2, 13, 3]. This paradigm benefits from the maturity of text language modeling and usually achieves robust sequence-level prediction. Nevertheless, the discretization process inevitably removes part of the acoustic details [23, 5], making it difficult to fully preserve subtle speaker characteristics, prosodic variations, and expressive nuances. In contrast, continuous-latent AR-DiT models directly operate on compressed continuous speech representations and employ a diffusion Transformer decoder to reconstruct local acoustic segments [11, 24, 18]. This design avoids the information bottleneck introduced by vector quantization and enables more realistic acoustic rendering. Among these models, DiTAR [11] is a particularly strong baseline because it introduces an autoregressive language model to predict compressed speech patches while using a local diffusion Transformer, namely LocDiT, to generate high-quality continuous acoustic latents conditioned on text features and local acoustic history. Such a hybrid design provides a promising balance between long-range sequence modeling and local high-fidelity acoustic generation. Despite these advantages, DiTAR-like AR-DiT models still face two critical limitations. First, the historical receptive field is inherently restricted by the local prediction scheme—LocDiT relies mainly on the immediately preceding patch as acoustic pre-context, offering only a narrow view of the acoustic trajectory. As utterances lengthen, remote speaker and prosodic information becomes inaccessible, causing cumulative error propagation and declining speaker consistency. Second, LocDiT develops an acoustic inertia dependency during iterative denoising. Shallow layers over-rely on local acoustic pre-context, taking shortcuts by copying or continuing previous acoustic patterns, which weakens the role of semantic conditions from the upstream language model. When handling complex linguistic structures or long dependencies, the decoder ignores semantic instructions and produces errors such as repetitions, omissions, incorrect pronunciation, or loop-like degeneration. Richer historical context only exacerbates this issue, as it reinforces acoustic continuation over semantic alignment. In this paper, we propose DiTAR+, a dual-optimization framework for robust continuous-latent AR-DiT speech generation. The proposed framework contains two complementary components: Dilated Context Sampling (DCS) and Hierarchical Acoustic Masking (HAM). First, to expand the historical receptive field without violating temporal continuity, we introduce DCS at the autoregressive front end. Instead of reordering the speech latent sequence, the proposed strategy preserves the immediately preceding patch for local boundary smoothness while additionally sampling remote historical patches with a dilated interval. In this way, the model obtains a wider macro-level acoustic view at negligible computational cost, while the absolute physical order of continuous speech latents remains intact. Second, to mitigate the acoustic inertia dependency of LocDiT, we introduce HAM at the diffusion back end. Specifically, the shallow layers of LocDiT are prevented from attending to acoustic pre-context, forcing the model to first establish semantic alignment based on linguistic conditions. The deeper layers then recover access to historical acoustic context for speaker-consistent and smooth acoustic rendering. This hierarchical design explicitly decouples semantic alignment from acoustic detail reconstruction, thereby improving robustness on difficult sentences. The key contributions of our work are summarized as follows: • We propose Dilated Context Sampling, a strategy that expands the historical receptive field without disrupting temporal continuity, thereby improving long-form generation stability and speaker consistency. • We design Hierarchical Acoustic Masking to suppress shallow acoustic shortcut learning, effectively decoupling semantic information from acoustic rendering and reducing the word error rate. • We present DiTAR+, a dual-optimization framework that integrates DCS and HAM, significantly outperforming baselines in challenging scenarios by achieving robust zero-shot generation on complex sentences and maintaining stable speaker similarity across long utterances. 2 Method 2.1 DiTAR Baseline As shown in Fig. 1(a), the standard DiTAR framework employs a divide-and-conquer paradigm that seamlessly amalgamates an autoregressive language model (LM) with a local diffusion decoder (LocDiT). Initially, the target speech waveform is compressed into a highly compact continuous latent space via a Variational Autoencoder (VAE). To facilitate efficient modeling, the continuous tokens are grouped into non-overlapping patches, each of which is mapped into a single aggregation embedding. The generation pipeline operates in two decoupled stages. Figure 1: The overall architecture of the proposed DiTAR+ framework. (a) The standard two-stage DiTAR baseline. (b) Hierarchical Acoustic Masking (HAM), which masks historical acoustic context in shallow layers to decouple semantic alignment and acoustic rendering. (c) Dilated Context Sampling (DCS), which expands the macro-receptive field via dilated sampling while preserving temporal continuity. Inter-Patch Semantic Generation. At the macroscopic level, a Causal Autoregressive Transformer models the global sequence dependency. Given the text condition and reference audio as prefix prompts, the LM processes the aggregated historical latent patches sequentially. For the i-th target patch, the LM autoregressively predicts a semantic hidden condition vector, denoted as hih_i. This vector hih_i encapsulates the contextual semantic and prosodic instructions required for the current patch generation. Intra-Patch Acoustic Generation. At the microscopic level, a Localized Diffusion Transformer with bidirectional attention is utilized to reconstruct the fine-grained continuous speech tokens within each patch. During the iterative denoising process, LocDiT takes the semantic vector hih_i, the diffusion timestep t, and the noisy current patch xtx_t as inputs. Crucially, to maintain acoustic boundary smoothness, the standard LocDiT explicitly incorporates the immediate preceding patch (i.e., Patch i−1i-1) as a localized historical context (pre_context). This pre_context is consistently utilized as prefix inputs across all layers of the diffusion decoder. 2.2 Dilated Context Sampling In the standard DiTAR framework, the local diffusion decoder (LocDiT) predicts each target patch using only its immediately preceding acoustic context. Specifically, when generating the i-th patch, the model typically conditions on the (i−1)(i-1)-th patch. While this strictly local prediction paradigm promotes smooth transitions between adjacent speech segments, it inherently limits the effective receptive field of the decoder. For long-form speech synthesis, such restricted historical context leads to progressive error accumulation. As generation proceeds, the model gradually loses access to distant acoustic information, making it increasingly difficult to maintain speaker characteristics and long-term acoustic consistency. This issue often manifests as speaker drift, where speaker similarity deteriorates with increasing utterance length. A straightforward solution is to enlarge the receptive field through patch reordering strategies such as Skip-Patchify [15]. However, such approaches disrupt the original temporal structure of speech representations and may break local acoustic continuity by separating neighboring patches that are naturally adjacent in time. To address this limitation, we propose Dilated Context Sampling (DCS), as shown in Fig. 1(c), a simple yet effective mechanism that expands the receptive field while preserving the temporal ordering of acoustic features. Specifically, when predicting the i-th target patch PiP_i, we always retain a small window of immediately preceding patches (e.g., [Pi−c,…,Pi−1][P_i-c,…,P_i-1]) as local context to ensure smooth boundary transitions. In addition, we employ a dilated sampling strategy to sparsely gather distant historical patches from earlier positions in the sequence with a fixed stride j. The local continuous context and the dilated distant context are then concatenated and jointly provided to LocDiT as conditioning information: Ci=[Pi−nj,…,Pi−2j,Pi−j,Pi−c,…,Pi−2,Pi−1],C_i=[P_i-nj,…,P_i-2j,P_i-j,P_i-c,…,P_i-2,P_i-1], (1) where CiC_i denotes the overall historical context used for predicting the i-th patch, c is the local context window size (e.g., c=3c=3), j represents the dilation stride (where j>cj>c), and n determines the maximum range of the macro-receptive field. To explicitly illustrate the data flow of the proposed sampling strategy, we outline the exact execution steps in Phase 1 of Algorithm 1. During the initial phase of each autoregressive decoding step, the model constructs the context dynamically rather than relying on a static prefix. As shown in the algorithm, it sequentially extracts a dense local context window (ClocalC_local) to maintain boundary smoothness, followed by a sparsely sampled distant context (CdilatedC_dilated). These two distinct components are then concatenated to assemble a unified historical acoustic context (XhistX_hist). This programmatic operation ensures that the macro-receptive field is substantially expanded while the absolute physical temporal timeline remains strictly intact prior to being fed into the LocDiT network. By incorporating sparsely sampled historical patches, DCS enables LocDiT to simultaneously access both short-range continuity cues and long-range acoustic information. Unlike patch reordering methods, DCS significantly enlarges the effective receptive field without altering the original temporal structure of speech. Consequently, it mitigates long-range error accumulation, improves speaker consistency in extended speech generation, and substantially reduces speaker drift while maintaining high-quality acoustic continuity. Algorithm 1 Forward pass of LocDiT with DCS and HAM Input: Current patch index i, generated historical patches P, semantic condition hih_i, current noisy patch XcurrX_curr. Hyperparameters: Local window size c, dilation stride j, max dilated patches n, total Transformer layers L, masked shallow layers N1N_1. 1: ⊳ Phase 1: Dilated Context Sampling (DCS) 2: Clocal←[Pi−c,…,Pi−2,Pi−1]C_local←[P_i-c,…,P_i-2,P_i-1] ⊳ Extract local continuous context 3: Cdilated←[Pi−nj,…,Pi−2j,Pi−j]C_dilated←[P_i-nj,…,P_i-2j,P_i-j] ⊳ Extract sparse distant context 4: Xhist←Concatenate(Cdilated,Clocal)X_hist (C_dilated,C_local) ⊳ Assemble context in temporal order 5: 6: ⊳ Phase 2: Hierarchical Acoustic Masking (HAM) 7: H(0)←Concatenate(hi,Xhist,Xcurr)H^(0) (h_i,X_hist,X_curr) ⊳ Construct initial input sequence 8: for l=1l=1 to L do 9: Q,K,V←LinearProjections(H(l−1))Q,K,V (H^(l-1)) 10: Initialize attention mask M(l)←0M^(l)← 0 for all token pairs 11: if l≤N1l≤ N_1 then 12: for each query qu∈Xcurrq_u∈ X_curr and key kv∈Xhistk_v∈ X_hist do 13: Mu,v(l)←−∞M^(l)_u,v←-∞ ⊳ Hide historical acoustic tokens 14: end for 15: end if 16: Attn_Out←Softmax(QK⊤d+M(l))VAttn\_Out ( QK d+M^(l) )V ⊳ Masked self-attention 17: H(l)←FeedForward(Attn_Out)+H(l−1)H^(l) (Attn\_Out)+H^(l-1) ⊳ Update hidden states 18: end for 19: 20: return Extracted representations corresponding to XcurrX_curr from H(L)H^(L) 2.3 Hierarchical Acoustic Masking In the standard DiTAR architecture, historical token patches are concatenated as a prefix and directly provided to LocDiT. While this design improves acoustic continuity, it also introduces a strong dependency on historical acoustic contexts. Specifically, when the diffusion model has unrestricted access to the entire pre-context in early layers, it tends to exploit a shortcut by extrapolating from previously generated acoustic patterns rather than relying on semantic guidance from the LM. As a result, semantic conditioning is weakened, leading to unstable decoding behaviors such as repetitive generation, looping artifacts, insertion errors, pronunciation mistakes, and a substantial increase in word error rate, particularly for long-form or semantically challenging utterances. To address this issue, we propose a Hierarchical Acoustic Masking (HAM) strategy within LocDiT. The key idea is to progressively expose semantic and acoustic conditions across network depth. Let XcurrX_curr denote the current noisy token sequence, XhistX_hist denote the historical acoustic context (i.e., the pre-context), and hih_i denote the semantic representation produced by the LM. The masked self-attention operation in the l-th Transformer block is formulated as Attention(Q,K,V)=Softmax(QK⊤d+M(l))V,Attention(Q,K,V)=Softmax ( QK d+M^(l) )V, (2) where M(l)M^(l) is a layer-dependent attention mask. For attention interactions between query tokens qu∈Xcurrq_u∈ X_curr and key tokens kv∈Xhistk_v∈ X_hist, we define Mu,v(l)=−∞,qu∈Xcurr,kv∈Xhist,l≤N1,0,otherwise,M_u,v^(l)= cases-∞,&q_u∈ X_curr,\;k_v∈ X_hist,\;l≤ N_1,\\ 0,&otherwise, cases (3) where N1N_1 denotes the number of masked shallow Transformer blocks. Consequently, historical acoustic tokens are completely invisible during the early stages of denoising, forcing LocDiT to establish semantic alignment based solely on the LM-provided conditioning hih_i. In the subsequent N2N_2 deeper layers, the mask is removed, allowing the model to incorporate historical acoustic information for fine-grained acoustic refinement. To provide a concrete implementation perspective, the layer-by-layer execution of this dynamic masking mechanism is formally detailed in Phase 2 of Algorithm 1. Prior to the Transformer forward pass, the model seamlessly concatenates the semantic condition (hih_i), the assembled historical context (XhistX_hist), and the current noisy patch (XcurrX_curr) into a single input sequence. As this sequence propagates through the network depth, a conditional loop dynamically enforces the masking rule: for any layer index l≤N1l≤ N_1, the attention weights corresponding to the historical acoustic keys are aggressively overridden with −∞-∞. This algorithmic constraint forcibly cuts off the local acoustic shortcut in shallow layers, dynamically lifting the restriction only in deeper layers (l>N1l>N_1) to finalize the fine-grained acoustic reconstruction. This progressive conditioning mechanism effectively decouples semantic alignment from acoustic rendering. By preventing shallow layers from overfitting to local acoustic continuity, the model is encouraged to first construct a semantically faithful representation before leveraging historical acoustic cues for prosody, timbre, and continuity modeling. Empirically, the proposed masking strategy substantially improves decoding robustness in challenging long-form synthesis scenarios, mitigating failure modes such as repetitive loops, insertion and deletion errors, and pronunciation inconsistencies, while maintaining high-fidelity speech generation. 3 Experiments 3.1 Experimental Setup Training Data and Hardware. All models are trained from scratch on the large-scale multilingual speech dataset, Emilia [8]. The training process is conducted on a computing cluster comprising 4 nodes, each equipped with 8 H20 GPUs, totaling 32 H20 GPUs. During training, the batch size is set to 8 per GPU. Inference Configurations. The inference process employs an Euler ODE sampler with 10 steps. We apply Classifier-Free Guidance (CFG) with a scale of α=2.0α=2.0, a flow-matching scale of 1.01.0, and sway-sampling to balance generation diversity and stability. The maximum prompt length is truncated to 75. During synthesis, the prompt audio is utilized as the speaker reference to generate the target text utterance by utterance. Evaluation Metrics. To comprehensively assess the performance of the generated speech, we employ both objective and subjective metrics. For objective evaluation, we measure the Word Error Rate (WER) to reflect linguistic intelligibility and Speaker Similarity (SIM) to evaluate voice cloning accuracy, strictly adhering to the standard seed-tts-eval11 1 https://github.com/BytedanceSpeech/seed-tts-eval protocol. For subjective evaluation, we conduct multi-dimensional assessments including Naturalness MOS (N-MOS), Quality MOS (Q-MOS), and Similarity MOS (S-MOS) [10, 1, 21]. We invited 10 professional evaluators to rate 50 randomly selected utterances per system on a standard 5-point scale. Additionally, we conduct Comparative MOS (C-MOS) tests, utilizing our proposed DiTAR+ as the benchmark anchor (scored at 0.0) to explicitly gauge human preference against other comparative systems. Evaluation Dataset. We construct a comprehensive evaluation benchmark covering both standard and extreme zero-shot synthesis scenarios. First, we adopt the standard English and Chinese zero-shot test sets from seed-tts-eval, denoted as Seed-EN and Seed-ZH, respectively. To rigorously evaluate model robustness against complex linguistic structures, we utilize 400 challenging utterances from the official test-zh-hard set, denoted as ZH-Hard. Furthermore, to explicitly assess long-term speaker consistency and the mitigation of temporal speaker drift, we curate ZH-Long, a specialized subset comprising 400 extended sentences with expected acoustic durations ranging from 25 to 35 seconds. 3.2 Compared Methods To comprehensively evaluate the superiority of our proposed DiTAR+ framework, we benchmark it against both our direct baseline and representative state-of-the-art (SOTA) models covering diverse speech generation paradigms: • DiTAR [11]: The standard continuous-latent autoregressive diffusion transformer. We include this to directly demonstrate the effectiveness of our proposed DCS and HAM modules in mitigating error accumulation. • CosyVoice 2 [6]: A leading discrete-token based autoregressive model, representing the paradigm of large-scale LM with discrete audio codecs. • MaskGCT [22]: A state-of-the-art non-autoregressive (NAR) model utilizing masked generative modeling over discrete tokens, offering strong bidirectional context understanding. • F5-TTS [4] and E2 TTS [7]: Recent highly successful pure continuous flow-matching (diffusion) models that generate speech without an autoregressive linguistic planner. 3.3 Objective Evaluation The objective evaluation results comparing our proposed DiTAR+ with the baseline and SOTA models are presented in Table 1. We analyze the performance across standard evaluation sets and extreme stress-test scenarios. Performance on Standard Scenarios. On the standard Seed-EN and Seed-ZH subsets, DiTAR+ demonstrates exceptional fundamental generation capabilities. It achieves the lowest WER among all evaluated systems, reaching 1.672% on Seed-EN and 0.985% on Seed-ZH. In terms of speaker similarity (SIM), DiTAR+ yields highly competitive scores (0.735 and 0.762 respectively), performing on par with or exceeding strong baselines like MaskGCT and the original DiTAR, confirming its ability to produce highly intelligible and speaker-consistent speech in normal contexts. Table 1: Objective evaluation results compared with the baseline and state-of-the-art models on the SeedTTS test set. WER is reported in %. The best result in each column is in bold and the second best is underlined System Seed-EN Seed-ZH ZH-Hard ZH-Long WER↓ SIM↑ WER↓ SIM↑ WER↓ SIM↑ WER↓ SIM↑ Ground Truth 2.060 0.730 1.254 0.750 - - - - CosyVoice 2 2.570 0.652 1.450 0.749 13.415 0.678 2.776 0.748 MaskGCT 2.623 0.717 2.273 0.774 14.085 0.688 2.954 0.740 F5-TTS 1.982 0.670 1.560 0.760 11.209 0.669 2.894 0.729 DiTAR 1.774 0.737 1.287 0.753 12.478 0.672 2.778 0.741 DiTAR+ 1.672 0.735 0.985 0.762 9.893 0.695 2.173 0.759 Robustness on Extreme Scenarios. The structural superiority of the proposed dual-optimization framework becomes prominently evident in the extreme stress-test subsets. On the linguistically complex ZH-Hard set, the standard DiTAR baseline suffers from a severe performance drop, yielding a high WER of 12.478%. By explicitly decoupling semantic alignment from acoustic rendering via HAM, DiTAR+ significantly suppresses acoustic hallucination and reduces the WER to 9.893%, outperforming all competing models by a large margin. Furthermore, on the extended-length ZH-Long set, DiTAR+ achieves both the highest SIM (0.759) and the lowest WER (2.173%). This validates that the DCS sampling mechanism successfully expands the macro-receptive field, effectively preventing speaker drift and cumulative decoding errors during long-form generation without compromising acoustic continuity. 3.4 Subjective Evaluation We conducted multi-dimensional subjective evaluations with 10 evaluators on a standard 5-point scale, using DiTAR+ as the C-MOS anchor. The detailed results are presented in Table 2. Table 2: Subjective evaluation results (MOS) on the SeedTTS test set. We compare DiTAR+ with the DiTAR baseline and two leading flow-matching systems. The best result in each column is in bold and the second best is underlined System N-MOS↑ Q-MOS↑ S-MOS↑ C-MOS↑ Ground Truth 3.88 3.60 3.57 +0.22 E2 TTS 3.25 3.42 3.17 -0.28 F5-TTS 3.37 3.59 3.32 -0.02 DiTAR 3.70 3.88 3.52 -0.06 DiTAR+ 3.71 3.82 3.58 0.00 Overall Performance. DiTAR+ demonstrates exceptional subjective quality, achieving the highest Naturalness MOS and Similarity MOS among all evaluated models. Notably, compared to recent highly successful pure continuous flow-matching models (e.g., E2 TTS and F5-TTS), our dual-optimization framework exhibits a clear superiority in capturing and maintaining speaker characteristics, as reflected in the S-MOS scores. Furthermore, all competing models yield negative C-MOS scores, underscoring a strong human preference for the speech synthesized by DiTAR+. Interestingly, DiTAR+ performs on par with or even marginally exceeds the Ground Truth in S-MOS. This can be attributed to the model’s capacity to generate a clean, idealized, and stable timbre representation, effectively filtering out the natural intra-speaker variances (e.g., micro-environmental noises) present in real human recordings. Analysis of the Q-MOS Trade-off. We do note that the standard DiTAR baseline yields a slightly higher Q-MOS compared to DiTAR+. This represents a deliberate and necessary architectural trade-off. The standard baseline heavily overfits to the immediate local acoustic history, which creates locally smooth acoustic textures (resulting in high Q-MOS) but ultimately causes catastrophic semantic hallucinations and speaker drift in extended generation. By integrating DCS and HAM, DiTAR+ explicitly breaks this pathological local inertia. It trades a negligible loss in micro-level acoustic smoothness for a substantial leap in macro-level prosodic naturalness, semantic robustness, and long-term speaker stability. 3.5 Ablation Study To evaluate the individual contributions of our proposed mechanisms, we conducted an ablation study on the challenging ZH-Hard and ZH-Long sets, with overall results detailed in Table 3. Furthermore, to granularly analyze the temporal dynamics of speaker drift, we visualize the chunk-level speaker similarity over continuous generation steps in Fig. 2. Table 3: Ablation study of the proposed DCS and HAM modules on the challenging ZH-Hard and ZH-Long sets. The best result in each column is in bold and the second best is underlined System ZH-Hard ZH-Long WER(%)↓ SIM↑ WER(%)↓ SIM↑ DiTAR 12.478 0.672 2.778 0.741 DiTAR + DCS 10.671 0.681 2.165 0.757 DiTAR + HAM 9.889 0.700 3.056 0.749 DiTAR + Skip 16.231 0.612 5.105 0.685 DiTAR+ 9.893 0.695 2.173 0.759 Figure 2: Chunk-level speaker similarity over continuous generation steps (3.0 s per chunk). The standard baseline experiences severe speaker drift in the long tail, whereas DCS effectively stabilizes the temporal coherence. Effectiveness of DCS. Integrating the Dilated Context Sampling (DiTAR + DCS) primarily enhances long-form generation stability. Compared to the standard baseline, it significantly reduces the WER on the ZH-Long set to 2.165% and boosts overall speaker similarity to 0.757. As clearly depicted in Fig. 2, the standard baseline suffers from severe speaker drift, degrading sharply after the fifth to sixth chunk (approximately 15 to 18 seconds) and bottoming out around 0.655 at the tail. In contrast, introducing DCS substantially slows this degradation, sustaining a markedly higher SIM than the baseline throughout the extreme tail of long utterances. This visually demonstrates that DCS successfully expands the macro-receptive field, effectively mitigating cumulative errors over extended durations. Effectiveness of HAM. Conversely, applying Hierarchical Acoustic Masking (DiTAR + HAM) yields substantial improvements on linguistically complex sentences. It achieves the lowest WER (9.889%) and the highest SIM (0.700) on the ZH-Hard set, validating that explicitly suppressing shallow-layer acoustic inertia forces the model to prioritize semantic alignment. Notably, as shown in Fig. 2, applying HAM alone yields a downward SIM trajectory that closely tracks the baseline in extended generation, converging to a similarly low SIM at the tail. This aligns with our architectural design: HAM is strictly formulated to decouple acoustic inertia for semantic accuracy, rather than extending the historical acoustic context window. Failure of Naive Reordering. As a negative control, we evaluated a naive non-linear feature reordering strategy (DiTAR + Skip). As expected, this approach results in severe performance degradation across all metrics, with the WER soaring to 16.231% on ZH-Hard and 5.105% on ZH-Long. This catastrophic drop provides strong empirical evidence that disrupting the absolute temporal ordering of continuous latents destroys the learned temporal prior, rendering the expanded context useless. The Full Framework. Finally, our complete framework (DiTAR+) seamlessly integrates both modules. By combining the complementary strengths of DCS and HAM, the full model achieves near-optimal, highly balanced performance across both extreme scenarios. As visualized in Fig. 2, the full DiTAR+ achieves the most stable and highest speaker similarity trajectory across the entire sequence, suggesting that the semantic regularization from HAM is largely orthogonal to the temporal coherence established by DCS. Figure 3: Mel-spectrogram comparison between the standard DiTAR baseline and the proposed DiTAR+ on a challenging utterance. Top: the baseline model falls into a runaway looping state (highlighted by the box), failing to terminate generation properly. Bottom: DiTAR+ with the HAM mechanism explicitly suppresses these acoustic shortcuts, yielding accurate semantic alignment. 3.6 Spectrogram Analysis To demonstrate the effectiveness of our proposed method in mitigating semantic hallucinations and decoding instability, we visualize the Mel-spectrograms of a challenging utterance generated by both the baseline and our model, as shown in Fig. 3. As depicted in the top panel, the standard DiTAR baseline suffers from severe “runaway looping” during the autoregressive generation process. Instead of following the semantic guidance from the language model, the diffusion decoder falls into a state of acoustic inertia, repeatedly copying previously generated acoustic patterns. This catastrophic failure results in an abnormally extended audio duration of 25.5 seconds and an exceptionally high WER of 52%, with the repetitive artifacts clearly visible inside the highlighted bounding box. In contrast, the bottom panel demonstrates the generation result of DiTAR+ equipped with the HAM mechanism. By aggressively masking historical acoustic context in shallow layers, HAM forces the model to strictly align with the semantic condition. As a result, DiTAR+ completely suppresses the looping artifacts, reconstructs clear and non-repetitive spectral details, and successfully terminates the generation on time at 14.0 seconds, drastically reducing the WER to 6%. This qualitative evidence suggests that decoupling semantic alignment from acoustic rendering is crucial for robust speech generation. 4 Conclusion In this paper, we proposed DiTAR+, a parameter-free dual-optimization framework designed to address the decoding instability of continuous-latent Autoregressive Diffusion Transformer (AR-DiT) models in zero-shot speech generation. To overcome the fundamental architectural bottlenecks of restricted receptive fields and acoustic inertia dependency, we introduced two complementary mechanisms: Dilated Context Sampling (DCS) and Hierarchical Acoustic Masking (HAM). DCS effectively expands the macro-level receptive field while strictly preserving the absolute physical timeline, thereby preventing cumulative speaker drift in long-form generation. Concurrently, HAM explicitly decouples semantic alignment from acoustic detail reconstruction by progressively exposing historical context, significantly mitigating semantic hallucinations in linguistically complex sentences. Extensive evaluations demonstrate that by combining the orthogonal strengths of DCS and HAM, DiTAR+ substantially outperforms state-of-the-art models, suggesting that temporal phase coherence and semantic-acoustic decoupling are essential design principles for robust continuous-latent speech synthesis. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References [1] P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, et al. (2024) Seed-TTS: a family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430. Cited by: §3.1. [2] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2523–2533. Cited by: §1. [3] Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi (2023) SoundStorm: efficient parallel audio generation. arXiv preprint arXiv:2305.09636. Cited by: §1. [4] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen (2025) F5-TTS: a fairytaler that fakes fluent and faithful speech with flow matching. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, p. 6255–6271. Cited by: 4th item. [5] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research. Cited by: §1. [6] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, et al. (2024) CosyVoice 2: scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. Cited by: 2nd item. [7] S. E. Eskimez, X. Wang, M. Thakker, C. Li, C. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tan, et al. (2024) E2 TTS: embarrassingly easy fully non-autoregressive zero-shot TTS. In 2024 IEEE Spoken Language Technology Workshop (SLT), p. 682–689. Cited by: 4th item. [8] H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, et al. (2024) Emilia: an extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), p. 885–890. Cited by: §3.1. [9] R. Huang, J. Huang, D. Yang, Y. Ren, L. Liu, M. Li, Z. Ye, J. Liu, X. Yin, and Z. Zhao (2023) Make-An-Audio: text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, p. 13916–13932. Cited by: §1. [10] ITU-T Recommendation P.800 (1996) Methods for subjective determination of transmission quality. Technical report International Telecommunication Union, Geneva. Cited by: §3.1. [11] D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang, et al. (2025) DiTAR: diffusion transformer autoregressive modeling for speech generation. arXiv preprint arXiv:2502.03930. Cited by: §1, §1, 1st item. [12] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang, et al. (2024) NaturalSpeech 3: zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100. Cited by: §1. [13] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour (2023) Speak, read and prompt: high-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics 11, p. 1703–1718. Cited by: §1. [14] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al. (2023) Voicebox: text-guided multilingual universal speech generation at scale. Advances in Neural Information Processing Systems 36, p. 14005–14034. Cited by: §1. [15] T. Li, Y. Tian, H. Li, M. Deng, and K. He (2024) Autoregressive image generation without vector quantization. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §2.2. [16] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley (2023) AudioLDM: text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503. Cited by: §1. [17] H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley (2024) AudioLDM 2: learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, p. 2871–2883. Cited by: §1. [18] L. Meng, L. Zhou, S. Liu, S. Chen, B. Han, S. Hu, Y. Liu, J. Li, S. Zhao, X. Wu, et al. (2024) Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551. Cited by: §1, §1. [19] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 4195–4205. External Links: Document Cited by: §1. [20] K. Shen, Z. Ju, X. Tan, E. Liu, Y. Leng, L. He, T. Qin, J. Bian, et al. (2024) NaturalSpeech 2: latent diffusion models are natural and zero-shot speech and singing synthesizers. In International Conference on Learning Representations, p. 698–722. Cited by: §1. [21] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al. (2023) Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111. Cited by: §1, §3.1. [22] Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu (2025) MaskGCT: zero-shot text-to-speech with masked generative codec transformer. In International Conference on Learning Representations, p. 47127–47150. Cited by: 3rd item. [23] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi (2021) SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, p. 495–507. Cited by: §1. [24] C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and O. Levy (2025) Transfusion: predict the next token and diffuse images with one multi-modal model. In International Conference on Learning Representations, p. 6446–6469. Cited by: §1, §1.