Paper deep dive
Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis
Zuda Yu, Qianhui Xu, Ting Chen, Junhui Zhang, Tao Fu, Hongjiang Yu, Qiangqing Wang, Yang Song
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 8:58:41 AM
Summary
The paper proposes a unified guidance framework to address two main bottlenecks in Flow Matching (FM) for speech synthesis: timbre leakage and high inference latency. The framework consists of two pillars: Data-guidance (DG), which uses a dual-stage heterogeneous perturbation strategy (model-driven cross-synthesis and signal-driven acoustic deformation) to decouple linguistic content from acoustic residue, and Enhanced Model-guidance (MG), which combines intrinsic guidance distillation with trajectory rectification to eliminate Classifier-Free Guidance (CFG) overhead. Experimental results show a 3.25x inference speedup and improved speaker similarity in both Voice Conversion and Text-to-Speech tasks.
Entities (7)
Relation Signals (5)
Data-guidance → addresses → timbre leakage
confidence 100% · Data-guidance (DG)... effectively mitigating timbre leakage.
Enhanced Model-guidance → addresses → inference latency
confidence 100% · Enhanced Model-guidance (MG)... addresses both trajectory curvature and guidance overhead simultaneously.
Enhanced Model-guidance → incorporates → trajectory rectification
confidence 100% · integrates intrinsic guidance distillation and trajectory rectification
Diffusion Transformer → ispartof → Flow Matching model
confidence 100% · our decoder adopts a pure Diffusion Transformer (DiT)
Data-guidance → uses → heterogeneous augmentation
confidence 100% · Data-guidance (DG): We propose a dual-stage heterogeneous perturbation strategy
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage. To address these bottlenecks, we propose a unified guidance framework that enhances generation efficiency and robustness through two complementary strategies. On the data front, we introduce Data-guidance via heterogeneous augmentation, encouraging the model to disentangle linguistic content from acoustic residue. In parallel, we propose an enhanced Model-guidance mechanism that synergizes trajectory rectification with a novel intrinsic guidance objective. This approach distills conditional knowledge into network weights and straightens inference trajectory path, thereby eliminating Classifier-Free Guidance (CFG) overhead. Experiments demonstrate that our framework accelerates inference by nearly three times while effectively improving speaker similarity compared to state-of-the-art baselines.
Tags
Links
- Source: https://arxiv.org/abs/2607.00363v1
- Canonical: https://arxiv.org/abs/2607.00363v1
Trouble viewing inline? Open PDF directly →
Full Text
30,179 characters extracted from source content.
Expand or collapse full text
Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis Zuda Yu ∗ , Qianhui Xu ∗ , Ting Chen ∗ , Junhui Zhang ∗ , Tao Fu ∗ , Hongjiang Yu ∗ , Qiangqing Wang ∗ , Yang Song ∗ Zuoyebang, China yuzuda, xuqianhui02, chenting08, zhangjunhui04, futao01, yuhongjiang, wangqiangqiang, songyang@zuoyebang.com Abstract Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage. To address these bottlenecks, we propose a unified guidance framework that enhances generation efficiency and robustness through two complementary strate- gies. On the data front, we introduce Data-guidance via hetero- geneous augmentation, encouraging the model to disentangle linguistic content from acoustic residue. In parallel, we pro- pose an enhanced Model-guidance mechanism that synergizes trajectory rectification with a novel intrinsic guidance objec- tive. This approach distills conditional knowledge into network weights and straightens inference trajectory path, thereby elim- inating Classifier-Free Guidance (CFG) overhead. Experiments demonstrate that our framework accelerates inference by nearly three times while effectively improving speaker similarity com- pared to state-of-the-art baselines. Audio samples are available here. 1 . Index Terms: speech synthesis, voice conversion, flow match- ing 1. Introduction Flow Matching (FM) [1] has rapidly advanced speech synthe- sis by modeling the continuous transformation from a sim- ple prior to complex data distributions. This paradigm has demonstrated remarkable potential across various speech gen- eration tasks. For instance, text-based approaches such as F5- TTS [2] and Matcha-TTS [3] achieve fully non-autoregressive speech generation by directly mapping text to acoustic fea- tures. Concurrently, discrete-token-based frameworks includ- ing the CosyVoice family [4–7], MaskGCT [8], and Fir- eRedTTS [9] apply FM to convert semantic representations into mel-spectrograms. Furthermore, recent audio-language models like Kimi-Audio [10], GLM-4-Voice [11], and Step-Audio [12] increasingly rely on FM as a high-fidelity detokenizer for wave- form reconstruction. Beyond speech synthesis, models such as Seed-VC [13] and StableVC [14] illustrate the capacity of FM to effectively disentangle speech attributes for voice con- version. Despite these impressive milestones, the widespread deployment of FM models in real-time scenarios remains fun- damentally constrained by two distinct yet critical bottlenecks. The first bottleneck, timbre leakage, compromises the ro- bustness of zero-shot generation. Token-based models rely on discrete semantic representations as conditioning inputs, yet * These authors contributed equally. ** indicates the corresponding author. 1 https://yuzuda283.github.io/unified-guidanc e-flow-matching/Interspeech2026_demo_samples/ extracting purely disentangled content is notoriously difficult. Traditional information bottlenecks such as Vector Quantization (VQ) [15] attempt to strip away speaker identity but often de- grade linguistic intelligibility and pronunciation quality. Conse- quently, data-driven approaches have gained prominence. Seed- TTS [16] mitigates leakage by retraining models on synthetic cross-speaker pairs generated via self-distillation, while Seed- VC [13] successfully employs an external generative model to perturb the source audio during training, forcing the network to align with the target prompt. These strategies inspire us to ex- tend single-stage perturbation into a more comprehensive data- guidance constraint. The second bottleneck, inference latency, stems from the intrinsic nature of ODE-based generation imposing two distinct computational costs. First, iterative ODE solvers require a high Number of Function Evaluations (NFE) along curved proba- bility paths. Extensive research has focused on accelerating this process through trajectory manipulation, with fundamen- tal techniques like Rectified Flow [17] attempting to straighten the probability paths. Similar concepts mapping noise to data in fewer steps are explored in InstaFlow [18] and Consistency Models [19]. In the speech domain specifically, acceleration has been adapted through progressive distillation in ProDiff [20], adversarial training in FastDiff [21], and consistency distilla- tion in CoMoSpeech [22]. While these trajectory-level accel- eration methods substantially reduce NFE, the secondary com- putational burden of Classifier-Free Guidance (CFG) [23] of- ten remains. CFG ensures strict prompt adherence but dou- bles generation cost by requiring conditional and unconditional passes per sampling step. Concurrently, recent work on Model- guidance [24] offers an inspiring direction by explicitly opti- mizing a guided velocity field to replace CFG. Motivated by these two distinct lines of acceleration, we seek to effectively synergize them to address both trajectory curvature and guid- ance overhead simultaneously. We argue that overcoming these dual challenges bene- fits from a unified approach: robust zero-shot generation de- mands severing the acoustic information shortcut during train- ing, while efficient generation demands a straight, guidance- aware flow trajectory. To this end, we propose a unified guid- ance framework that systematically enhances FM through two complementary pillars: • Data-guidance (DG): We propose a dual-stage heteroge- neous perturbation strategy that cascades model-driven cross- synthesis with signal-driven acoustic deformations. By delib- erately degrading the acoustic reliability of semantic tokens during training, the model is forced to rely more heavily on the target prompt for timbre generation, thereby effectively mitigating timbre leakage. arXiv:2607.00363v1 [cs.SD] 1 Jul 2026 • Enhanced Model-guidance (MG): We propose an advanced optimization mechanism that explicitly synergizes intrin- sic guidance distillation with trajectory rectification. Our method simultaneously distills the CFG-aware velocity field into the network weights and linearizes the ODE path on the fly. This strategy circumvents the CFG dual-pass overhead and enables high-fidelity sampling in minimal steps. Through this systematic optimization, our framework ac- celerates inference by nearly 3× while achieving superior speaker similarity and pronunciation stability compared to state-of-the-art baselines. 2. Methodology 2.1. Conditional Flow Matching Our method builds upon Conditional Flow Matching (CFM) [1], which constructs probability paths via linear Optimal Transport. Let x 1 denote the target speech sample and x 0 ∼ N(0,I) be the prior noise. The probability path x t is defined via linear interpolation: x t = (1− t)x 0 + tx 1 , t∈ [0, 1].(1) The corresponding target velocity field is given by u t = x 1 − x 0 . CFM trains a neural network v θ (x t ,t,c) to approximate this velocity field by minimizing: L CFM (θ) =E t,x 0 ,x 1 ,c ∥v θ (x t ,t,c)− (x 1 − x 0 )∥ 2 .(2) During inference, samples are generated by solving the ODE dx/dt = v θ (x t ,t,c) from t = 0 to 1.To strengthen the model’s adherence to the target condition c (e.g., se- mantic tokens and speaker prompts), Classifier-Free Guidance (CFG) [23] is widely adopted. It explicitly modifies the velocity field by extrapolating away from the unconditional prediction: ̃v(x t ,t,c) = (1 + w)v θ (x t ,t,c)− wv θ (x t ,t,∅),(3) where w is the guidance scale. While CFG effectively improves conditional fidelity, it requires two network forward passes per integration step, doubling the computational cost and exacer- bating inference latency. 2.2. Data-guidance via heterogeneous augmentation Timbre leakage fundamentally stems from an information rout- ing problem during model optimization. In conditional Flow Matching (FM) for zero-shot generation, the model is guided by two distinct prompts: semantic tokens (intended for linguistic content) and acoustic prompts (e.g., mel-spectrogram, speaker embedding). However, discrete semantic tokens are rarely pure; they inherently retain residual acoustic hints from the source au- dio. During standard reconstruction training, if the semantic to- kens and the target acoustic prompt share similar acoustic prop- erties, the network tends to take an information shortcut. In- stead of learning to properly disentangle the two inputs, the model passively extracts overlapping acoustic clues directly from the semantic tokens. Consequently, during zero-shot in- ference, this entangled dependency causes the residual source acoustics within the tokens to overpower the target prompt, leading to severe timbre leakage. To address this, we propose Data-guidance (DG), formu- lated as a Dual-Stage Heterogeneous Perturbation strategy. Our objective is not merely to purify the tokens, but to deliberately degrade their acoustic reliability during training, thereby mit- igating the acoustic shortcut. As illustrated in Figure 1, we achieve this by constructing heavily mismatched training pairs through two cascaded steps: • Model-driven cross-synthesis: Inspired by [13], we first feed the original source semantic tokens through a generative system (e.g., a pretrained VC or TTS model) to synthesize an intermediate speech waveform. While this step introduces a preliminary identity shift, existing systems possess limited disentanglement capabilities. Consequently, the intermediate waveform inevitably retains latent acoustic residue that the model could still exploit. • Signal-driven acoustic deformation: To further suppress these remaining acoustic shortcuts, we introduce a secondary perturbation. Building upon proven speech augmentation principles [25], we apply explicit signal-processing transfor- mations, randomized pitch shifting and energy scaling to the intermediate waveform. This deliberate deformation effec- tively disrupts the residual acoustic information without al- tering the underlying phonetics. The final semantic tokens, extracted from this deformed au- dio, serve as the augmented condition ̃c. During training, the FM model must reconstruct the target speech x 1 from the mis- matched ̃c and the clean acoustic prompt. Since the acoustic properties of ̃c have been systematically degraded, the network is encouraged to extract linguistic content from the semantic to- kens while relying primarily on the acoustic prompt for timbre rendering. 2.3. Enhanced Model-guidance for unified acceleration The slow inference of flow matching stems from two factors: the iteration overhead due to curved ODE paths (requiring many NFE) and the guidance overhead from CFG (requiring dual for- ward passes). Previous works [17, 18, 24] typically treat these as separate challenges. We bypass this complexity with an Enhanced Model- guidance (MG) mechanism that effectively integrates intrinsic guidance distillation [24] and trajectory rectification [17] within a single, online training loop. As illustrated in Figure 2, for each training batch, the optimization alternates between two sequen- tial steps utilizing the augmented condition ̃c. 2.3.1. Model-guidance distillation First, we infuse the CFG mechanism directly into the network weights. We formulate an intrinsic guidance objective that re- places the standard CFM target with a guided velocity field: v ′ target = (x 1 − x 0 ) + w· sg v θ (x t ,t, ̃c)− v θ (x t ,t,∅) , (4) where sg(·) denotes the stop-gradient operation. The network is updated via a single backward pass to minimize the distillation loss: L Distill (θ) =E t,x 0 ,x 1 , ̃c h v θ (x t ,t, ̃c)− v ′ target 2 i .(5) By optimizingL Distill , the network intrinsically learns the CFG- aware vector field, enabling it to generate highly condition- aligned results using only a single forward pass v θ (x t ,t, ̃c) at inference time. 2.3.2. Trajectory rectification Following the distillation step, we leverage the updated, CFG- distilled network to straighten the generation trajectory. Using Figure 1: Illustration of the Data-guidance (DG) strategy. DG employs a dual-stage heterogeneous perturbation pipeline. By cascading model-driven cross-synthesis with signal-driven acoustic deformation, it constructs severely mismatched training pairs. Figure 2: Overview of the Enhanced Model-guidance (MG) mechanism. MG operates within a unified batch-level training loop. It first distills the CFG-aware vector field directly into the network weights. Subsequently, it utilizes the updated model to integrate straight trajectory paths. the same batch of prior noise z 0 = x 0 and the augmented con- dition ̃c, we perform numerical ODE integration to predict the corresponding clean speech feature ˆz 1 : ˆz 1 = ODE(v θ ,z 0 , ̃c).(6) Because the current model v θ has already absorbed the CFG knowledge, this generation does not require double forward passes, making the on-the-fly simulation efficient. We then con- struct a rectified, straight probability path z t = (1−t)z 0 +tˆz 1 and update the network to track this linear trajectory: L Rectify (θ) =E t,z 0 ,ˆz 1 , ̃c ∥v θ (z t ,t, ̃c)− (ˆz 1 − z 0 )∥ 2 .(7) 3. Experiments 3.1. Experimental setup 3.1.1. Datasets We employ a two-stage data construction strategy to balance pre-training scale with optimization quality. First, to establish a robust initialization, the foundation model is pre-trained on 50k hours of English speech from the Emilia dataset [26] under a standard matched-condition setting. Second, to enforce the pro- posed Data-guidance (DG) constraints, we curate a high-quality subset of 30k hours by filtering for DNSMOS [27] scores above 3.2. We then apply our dual-stage heterogeneous perturbation pipeline (model-driven cross-synthesis followed by randomized pitch and energy shifting) to this subset. This generates an ad- ditional 30k hours of acoustically mismatched but linguistically identical pairs. The original and perturbed subsets are combined to form a 60k-hour mixed corpus, which serves as the training data for all subsequent Enhanced Model-guidance (MG) opti- mization stages. 3.1.2. Experimental configuration Model architecture: We build a token-based flow match- ing model inspired by CosyVoice2 [5]. Unlike its original convolution-augmented architecture, our decoder adopts a pure Diffusion Transformer (DiT) [28]. The model consists of 20 stacked DiT blocks, each configured with an attention dimen- sion of 1024 and a feed-forward network (FFN) dimension of 4096, totaling approximately 330M parameters. Instead of con- catenating speaker embeddings with the input tokens, we in- ject speaker information through Adaptive Layer Normalization (AdaLN) [29], which provides more effective and fine-grained timbre control in practice. Training details: All training processes are conducted on 16 NVIDIA H100 GPUs. Our foundation Flow Matching model is first pre-trained for 5 epochs. We optimize the network using AdamW, with the learning rate linearly warmed up to a peak of 1 × 10 −4 over the first 1,000 steps, followed by a cosine annealing decay to 1× 10 −5 . This pre-training stage takes ap- proximately 48 hours. Building upon this initialization, we deploy our unified op- timization pipeline on the 60k-hour mixed corpus. The En- hanced MG is executed for 2 epochs. Specifically, to achieve true joint optimization, we perform two distinct backward passes for each mini-batch: First, we compute and backprop- agate the Intrinsic Guidance Distillation loss (Equation 5) to internalize the CFG field. Second, we immediately utilize the updated weights to simulate straight trajectories on the fly with the exact same batch of noise, backpropagating the Rectification loss (Equation 7). During this unified optimization phase, we freeze all modules except the DiT decoder to ensure the model efficiently learns the straightened flow paths without catastroph- ically forgetting its pre-trained linguistic representations. Due to the online training, this stage takes approximately 90 hours. 3.2. Evaluation metrics To assess the performance of our guidance-enhanced Flow Matching, we consider the following objective metrics: Real-time factor (RTF): Measures inference speed. We report RTF for the Flow Matching model alone, isolating its impact from other pipeline components. Speaker similarity (SIM): Cosine similarity between speaker embeddings of generated speech and reference prompts. We extract embeddings using Cam++ model [30]. Word error rate (WER): Assesses linguistic intelligibility by comparing Whisper-Large ASR [31] transcriptions with the ground-truth text. 3.3. Results and discussion We report evaluation results separately for Voice Conversion (VC) and Text-to-Speech (TTS) to highlight task-specific per- formance. All samples are generated on an NVIDIA RTX 4090 GPU. VC is conditioned on semantic tokens from source speech, while TTS uses semantic features derived from text via the CosyVoice2 LLM [5] and synthesizes audio with the HiFT- Net Vocoder [32]. 3.3.1. Voice Conversion (VC) performance Table 1 presents the ablation results for Voice Conversion under both Parallel (same speaker, different audio) and Non-Parallel (cross-speaker) settings. We first isolate the effect of Data- guidance (DG). Applying DG alone yields the highest Speaker Similarity (SIM) scores, exceeding the Ground Truth SIM, sug- gesting that our heterogeneous perturbation effectively encour- ages the model to ignore source acoustic residue. This confirms that our heterogeneous perturbation successfully forces the model to ignore source acoustic residue. Conversely, address- ing only the inference bottleneck via Model-guidance (MG) re- duces latency but incurs trade-offs. While Vanilla MG inter- nalizes the CFG logic to lower the RTF from 0.078 to 0.058, our Enhanced MG further incorporates trajectory rectification to achieve an extreme 3-step inference. However, without data- level correction, this acceleration leads to a minor degradation in SIM. Finally, our unified guidance framework resolves this dilemma by combining the 3-step Enhanced MG with the DG strategy. Although its absolute SIM is marginally lower than the computationally heavy DG-only model, it achieves a sub- stantial 3.25× speedup, mitigating the performance drop seen in the pure Enhanced MG model and outperforming the 10-step Base baseline. Crucially, the Unified framework demonstrates strong zero-shot robustness: its Non-Parallel SIM on LibriTTS (0.808) not only surpasses the Base model but even exceeds the Ground Truth Parallel SIM (0.799). This confirms that despite operating under 3-step acceleration without CFG, our frame- work achieves highly effective content-timbre disentanglement, offering a practical solution for real-time deployment. 3.3.2. Text-to-Speech (TTS) performance To evaluate the generalization capability of our framework, we extend our experiments to the zero-shot TTS task. We utilize Table 1: Results for Voice Conversion on LibriTTS and Seed- TTS. The table demonstrates the individual and combined im- pact of the Data-guidance (DG) strategy and Enhanced Model- guidance (MG) mechanism. MethodRTF (↓) SIM (↑) ParallelNon-Parallel LibriTTSSeed-TTSLibriTTSSeed-TTS Reference (GT)-0.7990.7890.0730.128 Base Model (10 NFE)0.0780.8740.8000.7930.730 Data-guidance (DG) + DG (10 NFE)0.0780.8970.8220.8690.792 Model-guidance (MG) + Vanilla MG (10 NFE)0.0580.8850.8100.8130.744 + Enhanced MG (3 NFE) 0.0240.8700.7910.7920.722 Unified Guidance (3 NFE)0.0240.8870.8080.8500.767 Table 2: Text-to-Speech (TTS) performance on LibriTTS and Seed-TTS. MethodLibriTTSSeed-TTS WER (↓)SIM (↑)WER (↓)SIM (↑) Reference (GT)2.120.7991.820.789 CosyVoice22.570.8472.470.750 Base Model 2.570.8712.220.794 Unified Guidance2.600.8882.450.806 the CosyVoice2 LLM [5] to generate semantic tokens and ex- clusively swap the Flow Matching (FM) model. As reported in Table 2, our 3-step Unified FM backend maintains comparable intelligibility compared to the official CosyVoice2 backend. While extreme trajectory rectification introduces a slight degradation in WER, the linguistic stabil- ity remains robust. More importantly, when paired with the same LLM, our unified model achieves a higher SIM score outperforming the unoptimized Base FM model. This demon- strates that our Data-guidance strategy effectively filters out la- tent acoustic variance inherently predicted by the LLM, anchor- ing the timbre to the target prompt. Overall, our framework serves as a robust, efficient acoustic detokenizer for existing TTS systems. 4. Conclusion This paper presented a unified guidance framework that ad- dresses the distinct bottlenecks of timbre leakage and high inference latency in Flow Matching-based speech generation. On the data side, our Data-guidance strategy employs dual- stage heterogeneous perturbation to sever acoustic shortcuts during training, promoting robust content-timbre disentangle- ment. On the model side, our Enhanced Model-guidance mech- anism integrates intrinsic guidance distillation with online tra- jectory rectification, entirely eliminating CFG overhead. Ex- periments on both Voice Conversion and Text-to-Speech bench- marks show that the proposed framework achieves a 3.25× in- ference speedup while consistently improving zero-shot speaker similarity. Furthermore, when deployed as an acoustic backend for existing TTS systems, our framework maintains competitive intelligibility while delivering improved speaker fidelity. These results demonstrate that our unified approach offers a practical and effective solution for real-time speech generation. 5. Acknowledgments The authors thank the Zuoyebang Speech Team for providing the computing power and related platforms that made this re- search possible. We also acknowledge the open-source com- munity for the models and datasets used in this study. 6. Generative AI Use Disclosure The authors used generative AI tools (Gemini 3.0 pro) solely for the purpose of checking LaTeX formatting, correcting syntax errors, and refining the layout of this manuscript. No AI tools were used to generate scientific content, experimental data, or the intellectual ideas presented in this work. 7. References [1] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=PqvMRD CJT9t [2] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. JianZhao, K. Yu, and X. Chen, “F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds.Vienna, Austria: Association for Computational Linguistics, Jul. 2025, p. 6255–6271. [Online]. Available: https://aclanthology.org/2025.acl-long.313/ [3] S. Mehta, R. Tu, J. Beskow, ́ E. Sz ́ ekely, and G. E. Henter, “Matcha-TTS: A fast TTS architecture with conditional flow matching,” in Proc. ICASSP, 2024. [4] Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y. Yang, H. Hu, S. Zheng, Y. Gu, Z. Ma et al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv preprint arXiv:2407.05407, 2024. [5] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024. [6] Z. Du, C. Gao, Y. Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, X. Shi, K. An et al., “Cosyvoice 3: Towards in-the- wild speech generation via scaling-up and post-training,” arXiv preprint arXiv:2505.17589, 2025. [7] X. Lyu, Y. Wang, T. Zhao, H. Wang, H. Liu, and Z. Du, “Build llm-based zero-shot streaming tts system with cosyvoice,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, p. 1–2. [8] Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text- to-speech with masked generative codec transformer,” in ICLR. OpenReview.net, 2025. [9] H.-H. Guo, Y. Hu, K. Liu, F.-Y. Shen, X. Tang, Y.-C. Wu, F.- L. Xie, K. Xie, and K.-T. Xu, “Fireredtts: A foundation text-to- speech framework for industry-level generative speech applica- tions,” arXiv preprint arXiv:2409.03283, 2024. [10] D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tang et al., “Kimi-audio technical report,” arXiv preprint arXiv:2504.18425, 2025. [11] A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y. Dong, and J. Tang, “Glm-4-voice:Towards intelligent and human-like end-to-end spoken chatbot,” 2024. [Online]. Available: https://arxiv.org/abs/2412.02612 [12] A. Huang, B. Wu, B. Wang, C. Yan, C. Hu, C. Feng, F. Tian, F. Shen, J. Li, M. Chen et al., “Step-audio: Unified understanding and generation in intelligent speech interaction,” arXiv preprint arXiv:2502.11946, 2025. [13] S. Liu, “Zero-shot voice conversion with diffusion transformers,” 2024. [Online]. Available: https://arxiv.org/abs/2411.09943 [14] J. Yao, Y. Yuguang, Y. Pan, Z. Ning, J. Ye, H. Zhou, and L. Xie, “Stablevc: style controllable zero-shot voice conversion with conditional flow matching,” in Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, ser. AAAI’25/IAAI’25/EAAI’25.AAAI Press, 2025. [Online]. Available: https://doi.org/10.1609/aaai.v39i24.34 758 [15] T. V. Ho and M. Akagi, “Non-parallel Voice Conversion based on Hierarchical Latent Embedding Vector Quantized Variational Autoencoder,” in Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, 2020, p. 140–144. [16] P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao et al., “Seed-tts: A family of high-quality versatile speech generation models,” arXiv preprint arXiv:2406.02430, 2024. [17] X. Liu, C. Gong, and qiang liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=XVjTT1 nw5z [18] X. Liu, X. Zhang, J. Ma, J. Peng, and qiang liu, “Instaflow: One step is enough for high-quality diffusion-based text-to- image generation,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available:https: //openreview.net/forum?id=1k4yZbbDqX [19] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever, “Consistency models,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML’23. JMLR.org, 2023. [20] R. Huang, Z. Zhao, H. Liu, J. Liu, C. Cui, and Y. Ren, “Prodiff:Progressive fast diffusion model for high-quality text-to-speech,” in Proceedings of the 30th ACM International Conference on Multimedia, ser. M ’22.Association for Computing Machinery, 2022, p. 2595–2605. [Online]. Available: https://doi.org/10.1145/3503161.3547855 [21] R. Huang, M. W. Lam, J. Wang, D. Su, D. Yu, Y. Ren, and Z. Zhao, “Fastdiff: A fast conditional diffusion model for high- quality speech synthesis,” 2022. [22] Z. Ye, W. Xue, X. Tan, J. Chen, Q. Liu, and Y. Guo, “Comospeech: One-step speech and singing voice synthesis via consistency model,” in Proceedings of the 31st ACM International Conference on Multimedia.Association for Computing Machinery, 2023, p. 1831–1839. [Online]. Available: https://doi.org/10.1145/3581783.3612061 [23] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. [Online]. Available: https://openreview.net/forum?id=qw8AKxfYbI [24] Z. Tang, D. Chen, J. Bao, and B. Guo, “Diffusion models without classifier-free guidance,” 2025. [Online]. Available: https://openreview.net/forum?id=kkiLdrKk0G [25] E. Kharitonov, M. Rivi ` ere, G. Synnaeve, L. Wolf, P.-E. Mazar ́ e, M. Douze, and E. Dupoux, “Data augmenting contrastive learning of speech representations in the time domain,” in 2021 IEEE Spo- ken Language Technology Workshop (SLT), 2021, p. 215–222. [26] H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, p. 885–890. [27] C. K. A. Reddy, V. Gopal, and R. Cutler, “Dnsmos: A non- intrusive perceptual objective speech quality metric to evalu- ate noise suppressors,” in ICASSP 2021 - 2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, p. 6493–6497. [28] W. Peebles and S. Xie, “Scalable diffusion models with transform- ers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, p. 4195–4205. [29] J. Xu, X. Sun, Z. Zhang, G. Zhao, and J. Lin, “Understanding and improving layer normalization,” in Advances in Neural Informa- tion Processing Systems (NeurIPS), vol. 32, 2019. [30] H. Wang, S. Zheng, Y. Chen, L. Cheng, and Q. Chen, “CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking,” in Interspeech 2023, 2023, p. 5301– 5305. [31] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202.PMLR, 23– 29 Jul 2023, p. 28 492–28 518. [Online]. Available: https: //proceedings.mlr.press/v202/radford23a.html [32] Y. A. Li, C. Han, X. Jiang, and N. Mesgarani, “Hiftnet: A fast high-quality neural vocoder with harmonic-plus-noise filter and inverse short time fourier transform,” 2023. [Online]. Available: https://arxiv.org/abs/2309.09493