Paper deep dive
MOSS-TTSD: Text to Spoken Dialogue Generation
Yuqian Zhang, Donghua Yu, Zhengyuan Lin, Botian Jiang, Mingshu Chen, Yaozhou Jiang, Yiwei Zhao, Yiyang Zhang, Yucheng Yuan, Hanfu Chen, Kexin Huang, Jun Zhan, Cheng Chang, Zhaoye Fei, Shimin Li, Xiaogui Yang, Qinyuan Cheng, Xipeng Qiu
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/23/2026, 12:08:40 PM
Summary
MOSS-TTSD is a spoken dialogue synthesis model designed for expressive, multi-party, long-form conversations. It utilizes a Qwen3-8B-base LLM backbone and MOSS-Audio-Tokenizer to support up to 60 minutes of single-pass synthesis, zero-shot voice cloning, and multi-party interactions with up to 5 speakers. The authors also introduce TTSD-eval, an objective evaluation framework based on forced alignment to measure speaker attribution accuracy and similarity without relying on traditional diarization tools.
Entities (5)
Relation Signals (3)
MOSS-TTSD â usesbackbone â Qwen3-8B-base
confidence 100% · it uses Qwen3-8B-base [ 6] as the autoregressive backbone
MOSS-TTSD â usestokenizer â MOSS-Audio-Tokenizer
confidence 100% · together with MOSS-Audio-Tokenizer [7]
TTSD-eval â evaluates â MOSS-TTSD
confidence 95% · Both objective and subjective evaluation results show that MOSS-TTSD surpasses strong open-source and proprietary baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spoken dialogue generation is crucial for applications like podcasts, dynamic commentary, and entertainment content, but poses significant challenges compared to single-utterance text-to-speech (TTS). Key requirements include accurate turn-taking, cross-turn acoustic consistency, and long-form stability, which current models often fail to address due to a lack of dialogue context modeling. To bridge this gap, we present MOSS-TTSD, a spoken dialogue synthesis model designed for expressive, multi-party conversational speech across multiple languages. With enhanced long-context modeling, MOSS-TTSD generates long-form spoken conversations from dialogue scripts with explicit speaker tags, supporting up to 60 minutes of single-pass synthesis, multi-party dialogue with up to 5 speakers, and zero-shot voice cloning from a short reference audio clip. The model supports various mainstream languages, including English and Chinese, and is adapted to several long-form scenarios. Additionally, to address limitations of existing evaluation methods, we propose TTSD-eval, an objective evaluation framework based on forced alignment that measures speaker attribution accuracy and speaker similarity without relying on speaker diarization tools. Both objective and subjective evaluation results show that MOSS-TTSD surpasses strong open-source and proprietary baselines in dialogue synthesis.
Tags
Links
- Source: https://arxiv.org/abs/2603.19739v1
- Canonical: https://arxiv.org/abs/2603.19739v1
Trouble viewing inline? Open PDF directly â
Full Text
35,425 characters extracted from source content.
Expand or collapse full text
OpenMOSS MOSS-TTSD: Text to Spoken Dialogue Generation SII-OpenMOSS Team * Abstract Spoken dialogue generation is crucial for applications like podcasts, dynamic commentary, and en- tertainment content, but poses significant challenges compared to single-utterance text-to-speech (TTS). Key requirements include accurate turn-taking, cross-turn acoustic consistency, and long-form stability, which current models often fail to address due to a lack of dialogue context modeling. To bridge this gap, we present MOSS-TTSD, a spoken dialogue synthesis model designed for expressive, multi-partyconversationalspeechacrossmultiplelanguages. Withenhancedlong-contextmodeling, MOSS-TTSD generates long-form spoken conversations from dialogue scripts with explicit speaker tags, supporting up to 60 minutes of single-pass synthesis, multi-party dialogue with up to 5 speak- ers, and zero-shot voice cloning from a short reference audio clip. The model supports various main- stream languages, including English and Chinese, and is adapted to several long-form scenarios. Additionally, to address limitations of existing evaluation methods, we propose TTSD-eval, an ob- jective evaluation framework based on forced alignment that measures speaker attribution accuracy and speaker similarity without relying on speaker diarization tools. Both objective and subjective evaluation results show that MOSS-TTSD surpasses strong open-source and proprietary baselines in dialogue synthesis. a Code: https://github.com/OpenMOSS/MOSS-TTSD Model:https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0 Ă TTSD-eval:https://github.com/OpenMOSS/TTSD-eval 1Introduction Generating spoken dialogue is more challenging than monologue speech due to the need for accurate and natural turn-taking, robust speaker switching, and long-range contextual coherence across multiple turns. Models must maintain speaker-specific timbre and prosody within and across turns and produce long-form audio without stitching artifacts. While current text-to-speech synthesis systems (TTS) has made substantial progress for short, single-speaker utterances [1â5], gaps remain in coherence, spontaneity, and robustness for real-world conversational scenarios. To address these challenges, we introduce MOSS-TTSD (Text-to-Spoken Dialogue), a spoken dialogue syn- thesis model built for expressive, multi-party conversations. Since its initial release, MOSS-TTSD has evolved through multiple iterations from v0 to v1.0; this paper focuses on the latest release,MOSS-TTSD v1.0. Un- less otherwise specified, all descriptions and evaluations in this paper refer to MOSS-TTSD v1.0. MOSS-TTSD follows a fully discrete speech generation paradigm: it uses Qwen3-8B-base [ 6] as the autore- gressive backbone together with MOSS-Audio-Tokenizer [7]. Inspired by MusicGen, we adopt a multi-head â Full contributors can be found in the Contributors section. 1 arXiv:2603.19739v1 [cs.SD] 20 Mar 2026 delay pattern [8] to autoregressively predict RVQ codebook tokens, and we model only the first 16 RVQ lay- ers to enable robust long-context generation at a low bitrate. Conditioned on dialogue scripts with explicit speaker tags (e.g.,[S1]/[S2]) and optional per-speaker reference audio, MOSS-TTSD supports accurate turn-taking and zero-shot multi-speaker voice cloning, and scales to long-form synthesis, generating coher- ent conversations of up to 60 minutes in a single pass without stitching artifacts. The key features of MOSS-TTSD can be summarized as: âąLong-form script-to-conversation generation.MOSS-TTSD converts dialogue scripts into spoken con- versations and supports up to 60 minutes of single-pass generation. âąMulti-party voice cloning.MOSS-TTSD supports up to 5 speakers in a single dialogue session and provides zero-shot voice cloning from a short reference audio clip to improve speaker consistency. âąMultilingual and scenario coverage.MOSS-TTSD provides strong support for multiple languages, such as English, Chinese, Spanish, Portuguese, German, French, Japanese, Korean, and Russian. It is adapted to scenarios such as podcasts, dynamic commentary, and entertainment content like audio- books, dubbing, and crosstalk. Additionally, to address the limitations of existing evaluation metrics that rely on speaker diarization tools, we introduce TTSD-eval, an objective evaluation framework that leverages forced alignment to measure speaker attribution, speaker similarity, and intelligibility in long-form, multi-speaker scenarios. Compre- hensive evaluations with TTSD-eval, together with human subjective assessments, demonstrate that MOSS- TTSD outperforms strong open-source and proprietary baselines for spoken dialogue synthesis. 2MOSS-TTSD History Since June 2025, MOSS-TTSD has evolved through several iterations, progressively improving long-form stability, speaker consistency, and feature coverage. MOSS-TTSD v0 (2025-06).We released MOSS-TTSD v0 with bilingual support for Chinese and English, and up to 960s single-pass generation. In the data pipeline, we used WhisperD [ 9] for English ASR, and an internal diarization-enabled Whisper model fine-tuned from Whisper-large-v3 [10] for Chinese. Due to the ASR modelâs 30s per-request audio constraint, multi-speaker transcripts were constructed by combin- ing coarse-grained speaker segments from a diarization model with fine-grained, token-level speaker labels predicted by the ASR model, enabling explicit speaker annotation over long recordings. MOSS-TTSD v0.5 (2025-07).In v0.5, we improved timbre switching and zero-shot voice cloning robustness by fine-tuning on a high-quality subset of training data curated with a TTSD-eval-like pipeline. We also introduced a 32kHz variant of the XY-Tokenizer to support higher-fidelity synthesis. MOSS-TTSD v0.7 (2025-11).v0.7 substantially overhauled the data pipeline by adopting an early internal version of MOSS Transcribe Diarize [11] to generate ASR transcripts with explicit speaker tags end-to-end, reducing errors caused by inconsistencies between diarization outputs and transcript-level speaker labeling. In addition, we refined the XY-Tokenizer decoder so that decoding is explicitly conditioned on both the LLM-generated tokens and the speaker reference tokens, which further improved voice cloning capability and audio quality. These improvements also extended the maximum single-pass generation length from 960s to 1700s. MOSS-TTSD v1.0 (2026-02).MOSS-TTSD v1.0 further strengthens long-form synthesis, benefiting from scaled data and a stronger LLM backbone, as well as continued improvements in the audio tokenizer and data pipeline. It supports up to 60-minute single-session context and multi-party interactions with up to five speakers, and expands beyond Chinese and English to multiple additional mainstream languages. 3Data Engineering 2 Figure 1Overview of the MOSS-TTSD data pipeline: raw audio is normalized and diarized, then merged into clips with varying speaker counts and annotated with quality, language, and sampling-rate metadata; noise-heavy domains may undergo additional denoising. Clips are transcribed end-to-end with explicit speaker tags, followed by heuristic and audio/text language-consistency filtering to produce the final training set. 3.1Data Pipeline Figure1provides an overview of our data pipeline. MOSS-TTSD shares the same foundational data process- ing pipeline as MOSS-TTS [12], encompassing raw data collection, audio normalization, and speaker diariza- tion. Followingthisbaselineprocessing, weobtaindiarizationsegments. Continuoussegmentsderivedfrom the same raw recording are merged into single-speaker, two-speaker, or multi-speaker (3â5 speakers) clips, with a maximum duration capped at 3600s. Each clip is annotated with a DNSMOS score [13], a language label generated by Whisper-large-v3 [10], and the true sample rate estimated from the Mel-spectrogram, serving as criteria for subsequent filtering. Subsequently, leveraging its superior performance in long-context and multi-speaker scenarios, we employ MOSSTranscribeDiarize[11]astheASRmodeltoprocesseachclipinanend-to-endmanner, simultaneously yielding the ASR transcript and explicit speaker tags. We implement a set of heuristic rules to filter out low- quality audio and samples exhibiting ASR hallucinations. Additionally, a lightweight language detection tool, fastText [14,15], is used to categorize the language of the ASR transcripts; only samples where the audio language label matches the transcript language category are retained. Furthermore, for specific audio types characterized by significant background noisesuch as movies, TV shows, sports commentary, and esports commentarywe utilize MossFormer2 [16] for additional denoising. Finally, for the training dataset, we strictly retain samples with DNSMOSâ„2.8. 3.2Data Synthesis and Augmentation Voice cloning data.To equip the model with multi-speaker voice cloning capabilities, we construct a subset of data augmented with voice references. Specifically, for two-speaker and multi-speaker clips, we identify non-overlappingsingle-speakersegments extractedfrom thesame rawrecording. Basedon thespeakeriden- tities derived from the diarization annotations, we map these single-speaker segments to the corresponding speaker reference slots within the prompt template. The chat templates used for both the voice cloning data and the common TTS data are provided in AppendixA. Synthesis data.Accurately annotating speaker identities over long contexts in multi-speaker scenarios (3â 5 speakers) poses a significant challenge, both for traditional speaker diarization models and for end-to- end systems like MOSS Transcribe Diarize that simultaneously generate speaker tags and ASR transcripts. 3 Figure 2MOSS-TTSD inference for multi-speaker voice cloning. Given a text prompt with speaker tags, the model conditions on per-speaker reference audio and continues the spoken dialogue by generating discrete speech tokens, unifying reference-conditioned cloning with continuation-based cloning. Consequently, we filter our dataset, retaining only a small fraction of high-quality multi-speaker clips from real-world data. To enhance the modelâs speaker identity consistency over extended contexts, we augment the dataset with synthetic samples constructed by concatenating single-speaker clips. Specifically, single- speaker segments attributed to the same speaker by the diarization model are clustered into groups. These distinct speaker groups are then interleaved according to predefined rules to simulate the format of multi- speaker conversations. To ensure natural transitions in background acoustics when merging clips from di- verse raw recordings, we strictly select segments with a DNSMOS score ofâ„3.4and require all constituent clips within a single synthetic sample to share an identical sample rate. Text augmentation.Certain typographical symbols are relatively scarce in real-world speech datasets, and Automatic Speech Recognition (ASR) models typically exhibit inherent biases toward generating a limited set of common punctuation marks. To mitigate these biases and improve the modelâs robustness to diverse text inputs, we apply rule-based text augmentation to enhance punctuation diversity in the transcripts of a selected subset of training samples. The specific substitution rules are detailed in AppendixB. 4Method 4.1Architecture MOSS-TTSD and MOSS-TTS [12] share the same architecture and both adopt a fully discrete speech gener- ation approach. For sequence modeling, inspired by MusicGen, we use autoregressive generation with a multi-head delay pattern for speech tokens. We use Qwen3-8B-base [6] as the LLM backbone and MOSS-Audio-Tokenizer [7] as the audio tokenizer. Unlike MOSS-TTS, MOSS-TTSD models only the RVQ tokens from the first 16 layers of the audio tokenizer. Thanks to the strong performance of MOSS-Audio-Tokenizer at a low bitrate of 2 kbps with a 12.5 Hz frame rate, the LLM can robustly model audio sequences in long-context, multi-speaker settings, enabling training contexts of up to 3600 seconds and up to five speakers. 4 Figure 3Overview of TTSD-eval. Given the input script with explicit speaker tags and the generated audio, TTSD-eval uses forced alignment to obtain word-level timestamps and segments the audio into utterance fragments. ACC and SIM are computed from speaker-embedding similarities between each fragment and the reference voices. 4.2Curriculum Learning Starting from MOSS-TTS, MOSS-TTSD adopts a three-stage curriculum to shift from single-speaker synthesis to natural, high-quality, long-context multi-speaker dialogue generation. Stage 1.We perform continued pre-training from an intermediate MOSS-TTS checkpoint based on Qwen3- 8B-base, pretrained on single-speaker TTS data with a sequence length of 32,768 tokens. In this stage, we include all single and two-speaker data with DNSMOSâ„2.8as well as voice cloning data, and extend the se- quence length to 65,536 tokens. The goal is to adapt the model to longer contexts and to learn explicit control of speaker identity via speaker tags and reference audio, along with natural turn-taking in real dialogues. Stage 2.We restrict training to the subset of Stage 1 data with DNSMOSâ„3.4and true sample rateâ„24 kHz, reduce the sampling ratio of single-speaker data, and lower the learning rate. This stage strengthens high-fidelity dialogue synthesis while preserving single-speaker capability. Stage 3.On top of all Stage 2 data, we add multi-speaker data from real raw recordings and a suitable proportion of synthetic data. This enables the model to generate high-quality speech for 1â5 speakers and improves turn-switching stability and speaker attribution accuracy with almost no degradation in speech naturalness. 4.3Inference Explicitly constructed voice cloning data teach the model to clone timbre from the reference slots defined in the chat templates. In addition, autoregressive TTS models naturally support voice cloning by continuation. We find that combining these two paradigms, as illustrated in Figure 2, substantially improves voice cloning performance; detailed results are reported in AppendixC. We define this setting asvoice_clone_and_- continuationand use it as the default inference configuration. 5Evaluations 5 Table 1TTSD-eval results.ACCdenotes speaker attribution accuracy,SIMrepresents speaker voice similarity, andWERdenotes Word Error Rate (lower is better). Best results are inbold. Model ZH EN ACCâSIMâWERâACCâSIMâWERâ Comparison with Open-Source Models Higgs Audio V2 [17]--- 0.9025 0.6860 21.31% FireRedTTS-2 [18]0.9022 0.7383 7.68% --- VibeVoice 1.5B [19]0.8798 0.7415 8.18% 0.9353 0.6961 11.33% VibeVoice 7B [19]0.9222 0.7590 5.70% 0.9554 0.71409.46% MOSS-TTSD (ours)0.9587 0.7949 4.85% 0.9626 0.73269.88% Comparison with Proprietary Models Eleven V30.9653 0.69703.63%0.9498 0.67308.24% MOSS-TTSD (elevenlabs_voice) 0.9736 0.81653.91%0.9565 0.730410.05% gemini-2.5-pro-preview-tts--- 0.9537 0.67868.59% gemini-2.5-flash-preview-tts--- 0.9511 0.7194 8.71% MOSS-TTSD (gemini_voice)---0.9655 0.78939.84% Doubao_Podcast0.9606 0.80344.72%--- MOSS-TTSD (doubao_voice) 0.9630 0.82265.71% --- 5.1TTSD-eval Traditional evaluation of spoken dialogue generation typically reports cpWER [20,21] to indirectly assess speaker-attribution accuracy, and cpSIM to measure speaker similarity. However, both metrics are often constrained by the performance of speaker diarization models, and cpWER is further affected by the robust- ness of the ASR system. As the number of speakers increases from two to five or more, the error introduced by speaker diarization tends to grow substantially. To address these limitations, we propose TTSD-eval, an objective evaluation framework based on forced alignment. The overall TTSD-eval pipeline is illustrated in Figure3. Evaluation Metric.In TTSD-eval, we first employ MMS-FA (Metaâs Massively Multilingual Speech Forced Alignment) to obtain word-level alignments between the input script and the generated audio. We then segment the audio into sentence fragments according to punctuation in the source text and assign speaker identities directly from the explicit speaker tags in the input script. We adopt wespeaker-SimAMResNet100 as the speaker embedding model. For each generated fragment, we compute its similarity to the prompt audio of all candidate speakers and assign the speaker with the highest score as the prediction. By comparing these predicted labels with the ground-truth speaker tags in the input script, we calculate theSpeaker Attribution Accuracy (ACC). Furthermore, we defineSpeaker Similarity (SIM)as the similarity between each generated fragment and the prompt audio of its ground-truth speaker. In addition, we reportWord Error Rate (WER)to assess intelligibility. We use Whisper-large-v3 as the ASR system for both Chinese and English. To ensure a fair comparison, we strip all speaker tags and sound event tags, and further apply the text normalization procedure used in Seed-TTS-eva [ 3] to both the reference text and the ASR transcription before computing WER. Test Sets.Our test sets consist of both Chinese and English subsets, with 50 dialogue samples for each language. For both languages, we construct the corresponding subset using 20 pairs of speaker timbre refer- 6 MOSS-TTSDVibevoice 7BVibevoice 1.5 BFireRedTTS2Higgs Audio V2 ZH - ACC 900 950 1000 1050 1100 1028.6 MOSS-TTSD 998.1 Vibevoice 7B 991.1 Vibevoice 1.5 B 982.2 FireRedTTS2 ZH - SIM 900 950 1000 1050 1100 1043.5 MOSS-TTSD 1030.8 Vibevoice 7B 990.9 Vibevoice 1.5 B 934.8 FireRedTTS2 ZH - Rhythm 900 950 1000 1050 1100 1027.0 MOSS-TTSD 1028.9 Vibevoice 7B 1005.4 Vibevoice 1.5 B 938.7 FireRedTTS2 ZH - Overall 900 950 1000 1050 1100 1041.5 MOSS-TTSD 1032.6 Vibevoice 7B 985.5 Vibevoice 1.5 B 940.3 FireRedTTS2 EN - ACC 900 950 1000 1050 1100 1008.4 MOSS-TTSD 1000.0 Vibevoice 7B 1000.0 Vibevoice 1.5 B 991.6 Higgs Audio V2 EN - SIM 900 950 1000 1050 1100 1008.5 MOSS-TTSD 1024.3 Vibevoice 7B 977.4 Vibevoice 1.5 B 989.9 Higgs Audio V2 EN - Rhythm 900 950 1000 1050 1100 1010.1 MOSS-TTSD 1010.4 Vibevoice 7B 989.6 Vibevoice 1.5 B 989.9 Higgs Audio V2 EN - Overall 900 950 1000 1050 1100 1001.8 MOSS-TTSD 1026.0 Vibevoice 7B 973.9 Vibevoice 1.5 B 998.3 Higgs Audio V2 Figure4EloratingsandconfidenceintervalsofMOSS-TTSDandotheropen-sourcemodelsonhuman-perceivedspeaker attribution accuracy (ACC), voice similarity (SIM), rhythm, and overall quality. ences manually collected from the Internet and 30 reference pairs sampled from seed-tts-eval; all dialogue texts are generated using Gemini 2.5 Pro. Audio durations range from approximately 30 to 720 seconds, cov- ering diverse scenarios from short dialogues to long-form multi-turn interactions, including podcasts, film and TV dubbing, sports and esports commentary, variety shows, animation dialogue, and crosstalk. Experimental Setup and Results.For open-source models, we directly follow the standard TTSD-eval pipeline and use the same test sets. Due to context-length and GPU-memory constraints, we set Higgs Audio V2âsgeneration_chunk_buffer_sizeto 6. Generation is performed chunk by chunk, and each new chunk is conditioned on the prompt and only the six most recently generated chunks. Under the same constraint, we implementedgeneration_chunk_buffer_sizefor FireRedTTS2 following Higgs Audio V2 and also set it to 6. For proprietary models, due to the constraints of voice cloning, we use speakers from each modelâs private voice library to ensure a fair comparison. MOSS-TTSD uses the same speakers as reference and performs zero-shot voice cloning for evaluation. We denote MOSS-TTSD with different reference voices as MOSS- TTSD (elevenlabs_voice), MOSS-TTSD (gemini_voice), and MOSS-TTSD (doubao_voice); the corresponding voice information is provided in AppendixD. The texts used for evaluation are identical to those in the TTSD- eval test sets. Due to the Eleven V3 APIâs 5,000-character limit per request, we split the input into 10-turn chunks, generate audio per chunk, and concatenate them. Except for Eleven V3, all other models perform single-pass generation. The results are summarized in Table 1, showing that MOSS-TTSD outperforms leading open-source and proprietary models in terms of speaker consistency and intelligibility in dialogue generation. 7 ZH vs doubao-podcast 35.8%32.6%31.6% vs Eleven V3 44.4%20.2%35.4% EN vs gemini-2.5-flash-preview-tts 44%22%34% vs gemini-2.5-pro-preview-tts 45.5%21.2%33.3% vs Eleven V3 33.3%25.3%41.4% 020406080100 Percentage (%) WinTieLose Figure 5Subjective preference results between MOSS-TTSD and other proprietary models. Bars report win/tie/lose rates of MOSS-TTSD against proprietary baselines in Chinese (ZH) and English (EN), where annotators select the overall preferred sample. 5.2Subjective Evaluation The subjective evaluation is conducted on the same test sets as TTSD-eval. To prevent annotators from losing focus when assessing long audio samples, we apply word-level alignment to segment each audio into clips no longer than 90 seconds for evaluation, and ensure that, for each rated audio pair, both samples correspond to the same text and speaker references. For open-source models, annotators are asked to score each sample pair in terms of speaker attribution accuracy, voice similarity, rhythm, and overall quality. Following the methodology of the LMSYS Chatbot Arena [22], we compute Elo ratings and confidence intervals for each dimension. The results are shown in Figure4. For closed-source models, annotators are only asked to choose the overall preferred one in each pair, and we compute the win rate accordingly. The results are shown in Figure5. Subjective evaluation results demonstrate that MOSS-TTSD surpasses leading open-source and proprietary models in terms of human-perceived quality. Furthermore, the strong correlation between human percep- tion of speaker consistency and TTSD-eval scores validates the reliability of TTSD-eval as an effective objec- tive metric. 6Conclusion In this work, we presented MOSS-TTSD, a spoken dialogue synthesis model for expressive and long-form multi-partyconversations. BuiltonafullydiscreteparadigmwithQwen3-8B-baseandMOSS-Audio-Tokenizer, MOSS-TTSD adopts a multi-head delay pattern and models only the first 16 RVQ layers to enable robust long- 8 context generation. Conditioned on dialogue scripts with explicit speaker tags and optional per-speaker references, it supports natural turn-taking, zero-shot multi-speaker voice cloning, and up to 5 speakers, scal- ing to 60 minutes of single-pass synthesis without stitching artifacts. The model covers various mainstream languages and multiple long-form scenarios such as podcasts, commentary, dubbing, and entertainment content. We further proposed TTSD-eval, a forced-alignment-based framework that measures speaker at- tribution accuracy (ACC) and speaker similarity (SIM) without relying on diarization tools, together with intelligibility via WER. Both objective and subjective evaluations show that MOSS-TTSD consistently outper- forms strong open-source and proprietary baselines for spoken dialogue generation. We release our code and models to support future research and real-world deployment. Contributors Contributors: Yuqian Zhang * , Donghua Yu, Zhengyuan Lin, Botian Jiang, Mingshu Chen, Yaozhou Jiang, Yiwei Zhao, Yiyang Zhang, Yucheng Yuan, Hanfu Chen, Kexin Huang, Jun Zhan, Cheng Chang, Zhaoye Fei, Shimin Li, Xiaogui Yang, Qinyuan Cheng. Advisors: Xipeng Qiu â . Affiliations: Shanghai Innovation Institute MOSI Intelligence Fudan University 1 *yuqianzhang24@m.fudan.edu.cnâ Corresponding author:xpqiu@fudan.edu.cn 9 References [1]Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers.CoRR, abs/2301.02111, 2023. doi: 10.48550/ARXIV.2301.02111. URLhttps://doi.org/10.48550/ arXiv.2301.02111. [2]Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 6255â6271. Association for Computational Linguistics, 2025. URL https://aclanthology.org/2025.acl-long.313/. [3]PhilipAnastassiou, JiaweiChen, JitongChen, YuanzheChen, ZhuoChen, ZiyiChen, JianCong, LelaiDeng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-tts: A family of high-quality versatile speech generation models.CoRR, abs/2406.02430, 2024. doi: 10.48550/ARXIV.2406.02430. URL https://doi.org/10.48550/arXiv.2406.02430. [4]Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, Lei Xie, Yike Guo, and Wei Xue. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens.CoRR, abs/2503.01710, 2025. doi: 10.48550/ARXIV.2503.01710. URLhttps://doi.org/10.48550/arXiv.2503.01710. [5]Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, Hongzhan Lin, Jianyi Chen, Xingjian Du, Liumeng Xue, Yunlin Chen, Zhifei Li, Lei Xie, Qiuqiang Kong, Yike Guo, and Wei Xue. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis.CoRR, abs/2502.04128, 2025. doi: 10.48550/ARXIV.2502.04128. URLhttps://doi.org/10.48550/arXiv.2502.04128. [6]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report.CoRR, abs/2505.09388, 2025. doi: 10.48550/ARXIV.2505.09388. URLhttps://doi.org/10.48550/arXiv.2505.09388. [7]Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruix- iao Li, Qingyuan Cheng, Shimin Li, and Xipeng Qiu. Moss-audio-tokenizer: Scaling audio tokenizers for future audio foundation models, 2026. URLhttps://arxiv.org/abs/2602.10934. [8]JadeCopet, FelixKreuk, ItaiGat, TalRemez, DavidKant, GabrielSynnaeve, YossiAdi, andAlexandreDĂ©fossez. Sim- ple and controllable music generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URLhttp://papers. nips.c/paper_files/paper/2023/hash/94b472a1842cd7c56dcb125fb2765fbd-Abstract-Conference.html . [9]Jordan Darefsky, Ge Zhu, and Zhiyao Duan. Parakeet, 2024. URLhttps://jordandarefsky.com/blog/2024/ parakeet/. [10]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URLhttps://arxiv.org/abs/2212.04356. [11]MOSI. AI, Donghua Yu, Zhengyuan Lin, Chen Yang, Yiyang Zhang, Hanfu Chen, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Yitian Gong, Yuqian Zhang, Wenbo Zhang, SonglinWang, ZhiyuWu, ZhaoyeFei, QinyuanCheng, ShiminLi, andXipengQiu. Mosstranscribediarizetechnical report, 2026. URL https://arxiv.org/abs/2601.01554. 10 [12]Yitian Gong, Botian Jiang, Yiwei Zhao, Yucheng Yuan, Kuangwei Chen, Yaozhou Jiang, Cheng Chang, Dong Hong, Mingshu Chen, Ruixiao Li, Yiyang Zhang, Yang Gao, Hanfu Chen, Ke Chen, Songlin Wang, Xiaogui Yang, Yuqian Zhang, Kexin Huang, ZhengYuan Lin, Kang Yu, Ziqi Chen, Jin Wang, Zhaoye Fei, Qinyuan Cheng, Shimin Li, and Xipeng Qiu. Moss-tts technical report, 2026. URLhttps://arxiv.org/abs/2603.18090. [13]Chandan K. A. Reddy, Vishak Gopal, and Ross Cutler. DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors, 2021. URL https://arxiv.org/abs/2010.15258. [14]Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016. URLhttps://arxiv.org/abs/1607.01759. [15]Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, HervĂ© JĂ©gou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models.arXiv preprint arXiv:1612.03651, 2016. URLhttps://arxiv.org/abs/1612. 03651. [16]Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang, Hao Wang, Trung Hieu Nguyen, Kun Zhou, Jiaqi Yip, Dianwen Ng, and Bin Ma. Mossformer2: Combining transformer and rnn-free recurrent network for enhanced time-domain monaural speech separation, 2024. URLhttps://arxiv.org/abs/2312.11825. [17]Boson AI. Higgs Audio V2: Redefining Expressiveness in Audio Generation.https://github.com/boson-ai/ higgs-audio, 2025. GitHub repository. Release blog available athttps://w.boson.ai/blog/higgs-audio-v2. [18]Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025. [19]Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, et al. Vibevoice technical report.arXiv preprint arXiv:2508.19205, 2025. [20]Han Zhu, Wei Kang, Liyong Guo, Zengwei Yao, Fangjun Kuang, Weiji Zhuang, Zhaoqing Li, Zhifeng Han, Dong Zhang, Xin Zhang, Xingchen Song, Long Lin, and Daniel Povey. Zipvoice-dialog: Non-autoregressive spoken dialogue generation with flow matching, 2025. URLhttps://arxiv.org/abs/2507.09318. [21]Fan Yu, Tao Wang, You Wu, Lin Zhu, Wei Deng, Weisheng Han, Wenchao Wang, Lin Hu, Xiangyu Liang, Xiaodong He, Yankun Huang, Yu Gu, Yuan Liu, Yuxuan Wang, Zhangyu Xiao, Ziteng Wang, Boya Dong, Feng Dang, Jinming Chen, Jingdong Li, Jun Wang, Yechen Jin, Yuan Zhang, Zhengyan Sheng, and Xin Wang. Joyvoice: Long-context conditioning for anthropomorphic multi-speaker conversational synthesis, 2025. URL https://arxiv.org/abs/ 2512.19090. [22]Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132. 11 Appendix Contents A Chat Templates of Training Data13 B Text Augmentation Rules13 C Voice Clone and Continuation13 D Voice References used for Evaluation14 12 AChat Templates of Training Data This appendix details the instruction-following chat templates used to construct training samples for both voice cloning and common TTS tasks. To ensure consistent conditioning across speakers and turns, both templates use explicit speaker tags in the text prompts. In addition, the voice cloning template includes per-speaker reference audio to guide timbre, whereas the common TTS template omits this field. Table 2Chat templates for voice cloning data and common TTS data. Voice clone dataCommon TTS data <|im_start|>user <user_inst> Reference(s): [S1]:<audio 1 > [S2]:<audio 2 > ... [S n ]:<audio n > other fields: None Text: [S1]text 1 [S2]text 2 ...[S n ]text n </user_inst><|im_end|> <|im_start|>assistant <generated_audio> <|im_end|> <|im_start|>user <user_inst> Reference(s): None other fields: None Text: [S1]text 1 [S2]text 2 ...[S n ]text n </user_inst><|im_end|> <|im_start|>assistant <generated_audio> <|im_end|> BText Augmentation Rules We apply rule-based punctuation augmentation to a subset of transcripts to improve robustness to diverse input styles. For each rule, the Original Token is replaced, with probability Prob., by a token randomly selected from the corresponding Replacement Set in Table 3. Table 3Rule-based text augmentation for punctuation diversity. Original TokenProb. Replacement Set Comma ([,] or []) 10% [] [;] [] [] [:] [-] [_] [---] [--] [ïżż] [ ] [space] Period ([.] or []) 5% [......] [......] [...] [...] [] [~] [ ] [space] CVoice Clone and Continuation Table4reports Seed-TTS-eval results and Table5reports TTSD-eval results under the same three inference configurations:voice_clone(reference-conditioned cloning only),continuation(pure autoregressive con- tinuation), andvoice_clone_and_continuation(the default setting that combines both). FromtheTTSD-evalandSeed-TTS-evalresults, thecombinedvoice_clone_and_continuationsettingachieves the best speaker similarity and obtains comparable results on the other two metrics. 13 Table 4Seed-TTS-eval results.SIMdenotes speaker voice similarity (higher is better). Best results are inbold. ModelEN SIMâZH SIMâZH (hard case) SIMâ MOSS-TTSD(voice_clone)0.60750.71600.6964 MOSS-TTSD(continuation)0.65790.75130.7248 MOSS-TTSD(voice_clone_and_continuation)0.68280.75900.7401 Table 5TTSD-eval results for different voice cloning paradigms.ACCdenotes speaker attribution accuracy, SIMrepresents speaker voice similarity, andWERdenotes Word Error Rate (lower is better). Best results are inbold. Model ZHEN ACCâSIMâWERâACCâSIMâWERâ MOSS-TTSD(voice_clone)0.9387 0.7852 6.07%0.96800.7228 9.39% MOSS-TTSD(continuation)0.9254 0.7740 5.46% 0.9209 0.71079.07% MOSS-TTSD(voice_clone_and_continuation)0.9587 0.7949 4.85%0.96260.73269.88% DVoice References used for Evaluation Table6summarizes the proprietary voice references used in our evaluations. For Doubao, the official voice library does not provide speaker prompts forDoubao_Podcast; therefore, we use the official API to generate two corresponding speaker prompts as the references in our evaluation. Table 6Proprietary voice references used for evaluation. ReferenceVoice ZH elevenlabs_voice speaker1: Siqi Liu â Calm, Warm and Gentle (Voice ID: W8lBaQb9YIoddhxfQNLP) speaker2: Susan â Clear and Calm Storyteller (Voice ID: kAIqZ7fZv234ClKXwzDx) doubao_voice speaker1: zh_male_dayixiansheng_v2_saturn_bigtts speaker2: zh_female_mizaitongxue_v2_saturn_bigtts EN elevenlabs_voice speaker1: Henry â Deep, Professional, and Soothing (Voice ID: pVnrL6sighQX7hVz89cp) speaker2: Ava â Eager, Helpful and Understanding (Voice ID: gJx1vCzNCD1EQHT212Ls) gemini_voice speaker1: Puck speaker2: Zephyr 14