Paper deep dive
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis
Zhenqi Jia, Yuan Zhao, Aruukhan, Rui Liu, Haizhou Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 3:37:35 AM
Summary
The paper introduces AuEmoChat, a framework for Conversational Speech Synthesis (CSS) that addresses limitations in existing methods regarding authentic emotion representation and context redundancy. It proposes AuEmoCodec to learn discrete authentic emotion tokens via finite scalar quantization, AuEmoToMe to merge redundant multimodal tokens in dialogue history while preserving emotional context, and Authentic Emotion Flow Matching for speech rendering. Experiments on the NCSSD-EmCap dataset show superior performance over state-of-the-art baselines.
Entities (14)
Relation Signals (13)
AuEmoChat → includes → AuEmoToMe
confidence 98% · We further propose AuEmoToMe, an authentic-emotion-guided token merging algorithm
AuEmoChat → includes → AuEmoCodec
confidence 98% · First, we develop AuEmoCodec... We further propose AuEmoToMe... Finally, we propose Authentic Emotion Flow Matching
AuEmoChat → solvesproblem → Conversational Speech Synthesis
confidence 98% · Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression... To address these issues, we propose AuEmoChat
AuEmoChat → includes → Authentic Emotion Flow Matching
confidence 97% · Finally, we propose Authentic Emotion Flow Matching
AuEmoChat → solves → Conversational Speech Synthesis
confidence 97% · Conversational Speech Synthesis (CSS) aims to synthesize speech... To address these issues, we propose AuEmoChat
AuEmoChat → evaluatedon → NCSSD-EmCap
confidence 96% · Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines
AuEmoChat → includes → Authentic Emotion Flow Matching
confidence 95% · Finally, we propose Authentic Emotion Flow Matching, which renders speech...
AuEmoChat → includes →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding. To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization, enabling a more authentic emotion representation than limited basic emotion categories. We further propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. We integrate it into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens. Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors. Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech.
Tags
Links
- Source: https://arxiv.org/abs/2607.15755v1
- Canonical: https://arxiv.org/abs/2607.15755v1
Trouble viewing inline? Open PDF directly →
Full Text
67,767 characters extracted from source content.
Expand or collapse full text
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis Zhenqi Jia jiazhenqi7@163.com College of Computer ScienceInner Mongolia UniversityHohhotChina , Yuan Zhao zy404nf@163.com College of Computer ScienceInner Mongolia UniversityHohhotChina , Aruukhan aruukhanres@gmail.com College of Computer ScienceInner Mongolia UniversityHohhotChina , Rui Liu imucslr@imu.edu.cn College of Computer ScienceInner Mongolia UniversityHohhotChina and Haizhou Li haizhouli@cuhk.edu.cn SRIBD, School of Artificial IntelligenceThe Chinese University of Hong KongShenzhenChina (5 June 2009) Abstract. Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding. To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization, enabling a more authentic emotion representation than limited basic emotion categories. Furthermore, we propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. We integrate it into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens. Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors. Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech. The code and speech demos will be available at: https://github.com/anonymous-css/AuEmoChat. Conversational Speech Synthesis, User-Agent Interactions, Authentic Emotion, Token Merging, Flow Matching †copyright: acmlicensed†journalyear: 2026†doi: X.X†conference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, Brazil†isbn: 978-1-4503-X-X/2018/06†ccs: Information systems Multimedia content creation†ccs: Information systems Sentiment analysis†ccs: Human-centered computing Auditory feedback 1. Introduction Conversational Speech Synthesis (CSS) aims to leverage multimodal dialogue history to synthesize target speech with contextually appropriate affective prosody in User-Agent Interactions (UAI) (Guo et al., 2021; Liu et al., 2024b; Hu et al., 2025a). In UAI systems, the ability of the agent to generate speech that is appropriate for the dialogue context and conveys human-like emotional expression is vital for enhancing user experience. As UAI becomes increasingly prevalent, CSS has become a crucial component of intelligent interactive systems (Zhou et al., 2020; Seaborn et al., 2021; McTear, 2022) and plays an important role in embodied intelligence applications such as virtual assistants, voice agents, intelligent robots, smart home devices, and in-vehicle infotainment systems. Figure 1. Previous CSS models rely on a limited emotion label space and directly model redundant multimodal context tokens, resulting in limited emotional speech expression. In contrast, AuEmoChat introduces an authentic emotion token space and models the context based on merged tokens, thereby generating more authentic emotional speech. Previous CSS studies (Guo et al., 2021; Lee et al., 2023; Xue et al., 2023; Li et al., 2022a, b; Deng et al., 2024; Liu et al., 2024d, b; Jia and Liu, 2025; Jia et al., 2025; Liu et al., 2024c; Hu et al., 2025a, b) mainly rely on a set of predefined emotion labels (including anger, disgust, fear, happiness, sadness, surprise, and neutral) derived from Ekman’s basic emotion theory (Ekman, 1992), and directly model the multi-turn dialogue history for context modeling. For instance, ECSS (Liu et al., 2024b) encodes multimodal dialogue history into a heterogeneous graph to improve the emotional expressiveness of the synthesized target speech. GPT-Talker (Liu et al., 2024c) converts multimodal information (Ning et al., 2024; Fu et al., 2026; Meng et al., 2026) in the dialogue history into discrete token sequences containing semantic and style information, and utilizes an autoregressive model to predict target acoustic tokens for synthesizing emotional speech. Recently, Chain-Talker (Hu et al., 2025a) further generates empathetic captions based on these predefined emotion labels, adopting a three-stage framework that includes emotion understanding, semantic understanding, and empathy rendering to enhance the empathetic expression of the target speech. Despite the progress of these advanced CSS methods in speech quality and expressiveness, they still face the following limitations, as illustrated in Fig. 1 (top): 1) Limited Emotion Label Space: In real-world dialogues, emotional expressions are richer and broader (Zhou et al., 2022; Tang et al., 2023). For example, emotions related to “happiness” include excitement, pleasure, and tearful joy. Such a broad emotional space is more capable of reflecting humans’ authentic emotions (AuEmo). However, conventional predefined emotion labels cannot distinguish these richer and more diverse emotional states, limiting a model’s ability to learn authentic emotional expressions. 2) Redundant Multimodal Context Tokens: As dialogue turns increase, multimodal token sequences become increasingly long, inevitably introducing a large amount of redundant information (Liu et al., 2024c; Hu et al., 2025a). These redundant tokens introduce significant noise into contextual modeling, severely interfering with emotion understanding and high-quality speech generation for the target utterance (Zhang et al., 2025). To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. Specifically, we first develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech. AuEmoCodec quantizes emotional speech into an AuEmo token in an emotion codebook via Finite Scalar Quantization (FSQ) (Mentzer et al., 2023). The AuEmo token is then used to reconstruct the perceived emotion scores of the source speech for seven basic emotion axes, enabling the model to capture more authentic and fine-grained emotional states. Second, we design an AuEmo-guided token merging (AuEmoToMe) algorithm that merges redundant text and speech tokens in multimodal dialogue history. This design reduces the interference caused by redundant tokens during emotion modeling while preserving emotion-relevant contextual information. As a result, the model can better infer the target AuEmo token and speech tokens for the target utterance. Finally, we propose Authentic Emotion Flow Matching, which generates emotionally expressive speech conditioned on the merged dialogue context and the predicted AuEmo token. In addition, this mechanism incorporates AuEmo classifier guidance during the flow matching process, enabling the synthesized speech to better align with the target authentic emotion. In summary, the main contributions of this work are as follows: 1) We propose AuEmoChat, the first CSS framework dedicated to authentic emotion understanding and rendering. 2) We develop AuEmoCodec, which leverages finite scalar quantization to learn an authentic emotion token space from large-scale emotional speech, thereby providing more authentic emotional representations. 3) We design AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant multimodal context tokens while preserving emotion-relevant context. Furthermore, we propose Authentic Emotion Flow Matching to improve context-consistent emotional speech rendering. 4) Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat significantly outperforms advanced baselines in both emotional expressiveness and speech quality. 2. Related Works 2.1. Speech Neural Codec Speech neural codecs convert continuous speech waveforms into discrete tokens, providing the foundation for token-based speech modeling and large language model (LLM)-based audio processing (Van Den Oord et al., 2017; Liu et al., 2024a). SpeechTokenizer (Zhang et al., 2023) uses an encoder-decoder architecture with residual vector quantization (RVQ) to model semantic and acoustic information in speech jointly. SemantiCodec (Liu et al., 2024a) adopts a dual-encoder architecture to extract ultra-low-bitrate and semantically rich audio tokens for speech, general sounds, and music. FACodec (Ju et al., 2024) decomposes speech into content, prosody, timbre, and acoustic-detail subspaces through factorized vector quantization (FVQ), enabling separate generation of different speech attributes. TF-Codec (Jiang et al., 2023) integrates latent-domain predictive coding and learnable time-frequency compression into VQ for low-latency speech coding. CosyVoice (Du et al., 2024a) extracts semantically rich speech tokens with an Automatic Speech Recognition (ASR)-based self-supervised method, while CosyVoice2 (Du et al., 2024b) further improves token representation by introducing FSQ. Our AuEmoCodec has the following key distinctions: 1) It quantizes emotional speech into fixed AuEmo tokens rather than semantic or acoustic-related tokens. 2) Unlike conventional codecs that rely on ASR supervision to learn semantic information or reconstruct the original speech waveform, AuEmoCodec is trained to reconstruct quantized AuEmo tokens into a richer emotion perception space by modeling perceived scores across seven basic emotion axes, allowing the tokens to capture authentic human-like emotion. 2.2. Emotion Label Space The emotion label space is commonly used to quantify emotional representations in speech and serves as a control condition for emotional speech synthesis (Liu et al., 2024b; Hu et al., 2025b). Traditional CSS methods learn emotions based on Ekman’s basic emotion theory (Ekman, 1992), which categorizes emotions into six basic types: anger, disgust, fear, happiness, sadness, and surprise. Emotions that cannot be clearly categorized are usually labeled as neutral. This type of model aligns better with people’s intuitive understanding of emotions in daily life. However, its label space is limited and cannot fully cover the rich and subtle emotional states in real dialogues. Plutchik (Plutchik, 2001) points out that humans can express about 34,000 distinct emotions, which further indicates that traditional discrete models with limited categories cannot fully represent the authentic emotion space. To address the limitation of label space, Lian et al. (Lian et al., 2024) introduce open-vocabulary emotion recognition, which expands emotion categories through open-vocabulary emotion labels. However, although it increases the number of categories, it also introduces many semantically similar or synonymous labels, failing to effectively disentangle the emotion representation space and thus increasing modeling difficulty. To construct a more authentic emotion token space, we use AuEmoCodec to learn AuEmo tokens from large-scale emotional speech. Specifically: 1) We build a discrete emotion codebook, where each token represents a distinct authentic emotional state, providing a finer-grained representation than traditional emotion categories. 2) Instead of relying on predefined emotion labels, AuEmoCodec learns the AuEmo token representation directly from large-scale emotional speech. Figure 2. The left side illustrates the overall framework of the proposed AuEmoChat, which includes: Multimodal Dialogue Tokenization, AuEmoToMe-based Authentic Emotion Understanding, and Merged Context-Aware Authentic Emotion Rendering. The right side illustrates the overall framework of the proposed AuEmoCodec. 2.3. Token Merging Token merging improves model efficiency and performance by combining redundant tokens in multimodal sequences (Bolya et al., 2022). For instance, ToMe (Bolya et al., 2022) gradually merges similar tokens within the Transformer, enhancing inference speed while largely preserving performance. Token Merger (Feng and Zhang, 2023) identifies meta tokens representing meaningful image cues and merges similar tokens adaptively, better preserving semantic context and generalizing across vision tasks. FastAST (Behera et al., 2024) applies token merging to audio spectrograms, improving inference speed while maintaining recognition accuracy. In multimodal dialogue history, speech tokens typically have high bitrate, and long conversations further increase sequence length, making emotional information highly redundant. Redundant tokens dilute model attention to key emotional cues, hindering accurate emotion understanding. Inspired by these works, we propose AuEmoToMe in AuEmoChat: 1) It is the first attempt to introduce token merging into AR-based CSS for modeling multimodal dialogue context. 2) It computes intra-modal similarity among text tokens and speech tokens in the multimodal dialogue history and merges redundant tokens. 3) To preserve emotional information, AuEmoToMe adopts an AuEmo-guided strategy that prioritizes merging redundant tokens while minimally affecting emotion-related key tokens. 3. Task Definition A dialogue can be defined as a sequence of alternating interactions between the user and the agent, denoted as U1,U2,…,U,U\U_1,U_2,…,U_J,U_N\, where U1,U2,…,U\U_1,U_2,…,U_J\ denotes the dialogue history, and U_N denotes the target utterance to be processed by the agent. For each historical utterance UiU_i, it contains multiple modalities Pi,Ti,Si\P_i,T_i,S_i\, which represent the speaker, text, and speech of the i-th turn, respectively. For the target utterance U_N, the speaker PP_N and text TT_N are given, and the goal is to generate the corresponding speech SS_N. CSS based on authentic emotion understanding and rendering needs to address the following points: 1) How to more accurately represent the authentic emotion of each utterance in the dialogue. 2) How to effectively infer the authentic emotion token and speech token of the target utterance from a long multimodal dialogue history. 3) How to synthesize speech with authentic emotion expression that is consistent with the current dialogue context. 4. Methodology: AuEmoChat Fig. 2 (left) illustrates the overall architecture of AuEmoChat. The Multimodal Dialogue Tokenization module extracts text, speech, speaker, and AuEmo tokens from the input dialogue. The AuEmoToMe-based Authentic Emotion Understanding module incorporates the proposed AuEmoToMe into the LLM to merge redundant text and speech tokens in the dialogue history, based on which the LLM predicts the AuEmo token and speech tokens of the target utterance. The Merged Context-Aware Authentic Emotion Rendering module conditions on the merged history tokens, the predicted AuEmo token, and the speech tokens to synthesize authentic emotional speech that is consistent with the context through an authentic emotion flow matching mechanism. In addition, Fig. 2 (right) illustrates the training process of AuEmoCodec, which will be described in detail at the end of this section. 4.1. Multimodal Dialogue Tokenization In this section, we introduce how multimodal dialogue data are processed to extract the corresponding multimodal tokens. The implementation details are as follows. Text Tokenizer. We adopt a BPE-based Text Tokenizer (Gage, 1994) to tokenize the dialogue text. Specifically, as shown in Eq. 1, the input dialogue text sequence T1→T_1 is processed by the Text Tokenizer to obtain the corresponding text token embeddings: (1) f1→,1→maxtT=Text Tokenizer(T1→)f^T_1 ,1→ max_t=Text Tokenizer(T_1 ) where “1→1 ” denotes the dialogue turn indices, and “1→maxt1→ max_t” denotes the text token indices within each utterance. Speaker Embedding. We use Speaker Embedding to encode the discrete identities of different speakers. Specifically, the speaker sequence P1→P_1 of the current dialogue is processed by the Speaker Embedding to obtain the corresponding speaker token embeddings: (2) f1→P=Speaker Embedding(P1→)f^P_1 =Speaker Embedding(P_1 ) Speech Tokenizer. We adopt the Speech Tokenizer proposed in CosyVoice2 (Du et al., 2024b) to discretize the speech signals in the dialogue history. Specifically, the dialogue history speech sequence S1→S_1 is processed by the Speech Tokenizer to obtain the corresponding speech token embeddings: (3) f1→,1→maxsS=Speech Tokenizer(S1→)f^S_1 ,1→ max_s=Speech Tokenizer(S_1 ) where “1→maxs1→ max_s” denotes the speech token indices within each utterance. AuEmo Tokenizer. To represent authentic emotional expressions in dialogue, we obtain an AuEmo Tokenizer by training AuEmoCodec, as shown in Fig. 2 (right). AuEmoCodec is trained on large-scale emotional speech data to learn a discrete authentic emotion token space via FSQ. We construct an emotion codebook of size 1000, from which 750 tokens are activated after training and used as AuEmo tokens. The detailed training procedure of AuEmoCodec is described in Sec. 4.4. The dialogue history speech sequence S1→S_1 is then processed by the AuEmo Tokenizer to obtain the corresponding AuEmo token embeddings: (4) f1→E=AuEmo Tokenizer(S1→)f^E_1 =AuEmo Tokenizer(S_1 ) 4.2. AuEmoToMe-based Authentic Emotion Understanding In AR-based CSS, multimodal dialogue history is typically organized as a unified token sequence for contextual modeling. As dialogue turns increase, text and speech tokens accumulate rapidly, resulting in a long multimodal input sequence with many redundant tokens. To address this issue, we design the AuEmoToMe-based Authentic Emotion Understanding module, which merges redundant emotion-related text and speech tokens in the dialogue history and predicts the target AuEmo and speech tokens from the merged history representation. The module consists of three execution steps: Multimodal Dialogue Token Sequence Construction. Following the order of dialogue turns, the dialogue history is constructed as the following multimodal dialogue history token sequence: (5) U1→= U_1 =\ f1P,f1,1→maxtT,f1E,f1,1→maxsS, f^P_1,f^T_1,1→ max_t,f^E_1,f^S_1,1→ max_s, …, …, fP,f,1→maxtT,fE,f,1→maxsS f^P_J,f^T_J,1→ max_t,f^E_J,f^S_J,1→ max_s\ where fiPf^P_i, fi,1→maxtTf^T_i,1→ max_t, fiEf^E_i, and fi,1→maxsSf^S_i,1→ max_s denote the speaker token, text tokens, AuEmo token, and speech tokens of the i-th utterance. For the target utterance, we only use the speaker token and text tokens as conditional inputs to construct the target utterance token sequence: (6) U=S,fP,f,1→maxtTU_N=\ to12pt to12pt -5.9981pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto5.7981pt0.0pt @curveto5.7981pt3.20224pt3.20224pt5.7981pt0.0pt5.7981pt @curveto-3.20224pt5.7981pt-5.7981pt3.20224pt-5.7981pt0.0pt @curveto-5.7981pt-3.20224pt-3.20224pt-5.7981pt0.0pt-5.7981pt @curveto3.20224pt-5.7981pt5.7981pt-3.20224pt5.7981pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-2.77779pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke S @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope ,f^P_N,f^T_N,1→ max_t\ where S denotes the start token of the target utterance. Authentic Emotion-driven Token Merging (AuEmoToMe). To reduce the interference of redundant information in multi-turn multimodal dialogue history on target emotion modeling, we design AuEmoToMe based on ToMe (Bolya et al., 2022). It computes cosine similarity between tokens to identify redundant text and speech tokens, and merges them into a more compact merged history tokens sequence. To preserve authentic emotional information during compression, AuEmoToMe employs an AuEmo-guided merging strategy, where the AuEmo token of each historical utterance acts as an emotion anchor, guiding the weighted aggregation of matched text and speech redundant tokens separately. Thus, AuEmoToMe preserves more emotion-relevant information while merging redundant tokens. Input: Dialogue history tokens U1→U_1 , merge ratio ρ Output: Merged history tokens fmergedhisf^his_merged Initialize fmergedhis←∅f^his_merged← ; for each utterance UiU_i in U1→U_1 do 1. Extract text tokens fi,1→maxtTf^T_i,1→ max_t, speech tokens fi,1→maxsSf^S_i,1→ max_s, and AuEmo token fiEf^E_i. 2. Construct token sequence XiMX^M_i, where M∈T,SM∈\T,S\ denotes text or speech. 3. Partition XiMX^M_i into source set AMA^M and destination set BMB^M, where |BM|=⌊|XiM|(1−ρ)⌋|B^M|= |X^M_i|(1-ρ) and AM=XiM∖BMA^M=X^M_i B^M. 4. Compute cosine similarity between tokens in AMA^M and tokens in BMB^M. 5. Match each token in AMA^M to its similar token in BMB^M. 6. Select the top-k pairs with the highest similarity. for each selected pair (au,bv)∈(AM,BM)(a_u,b_v)∈(A^M,B^M) do Compute cosine similarities sa=sim(au,fiE)s_a=sim(a_u,f^E_i) and sb=sim(bv,fiE)s_b=sim(b_v,f^E_i). Normalize them into fusion weights: wa=exp(sa)exp(sa)+exp(sb)w_a= (s_a) (s_a)+ (s_b), wb=exp(sb)exp(sa)+exp(sb)w_b= (s_b) (s_a)+ (s_b). Update token representation: bv←waau+wbbvb_v← w_aa_u+w_bb_v. Update UiU_i with the merged tokens XiMX^M_i. Append the updated utterance UiU_i to fmergedhisf^his_merged. Algorithm 1 AuEmoToMe Specifically, as shown in Algorithm 1, AuEmoToMe takes the dialogue history token sequence U1→U_1 as input and performs token merging for each historical utterance. For each utterance UiU_i, its text tokens fi,1→maxtTf^T_i,1→ max_t and speech tokens fi,1→maxsSf^S_i,1→ max_s are separately used to construct token sequences XiMX^M_i, where M∈T,SM∈\T,S\ denotes text or speech. The AuEmo token fiEf^E_i serves as an emotion anchor. Given the merging ratio ρ, XiMX^M_i is partitioned into a source set AMA^M and a destination set BMB^M. Cosine similarities between tokens in AMA^M and BMB^M are computed to select the top-k matched pairs for merging. The selected pairs are then weighted by their similarities to the AuEmo anchor and aggregated accordingly. Finally, the merged tokens are restored to their original temporal order, and the merged tokens of all utterances are concatenated to form the merged dialogue history tokens fmergedhisf^his_merged. Target AuEmo Token and Speech Token Inference. The merged history tokens are concatenated with the target utterance token sequence: (7) fmergedhis,S,fP,f,1→maxtT\f^his_merged, to12pt to12pt -5.9981pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto5.7981pt0.0pt @curveto5.7981pt3.20224pt3.20224pt5.7981pt0.0pt5.7981pt @curveto-3.20224pt5.7981pt-5.7981pt3.20224pt-5.7981pt0.0pt @curveto-5.7981pt-3.20224pt-3.20224pt-5.7981pt0.0pt-5.7981pt @curveto3.20224pt-5.7981pt5.7981pt-3.20224pt5.7981pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-2.77779pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke S @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope ,f^P_N,f^T_N,1→ max_t\ The model first predicts the AuEmo token of the target utterance fEf^E_N, and then generates the speech token sequence f,1→maxsSf^S_N,1→ max_s, which is terminated by an end token E. Therefore, the target output sequence can be defined as: (8) =fE,f,1→maxsS,EY_N=\f^E_N,f^S_N,1→ max_s, to12.82pt to12.82pt -6.4124pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto6.2124pt0.0pt @curveto6.2124pt3.43105pt3.43105pt6.2124pt0.0pt6.2124pt @curveto-3.43105pt6.2124pt-6.2124pt3.43105pt-6.2124pt0.0pt @curveto-6.2124pt-3.43105pt-3.43105pt-6.2124pt0.0pt-6.2124pt @curveto3.43105pt-6.2124pt6.2124pt-3.43105pt6.2124pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-3.40279pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke E @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope \ The generation process follows a conditional autoregressive formulation: (9) p(∣fmergedhis,S,fP,f,1→maxtT;Θ) p(Y_N f^his_merged, to12pt to12pt -5.9981pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto5.7981pt0.0pt @curveto5.7981pt3.20224pt3.20224pt5.7981pt0.0pt5.7981pt @curveto-3.20224pt5.7981pt-5.7981pt3.20224pt-5.7981pt0.0pt @curveto-5.7981pt-3.20224pt-3.20224pt-5.7981pt0.0pt-5.7981pt @curveto3.20224pt-5.7981pt5.7981pt-3.20224pt5.7981pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-2.77779pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke S @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope ,f^P_N,f^T_N,1→ max_t; ) where Θ denotes the model parameters. To explicitly model the authentic emotion understanding process, the prediction of the target AuEmo token is formulated as: (10) p(fE∣fmergedhis,S,fP,f,1→maxtT;Θ)p(f^E_N f^his_merged, to12pt to12pt -5.9981pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto5.7981pt0.0pt @curveto5.7981pt3.20224pt3.20224pt5.7981pt0.0pt5.7981pt @curveto-3.20224pt5.7981pt-5.7981pt3.20224pt-5.7981pt0.0pt @curveto-5.7981pt-3.20224pt-3.20224pt-5.7981pt0.0pt-5.7981pt @curveto3.20224pt-5.7981pt5.7981pt-3.20224pt5.7981pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-2.77779pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke S @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope ,f^P_N,f^T_N,1→ max_t; ) and the speech token sequence is then generated conditioned on the predicted emotion token: (11) p(f,1→maxsS∣fmergedhis,S,fP,f,1→maxtT,fE;Θ)p(f^S_N,1→ max_s f^his_merged, to12pt to12pt -5.9981pt to0.0pt @beginscope @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke @setlinewidth @invoke to0.0pt @beginscope @invoke @beginscope @invoke @moveto5.7981pt0.0pt @curveto5.7981pt3.20224pt3.20224pt5.7981pt0.0pt5.7981pt @curveto-3.20224pt5.7981pt-5.7981pt3.20224pt-5.7981pt0.0pt @curveto-5.7981pt-3.20224pt-3.20224pt-5.7981pt0.0pt-5.7981pt @curveto3.20224pt-5.7981pt5.7981pt-3.20224pt5.7981pt0.0pt @closepath @moveto0.0pt0.0pt @stroke @invoke @beginscope @invoke @transformcm1.00.00.01.0-2.77779pt-3.41666pt @invoke pgfstrokecolorrgb0,0,0 @color@rgb@stroke000 @invoke @color@rgb@fill000 @invoke S @invoke @endscope @invoke @endscope @invoke @endscope @discardpath @invoke @endscope ,f^P_N,f^T_N,1→ max_t,f^E_N; ) 4.3. Merged Context-Aware Authentic Emotion Rendering This module aims to generate conversational speech with authentic emotional expression by jointly conditioning on the merged history tokens fmergedhisf^his_merged, the target AuEmo token fEf^E_N, and the target speech token sequence f,1→maxsSf^S_N,1→ max_s. Merged Context Encoder. To obtain a context condition for Flow Matching, we feed fmergedhisf^his_merged into the Merged Context Encoder. This encoder uses a bidirectional GRU (Chung et al., 2014) to model the merged history tokens and capture the global dialogue context. The resulting context condition can be formulated as: (12) ℱctx=Merged Context Encoder(fmergedhis)F_ctx=Merged Context Encoder(f^his_merged) AuEmo Encoder. To obtain an authentic emotion condition for Flow Matching, we feed the predicted AuEmo token fEf^E_N into the AuEmo Encoder. This encoder projects the AuEmo token into the conditioning space of the Flow Matching model. The resulting authentic emotion condition can be formulated as: (13) ℱe=AuEmo Encoder(fE)F_e=AuEmo Encoder(f^E_N) Authentic Emotion Flow Matching. Given the predicted target speech token sequence f,1→maxsSf^S_N,1→ max_s, we first extract its acoustic prior representation, denoted as μ. The context condition ℱctxF_ctx and the target emotion condition ℱeF_e are then injected into the Flow Matching network together with μ, such that the generation process is jointly constrained by the target speech prior, the merged dialogue context, and the target authentic emotion. Following conditional Flow Matching, let x1x_1 denote the ground-truth mel-spectrogram of the target utterance. Let x0∼(0,I)x_0 (0,I) denote the Gaussian noise. We define the interpolation path between x0x_0 and x1x_1 as: (14) ϕt(x)=(1−(1−σmin)t)x0+tx1,t∈[0,1] _t(x)= (1-(1- _min)t )x_0+tx_1, t∈[0,1] where σmin=10−6 _ =10^-6, following CosyVoice2 (Du et al., 2024b). The corresponding target vector field is: (15) ut(ϕt(x)∣x1)=x1−(1−σmin)x0u_t( _t(x) x_1)=x_1-(1- _min)x_0 Conditioned on μ, ℱctxF_ctx, and ℱeF_e, the Flow Matching network predicts the vector field: (16) vt(ϕt(x)∣μ,ℱctx,ℱe;θ)v_t( _t(x) μ,F_ctx,F_e;θ) where θ denotes the model parameters. The main objective is to make the predicted vector field match the target transport direction from noise to the real mel-spectrogram. Therefore, the Flow Matching loss (ℒfmL_fm) is defined as: (17) ℒfm=x0,x1,t[∥vt(ϕt(x)∣μ,ℱctx,ℱe;θ)−ut(ϕt(x)∣x1)∥22]L_fm=E_x_0,x_1,t [ \|v_t( _t(x) μ,F_ctx,F_e;θ)-u_t( _t(x) x_1) \|_2^2 ] Although the ℒfmL_fm learns the acoustic transport path, it does not explicitly constrain the intermediate mel states to preserve the target authentic emotion. To address this issue, we further introduce an AuEmo classifier guidance mechanism. Specifically, an auxiliary AuEmo classifier ψ is applied to the intermediate state ϕt(x) _t(x) to estimate the probability that the current mel state matches the target AuEmo token fEf^E_N. Based on this classifier, the guided vector field can be written as: (18) v~t=vt(ϕt(x)∣μ,ℱctx,ℱe;θ)+(t>r)⋅∇ϕt(x)logpψ(fE∣ϕt(x)) v_t=v_t( _t(x) μ,F_ctx,F_e;θ)+I(t>r)· _ _t(x) p_ψ (f^E_N _t(x) ) where (⋅)I(·) is the indicator function, r is a activation threshold set to 0.70.7, and ψ denotes the pre-trained AuEmoCodec. The purpose is to guide the intermediate mel state toward a direction that is more consistent with the target authentic emotion. In this way, the optimization trajectory is guided not only by the acoustic transport objective, but also by the target authentic emotion. Notably, we activate this guidance only when t>rt>r, because the intermediate state is still close to Gaussian noise when t is small and does not yet contain stable emotion-related spectro-temporal patterns. Applying emotion guidance too early may introduce noisy gradients and weaken the learning of the basic transport path. When t>rt>r, the intermediate mel state becomes more structured, and the emotion classifier can provide more reliable guidance. During inference, the model starts from Gaussian noise x0∼(0,I)x_0 (0,I) and solves the following ordinary differential equation: (19) dϕt(x)dt=vt(ϕt(x)∣μ,ℱctx,ℱe;θ),t∈[0,1] d _t(x)dt=v_t( _t(x) μ,F_ctx,F_e;θ), t∈[0,1] The final output x1x_1 is the generated mel-spectrogram, which is then converted into a waveform by the HiFi-GAN vocoder (Kong et al., 2020). 4.4. Training Process of AuEmoCodec AuEmoCodec aims to learn discrete AuEmo tokens from large-scale emotional speech data that are more discriminative and more consistent with real human emotional expression. Specifically, given an emotional speech segment SiS_i, we first use a Speech Emotion Encoder to extract its emotion representation. We then map this representation into a low-rank quantization space through FSQ-Down and discretize it with a bounded rounding operation. In our implementation, the quantization levels are set to levels=[8,5,5,5]levels=[8,5,5,5]. The resulting discrete code is defined as the AuEmo token: (20) fiE=ROUND(FSQ-Down(Speech Emotion Encoder(Si)))f^E_i=ROUND(FSQ -Down(Speech Emotion Encoder(S_i))) To enable AuEmoCodec to focus exclusively on emotion representation learning, we do not train it to reconstruct the original waveform. Instead, motivated by the fact that authentic human emotion is often expressed as a complex mixture of multiple affective states (Zhou et al., 2022; Tang et al., 2023), we use the perceived scores of speech sample SiS_i over seven basic emotion axes as the reconstruction target. Specifically, the quantized AuEmo token fiEf^E_i is first mapped back to a high-dimensional representation through FSQ-Up, from which the model reconstructs the multi-axis perceived emotion scores, denoted as PiaeP^ae_i. To obtain supervision targets, we use Gemini-2.5-Flash (Comanici et al., 2025) to annotate the perceived scores of input speech along each emotion axis, denoted as GTiaeGT_i^ae. The prompt used for Gemini-2.5-Flash, together with representative examples of the resulting perceived scores over the seven emotion axes, is provided in Appendix A. AuEmoCodec is then trained by minimizing the mean squared error (MSE) between PiaeP_i^ae and GTiaeGT_i^ae, which encourages the learned AuEmo tokens to capture emotional nuances and thereby establish a more effective authentic emotion token space for CSS. 5. Experiments This section presents the experimental setup, baseline and ablation models, and the evaluation metrics used to assess speech quality and emotional expressiveness. Additional experimental details are provided in Appendix B. 5.1. Experimental Setup We use the open-source NCSSD-EmCap (Hu et al., 2025a) dataset to evaluate the effectiveness of AuEmoChat. This dataset integrates three high-quality dialogue speech datasets: DailyTalk (Lee et al., 2023), NCSSD (Liu et al., 2024c), and MultiDialog (Park et al., 2024). Specifically, NCSSD-EmCap contains approximately 384 hours of speech data. It comprises 18,580 dialogues with a total of 245,984 utterances, averaging 13.2 turns per dialogue, and covers 25 speakers. Among them, 124,599 utterances are from male speakers and 121,385 utterances are from female speakers. The dataset is divided into training, validation, and test sets with a ratio of 8:1:1, and the same split is used for training both AuEmoCodec and AuEmoChat. For training, AuEmoChat uses 4 NVIDIA A100 GPUs with a batch size of 4 and 8 gradient accumulation steps. Table 1. Subjective (95% confidence interval (Yasuda and Toda, 2023)) and objective results with different comparative models. Blue indicates the best performance. Methods N-DMOS (↑ ) E-DMOS (↑ ) WER (↓ ) MCD (↓ ) SpkSIM (↑ ) EmoACC (↑ ) AuEmoACC (↑ ) BaseCSS (Lee et al., 2023) (ICASSP 2023) 3.431 ± 0.023 3.299 ± 0.024 27.28 9.359 70.75 45.62 19.12 ECSS (Liu et al., 2024b) (AAAI 2024) 3.675 ± 0.022 3.509 ± 0.022 25.10 8.568 72.51 49.85 22.31 GPT-Talker (Liu et al., 2024c) (M 2024) 3.812 ± 0.023 3.654 ± 0.022 20.49 8.061 77.68 57.08 23.45 Chain-Talker (Hu et al., 2025a) (ACL 2025) 3.955 ± 0.025 3.756 ± 0.018 15.07 7.686 77.56 57.16 24.12 AuEmoChat (Ours) 4.171 ± 0.026 3.979 ± 0.021 9.14 6.847 78.03 61.04 28.71 5.2. Baseline and Ablation Models Baseline Models. To evaluate the effectiveness of AuEmoChat, we compare it with state-of-the-art (SOTA) emotional CSS systems, including 1) BaseCSS (Guo et al., 2021; Lee et al., 2023), 2) ECSS (Liu et al., 2024b), 3) GPT-Talker (Liu et al., 2024c), and 4) Chain-Talker (Hu et al., 2025a). For fair comparison, all baseline models are configured to use the authentic emotion token space. Ablation Models. To assess the contribution of each component in AuEmoChat, we conduct three groups of ablation studies. 1) Ablations on AuEmo Tokenizer: We replace AuEmo Tokenizer with three alternative emotion representations to examine the role of the authentic emotion token space in supporting emotion understanding and rendering: Abl.1 w/ LimEmo, which uses a traditional limited emotion label space. Abl.2 w/ OVEmo, which uses an open-vocabulary (OV) emotion label space. Abl.3 w/ EmoCap, which uses emotion captions. 2) Ablations on AuEmoToMe: We remove AuEmoToMe or its AuEmo-guided merging strategy to investigate its importance for target-utterance emotion understanding and speech token prediction: Abl.4 w/o AuEmoToMe removes the entire AuEmoToMe module. Abl.5 w/o AuEmo-guided Strategy removes the AuEmo-guided token merging strategy. 3) Ablations on Authentic Emotion Flow Matching: We remove different conditioning components to analyze the roles of merged dialogue context and authentic emotion constraints in context-consistent emotional speech rendering: Abl.6 w/o FM-AuEmo removes the AuEmo condition. Abl.7 w/o FM-ACG removes AuEmo classifier guidance (ACG). Abl.8 w/o FM-Context removes merged context condition. 5.3. Evaluation Metrics We use the following metrics to evaluate each model’s performance: Subjective Metrics: Naturalness-DMOS (N-DMOS) (Streijl et al., 2016; Liu et al., 2024d, 2025) assesses the naturalness and overall quality of synthesized speech. Emotion-DMOS (E-DMOS) (Liu et al., 2024b; Hu et al., 2025a) measures the emotional expressiveness and consistency with ground truth. A total of 30 trained, English-proficient evaluators participated in the subjective evaluation. Each evaluator reviewed the dialogue context and listened to each synthesized sample at least three times before rating its naturalness and emotional consistency. Objective Metrics: We employ Word Error Rate (WER) (Morris et al., 2004), Mel Cepstral Distortion (MCD) (Kubichek, 1993; Chen et al., 2022), and Speaker Similarity (SpkSIM) (Cong et al., 2025; Zhang et al., 2024) to evaluate the quality of the synthesized speech. For emotion expressiveness, Emotion Accuracy (EmoACC) evaluates performance under a seven-category basic emotion space using emotion2vec (Ma et al., 2024), while Authentic Emotion Accuracy (AuEmoACC) measures consistency in the authentic emotion token space using the trained AuEmoCodec. AuEmoCodec Analysis Metrics: To validate the training strategy and hyperparameter design of AuEmoCodec, we employ the following metrics: 1) Tolerance at m axes (Tol@m) evaluates the reconstruction accuracy of multi-axis perceived emotion scores while allowing at most m mismatched axes. 2) Strict Emotion Axis Alignment (S-EAA) measures the reconstruction accuracy of perceived emotion scores across all emotion axes. 3) Dominant Emotion Axis Alignment (D-EAA) evaluates the accuracy of the reconstructed dominant emotion axis. 4) Usage measures codebook utilization. 6. Results and Discussion In this section, we comprehensively evaluate AuEmoChat through baseline comparisons, component ablations, analysis of the AuEmoCodec architecture, and analysis of the context token merging rate. Additional experimental results are provided in Appendix C. Table 2. Subjective (95% confidence interval) and objective results with different ablation models. Blue indicates the best performance, and underlined indicates the worst performance. Methods N-DMOS (↑ ) E-DMOS (↑ ) WER (↓ ) MCD (↓ ) SpkSIM (↑ ) EmoACC (↑ ) AuEmoACC (↑ ) Ablations on AuEmo Tokenizer Abl.1 w/ LimEmo 3.983 ± 0.020 3.736 ± 0.025 9.69 6.921 77.59 58.46 20.38 Abl.2 w/ OVEmo 3.889 ± 0.024 3.859 ± 0.022 9.74 6.971 77.46 59.19 24.57 Abl.3 w/ EmoCap 3.979 ± 0.022 3.765 ± 0.021 9.43 6.936 77.46 58.54 24.15 Ablations on AuEmoToMe Abl.4 w/o AuEmoToMe 3.817 ± 0.027 3.750 ± 0.024 9.55 7.274 77.84 57.97 25.72 Abl.5 w/o AuEmo-guided Strategy 3.948 ± 0.022 3.815 ± 0.027 9.37 6.905 77.53 59.81 27.02 Ablations on Authentic Emotion Flow Matching Abl.6 w/o FM-AuEmo 3.982 ± 0.023 3.774 ± 0.024 9.15 6.865 77.18 58.33 23.88 Abl.7 w/o FM-ACG 4.008 ± 0.023 3.820 ± 0.022 9.27 6.989 77.71 59.84 25.47 Abl.8 w/o FM-Context 3.863 ± 0.024 3.873 ± 0.019 13.63 7.610 72.61 60.17 26.31 AuEmoChat (Ours) 4.171 ± 0.026 3.979 ± 0.021 9.14 6.847 78.03 61.04 28.71 6.1. AuEmoChat vs. Baseline Models Table 1 presents the subjective and objective evaluation results comparing AuEmoChat with SOTA CSS baselines. Overall, AuEmoChat achieves the best performance across all metrics, demonstrating its superiority in both speech quality and authentic emotion modeling. For subjective evaluations, AuEmoChat achieves the highest scores in both N-DMOS (4.171) and E-DMOS (3.979), surpassing the strongest baseline Chain-Talker by 0.216 and 0.223, respectively. This indicates that AuEmoChat generates more natural speech with more accurate and context-consistent emotional expression. For objective evaluations, AuEmoChat consistently outperforms all baselines in both speech quality and emotion expressiveness. Compared with the strongest baseline Chain-Talker, AuEmoChat reduces WER from 15.07 to 9.14 and MCD from 7.686 to 6.847, indicating improved pronunciation accuracy and acoustic fidelity. It also achieves the highest speaker similarity (78.03), showing better preservation of speaker identity. In terms of emotional expression, AuEmoChat improves EmoACC from 57.16 to 61.04 and AuEmoACC from 24.12 to 28.71. The improvement in AuEmoACC further demonstrates the effectiveness of the proposed AuEmoChat in capturing authentic emotional states beyond basic emotion labels. Overall, these results confirm that AuEmoChat effectively addresses the limitations of existing CSS systems by improving authentic emotion understanding, reducing redundant context interference, and generating more expressive and context-consistent speech. 6.2. Ablation Results of Key Components Table 2 presents the ablation results obtained by removing or replacing key components in AuEmoChat. Overall, all ablation variants perform worse than the full AuEmoChat model across both speech quality and emotion expressiveness metrics, confirming the necessity of each proposed component. In Abl.1-Abl.3, replacing AuEmo Tokenizer with alternative emotion representations leads to clear degradation in emotional expressiveness. This indicates that the learned authentic emotion token space is more effective than limited emotion labels, open-vocabulary emotion labels, and emotion captions, enabling the model to better capture emotional states in conversational speech. In Abl.4-Abl.5, removing AuEmoToMe or its AuEmo-guided merging strategy degrades both speech quality and emotional expressiveness. This suggests that redundant dialogue tokens hinder effective context modeling, whereas AuEmoToMe improves target emotion understanding and speech token prediction by reducing irrelevant information. In Abl.6-Abl.8, removing the AuEmo condition, AuEmo classifier guidance, or merged dialogue context degrades both speech quality and emotional expressiveness. This demonstrates that jointly modeling authentic emotion and merged dialogue context is essential for generating context-consistent and emotionally expressive speech. Notably, Abl.1, which uses traditional limited emotion labels, achieves the worst performance on both E-DMOS and AuEmoACC, highlighting the importance of modeling an authentic emotion. In Abl.8, removing merged dialogue context in flow matching significantly degrades speech quality, with WER, MCD, and SpkSIM reaching the worst values, indicating that contextual information plays a critical role in speech generation. Table 3. Analysis results of key design choices in AuEmoCodec. Blue indicates the best performance, and pink indicates the second-best performance. Methods Tol@1 Tol@2 S-EAA D-EAA Usage Comparative Study of Quantization Paradigms VQ 35.34 62.55 10.85 47.46 55.10% RVQ 54.39 81.07 18.53 71.25 61.10% R-FSQ 53.97 81.69 16.25 66.15 87.30% G-FSQ 54.46 82.24 18.30 70.81 72.10% FSQ 55.04 81.17 18.61 71.77 75.00% Comparative Study of Codebook Capacity Codebook Size (200) 45.82 79.52 10.27 46.44 76.00% Codebook Size (600) 46.62 79.75 10.01 46.96 83.16% Codebook Size (1000) 55.04 81.17 18.61 71.77 75.00% Codebook Size (1400) 52.26 81.00 17.21 70.50 67.28% Codebook Size (2000) 52.82 80.99 17.73 70.59 52.10% Codebook Size (4000) 54.28 81.21 18.30 70.30 41.42% Comparative Study of Different LLM-as-a-Judge Models Qwen2-Audio 39.99 74.12 15.17 54.44 73.40% Kimi-Audio 43.25 78.54 17.25 64.07 74.10% Qwen3-Omni-Captioner 43.04 81.51 17.88 68.51 75.90% Gemini-2.5-Flash 55.04 81.17 18.61 71.77 75.00% 6.3. Analysis of AuEmoCodec Architecture In this section, we analyze the key design choices of AuEmoCodec from three aspects: quantization paradigms, codebook capacity, and the judge model, as shown in Table 3. For quantization paradigms, we evaluate Vector Quantization (VQ) (Van Den Oord et al., 2017), Residual VQ (RVQ), Finite Scalar Quantization (FSQ) (Mentzer et al., 2023), Residual FSQ (R-FSQ), and Grouped FSQ (G-FSQ). FSQ achieves the best performance on Tol@1, S-EAA, and D-EAA, and shows the most stable overall performance. Although its codebook usage is slightly lower than R-FSQ, it achieves stronger authentic emotion modeling, indicating that FSQ provides a better balance between representation capacity and efficiency. For codebook capacity, as the codebook size increases from 200 to 1000, the performance on Tol@1, S-EAA, and D-EAA consistently improves, indicating that a larger codebook enhances the discrimination ability and representation precision of authentic emotions. However, as the codebook size increases further, codebook usage decreases and performance degrades, suggesting that an excessively large codebook reduces effective utilization and impacts authentic emotion modeling performance. Overall, a codebook size of 1000 achieves the best balance between performance and usage. For the judge model selection, we evaluate Qwen2-Audio (Chu et al., 2024), Kimi-Audio (Ding et al., 2025), Qwen3-Omni-Captioner (Xu et al., 2025), and Gemini-2.5-Flash (Comanici et al., 2025). Gemini-2.5-Flash achieves the best performance on Tol@1, S-EAA, and D-EAA, demonstrating stronger authentic emotion perception ability. In contrast, Qwen3-Omni-Captioner performs better on Tol@2 and usage. Overall, Gemini-2.5-Flash provides more accurate emotion supervision, which helps learn high-quality emotion representations. Based on the above analysis, we adopt FSQ as the quantization method, set the codebook size to 1000, and use Gemini-2.5-Flash as the judge model. Figure 3. Analysis results of different token merging rates on speech quality and emotion expressiveness. 6.4. Analysis of Context Token Merging Rate To analyze the impact of the token merging rate in AuEmoToMe, we conduct experiments with merging ratios of 0%, 15%, 30%, 45%, 60%, and 75%. As shown in Fig. 3, we report the performance on AuEmoACC, EmoACC, WER, MCD, and SpkSIM, where the five sectors correspond to different evaluation metrics. Overall, a token merging rate of 30% achieves the best overall performance. Under this setting, the model attains the best results on AuEmoACC, EmoACC, WER, and MCD, while maintaining near-best performance on SpkSIM. For emotion metrics, as the merging ratio increases from 0% to 30%, both AuEmoACC and EmoACC consistently improve. This indicates that moderate token merging effectively reduces the interference of redundant tokens and enhances the model’s ability to capture the authentic emotion of the target utterance. When the merging ratio is further increased, the performance degrades, suggesting that excessive merging impairs emotional expressiveness. For speech quality metrics, WER and MCD achieve the best performance at 30%, indicating that appropriate token merging alleviates the interference of redundant context in speech modeling, thereby improving pronunciation accuracy. At higher merging ratios, the performance degrades, further confirming the negative impact of information loss on speech generation. Finally, we adopt a token merging rate of 30% during inference to achieve the best balance between emotional expressiveness and speech quality. 7. Conclusion and Future Work In this work, we propose AuEmoChat, a novel CSS framework for authentic emotion understanding and rendering. AuEmoChat introduces AuEmoCodec to learn an authentic emotion token space, AuEmoToMe to merge redundant multimodal context tokens while preserving emotion context information, and Authentic Emotion Flow Matching to generate context-consistent authentic emotional speech. To the best of our knowledge, AuEmoChat is the first CSS system explicitly devoted to authentic emotion modeling. We hope this work can inspire further research on more human-like emotional speech synthesis for conversational agents. In future work, we will extend AuEmoChat to multilingual settings, scale up AuEmoCodec training, and further improve the interpretability of authentic emotion for more comprehensive emotional CSS. More detailed limitations and future works are provided in Appendix D. References S. R. Behera, A. Dhiman, K. Gowda, and A. S. Narayani (2024) Fastast: accelerating audio spectrogram transformer via token merging and cross-model knowledge distillation. arXiv preprint arXiv:2406.07676. Cited by: §2.3. D. Bolya, C. Fu, X. Dai, P. Zhang, C. Feichtenhofer, and J. Hoffman (2022) Token merging: your vit but faster. arXiv preprint arXiv:2210.09461. Cited by: §2.3, §4.2. Q. Chen, M. Tan, Y. Qi, J. Zhou, Y. Li, and Q. Wu (2022) V2C: visual voice cloning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 21242–21251. Cited by: §5.3. Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, et al. (2024) Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. Cited by: §6.3. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. Cited by: §4.3. G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §4.4, §6.3. G. Cong, J. Pan, L. Li, Y. Qi, Y. Peng, A. Van Den Hengel, J. Yang, and Q. Huang (2025) Emodubber: towards high quality and emotion controllable movie dubbing. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 15863–15873. Cited by: §5.3. Y. Deng, J. Xue, Y. Jia, Q. Li, Y. Han, F. Wang, Y. Gao, D. Ke, and Y. Li (2024) Concss: contrastive-based context comprehension for dialogue-appropriate prosody in conversational speech synthesis. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 10706–10710. Cited by: §1. D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tang, et al. (2025) Kimi-audio technical report. arXiv preprint arXiv:2504.18425. Cited by: §6.3. Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y. Yang, H. Hu, S. Zheng, Y. Gu, Z. Ma, et al. (2024a) Cosyvoice: a scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407. Cited by: §2.1. Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, et al. (2024b) Cosyvoice 2: scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. Cited by: §2.1, §4.1, §4.3. P. Ekman (1992) An argument for basic emotions. Cognition & emotion 6 (3-4), p. 169–200. Cited by: §1, §2.2. Z. Feng and S. Zhang (2023) Efficient vision transformer via token merger. IEEE Transactions on Image Processing 32, p. 4156–4169. Cited by: §2.3. R. Fu, Z. Wang, C. Meng, J. Lu, J. Wu, K. Qian, H. Zhang, and S. Fong (2026) Missing-by-design: certifiable modality deletion for revocable multimodal sentiment analysis. arXiv preprint arXiv:2602.16144. Cited by: §1. P. Gage (1994) A new algorithm for data compression. The C Users Journal 12 (2), p. 23–38. Cited by: §4.1. H. Guo, S. Zhang, F. K. Soong, L. He, and L. Xie (2021) Conversational end-to-end tts for voice agents. In 2021 IEEE Spoken Language Technology Workshop (SLT), p. 403–409. Cited by: §1, §1, §5.2. Y. Hu, R. Liu, Y. Ren, X. Yin, and H. Li (2025a) Chain-talker: chain understanding and rendering for empathetic conversational speech synthesis. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 1988–2003. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §1, §1, §1, §5.1, §5.2, §5.3, Table 1. Y. Hu, R. Liu, Y. Ren, X. Yin, and H. Li (2025b) UniTalker: conversational speech-visual synthesis. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 10248–10257. Cited by: §1, §2.2. Z. Jia, R. Liu, B. Sisman, and H. Li (2025) Multimodal fine-grained context interaction graph modeling for conversational speech synthesis. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 8863–8869. Cited by: §1. Z. Jia and R. Liu (2025) Intra-and inter-modal context interaction modeling for conversational speech synthesis. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1. X. Jiang, X. Peng, H. Xue, Y. Zhang, and Y. Lu (2023) Latent-domain predictive neural speech coding. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2111–2123. Cited by: §2.1. Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang, et al. (2024) Naturalspeech 3: zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100. Cited by: §2.1. J. Kong, J. Kim, and J. Bae (2020) Hifi-gan: generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems 33, p. 17022–17033. Cited by: §4.3. R. Kubichek (1993) Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE pacific rim conference on communications computers and signal processing, Vol. 1, p. 125–128. Cited by: §5.3. K. Lee, K. Park, and D. Kim (2023) Dailytalk: spoken dialogue dataset for conversational text-to-speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1, §5.1, §5.2, Table 1. J. Li, Y. Meng, C. Li, Z. Wu, H. Meng, C. Weng, and D. Su (2022a) Enhancing speaking styles in conversational text-to-speech synthesis with graph-based multi-modal context modeling. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 7917–7921. Cited by: §1. J. Li, Y. Meng, X. Wu, Z. Wu, J. Jia, H. Meng, Q. Tian, Y. Wang, and Y. Wang (2022b) Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks. In Proceedings of the 30th ACM International Conference on Multimedia, p. 5811–5820. Cited by: §1. Z. Lian, H. Sun, L. Sun, L. Chen, H. Chen, H. Gu, Z. Wen, S. Chen, Z. Siyuan, H. Yao, et al. (2024) Open-vocabulary multimodal emotion recognition: dataset, metric, and benchmark. Cited by: §2.2. H. Liu, X. Xu, Y. Yuan, M. Wu, W. Wang, and M. D. Plumbley (2024a) Semanticodec: an ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing 18 (8), p. 1448–1461. Cited by: §2.1. R. Liu, Y. Hu, Y. Ren, X. Yin, and H. Li (2024b) Emotion rendering for conversational speech synthesis with heterogeneous graph-based context modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 18698–18706. Cited by: §1, §1, §2.2, §5.2, §5.3, Table 1. R. Liu, Y. Hu, Y. Ren, X. Yin, and H. Li (2024c) Generative expressive conversational speech synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 4187–4196. Cited by: §1, §1, §5.1, §5.2, Table 1. R. Liu, Z. Jia, F. Bao, and H. Li (2025) Retrieval-augmented dialogue knowledge aggregation for expressive conversational speech synthesis. Information Fusion, p. 102948. Cited by: §5.3. R. Liu, Z. Jia, J. Yang, Y. Hu, and H. Li (2024d) Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling. arXiv preprint arXiv:2410.09524. Cited by: §1, §5.3. Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen (2024) Emotion2vec: self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, p. 15747–15760. Cited by: §5.3. M. McTear (2022) Conversational ai: dialogue systems, conversational agents, and chatbots. Springer Nature. Cited by: §1. C. Meng, J. Luo, Z. Yan, Z. Yu, R. Fu, Z. Gan, and C. Ouyang (2026) Tri-subspaces disentanglement for multimodal sentiment analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8791–8800. Cited by: §1. F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen (2023) Finite scalar quantization: vq-vae made simple. arXiv preprint arXiv:2309.15505. Cited by: §1, §6.3. A. C. Morris, V. Maier, and P. D. Green (2004) From wer and ril to mer and wil: improved evaluation measures for connected speech recognition.. In Interspeech, p. 2765–2768. Cited by: §5.3. J. Ning, Y. Sun, B. Xu, Z. Yang, L. Luo, and H. Lin (2024) Breaking the boundaries: a unified framework for chinese named entity recognition across text and speech. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1250–1260. Cited by: §1. S. Park, C. Kim, H. Rha, M. Kim, J. Hong, J. Yeo, and Y. Ro (2024) Let’s go real talk: spoken dialogue model for face-to-face conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16334–16348. Cited by: §5.1. R. Plutchik (2001) The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist 89 (4), p. 344–350. Cited by: §2.2. K. Seaborn, N. P. Miyake, P. Pennefather, and M. Otake-Matsuura (2021) Voice in human–agent interaction: a survey. ACM Computing Surveys (CSUR) 54 (4), p. 1–43. Cited by: §1. R. C. Streijl, S. Winkler, and D. S. Hands (2016) Mean opinion score (mos) revisited: methods and applications, limitations and alternatives. Multimedia Systems 22 (2), p. 213–227. Cited by: §5.3. H. Tang, X. Zhang, J. Wang, N. Cheng, and J. Xiao (2023) EmoMix: emotion mixing via diffusion models for emotional speech synthesis. In Interspeech 2023, p. 12–16. External Links: Document, ISSN 2958-1796 Cited by: §1, §4.4. A. Van Den Oord, O. Vinyals, et al. (2017) Neural discrete representation learning. Advances in neural information processing systems 30. Cited by: §2.1, §6.3. J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, et al. (2025) Qwen3-omni technical report. arXiv preprint arXiv:2509.17765. Cited by: §6.3. J. Xue, Y. Deng, F. Wang, Y. Li, Y. Gao, J. Tao, J. Sun, and J. Liang (2023) M 2-ctts: end-to-end multi-scale multi-modal conversational text-to-speech synthesis. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §1. Y. Yasuda and T. Toda (2023) Analysis of Mean Opinion Scores in Subjective Evaluation of Synthetic Speech Based on Tail Probabilities. In Proc. INTERSPEECH 2023, p. 5491–5495. External Links: Document, ISSN 2308-457X Cited by: Table 1. X. Zhang, F. Zeng, Y. Quan, Z. Hui, and J. Yao (2025) Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 10203–10211. Cited by: §1. X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu (2023) Speechtokenizer: unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692. Cited by: §2.1. Z. Zhang, L. Li, G. Cong, H. Yin, Y. Gao, C. Yan, A. v. d. Hengel, and Y. Qi (2024) From speaker to dubber: movie dubbing with prosody and duration consistency learning. In Proceedings of the 32nd ACM international conference on multimedia, p. 7523–7532. Cited by: §5.3. K. Zhou, B. Sisman, R. Rana, B. W. Schuller, and H. Li (2022) Speech synthesis with mixed emotions. IEEE Transactions on Affective Computing 14 (4), p. 3120–3134. Cited by: §1, §4.4. L. Zhou, J. Gao, D. Li, and H. Shum (2020) The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics 46 (1), p. 53–93. Cited by: §1.