Paper deep dive

EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis

Tianchi Liu, Zeyang Song, Tianrui Wang, Zhipeng Li, Chenglin Xu, Yiwen Guo

Year: 2026Venue: arXiv preprintArea: eess.ASType: PreprintEmbeddings: 89

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/26/2026, 4:34:30 AM

Summary

The paper introduces EmoTra-TTS, a text-to-speech system designed to generate smooth intra-utterance emotion transitions. It addresses the limitation of current TTS systems that use static emotion labels by employing a multi-pass flow blending pipeline to synthesize frame-aligned transition audio, dual-stage Valence-Arousal-Dominance (VAD) conditioning for prosodic planning and acoustic realization, and a direction-magnitude decoupled injection mechanism to prevent content degradation. The system achieves significant improvements in emotion transition quality with minimal parameter overhead.

Entities (7)

EmoTra-TTS · model · 100%Valence-Arousal-Dominance · concept · 95%Direction-magnitude decoupled injection · method · 90%Flow Matching · method · 90%Multi-pass flow blending · method · 90%CosyVoice2 · model · 85%LayerNorm · component · 85%

Relation Signals (6)

EmoTra-TTS → uses → Multi-pass flow blending

confidence 95% · a multi-pass flow blending pipeline synthesizes frame-aligned transition audio

EmoTra-TTS → uses → Valence-Arousal-Dominance

confidence 95% · dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning

EmoTra-TTS → uses → Direction-magnitude decoupled injection

confidence 95% · direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude

EmoTra-TTS → isbasedon → CosyVoice2

confidence 90% · CosyVoice2 Du et al. (2024), our backbone

CosyVoice2 → uses → Flow Matching

confidence 90% · employs supervised semantic tokens with a causal conditional flow matching (CFM) decoder

Direction-magnitude decoupled injection → uses → LayerNorm

confidence 90% · LayerNorm + fixed scale decouples emotion direction from injection magnitude

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence-Arousal-Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction-magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS adds only +0.43% parameters with no latency overhead, achieves 30%-87% relative improvement on emotion transition quality, corroborated by 64.4%-79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

85,456 characters extracted from source content.

Expand or collapse full text

EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesis Tianchi Liu Affiliation: LIGHTSPEED Zeyang Song Affiliation: National University of Singapore Tianrui Wang Affiliation: Nanyang Technological University Zhipeng Li Affiliation: LIGHTSPEED Chenglin Xu Affiliation: LIGHTSPEED Yiwen Guo Affiliation: Independent Researcher †Corresponding Authors:tianchi_liu, zeyang_song@u.nus.edu Abstract Psychological research on emotion dynamics has established that human affect is a continuous, evolving process: emotions rise, decay, and transition within seconds. Current emotional text-to-speech (TTS) systems, however, condition on a single discrete label or static embedding per utterance, fundamentally misaligning with the temporal nature of affect. While recent LLM-based TTS systems may implicitly vary prosody through text understanding, such variation is neither explicitly controllable nor precise enough for targeted intra-utterance transitions. We address three challenges: (1) a multi-pass flow blending pipeline synthesizes frame-aligned transition audio, circumventing the scarcity of natural intra-utterance transitions; (2) dual-stage Valence–Arousal–Dominance (VAD) conditioning guides prosodic planning in the LLM and acoustic realization in the flow decoder via frame-level VAD embeddings; (3) direction–magnitude decoupled injection structurally separates emotion direction from injection magnitude, preventing content degradation. EmoTra-TTS11 1 Demo page: https://liu-tianchi.github.io/EmoTra_DemoPage/; The complete code, including data synthesis and training pipelines: https://github.com/Liu-Tianchi/EmoTra-TTS adds only +0.43%+0.43\% parameters with no latency overhead, achieves 30%–87% relative improvement on emotion transition quality, corroborated by 64.4%–79.5% overall win rates in pairwise preference tests against four SOTA baselines and two commercial systems. 1 Introduction Human emotion is not a static label but a dynamic process that unfolds over time. Research on emotion dynamics Davidson (1998); Kuppens and Verduyn (2017) has established that affective states fluctuate continuously, exhibiting inertia, gradual decay, and smooth transitions between categories Verduyn et al. (2009); Cowen and Keltner (2017), with temporal envelopes that carry communicative intent Trampe et al. (2015). Russell’s circumplex model Russell (1980) and the PAD framework Mehrabian (1996) formalize this continuity by representing emotions in a continuous VAD space, supported by neuroscientific evidence Posner et al. (2005) and continuous gradients between categories Cowen and Keltner (2017). These findings motivate emotion control that is continuous in both emotion space and time, especially for expressive speech scenarios such as storytelling, dubbing, and audiobook narration, where abrupt transitions are highly perceptible. Despite this foundation, emotional TTS predominantly assigns one categorical label per utterance Lei et al. (2022). Recent systems offer finer control: EmoSphere-TTS Cho et al. (2024) uses spherical VAD vectors, EmoKnob Chen et al. (2024) provides disentangled style knobs, and WeSCon Wang et al. (2025a) explores discrete word-level control. Yet these utterance- and word-level systems do not model the continuous temporal dynamics of emotion: the rise, transition, and decay that define natural affective experience. Recent LLM-based TTS systems Hu et al. (2026); Du et al. (2025); Hussain et al. (2025); Leng et al. (2024); Zhou et al. (2025); Li et al. (2026); Wang et al. (2026c); Song et al. (2026) may implicitly vary prosody through text understanding, but such variation is neither user-controllable nor sufficiently precise for targeted emotion transitions within an utterance. Bridging this gap requires addressing three interconnected challenges: (1) Data scarcity: naturally occurring intra-utterance emotion transitions are severely underrepresented in emotional speech corpora, and eliciting them from actors is notoriously difficult Busso et al. (2008); Zhou et al. (2022); Michel et al. (2026). (2) Dual-level conditioning: modern LLM-based TTS Du et al. (2024); Wang et al. (2023); Anastassiou et al. (2024) decomposes synthesis into prosodic planning (LLM) and acoustic realization (flow decoder), each requiring temporally aligned emotion signals Scherer (2003). (3) Magnitude control: injecting conditioning into a pretrained decoder risks pushing it outside its operating range; learnable constraints are circumvented by co-adaptation Locatello et al. (2019). To address these challenges, we propose EmoTra-TTS built upon three key components: • Synthetic emotion transition data (§3.2): a multi-pass flow blending pipeline generates frame-aligned transition audio, circumventing the difficulty of recording natural transitions. • Dual-stage VAD conditioning (§3.3, §3.4): VAD tokens condition the LLM for prosodic planning; frame-level VAD embeddings modulate the flow decoder’s speaker pathway. • Direction–magnitude decoupled injection (§3.4.4): LayerNorm + fixed scale decouples emotion direction (learned) from injection magnitude (kept bounded via re-normalization), mitigating the content-expressiveness trade-off. 2 Related Work 2.1 Emotion Dynamics and Emotional TTS Emotion dynamics research Davidson (1998); Kuppens and Verduyn (2017) has established that affect possesses an intrinsic temporal microstructure. Dimensional models Mehrabian (1996) with acoustic correlates Scherer (2003); Banse and Scherer (1996) motivate VAD transitions, not merely VAD points, as conditioning signals. Emotional TTS has evolved from discrete emotion embeddings Wang et al. (2018) and prosody transfer Skerry-Ryan et al. (2018) to continuous VAD control Cho et al. (2024); Inoue et al. (2025); Zhou et al. (2024). EmoShift Zhou et al. (2026) and CoCoEmo Wang et al. (2026b) explore activation steering, while EMORL-TTS Li et al. (2025) applies reinforcement learning for emotion control. Recent efforts further explore finer-grained control: Wu et al. (2024) condition a flow-matching decoder on frame-level arousal–valence from reference audio, Liang et al. (2026) explore segment-level modeling for intra-utterance emotion variation, and WeSCon Wang et al. (2025a) introduces word-level discrete control. While these works address related forms of fine-grained control, explicit and controllable intra-utterance transitions along continuous VAD dimensions, specified as a user-defined trajectory rather than reference-driven or discrete control, remain underexplored Cui et al. (2025b); Xie et al. (2025). Recent benchmarks report the same limitation for commercial systems: InstructTTSEval Huang et al. (2025) shows Gemini and VoxInstruct handle only rudimentary changes (e.g., voice rise), not genuine emotion change; EmergentTTS-Eval Manku et al. (2025) lists natural shifts to contrasting emotions among its most challenging cases; EmoS Wang et al. (2026a) finds commercial models (e.g., GPT-4o-Audio, Gemini 2.5 Pro) lag far behind humans on cross-turn emotion transitions; and MINT-Bench Chen et al. (2026) identifies paralinguistic control, including explicit intra-utterance emotion transitions, as a major bottleneck for commercial systems (e.g., Gemini, GPT-4o mini TTS, ElevenLabs). In contrast, EmoTra-TTS models such continuous transitions. 2.2 Synthetic Data for Speech Tasks Synthetic data has proven effective for speech tasks: TTS-augmented ASR training Yang et al. (2025c), speech translation Zhao et al. (2023), speech security Hwang et al. (2026); Liu et al. (2024b); Liu et al. (2025b), speaker recognition Tao et al. (2025); Liu et al. (2025a), and emotion recognition Ma et al. (2024a); Ma et al. (2024b). We extend this paradigm to within-utterance transition synthesis via mel-space flow blending, which existing corpora Busso et al. (2008); Zhou et al. (2022) cannot provide. 2.3 LLM-Based TTS and Flow Matching VALL-E Wang et al. (2023) pioneered LLM-based zero-shot TTS, followed by Seed-TTS Anastassiou et al. (2024), NaturalSpeech 3 Ju et al. (2024), MaskGCT Wang et al. (2025c), Llasa Ye et al. (2025), Spark-TTS Wang et al. (2025b), Sticker-TTS Chen et al. (2025a), and GLM-TTS Cui et al. (2025a). CosyVoice2 Du et al. (2024), our backbone, employs supervised semantic tokens with a causal conditional flow matching (CFM) Lipman et al. (2023) decoder. CFM has demonstrated strong performance Mehta et al. (2024); Chen et al. (2025b); Yang et al. (2025a). Our work extends CFM conditioning to support time-varying emotion. 2.4 Conditioning Injection in Frozen Models Injecting auxiliary signals into frozen models is a growing paradigm. In audio, FiLM Perez et al. (2018) conditioning appears in WaveGrad Chen et al. (2021) and DiffWave Kong et al. (2021); LoRA Hu et al. (2022) and DoRA Liu et al. (2024a) enable efficient adaptation. A common thread is the need to control influence strength: e.g., ControlNet Zhang et al. (2023) uses zero-initialization. Our direction–magnitude decoupling takes a different route: unlike soft penalties that can be circumvented Locatello et al. (2019), LayerNorm + fixed scale constrains the input-dependent MLP output through re-normalization, so that the injection magnitude remains bounded and close to the speaker-embedding norm. Figure 1: Synthetic emotion transition data generation pipeline. Samples are available on the demo page. 3 Method Research on emotion dynamics shows that affective episodes unfold in phases: from rising intensity to transition or decay Davidson (1998); Verduyn et al. (2009); Kuppens and Verduyn (2017). Within a single utterance, this manifests as a shift from one affective state to another. We model this minimal temporal unit as a transition from an initial emotion init v_init to a resultant emotion res v_res, representing the non-trivial dynamic pattern that existing single-emotion TTS systems cannot express. 3.1 Problem Formulation Given text T, an initial emotion init=(Vinit,Ainit,Dinit)∈[0,1]3 v_init=(V_init,A_init,D_init)∈[0,1]^3, and a resultant emotion res∈[0,1]3 v_res∈[0,1]^3, we seek to synthesize speech where emotion transitions smoothly from init v_init to res v_res over the utterance duration. Framed in terms of emotion dynamics Davidson (1998); Kuppens and Verduyn (2017), the system must model the temporal transition of an emotional episode, not merely its static endpoint. We adopt the VAD dimensional model Russell (1980); Mehrabian (1996) for three reasons: (a) it subsumes discrete emotion categories as specific points in the continuous space Posner et al. (2005); (b) it naturally supports interpolation for constructing smooth transitions, mirroring gradual emotion shifts in emotion dynamics research Verduyn et al. (2009); Cowen and Keltner (2017); (c) its axes have well-established acoustic correlates Scherer (2003); Banse and Scherer (1996), ensuring that changes in VAD space correspond to perceptually meaningful acoustic variations. 3.2 Synthetic Emotion Transition Data 3.2.1 Motivation Existing emotional speech corpora are predominantly annotated with single emotion labels Busso et al. (2008); Zhou et al. (2022). Eliciting natural within-utterance transitions from actors requires precisely controlling onset timing, direction, and intensity while maintaining natural prosody and content. These are demands that professional actors find challenging. This data bottleneck has limited prior work on intra-utterance emotion dynamics. We circumvent this limitation by generating synthetic emotion transition data from a pretrained zero-shot TTS system. The key insight is that flow-based decoders condition on speaker/emotion embeddings independently of linguistic content: the same speech token sequence can be decoded under different emotion conditionings. This produces mel spectrograms that are temporally aligned but differ in emotion. This content-invariance property enables smooth blending in the mel spectrogram space without temporal misalignment. 3.2.2 Multi-Pass Flow Blending For a given text and emotion pair (initial, resultant), the pipeline proceeds as follows: (1) Shared token generation. The LLM generates a single speech token sequence init=[s1,…,sM] s_init=[s_1,…,s_M] conditioned on the initial emotion. This sequence encodes the prosodic structure and is shared across all subsequent flow passes, ensuring frame-level temporal alignment. (2) Multi-pass flow decoding. Three independent flow passes produce emotion-specific mel spectrograms (Fig. 1, Part 1 to Part 3): melinit _init =Flow(init,init,init) =Flow( s_init,\; e_init,\; c_init) (1) meltra _tra =Flow(init,res,init) =Flow( s_init,\; e_res,\; c_init) (2) melres _res =Flow(init,res,res) =Flow( s_init,\; e_res,\; c_res) (3) where e denotes the speaker embedding and c the prompt condition, both derived from the target speaker. The intermediate meltramel_tra uses the initial emotion’s masked mel spectrogram context but the resultant emotion’s speaker embedding, forming a transition bridge between the two emotional states. (3) Sigmoid crossfade blending. The three mel spectrograms are blended via sigmoid crossfades: σ⁡(t)=11+e−κ⁡(t−tc)σ(t)= 11+e^-κ(t-t_c) (4) with steepness κ and center tct_c, providing near-binary blending at the edges (preserving pure emotion regions) with a smooth crossfade in the center, avoiding audible artifacts from linear blending or hard concatenation. The choice of the crossfade curve among several candidates, as shown in the upper part of Fig. 1, is analyzed in Appendix F. After vocoding, a fourth part appends a pure resultant-emotion utterance (Part 4 in Fig. 1). (4) Quality filtering. Whisper ASR verifies content integrity; samples exceeding CER threshold τCER _CER or WER threshold τWER _WER are discarded, as the blending process can occasionally produce phonetically ambiguous regions near transition boundaries. 3.2.3 Advantages over Recordings This pipeline offers three advantages. First, it produces arbitrary emotion pairs: any combination of available discrete emotions can be synthesized. Second, transition timing and shape are precisely controlled via segment ratios and crossfade parameters, providing consistent supervision. Third, it is scalable: multi-GPU parallel synthesis generates thousands of transition samples per hour, whereas studio recording of emotion transitions requires extensive actor preparation and multiple takes. Figure 2: Overview of the EmoTra-TTS. Lower part: LLM stage with temporal VAD tokens (§3.3). Upper part: Flow decoder with frame-level emotion injection via direction–magnitude decoupled injection (§3.4). 3.2.4 VAD Annotation VAD values for the initial and resultant emotions are obtained by annotating their reference speech with a pretrained wav2vec2-based speech emotion recognition model Wagner et al. (2023). 3.3 LLM with Temporal VAD Tokens 3.3.1 VAD Projection A projection network ϕproj:ℝ3→ℝdLLM _proj:R^3 ^d_LLM maps VAD vectors to the LLM embedding space: ϕproj()=W2⋅ReLU(LN(W1+b1))+b2 _proj( v)=W_2·ReLU(LN(W_1 v+b_1))+b_2 (5) where W1∈ℝdh×3W_1 ^d_h× 3, W2∈ℝdLLM×dhW_2 ^d_LLM× d_h, and LN denotes LayerNorm. The intermediate normalization stabilizes training given the large dimensionality expansion from 3 to dLLMd_LLM, following best practices for cross-modal projection in multimodal learning Li et al. (2023). 3.3.2 Temporal Token Design A key design question is how to encode a continuous VAD transition in a discrete token sequence. A single VAD embedding provides only a static global label, while using only the two endpoints init v_init and res v_res leaves the transition trajectory implicit. In our preliminary experiments, this produced perceptibly abrupt transitions. We therefore uniformly sample NtraN_tra intermediate points along the line segment between them: tra,k=ϕproj((1−αk)init+αkres),αk=kNtra+1 q_tra,k= _proj ((1- _k) v_init+ _k v_res ), _k= kN_tra+1 (6) for k=1,2,…,Ntrak=1,2,…,N_tra. Together with the endpoint tokens init q_init and res q_res, the Ntra+2N_tra+2 VAD tokens are prepended to the text tokens. This balances transition resolution with avoiding excessive conditioning that may dilute the LLM’s attention to text. 3.3.3 Training Objective The input sequence is [SOS][inittra,1⋯[SOS][ q_init\; q_tra,1·s tra,Ntrares][text][task][speech][EOS] q_tra,N_tra\; q_res][text][task][speech][EOS], with the loss: ℒLLM=ℒCE+λ⋅ℒhiddenL_LLM=L_CE+λ·L_hidden (7) where ℒCEL_CE is the cross-entropy loss on speech tokens. The auxiliary hidden-state reconstruction loss provides perceptual grounding. Without it, ϕproj _proj could map VAD vectors to arbitrary embedding regions that are numerically distinct but acoustically meaningless. ^init h_init =ϕrecon(init),^res=ϕrecon(res) = _recon( q_init), h_res= _recon( q_res) (8) ℒhidden=‖^init−init‖1+‖^res−res‖1L_hidden=\| h_init- h_init\|_1+\| h_res- h_res\|_1 (9) where ϕrecon:ℝdLLM→ℝdSER _recon:R^d_LLM ^d_SER maps to the wav2vec2 feature space Wagner et al. (2023), anchoring VAD embeddings to perceptually meaningful prosodic and affective representations rather than arbitrary embedding space regions. 3.4 Flow Decoder with Frame-Level Emotion Injection 3.4.1 Emotion via the Speaker Pathway The backbone flow decoder receives four-channel input [t,,,][ x_t, μ, e, c] Du et al. (2024), each ∈ℝdmel ^d_mel, where μ encodes linguistic content and e encodes speaker identity. We inject emotion through the speaker channel based on two observations. First, speaker identity and emotion both modulate how speech sounds without changing what is said; they share the same conditioning role relative to content Kang et al. (2023). Second, additive injection through an existing channel requires no architectural modification to the pretrained decoder, following the minimal intervention principle that has proven effective in adapter-based approaches Zhang et al. (2023); Li et al. (2024): ^(t)=+emo(t) e(t)= e+ e_emo(t) (10) This extends the static 2D speaker tensor to a time-varying 3D tensor ^∈ℝB×dmel×T e ^B× d_mel× T. 3.4.2 Frame-Level VAD Interpolation The frame-level VAD ⁡(t) v(t) is constructed via piecewise linear interpolation: α⁡(t)=0α(t)=0 for t<rinit⋅Tt<r_init· T, linearly interpolated during [rinit⋅T,(rinit+rtra)⋅T)[r_init· T,\,(r_init+r_tra)· T), and α⁡(t)=1α(t)=1 thereafter. This mirrors the temporal structure of the synthetic training data (§3.2), ensuring consistency between training supervision and inference-time conditioning. 3.4.3 VAD Embedding Pipeline The emotion embedding reuses frozen LLM-stage modules for cross-stage consistency: emo(t)=finject(ψdown(ϕrecon(ϕproj(⁡(t))))) e_emo(t)=f_inject ( _down( _recon( _proj( v(t)))) ) (11) where ϕproj _proj and ϕrecon _recon are frozen from Stage 1, and ψdown:ℝdSER→ℝdmel _down:R^d_SER ^d_mel is a trainable MLP with zero-initialized output layer. Freezing and reusing VAD modules ensures the same VAD vector produces identical intermediate representations at both stages, preserving the perceptual grounding learned during Stage 1 without additional supervision. 3.4.4 Direction–Magnitude Decoupled Injection The choice of finjectf_inject is the central design challenge. An MLP conditioning vector entangles two semantically distinct factors: a direction (specifying which emotion) and a magnitude (specifying how strongly it modulates the decoder). Our ablation (§5.1 and Appendix E) shows that unconstrained training allows MLP output norms to exceed the speaker embedding norm, causing content degradation. Learnable constraints (e.g., FiLM Perez et al. (2018)) fail because the MLP co-adapts to circumvent them, paralleling the well-documented insufficiency of soft penalties for disentanglement Locatello et al. (2019). The flow matching loss, averaging over all mel frames, provides no gradient signal that explicitly constrains the emotion norm, creating exactly the conditions under which soft constraints fail. We enforce a hard separation through architectural constraints: finject()=ϵ⋅LayerNorm()f_inject( x)=ε·LayerNorm( x) (12) LayerNorm re-normalizes the MLP output, so the MLP learns the emotion direction while the injection magnitude stays controlled; the trainable affine parameters (γ, β) are few (160) and identity-initialized, and the measured post-training norm is reported in §5.4. The non-trainable scalar ϵε independently controls injection strength, with no gradient path connecting the two factors. Zero-initialized output layers ensure emo(t)= e_emo(t)= 0 at training start Zhang et al. (2023). This design mirrors the direction–magnitude decomposition in DoRA Liu et al. (2024a), which shows that fine-tuning primarily changes weight directions. The complete injection is: ^(t)=+ϵ⋅LN(ψdown(ϕrecon(ϕproj(⁡(t))))) e(t)= e+ε·LN ( _down( _recon( _proj( v(t)))) ) (13) All decoder parameters are frozen; only ψdown _down and LayerNorm affine parameters are trainable. Speech tokens are pre-computed from Stage 1, fully decoupling the two training stages. 3.4.5 Flow Matching Training Objective The flow decoder is trained with a conditional flow matching (CFM) Lipman et al. (2023) objective. Given target mel spectrogram 1∈ℝdmel×T x_1 ^d_mel× T, noise ∼⁡(,) z ( 0, I), and time t∼⁡(0,1)t (0,1), we construct the interpolated sample t=(1−(1−σmin)t)+t1 x_t=(1-(1- _min)t)\, z+t\, x_1 and ground-truth velocity =1−(1−σmin) u= x_1-(1- _min)\, z, with σmin=10−6 _min=10^-6. The loss minimizes: ℒFlow=t,1,[‖^θ(t,t,,^(t))−‖2]L_Flow=E_t,\, x_1,\, z\! [ \| u_θ\! ( x_t,\,t,\, μ,\, e(t) )- u \|^2 ] (14) where ^θ u_θ is the velocity estimator and ^(t) e(t) is from Eq. (13). Classifier-free guidance Ho and Salimans (2021) is applied by randomly dropping conditioning with probability pcfg=0.2p_cfg=0.2. 4 Experimental Setup Base model. CosyVoice2-0.5B Du et al. (2024). Source corpus. EmoVoice-DB Yang et al. (2025b), containing five speakers and seven emotion categories (neutral, happy, sad, angry, fearful, disgusted, surprised). We apply cross-validated filtering: a pretrained wav2vec2-based VAD predictor Wagner et al. (2023)22 2 https://github.com/audeering/w2v2-how-to estimates continuous scores, and only samples falling within literature-informed acceptance ranges Russell (1980); Mehrabian (1996); Fontaine et al. (2007) (Table 6 in Appendix C) are retained, removing mislabeled or acoustically ambiguous samples. Synthetic data. Emotion transition pairs are generated via multi-pass flow blending (§3.2), covering all emotion pair combinations (∼ 100K utterances). Quality filtering uses Whisper large-v3 Radford et al. (2023) with τCER=0.10 _CER=0.10; minimum VAD change threshold δvad=0.35 _vad=0.35 (Chebyshev distance). Data split: 90/10 train/validation. Training. Stage 1 (LLM) fine-tunes the VAD projection and reconstruction modules; Stage 2 (Flow) freezes all decoder parameters and trains only the injection MLP and LayerNorm affine (∼ 280K parameters). Fixed emotion scale ϵ=0.07ε\!=\!0.07 yields an effective norm matched to the speaker embedding norm (‖2≈0.622\| e\|_2≈ 0.622). Full hyperparameters are provided in Appendix A. Evaluation. We conduct a blind listening test with a professional panel of 21 raters. To enable a more fine-grained and independent evaluation of different aspects, we decompose MOS into three metrics: MOS-Qua, for speech naturalness and quality (ignoring emotional correctness); MOS-Emo for emotion adequacy; and MOS-Tra for emotion transition smoothness. We compute speaker similarity (SIM)33 3 https://github.com/BytedanceSpeech/seed-tts-eval, with the reference defined as the average embedding of 200 samples covering all emotions. Word error rate (WER) is evaluated by Whisper large-v344 4 https://huggingface.co/openai/whisper-large-v3 Radford et al. (2023). We further include two pairwise preference tests with partially overlapping rater pools: one against the open-source baselines (11 raters, 160 pairs, ∼ 6× coverage; §5.5, Appendix B.2) and a supplementary one against two commercial systems (12 raters, 80 pairs, ∼ 8× coverage; §5.6, Appendix B.3); both report win rates, 95% CIs, and inter-rater agreement. In addition, we compute the Prosodic Jerk Ratio (JR-F0) in Appendix D as an objective metric for transitions. 5 Results and Analysis 5.1 Injection Architecture Comparison Table 1: Injection architecture comparison. Variant Quality Evaluation Emotion Evaluation MOS-Qua SIM WER MOS-Emo MOS-Tra Linear 3.60± 0.12 0.707 1.65 3.39± 0.15 3.33± 0.16 Dir–Mag (ours) 3.54± 0.13 0.701 1.77 3.52± 0.15 3.63± 0.17 Table 2: Comparison with existing TTS systems on the test set. SIM: speaker similarity. WER: word error rate (%). We decompose MOS into MOS-Qua, MOS-Emo, and MOS-Tra for a more fine-grained and independent evaluation of speech quality, emotion rendering adequacy, and intra-utterance emotion transition smoothness, respectively. Subjective evaluation results are reported as means with 95% CIs. Our base system is CosyVoice2. Model Emotion Conditioning Quality Evaluation Emotion Rendering Emotion Transition MOS-Qua↑ SIM↑ WER↓ MOS-Emo↑ MOS-Tra↑ (a) Neutral prompt speech; no emotion conditioning Qwen3-TTS Hu et al. (2026) — 4.02± 0.09 0.726 1.05 2.72± 0.17 2.40± 0.18 MOSS-TTS MOSS-TTS Team (2026) — 3.98± 0.09 0.754 1.01 2.76± 0.18 2.36± 0.17 CosyVoice2 Du et al. (2024) — 3.87± 0.10 0.711 1.21 2.53± 0.16 2.32± 0.17 (b) Natural-language emotion instruction (e.g., “speak happily then sadly”) Qwen3-TTS Hu et al. (2026) Instruct 3.79± 0.10 N/A 1.28 3.33± 0.15 2.67± 0.19 MOSS-TTS MOSS-TTS Team (2026) Instruct 3.22± 0.15 N/A 2.55 2.80± 0.17 2.13± 0.16 (c) Natural-language emotion instruction with neutral prompt speech CosyVoice2 Du et al. (2024) Instruct 3.63± 0.12 0.634 1.60 2.64± 0.14 2.15± 0.16 EmoVoice Yang et al. (2025b) Instruct 3.59± 0.13 0.640 1.42 2.48± 0.15 2.31± 0.17 (d) Emotional prompt speech: generate each emotion segment separately, then concatenate Qwen3-TTS Hu et al. (2026) Prompt 3.46± 0.14 0.730 0.92 3.70± 0.14 2.80± 0.18 MOSS-TTS MOSS-TTS Team (2026) Prompt 2.85± 0.17 0.735 2.64 3.89± 0.12 1.94± 0.16 CosyVoice2 Du et al. (2024) Prompt 2.97± 0.15 0.703 1.83 3.82± 0.12 2.26± 0.17 (e) Emotional prompt speech with word-level emotion generation control WeSCon Wang et al. (2025a) Prompt 2.80± 0.15 0.649 2.92 3.53± 0.14 2.39± 0.17 (f) Our synthetic training data: multi-pass flow blending (§3.2), decoded by base CosyVoice2 Synth. Data (Ours) Prompt 3.58± 0.11 0.707 1.49 4.05± 0.12 3.55± 0.15 (g) Proposed: continuous VAD conditioning with smooth intra-utterance transition EmoTra-TTS (Ours) VAD 3.54± 0.13 0.701 1.77 3.52± 0.15 3.63± 0.17 – w/o flow SFT VAD 3.62± 0.12 0.709 1.65 2.70± 0.17 2.58± 0.18 We evaluate seven injection architectures for Stage 2 (Table 1; full results and failure analysis in Appendix E, Table 8). Linear yields the best speech quality (MOS-Qua 3.60, SIM 0.707, WER 1.65%) but weaker emotion (MOS-Emo 3.39, MOS-Tra 3.33), suggesting that simple additive injection preserves content but lacks expressiveness. Our Dir–Mag design structurally decouples emotion direction from injection magnitude, achieving the best emotion performance (MOS-Tra 3.63, MOS-Emo 3.52) with acceptable speech quality trade-off (MOS-Qua 3.54, SIM 0.701, WER 1.77%). 5.2 Comparison with Open-source Systems Table 2 compares EmoTra-TTS against baselines spanning seven conditioning strategies. These baselines are all recent open-source systems, including CosyVoice2 Du et al. (2024), EmoVoice Yang et al. (2025b), WeSCon Wang et al. (2025a), Qwen3-TTS Hu et al. (2026), and MOSS-TTS MOSS-TTS Team (2026), which together reasonably represent the current instruction-following capability of open-source TTS. Closed-source commercial systems are treated separately: recent benchmarks report the same intra-utterance transition limitation (§2), and we provide a direct comparison against two of them (GPT-4o mini TTS and ElevenLabs v3) in §5.6. Neutral baselines (a) achieve the highest quality (up to MOS-Qua 4.02), but produce minimal emotion rendering and transition quality. MOS-Qua primarily reflects speech quality rather than emotional correctness; therefore, high MOS-Qua scores under neutral conditions are expected. Instruction-based systems (b, c) show that current TTS systems cannot translate natural-language emotion instructions into precise intra-utterance emotion transitions. The instruction-following capability determines emotion rendering (MOS-Emo ranging from 2.48 to 3.33); adding a neutral prompt (c) preserves speaker similarity (SIM 0.634–0.640) but anchors the decoder to neutral prosody, further limiting emotion expressiveness. Prompt-based concatenation (d) achieves the strongest baseline emotion rendering (up to MOS-Emo 3.89) since each segment is cloned from a matching emotional reference. However, stronger emotional expressiveness increases perceptual discontinuity at boundaries. MOSS-TTS exemplifies this: it achieves the highest baseline MOS-Emo (3.89) yet the lowest MOS-Tra (1.94), as vivid emotion makes boundary artifacts more noticeable. High Failure Cost. We further quantify this effect. When discrete baselines such as MOSS-TTS produce abrupt emotion shifts (Category d), naturalness drops significantly compared to neutral conditions (Category a), showing that a single failed transition can degrade overall quality. Moreover, this degradation increases with higher emotion expressiveness (MOS-Emo), indicating that failure cost grows with expressiveness. This emphasizes the importance of smooth transition modeling in expressive TTS due to its high perceptual impact. WeSCon (e) controls intra-utterance emotion variation in a single decoding pass via word-level prompt conditioning. While this enables finer-grained control, its discrete per-word assignment lacks smooth transition modeling between segments. As a result, it introduces boundary discontinuities, leading to lower MOS-Tra (2.39). All above systems achieve MOS-Tra ≤ 2.80, indicating that controllable smooth intra-utterance emotion transition remains an open problem. Our synthetic data (f) achieves the highest emotion rendering performance (MOS-Emo 4.05) and a MOS-Tra of 3.55, validating the effectiveness of the synthetic data. While outperforming all baselines, the multi-pass flow blending design substantially reduces synthesis efficiency (see §5.7), motivating the training of our proposed EmoTra-TTS. EmoTra-TTS (g) synthesizes the utterance in a single decoding pass with continuous frame-level VAD interpolation, enabling smooth rather than abrupt transitions, yielding the best MOS-Tra (3.63) with competitive results on other metrics. Among systems with explicit emotion rendering (Categories d, e), EmoTra-TTS achieves the highest MOS-Qua, corroborated by an objective metric (Appendix D) showing lower F0 jerk at transitions. 5.3 Ablation: Dual-Stage SFT We compare three configurations in Table 2: baseline CosyVoice2 (a), LLM-only SFT (“w/o flow SFT” in group g), and full EmoTra-TTS (g). LLM-only SFT improves emotion performance (MOS-Emo →2.702.53\!→\!2.70, MOS-Tra →2.582.32\!→\!2.58) while preserving speech quality, confirming that temporal VAD tokens provide meaningful prosodic planning signals. Adding flow SFT yields a further substantial gain (MOS-Emo →3.522.70\!→\!3.52, MOS-Tra →3.632.58\!→\!3.63), demonstrating complementary contributions: the LLM plans emotion-appropriate prosody, while the flow stage renders acoustic realization through frame-level VAD modulation. 5.4 Ablation: Fixed Scale ϵε Table 3: Ablation on fixed scale ϵε. The effective emotion norm is ϵ⋅dmelε· d_mel; the speaker embedding norm is ‖2≈0.622\| e\|_2≈ 0.622. ϵε Eff. norm Quality Evaluation Emotion Evaluation MOS-Qua SIM WER MOS-Emo MOS-Tra 0.04 0.358 3.60± 0.12 0.706 1.69 3.46± 0.15 3.31± 0.16 0.07 0.626 3.54± 0.13 0.701 1.77 3.52± 0.15 3.63± 0.17 0.10 0.894 2.59± 0.18 0.491 5.30 2.34± 0.17 1.85± 0.17 The fixed scale ϵε controls the emotion-to-speaker norm ratio r=∥emo∥2/∥2r= e_emo _2\,/\, e _2 (∥2≈0.622 e _2≈ 0.622), where r<1r<1 means speaker identity dominates, r>1r>1 means emotion dominates, and r≈1r≈ 1 is parity. The results reveal a sensitivity boundary: ϵ=0.04ε=0.04 (r=0.58r=0.58) preserves speech quality (MOS-Qua 3.60) but attenuates emotion; ϵ=0.10ε=0.10 (r=1.44r=1.44) triggers catastrophic degradation as emotion injection overwhelms content and speaker identity. The optimal ϵ=0.07ε=0.07 (r≈1.0r≈ 1.0) achieves the best transition quality (MOS-Tra 3.63), confirming that parity between emotion and speaker norms defines the safe operating regime. At the deployed checkpoint (ϵ=0.07ε\!=\!0.07), the measured post-training injection norm is ‖emo(t)‖2≈0.667\| e_emo(t)\|_2≈ 0.667, about 1.07×1.07× the speaker-embedding norm (≈0.622≈ 0.622); the 0.6260.626 in Table 3 is the nominal ϵ⋅dmelε· d_mel. That the measured norm stays close to the speaker norm confirms that the trainable LayerNorm affine parameters (γ,βγ,β; 160 in total, identity-initialized) do not inflate the injection magnitude, keeping it far below the failure regime, where an input-dependent MLP inflates the norm to ≈2×≈ 2× (MLP-Add) or 33–5×5× (FiLM) and collapses content (Appendix E). 5.5 Pairwise Preference Evaluation Figure 3: Paired preference of EmoTra-TTS over four baselines in Categories (d) and (e) of Table 2. Beyond the per-dimension MOS above, we conduct a paired preference test for an overall A/B comparison. Figure 3 shows EmoTra-TTS is preferred over all four baselines with win rates of 67.5%–79.5%, and every 95% CI lower bound exceeds 60%, well above the 50% chance level. Inter-rater agreement of 63.5%–71.6% supports these preferences. 5.6 Comparison with Commercial Systems As a supplementary study added during the revision period, we additionally compare EmoTra-TTS against two closed-source commercial systems spanning the two widely adopted commercial control paradigms. This study uses a partially overlapping but not identical rater pool and set of pairs; it therefore constitutes a valid EmoTra-vs.-commercial comparison but is not directly comparable to the paired results in §5.5 and Figure 3. The full protocol is provided in Appendix B.3. ElevenLabs v3 (latest), from one of the most widely used commercial TTS providers, controls emotion through discrete audio tags rather than natural-language prompts. GPT-4o mini TTS, from OpenAI’s GPT-4o audio family, is controlled via a natural-language instruction prompt; as it is the same family used to synthesize our source corpus EmoVoice-DB Yang et al. (2025b), the comparison is especially stringent. The two systems thus span the natural-language instruction setting (GPT-4o mini TTS) and the discrete-tag setting (ElevenLabs v3) available in commercial products; configuration and prompting details are given in Appendix B.3. Table 4: Comparison with commercial instruction-following systems. Values are the percentage of pairs in which EmoTra-TTS is preferred (win rate), with 95% CIs. “Emo. & Tra.” evaluates emotion rendering and transition naturalness; “Overall” is overall preference. “Agr.” denotes inter-rater agreement. EmoTra-TTS vs. Emo. & Tra. Overall Win 95% CI Agr. Win 95% CI Agr. GPT-4o mini TTS 74.5% [69.5, 78.9] 67.7% 72.0% [66.9, 76.6] 63.8% ElevenLabs v3 70.5% [65.2, 75.3] 63.5% 64.4% [59.0, 69.5] 60.7% As shown in Table 4, EmoTra-TTS is preferred over both commercial systems in overall preference (72.0% and 64.4%), with larger margins on emotion accuracy and transition naturalness (74.5% and 70.5%); every 95% CI lower bound exceeds the 50% chance level. 5.7 Inference Efficiency Table 5: Inference efficiency comparison. RTF: real-time factor. Mean ± std reported. Mem.: GPU memory. System RTF↓ Latency (s)↓ Params Δ Mem. CosyVoice2 0.270±.022 1.75±.39 639M – 3.04 GB Synth. Data 0.397±.045 2.92±.47 639M +0.00% 5.09 GB EmoTra-TTS 0.244±.010 1.74±.42 642M +0.43% 3.05 GB EmoTra-TTS’s latency is close to the baseline, indicating negligible overhead from the additional 0.43% parameters. The lower RTF results from longer generated audio rather than faster synthesis. The synthetic data pipeline requires 4×4× flow inference and is intended for offline generation. See Appendix B.4 for test set details. 6 Conclusion Emotion dynamics research has long established that affect unfolds continuously over time, yet explicit, controllable modeling of the temporal trajectory of emotion within a single utterance is still only partially addressed by existing systems. EmoTra-TTS bridges this gap through multi-pass flow blending for synthetic transition data, dual-stage VAD conditioning for complementary prosodic and acoustic control, and direction–magnitude decoupled injection that constrains the injection magnitude to alleviate the content-expressiveness trade-off. The system adds only +0.43%+0.43\% parameters with no latency overhead and achieves the best perceptual scores on emotion transition smoothness among all evaluated systems, with 30%–87% relative improvement over multiple SOTA TTS baselines and commercial systems, and 64.4%–79.5% win rates in pairwise preference tests. We hope this work offers a useful step toward modeling emotion as a temporal process rather than a static attribute, and encourages further exploration of affective speech synthesis. Limitations Our work has several limitations: First, the current experiments are conducted on EmoVoice-DB, which provides limited speaker diversity and language coverage. While we validate EmoTra-TTS on multiple speakers from EmoVoice-DB, a separate model is trained per voice, and zero-shot generalization to arbitrary unseen speakers is not yet demonstrated. This limitation, however, mainly concerns speaker and linguistic generalization rather than the proposed intra-utterance emotion transition mechanism itself. Extending the framework to larger multilingual multi-speaker emotional corpora and incorporating prompt-based speaker conditioning remain important directions for future work. Second, emotion control stability can be inconsistent for subtle VAD differences, likely improvable with more diverse training data. Third, the current piecewise linear VAD interpolation provides only a first-order approximation of natural emotion dynamics Davidson (1998). More complex nonlinear trajectories may better capture realistic affective evolution within an utterance. Fourth, a formal characterization of conditioning tolerance boundaries in frozen generative models remains an open research problem. Ethical Statement Human annotation and fair compensation. All evaluators involved in the blind listening test were either formally employed researchers or graduate students supported by institutional scholarships. The blind listening test work was compensated in accordance with local labor regulations and institutional guidelines, consistent with ACL requirements regarding fair treatment and remuneration of human participants. No personally identifiable data were collected from evaluators, and the task involved only subjective quality ratings of AI-synthesized speech; our institution classifies this as minimal-risk research exempt from formal ethics board review. Data privacy and consent. All training data are synthesized from EmoVoice-DB Yang et al. (2025b), a publicly available emotional speech corpus released for research purposes. EmoVoice-DB is itself entirely AI-generated and contains no human recordings. The synthetic emotion transition data are generated entirely from this public dataset via our multi-pass flow blending pipeline; no private, user-uploaded, or personally identifiable data are used at any stage. The released data and model checkpoints do not contain any personally identifiable information or sensitive user data. Licensing and responsible use. The complete code, including both the synthetic data pipeline and model training pipelines, will be released under an open-source license upon acceptance, restricted to non-commercial academic research. We acknowledge that controllable emotional speech synthesis carries potential risks of misuse, including generating deceptive or manipulative audio content. We emphasize that EmoTra-TTS is intended as a research contribution to advance affectively dynamic speech synthesis, and we encourage responsible use with appropriate human oversight in any downstream application. Usage of AI assistants. AI language models were used in two capacities: (1) generating test set sentences with emotion transition annotations (Appendix B.5), and (2) language polishing during paper writing. All experimental design, model development, analysis, and scientific conclusions were made by the authors. References Alain and Bengio (2016) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. Cited by: §E.2. Anastassiou et al. (2024) P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, et al. Seed-tts: a family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430. Cited by: §1, §2.3. Banse and Scherer (1996) R. Banse and K. R. Scherer Acoustic profiles in vocal emotion expression.. Journal of personality and social psychology 70 (3), p. 614. Cited by: Appendix C, §2.1, §3.1. Busso et al. (2008) C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan IEMOCAP: interactive emotional dyadic motion capture database. Language resources and evaluation 42 (4), p. 335–359. Cited by: §1, §2.2, §3.2.1. Chen et al. (2024) H. Chen, R. Chen, and J. Hirschberg EmoKnob: enhance voice cloning with fine-grained emotion control. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 8170–8180. Cited by: §1. Chen et al. (2026) H. Chen, J. Hu, L. Xue, Q. Zhan, W. Li, G. Ma, H. Xie, D. Guo, L. Ma, Y. Jiang, et al. MINT-bench: a comprehensive multilingual benchmark for instruction-following text-to-speech. arXiv preprint arXiv:2604.17958. Cited by: §2.1. Chen et al. (2025a) J. Chen, J. Jiang, Y. Min, Z. Dong, S. Wang, W. X. Zhao, and J. Wen Sticker-TTS: learn to utilize historical experience with a sticker-driven test-time scaling framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 12328–12338. Cited by: §2.3. Chen et al. (2021) N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan WaveGrad: estimating gradients for waveform generation. In International Conference on Learning Representations (ICLR), Cited by: §2.4. Chen et al. (2025b) Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen F5-TTS: a fairytaler that fakes fluent and faithful speech with flow matching. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §2.3. Cho et al. (2024) D. Cho, H. Oh, S. Kim, S. Lee, and S. Lee EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech. In Interspeech 2024, p. 1810–1814. Cited by: §1, §2.1. Cowen and Keltner (2017) A. S. Cowen and D. Keltner Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences 114 (38), p. E7900–E7909. Cited by: §1, §3.1. Cui et al. (2025a) J. Cui, Z. Yang, N. Li, J. Tian, X. Ma, Y. Zhang, G. Chen, R. Yang, Y. Cheng, Y. Zhou, et al. Glm-tts technical report. arXiv preprint arXiv:2512.14291. Cited by: §2.3. Cui et al. (2025b) W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, S. Y. Guo, and I. King Recent advances in speech language models: a survey. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), p. 13943–13970. Cited by: §2.1. Davidson (1998) R. J. Davidson Affective style and affective disorders: perspectives from affective neuroscience. Cognition & emotion 12 (3), p. 307–330. Cited by: §1, §2.1, §3.1, §3, Limitations. Du et al. (2025) Z. Du, C. Gao, Y. Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, C. Ni, X. Shi, et al. Cosyvoice 3: towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589. Cited by: §1. Du et al. (2024) Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, et al. Cosyvoice 2: scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. Cited by: Table 7, §1, §2.3, §3.4.1, §4, §5.2, Table 2, Table 2, Table 2. Fontaine et al. (2007) J. R. Fontaine, K. R. Scherer, E. B. Roesch, and P. C. Ellsworth The world of emotions is not two-dimensional. Psychological science 18 (12), p. 1050–1057. Cited by: §4. Ho and Salimans (2021) J. Ho and T. Salimans Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, Cited by: §3.4.5. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: §E.3, §2.4. Hu et al. (2026) H. Hu, X. Zhu, T. He, D. Guo, B. Zhang, X. Wang, Z. Guo, Z. Jiang, H. Hao, Z. Guo, et al. Qwen3-tts technical report. arXiv preprint arXiv:2601.15621. Cited by: Table 7, §1, §5.2, Table 2, Table 2, Table 2. Hu (2024) L. Hu Animate anyone: consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 8153–8163. Cited by: §E.3. Huang et al. (2025) K. Huang, Q. Tu, L. Fan, C. Yang, D. Zhang, S. Li, Z. Fei, Q. Cheng, and X. Qiu Instructttseval: benchmarking complex natural-language instruction following in text-to-speech systems. arXiv preprint arXiv:2506.16381. Cited by: §2.1. Hussain et al. (2025) S. S. Hussain, P. Neekhara, X. Yang, E. Casanova, S. Ghosh, R. Fejgin, M. T. Desta, R. Valle, and J. Li Koel-TTS: enhancing LLM based speech generation with preference alignment and classifier free guidance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 21219–21234. Cited by: §1. Hwang et al. (2026) C. Hwang, S. Kim, Y. K. Tan, T. Liu, S. Paik, D. Kim, M. Soumik, K. M. M. Aung, and J. H. Seo Scores know bobs voice: speaker impersonation attack. arXiv preprint arXiv:2603.02781. Cited by: §2.2. Inoue et al. (2025) S. Inoue, K. Zhou, S. Wang, and H. Li Hierarchical control of emotion rendering in speech synthesis. IEEE Transactions on Affective Computing 16 (4), p. 3316–3328. Cited by: §2.1. Ju et al. (2024) Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang, Z. Wu, T. Qin, X. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao NaturalSpeech 3: zero-shot speech synthesis with factorized codec and diffusion models. In Forty-first International Conference on Machine Learning (ICML), Cited by: §2.3. Kang et al. (2023) M. Kang, D. Min, and S. J. Hwang Grad-stylespeech: any-speaker adaptive text-to-speech synthesis with diffusion models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1–5. Cited by: §3.4.1. Kong et al. (2021) Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro DiffWave: a versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR), Cited by: §2.4. Kuppens and Verduyn (2017) P. Kuppens and P. Verduyn Emotion dynamics. Current opinion in psychology 17, p. 22–26. Cited by: §1, §2.1, §3.1, §3. Lei et al. (2022) Y. Lei, S. Yang, X. Wang, and L. Xie MsEmoTTS: multi-scale emotion transfer, prediction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (), p. 853–864. Cited by: §1. Leng et al. (2024) Y. Leng, Z. Guo, K. Shen, Z. Ju, X. Tan, E. Liu, Y. Liu, D. Yang, L. Zhang, K. Song, L. He, X. Li, sheng zhao, T. Qin, and J. Bian PromptTTS 2: describing and generating voices with text prompt. In The Twelfth International Conference on Learning Representations (ICLR), Cited by: §1. Li et al. (2026) H. Li, C. Jin, C. Li, W. Guan, Z. Huang, and X. Chen ReStyle-tts: relative and continuous style control for zero-shot speech synthesis. arXiv preprint arXiv:2601.03632. Cited by: §1. Li et al. (2025) H. Li, Y. Liu, Y. Sun, H. Shi, L. Qu, and T. Li EMORL-tts: reinforcement learning for fine-grained emotion control in llm-based tts. arXiv preprint arXiv:2510.05758. Cited by: §2.1. Li et al. (2023) J. Li, D. Li, S. Savarese, and S. Hoi BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), p. 19730–19742. Cited by: §3.3.1. Li et al. (2024) Y. Li, R. Bhardwaj, A. Mehrish, B. Cheng, and S. Poria HYPERTTS: parameter efficient adaptation in text to speech using hypernetworks. In LREC-COLING, p. 8516–8527. Cited by: §3.4.1. Liang et al. (2026) Q. Liang, Y. Liu, R. Wei, N. Lu, J. Zhao, and Y. Wang Segment-aware conditioning for training-free intra-utterance emotion and duration control in text-to-speech. arXiv preprint arXiv:2601.03170. Cited by: §2.1. Lin et al. (2024) H. Lin, J. Cho, A. Zala, and M. Bansal Ctrl-adapter: an efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arXiv:2404.09967. Cited by: §E.3. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In ICLR, Cited by: §2.3, §3.4.5. Liu et al. (2024a) S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen DoRA: weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning (ICML), Cited by: §2.4, §3.4.4. Liu et al. (2024b) T. Liu, I. Kukanov, Z. Pan, Q. Wang, H. B. Sailor, and K. A. Lee Towards quantifying and reducing language mismatch effects in cross-lingual speech anti-spoofing. In IEEE Spoken Language Technology Workshop (SLT), Vol. , p. 1185–1192. Cited by: §2.2. Liu et al. (2025a) T. Liu, R. Tao, Q. Wang, Y. Jiang, H. B. Sailor, K. Zhang, J. Lin, and H. Li Interpolating speaker identities in embedding space for data expansion. In Asia Pacific Signal and Information Processing Association Annual Summit and Conference, Vol. , p. 589–594. Cited by: §2.2. Liu et al. (2025b) T. Liu, D. Truong, R. Kumar Das, K. Aik Lee, and H. Li Nes2Net: a lightweight nested architecture for foundation model driven speech anti-spoofing. IEEE Transactions on Information Forensics and Security 20 (), p. 12005–12018. Cited by: §2.2. Locatello et al. (2019) F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, p. 4114–4124. Cited by: §E.5, §1, §2.4, §3.4.4. Ma et al. (2024a) Z. Ma, W. Wu, Z. Zheng, Y. Guo, Q. Chen, S. Zhang, and X. Chen Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 11146–11150. Cited by: §2.2. Ma et al. (2024b) Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen Emotion2vec: self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, p. 15747–15760. Cited by: §2.2. Manku et al. (2025) R. R. Manku, Y. Tang, X. Shi, M. Li, and A. Smola EmergentTTS-eval: evaluating TTS models on complex prosodic, expressiveness, and linguistic challenges using model-as-a-judge. In The Conference on Neural Information Processing Systems, Cited by: §2.1. Mehrabian (1996) A. Mehrabian Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Current psychology 14 (4), p. 261–292. Cited by: Appendix C, §1, §2.1, §3.1, §4. Mehta et al. (2024) S. Mehta, R. Tu, J. Beskow, É. Székely, and G. E. Henter Matcha-tts: a fast tts architecture with conditional flow matching. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 11341–11345. Cited by: §2.3. Michel et al. (2026) G. Michel, E. V. Epure, and C. Cerisara Computational narrative understanding for expressive text-to-speech. In Findings of the Association for Computational Linguistics: ACL 2026, p. 6194–6215. Cited by: §1. MOSS-TTS Team (2026) MOSS-TTS Team MOSS-TTS Family: an open-source speech and sound generation model family. Cited by: Table 7, §5.2, Table 2, Table 2, Table 2. Perez et al. (2018) E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville Film: visual reasoning with a general conditioning layer. In Proceedings of AAAI, Cited by: §E.5, §2.4, §3.4.4. Posner et al. (2005) J. Posner, J. A. Russell, and B. S. Peterson The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology 17 (3), p. 715–734. Cited by: §1, §3.1. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. In ICML, p. 28492–28518. Cited by: §4, §4. Russell (1980) J. A. Russell A circumplex model of affect.. Journal of personality and social psychology 39 (6), p. 1161. Cited by: Appendix C, §1, §3.1, §4. Scherer (2003) K. R. Scherer Vocal communication of emotion: a review of research paradigms. Speech communication 40 (1-2), p. 227–256. Cited by: Appendix C, §1, §2.1, §3.1. Skerry-Ryan et al. (2018) R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In ICML, p. 4693–4702. Cited by: §2.1. Song et al. (2026) Z. Song, T. Liu, T. Wang, C. Xu, Y. Guo, and H. Li Diagnose, then refine: a closed-loop TTS system with AudioLLM-guided correction. arXiv preprint. Cited by: §1. Tao et al. (2025) R. Tao, Z. Shi, Y. Jiang, T. Liu, and H. Li Voice conversion augmentation for speaker recognition on defective datasets. In Asia Pacific Signal and Information Processing Association Annual Summit and Conference, Vol. , p. 2529–2534. Cited by: §2.2. Trampe et al. (2015) D. Trampe, J. Quoidbach, and M. Taquet Emotions in everyday life. PloS one 10 (12), p. e0145450. Cited by: §1. Verduyn et al. (2009) P. Verduyn, E. Delvaux, H. Van Coillie, F. Tuerlinckx, and I. Van Mechelen Predicting the duration of emotional experience: two experience sampling studies.. Emotion 9 (1), p. 83. Cited by: §1, §3.1, §3. Wagner et al. (2023) J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller Dawn of the transformer era in speech emotion recognition: closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (9), p. 10745–10759. Cited by: Appendix C, §3.2.4, §3.3.3, §4. Wang et al. (2023) C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111. Cited by: §1, §2.3. Wang et al. (2026a) J. Wang, S. Zhang, P. Jiang, J. Zong, J. Zhang, T. Wang, Y. Lin, Z. Chen, S. Xie, Z. Ma, et al. EmoS: a theory-grounded framework for evaluating and aligning emotional intelligence in spoken language models. arXiv preprint arXiv:2608.09189. Cited by: §2.1. Wang et al. (2026b) S. Wang, S. Tan, S. Liu, H. Jia, G. Huang, J. Bailey, and T. Dang CoCoEmo: composable and controllable human-like emotional tts via activation steering. arXiv preprint arXiv:2602.03420. Cited by: §2.1. Wang et al. (2026c) T. Wang, Z. Ma, Y. Peng, H. Wang, Z. Niu, Z. Huang, Y. Wu, Y. Chao, Y. Jiang, Y. Lu, G. Yang, X. Li, H. Liu, C. Qiang, C. Gong, Y. Yang, T. Liu, J. Wang, N. Hou, M. Ge, F. You, Y. Wei, Z. Sun, H. Haifeng, X. Wang, E. S. Chng, X. Chen, L. Wang, and J. Dang Evaluating the expressive appropriateness of speech in rich contexts. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9088–9106. Cited by: §1. Wang et al. (2025a) T. Wang, H. Wang, M. Ge, C. Gong, C. Qiang, Z. Ma, Z. Huang, G. Yang, X. Wang, E. Chng, X. Chen, L. Wang, and J. Dang Word-level emotional expression control in zero-shot text-to-speech synthesis. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Table 7, §1, §2.1, §5.2, Table 2. Wang et al. (2025b) X. Wang, M. Jiang, Z. Ma, Z. Zhang, S. Liu, L. Li, Z. Liang, Q. Zheng, R. Wang, X. Feng, et al. Spark-tts: an efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710. Cited by: §2.3. Wang et al. (2025c) Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu MaskGCT: zero-shot text-to-speech with masked generative codec transformer. In The Thirteenth International Conference on Learning Representations (ICLR), Cited by: §2.3. Wang et al. (2018) Y. Wang, D. Stanton, Y. Zhang, R. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous Style tokens: unsupervised style modeling, control and transfer in end-to-end speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (ICML), p. 5180–5189. Cited by: §2.1. Wu et al. (2024) H. Wu, X. Wang, S. E. Eskimez, M. Thakker, D. Tompkins, C. Tsai, C. Li, Z. Xiao, S. Zhao, J. Li, and N. Kanda Laugh now cry later: controlling time-varying emotional states of flow-matching-based zero-shot text-to-speech. In IEEE Spoken Language Technology Workshop (SLT), Vol. , p. 690–697. Cited by: §2.1. Xie et al. (2025) T. Xie, Y. Rong, P. Zhang, W. Wang, and L. Liu Towards controllable speech synthesis in the era of large language models: a systematic survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 764–791. Cited by: §2.1. Yang et al. (2025a) D. Yang, Y. Cai, Y. Saito, L. Wang, and H. Saruwatari Shallow flow matching for coarse-to-fine text-to-speech synthesis. arXiv preprint arXiv:2505.12226. Cited by: §2.3. Yang et al. (2025b) G. Yang, C. Yang, Q. Chen, Z. Ma, W. Chen, W. Wang, T. Wang, Y. Yang, Z. Niu, W. Liu, F. Yu, Z. Du, Z. Gao, S. Zhang, and X. Chen EmoVoice: llm-based emotional text-to-speech model with freestyle text prompting. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 10748–10757. Cited by: §B.1, §4, §5.2, §5.6, Table 2, Ethical Statement. Yang et al. (2025c) G. Yang, F. Yu, Z. Ma, Z. Du, Z. Gao, S. Zhang, and X. Chen Enhancing low-resource asr through versatile tts: bridging the data gap. In IEEE International Conference on Acoustics, Speech and Signal Processing, Vol. , p. 1–5. Cited by: §2.2. Ye et al. (2025) Z. Ye, X. Zhu, C. Chan, X. Wang, X. Tan, J. Lei, Y. Peng, H. Liu, Y. Jin, Z. Dai, et al. Llasa: scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04128. Cited by: §2.3. Zhang et al. (2023) L. Zhang, A. Rao, and M. Agrawala Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3836–3847. Cited by: §E.5, §2.4, §3.4.1, §3.4.4. Zhao et al. (2023) J. Zhao, G. Haffari, and E. Shareghi Generating synthetic speech from SpokenVocab for speech translation. In Findings of the Association for Computational Linguistics: EACL 2023, p. 1975–1981. Cited by: §2.2. Zhou et al. (2022) K. Zhou, B. Sisman, R. Liu, and H. Li Emotional voice conversion: theory, databases and ESD. Speech Communication 137, p. 1–18. Cited by: §1, §2.2, §3.2.1. Zhou et al. (2024) K. Zhou, Y. Zhang, S. Zhao, H. Wang, Z. Pan, D. Ng, C. Zhang, C. Ni, Y. Ma, T. H. Nguyen, et al. Emotional dimension control in language model-based text-to-speech: spanning a broad spectrum of human emotions. arXiv preprint arXiv:2409.16681. Cited by: §2.1. Zhou et al. (2026) L. Zhou, H. Jiang, J. Li, T. Wang, and H. Li EmoShift: lightweight activation steering for enhanced emotion-aware speech synthesis. arXiv preprint arXiv:2601.22873. Cited by: §2.1. Zhou et al. (2025) S. Zhou, Y. Zhou, Y. He, X. Zhou, J. Wang, W. Deng, and J. Shu Indextts2: a breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. arXiv preprint arXiv:2506.21619. Cited by: §1. Figure 4: Rating criteria for ‘Naturalness & Audio Quality’ (MOS-Qua), ‘Emotion Accuracy’ (MOS-Emo), and ‘Emotion Transition Naturalness’ (MOS-Tra). Figure 5: The blind test evaluation interface. Evaluators listen to each sample and rate ‘MOS Naturalness’ (MOS-Qua), ‘Emotion Accuracy’ (MOS-Emo), and ‘Emotion Transition Naturalness’ (MOS-Tra) on 1–5 Likert scales. System identities are hidden, and the system order is randomly shuffled for each sample. Appendix A Training Details Synthetic data generation: Crossfade steepness κ=12κ=12; segment ratios rinit/rtra/rres=60%/15%/25%r_init/r_tra/r_res=60\%/15\%/25\%; crossfade duration min⁡(0.4s,0.1×Tdur) (0.4s,0.1× T_dur). ASR quality filter: Whisper large-v3 with CER threshold τCER=0.10 _CER=0.10. Stage 1 training (LLM): Adam optimizer, learning rate =1.0×10−5=1.0× 10^-5, gradient clipping at 5.05.0. Ntra=3N_tra=3 intermediate VAD tokens (Ntra+2=5N_tra+2=5 total), dh=256d_h=256, dLLM=896d_LLM=896, dSER=1024d_SER=1024, λ=0.1λ=0.1. Stage 2 training (Flow): All decoder parameters frozen. Adam optimizer, gradient accumulation factor of 2. Segment ratios: rinit=0.4r_init=0.4, rtra=0.3r_tra=0.3, rres=0.3r_res=0.3. Fixed emotion scale ϵ=0.07ε=0.07, yielding effective norm ≈ϵ⋅dmel=0.07×80≈0.626≈ε· d_mel=0.07× 80≈ 0.626, matched to the observed speaker embedding norm (‖2≈0.622\| e\|_2≈ 0.622). ψdown _down: 1024→256→ReLU→801024→ 256 → 80 (last layer zero-initialized). LayerNorm: dmel=80d_mel=80. Evaluation cost. The two commercial systems used for the paired comparison in §5.6 incurred a total cost of USD 11 (ElevenLabs subscription: USD 6; GPT-4o mini TTS API: USD 5). Appendix B Evaluation Protocol We conduct blind listening evaluations using a professional paid blind-test team with established quality control procedures. In total, 21 raters participated across all evaluations, including researchers and non-technical listeners. B.1 MOS-based Evaluation Protocol For the MOS-based evaluation, 10 evaluators (3 researchers and 7 non-technical listeners) participated to capture both expert and general-listener perspectives. Each system contains 200 utterances, and each evaluator rated about 25 randomly sampled utterances per system, resulting in approximately 250 ratings per system and about 25% repeated coverage. Figure 6: Screenshot of the paired preference evaluation interface. System identities are hidden, and the system order is randomly shuffled for each sample. These test sample texts and target emotions are generated by ChatGPT-5.2 55 5 https://help.openai.com/en/articles/6825453-chatgpt-release-notes using a structured prompt (Appendix B.5) that enforces event-driven narratives, balanced emotion-pair coverage, and VAD annotations within tight per-category ranges. The resulting test set is independent of the EmoVoice-DB Yang et al. (2025b) training set. Notably, we prioritize the quality of the blind test over its quantity. With a carefully selected panel, we emphasize careful, attentive ratings rather than maximizing raw throughput, while still achieving a high overall completion rate of ~94%. All samples are randomly shuffled and system identities are hidden from evaluators. Three metrics are collected on a 1–5 Likert scale: • MOS-Qua: speech naturalness and quality, without considering emotional correctness. • MOS-Emo: adequacy of emotion rendering relative to the target specification, evaluating whether both the initial and resultant emotions are correctly expressed. • MOS-Tra: smoothness and naturalness of intra-utterance emotion transition, assessing whether the shift between emotions sounds gradual and natural rather than abrupt or discontinuous. The detailed rating criteria for each metric are shown in Figure 4, and an example of the blind evaluation interface is provided in Figure 5. Figure 7: Rating criteria for the paired evaluation. B.2 Paired Preference Evaluation Protocol For the paired preference evaluation in §5.5, we retain 4 raters from the original pool in Appendix B.1 and include 7 additional raters, resulting in 11 raters for this study. For each system pair, 40 utterances were randomly sampled from the 200 utterances mentioned above, forming four paired comparison groups (160 pairs in total). Each rater completed about 85 comparisons on average, yielding about 6-fold repeated coverage per pair. The detailed rating criteria for each metric are shown in Figure 7, and an example of the blind evaluation interface is provided in Figure 6. B.3 Comparison with Commercial Systems Figure 8: ElevenLabs v3 inference interface used in our comparison, showing the manually inserted discrete audio tags (e.g., [sad], [angry]) and the “Stability” control set to “Creative”. Figure 9: Rating criteria for the comparison with commercial systems. Figure 10: Screenshot of the paired preference evaluation interface. System identities are hidden, and the system order is randomly shuffled for each sample. This appendix details the two commercial systems compared in §5.6 and the evaluation protocol. GPT-4o mini TTS (OpenAI’s GPT-4o audio family) is controlled via a natural-language instruction prompt. ElevenLabs v3 does not support natural-language prompts for emotion; instead, the user inserts discrete audio tags (e.g., [happy], [sad]) at chosen positions66 6 Prompting guide: https://elevenlabs.io/docs/overview/capabilities/text-to-speech/best-practices#prompting-eleven-v3-alpha77 7 (updated Jul 5, 2026): https://elevenlabs.io/blog/eleven-v3-situational-awareness, which requires an additional human or LLM step to place the tags and specifies discrete switches rather than continuous transitions. To present v3 in its best light, we placed the tags manually and set the “Stability” control to “Creative” (Figure 8), the most expressive setting per the official guidance.88 8 https://elevenlabs.io/app/speech-synthesis/text-to-speech We ran a paired evaluation on 40 sentences per system (80 pairs in total) with 12 raters (4 newly recruited). Each rater evaluated about 53 pairs, and each pair was rated about 8 times. To avoid conflating synthesis quality with transition quality, we collected dimension-specific preferences: (i) emotion accuracy and transition smoothness, and (i) overall preference as shown in Figure 9 and Figure 10. B.4 Inference Efficiency Evaluation Setup This appendix summarizes the test set used for the inference efficiency evaluation. The evaluation set contains 100 sentences, with 5 additional warm-up sentences excluded from the statistics. The average sentence length is 13.9 words, with a standard deviation of 3.4 words. Sentence lengths range from 9 to 21 words, with a median of 13 words. In total, the test set contains 1,391 words. B.5 Test Set Generation Prompt The following prompt was used with ChatGPT-5.2 to generate 200 test sentences with emotion transition annotations. Prompt for Test Set Generation Task: Generate English sentences that describe a transition from one emotional state to another within a short narrative. Each sample must include: the sentence, the starting emotion, the ending emotion, and the VAD (Valence, Arousal, Dominance) values for both emotions. EMOTION CATEGORIES angry, happy, sad, surprised, fearful, neutral Transitions may occur between any pair of different emotions (30 possible pairs). SENTENCE REQUIREMENTS Each sentence should describe a scenario or event that naturally causes an emotional shift. The narrative must clearly reflect the transition so that a speaker reading it aloud could express the emotional shift through prosody and tone. AVOID sentences that just name emotions. Instead, describe an event or realization that triggers the shift. BAD: ‘I was sad then I became happy’ GOOD: ‘The long empty street made him feel alone until distant music drifted through the night’ Sentence length: 12--30 words. Natural pauses allowed (commas, conjunctions). May include connectors such as: but, until, then, suddenly, moments later, after that, and then, when. Sentences should include diverse narrative contexts such as: daily life, workplace, social interaction, unexpected discoveries, environmental changes, memories, misunderstandings, surprising events, physical sensations, weather, animals, travel, etc. Avoid repeating similar structures, scenarios, or phrasings. VAD ANNOTATION Each emotion must include VAD values: (Valence, Arousal, Dominance). All values should be formatted to two decimal places. Target VAD Ranges: angry: V (0.18--0.26) A (0.82--0.90) D (0.78--0.89) happy: V (0.80--0.92) A (0.76--0.88) D (0.70--0.88) sad: V (0.18--0.26) A (0.18--0.28) D (0.18--0.26) surprised: V (0.60--0.80) A (0.80--0.89) D (0.50--0.68) fearful: V (0.18--0.26) A (0.78--0.88) D (0.18--0.28) Important: Values should reflect emotionally saturated states for non-neutral categories. Vary the values within the range; do not always use the same numbers. The VAD distance between starting and ending emotion must be clearly large enough to be perceptible. Avoid transitions where both emotions have very similar VAD values. DISTRIBUTION CONSTRAINT Cover at least 25 unique emotion pairs out of 30. No single pair more than 15 times in 200 samples. Roughly balanced distribution. FEW-SHOT EXAMPLES The loud crack startled him and then a creeping dread filled his chest | surprised | fearful | (0.68,0.86,0.60) | (0.20,0.84,0.22) The cheerful laughter at the party was cut short when she saw the broken vase on the floor | happy | angry | (0.88,0.82,0.78) | (0.20,0.88,0.85) A sudden gust of wind knocked the tent down and he scrambled in panic, but then burst into laughter | fearful | happy | (0.22,0.82,0.20) | (0.85,0.80,0.75) She cursed the delayed train and then sat down feeling deeply miserable | angry | sad | (0.24,0.87,0.83) | (0.19,0.23,0.22) OUTPUT FORMAT Each line: Sentence | emotion1 | emotion2 | (V,A,D) | (V,A,D) Use | as separator. No numbering, no headers, no blank lines. TARGET: Generate 200 samples. Appendix C Emotion-Specific VAD Ranges Table 6: Emotion-specific VAD acceptance ranges for source corpus filtering. All values are on a [0,1][0,1] scale. Ranges are intentionally broad to accommodate natural within-category variability. Emotion Valence Arousal Dominance Angry [0.00, 0.35] [0.75, 1.00] [0.70, 1.00] Happy [0.65, 1.00] [0.50, 1.00] [0.55, 1.00] Sad [0.00, 0.35] [0.00, 0.50] [0.00, 0.50] Surprised [0.35, 0.80] [0.60, 1.00] [0.35, 0.75] Fearful [0.00, 0.45] [0.40, 0.90] [0.10, 0.55] Disgusted [0.00, 0.35] [0.30, 0.70] [0.30, 0.65] Neutral [0.40, 0.60] [0.40, 0.60] [0.40, 0.60] Table 6 lists the VAD acceptance ranges used for cross-validated filtering of the source corpus (§4). These ranges are intentionally set broader than prototypical VAD centroids Russell (1980); Mehrabian (1996) for two reasons: (1) natural emotional speech exhibits substantial within-category variability Scherer (2003); Banse and Scherer (1996), and (2) the VAD predictor Wagner et al. (2023) has estimation uncertainty, so tight boundaries would conflate predictor error with genuine mismatch. The filtering retains samples only if their categorical label and predicted VAD are mutually consistent, removing mislabeled or acoustically ambiguous utterances. Appendix D Objective Smoothness Evaluation Table 7: Prosodic Jerk Ratio for the proposed EmoTra-TTS and baselines in Categories (d) and (e) of Table 2. All differences between EmoTra-TTS and the baselines are statistically significant (p<0.001p<0.001). System JR-F0 ↓ EmoTra-TTS (Ours) 0.025 ± 0.009 Synthetic data (Ours) 0.030 ± 0.014 Qwen3-TTS Hu et al. (2026) 0.036 ± 0.016 WeSCon Wang et al. (2025a) 0.038 ± 0.017 MOSS-TTS MOSS-TTS Team (2026) 0.038 ± 0.017 CosyVoice2 Du et al. (2024) 0.039 ± 0.018 To complement the perceptual evaluation with an objective measure, we introduce the Prosodic Jerk Ratio (JR-F0), a signal-level metric computed directly from f0f_0 via Praat. Given a voiced f0f_0 contour, we measure the second-order difference Δ2f0[n]=f0[n+1]−2f0[n]+f0[n−1] ^2f_0[n]=f_0[n+1]-2f_0[n]+f_0[n-1] and define JR-F0=#⁡n:|Δ2f0[n]|>τ#n:f0[n]voiced,JR -F0\;=\; \#\\,n:| ^2f_0[n]|>τ\,\\#\\,n:f_0[n]~voiced\,\, (15) i.e. the fraction of voiced frames with abrupt pitch discontinuities, where τ is an adaptive per-utterance threshold. Lower JR-F0 indicates smoother prosodic evolution. Table 8: Full injection architecture comparison across all seven variants. SIM: speaker similarity. WER: word error rate (%). We decompose MOS into MOS-Qua, MOS-Emo, and MOS-Tra for a more fine-grained and independent evaluation of speech naturalness, emotion rendering adequacy, and intra-utterance emotion transition smoothness, respectively. †Finetune is excluded from blind test evaluation due to severely degraded content (decoder drift renders most outputs unintelligible), precluding meaningful perceptual assessment. Variant Injection Δ Decoder Quality Evaluation Emotion Evaluation Key finding MOS-Qua↑ SIM↑ WER↓ MOS-Emo↑ MOS-Tra↑ Finetune† Linear 72K Trainable — — — — — Decoder drift Linear Linear, additive 82K Frozen 3.60± 0.12 0.707 1.65 3.39± 0.15 3.33± 0.16 Emotion Capacity limit MLP+LoRA MLP+LoRA 2.9M LoRA 1.03± 0.05 0.627 107.15 1.69± 0.13 1.42± 0.11 Content collapse MLP-Add MLP, additive 280K Frozen 2.10± 0.17 0.595 20.09 2.49± 0.17 2.17± 0.17 Magnitude mismatch FiLM FiLM modulation 560K Frozen 1.96± 0.16 0.400 53.35 2.44± 0.17 1.82± 0.16 MLP compensation Norm Clamp Norm Clamp 280K Frozen 3.29± 0.15 0.692 3.89 3.30± 0.15 3.02± 0.16 Gradient suppression Dir–Mag (ours) LN+fixed ϵε 280K Frozen 3.54± 0.13 0.701 1.77 3.52± 0.15 3.63± 0.17 Dir–mag decoupled As shown in Table 7, EmoTra-TTS attains the lowest JR-F0, corroborating the perceptual transition-naturalness results with a fully signal-level measure. Notably, EmoTra-TTS is even smoother than its own synthetic training data (0.0250.025 vs. 0.0300.030): the synthetic corpus retains residual discontinuities from four-segment blending (§3.2.2), whereas frame-level interpolated VAD conditioning yields globally continuous prosody. Appendix E Detailed Failure Analysis This appendix documents the iterative development across all seven injection architectures (Table 8), each failure revealing a distinct phenomenon in auxiliary conditioning of frozen generative models. E.1 Decoder Drift (Finetune) A linear projection (→80896\!→\!80; ∼ 72K parameters) injects emotion through the conditioning channel c with the decoder jointly trained. Strong emotion is achieved, but the decoder’s ∼ 639M parameters absorb the adaptation by shifting pretrained weights, degrading content fidelity. This establishes the decoder freezing principle for all subsequent variants. E.2 Emotion Capacity Limit (Linear) A single linear layer (→801024\!→\!80; 82K parameters) with frozen decoder and additive injection preserves content (MOS-Qua 3.60) but produces only moderate emotion (MOS-Emo 3.39, MOS-Tra 3.33). The linear mapping lacks capacity for the nonlinear dSERd_SER-to-dmeld_mel transformation Alain and Bengio (2016), establishing a lower bound on model complexity for expressive conditioning. E.3 Content Collapse (MLP+LoRA) A two-layer MLP (→801024\!→\!256\!→\!80; ∼ 280K) with LoRA Hu et al. (2022) rank-8 adapters on all 336 decoder layers (∼ 2.9M total) yields emotion but zero intelligible content (MOS-Qua 1.03, WER 107.15%, SIM 0.627). LoRA’s global perturbation shifts the flow decoder’s content–style equilibrium, catastrophically amplifying style at the expense of content, paralleling findings that adapter placement critically affects multi-conditional generation Hu (2024); Lin et al. (2024). This establishes that decoder weights should not be modified for single-axis adaptation, even with parameter-efficient methods Hu et al. (2022). E.4 Magnitude Mismatch (MLP-Add) The same MLP without LoRA (∼ 280K parameters, decoder frozen) achieves strong emotion but content failure (MOS-Qua 2.10, WER 20.09%) with a diagnostic bimodal pattern: emotion is reflected in both successful and garbled samples, indicating a distributional out-of-range problem. Norm measurements reveal the cause: ‖emo(t)‖2≈1.248≈2×‖2\| e_emo(t)\|_2≈ 1.248≈ 2×\| e\|_2, pushing the combined embedding 3×3× beyond the decoder’s pretrained operating range. We isolate magnitude as the causal factor via inference-time manual scaling λ on frozen MLP weights. Content failure is monotonically related to the emotion-to-speaker norm ratio: at λ=0.5λ\!=\!0.5 (ratio ≈1.0≈ 1.0), content is fully preserved; at λ=1.0λ\!=\!1.0 (ratio ≈2.0≈ 2.0), only a small portion of samples remain intelligible. Emotion direction is correct even in garbled samples, confirming that the problem is exclusively one of magnitude. This directly validates our Dir–Mag design and generalizes: for any additive conditioning of frozen models, the conditioning-to-existing-signal norm ratio must stay within the pretrained operating range. E.5 MLP Compensation (FiLM) FiLM Perez et al. (2018) modulation (^(t)=⁡(t)⊙+⁡(t) e(t)= γ(t) e+ β(t); ∼ 560K parameters, identity-initialized Zhang et al. (2023), decoder frozen) is designed to provide learnable per-dimension scale and shift. However, FiLM fails even more severely than MLP-Add (MOS-Qua 1.96, WER 53.35%, SIM 0.400): the MLP compensates by inflating output magnitude 33–5×5× during joint optimization. This demonstrates that soft constraints on entangled factors are insufficient Locatello et al. (2019). The flow loss, averaging over all frames, provides no gradient penalizing magnitude growth, and the MLP and FiLM parameters co-adapt to circumvent the constraint. E.6 Gradient Suppression (Norm Clamp) Hard norm clamping (τ=0.70τ\!=\!0.70, derived from the manual scaling experiment; ∼ 280K parameters, decoder frozen) substantially improves content over MLP-Add (MOS-Qua 3.29, WER 3.89%), confirming magnitude control as the key factor. However, emotion is only moderate (MOS-Emo 3.30) due to gradient suppression: beyond τ, clamping zeros the radial gradient component, and the discontinuous transition between clamped and unclamped regimes causes the MLP to oscillate near the boundary rather than converging smoothly. This demonstrates that even a correct threshold is insufficient without a smooth normalization mechanism. E.7 Success: Direction–Magnitude Decoupled The same MLP followed by LayerNorm(dmel=80d_mel\!=\!80) and fixed ϵ=0.07ε\!=\!0.07 (∼ 280K + 160 parameters, decoder frozen) achieves content fidelity with strong emotion (MOS-Qua 3.54, MOS-Emo 3.52, MOS-Tra 3.63). LayerNorm locks magnitude at dmel≈8.94 d_mel≈ 8.94; the effective norm ϵ⋅dmel≈0.626ε· d_mel≈ 0.626 matches the speaker norm (‖2≈0.622\| e\|_2≈ 0.622). The MLP learns emotion directions freely but cannot inflate magnitude, confirming that structural constraints are necessary for safe auxiliary conditioning. Appendix F Crossfade Curve Selection Table 9: Crossfade curves evaluated for mel-level blending. σκ(t)=1/(1+e−κ⁡(t−0.5)) _κ(t)=1/(1+e^-κ(t-0.5)), normalized to [0,1][0,1]; s(t)=12(1−cos⁡πt)s(t)= 12(1- π t). Act. fr. indicates active frames. Curve wA(t)w_A(t) wB(t)w_B(t) Act. fr. wA2+wB2=1w_A^2+w_B^2=1? Linear 1−t1-t t 200 ✗ Equal-power cosine cos⁡(π2s(t)) \! ( π2\,s(t) ) sin⁡(π2s(t)) \! ( π2\,s(t) ) ∼ 150 ✓ Sigmoid symmetric 1−σ12(t)1- _12(t) σ12(t) _12(t) ∼ 34 ✗ Sigmoid asymmetric 1−σ12(t) 1- _12(t) σ12(t)2 _12(t)^2 ∼ 34 ✗ Ultra-steep sigmoid 1−σ100(t) 1- _100(t) σ100(t)1.5 _100(t)^1.5 ∼ 4 ✗ This appendix supplements the mel-spectrogram-level crossfade in §3.2.2 (Fig. 1) by documenting the curve selection process. The crossfade curve is not a core contribution; we include this analysis for interested readers. The selection was based on informal listening by the authors (∼ 20 samples per curve, not a formal blind test). Within the transition region, blending follows mel(t)=wA(t)melA(t)+wB(t)melB(t)mel(t)=w_A(t)\,mel_A(t)+w_B(t)\,mel_B(t) on normalized t∈[0,1]t\!∈\![0,1]. Since the quality of synthetic training data directly depends on the blending curve, we evaluated five candidates (Table 9) to find the best trade-off between transition smoothness and emotion purity for data generation. Linear blending made the splice point audible in a subset of samples. Equal-power cosine removed the splice artifact but noticeably degraded quality in the transition region. Ultra-steep sigmoid (κ=100κ=100) compressed blending to ∼ 4 frames, yielding no artifacts but sounding perceptually identical to a hard switch and losing all sense of gradual transition. Both sigmoid variants (κ=12κ=12) concentrate blending in ∼ 34 central frames while keeping edges binary and sound largely identical in practice. The asymmetric variant exhibited mild spectral artifacts in only a few samples; the symmetric version avoided these, so we adopted it for consistency. We adopted the sigmoid symmetric with κ=12κ=12 (Eq. (4)). The key finding is that κ governs the smoothness–purity trade-off: too gradual (linear, cosine) sacrifices emotion contrast; too steep (κ≥100κ≥100) collapses to a hard switch.

Instant research discovery