Paper deep dive
A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech
Jia-Hong Huang, Seulgi Kim, Yi Chieh Liu, Yixian Shen, Hongyi Zhu, Prayag Tiwari, Stevan Rudinac, Evangelos Kanoulas
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/10/2026, 3:35:16 AM
Summary
The paper introduces an automatic framework for detecting 'speaker drift'—a gradual, subtle shift in speaker identity within a single synthesized utterance. The method uses pairwise cosine similarity scores of speaker embeddings from overlapping speech segments as input for Large Language Models (LLMs) to perform binary classification. The authors provide theoretical guarantees for this approach and validate it using a new synthetic benchmark dataset, demonstrating that the embedding-to-reasoning pipeline outperforms traditional threshold-based and PCA-based baselines.
Entities (5)
Relation Signals (3)
Speaker Drift Detection Framework → employs → LLM
confidence 100% · prompts large language models (LLMs) with structured representations to assess drift
Speaker Drift Detection Framework → uses → Cosine Similarity
confidence 100% · Our method computes cosine similarity across overlapping segments of synthesized speech
Wav2Vec2 → extracts → Speaker Embeddings
confidence 95% · extract fixed-dimensional speaker embeddings e i = f embed (s i ) ∈ R d using a pre-trained model (e.g., Wav2Vec2)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by formulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similarity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured representations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high-quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the viability of this embedding-to-reasoning pipeline. Our work establishes speaker drift as a standalone research problem and bridges geometric signal analysis with LLM-based perceptual reasoning in modern TTS.
Tags
Links
- Source: https://arxiv.org/abs/2604.06327v1
- Canonical: https://arxiv.org/abs/2604.06327v1
Trouble viewing inline? Open PDF directly →
Full Text
35,392 characters extracted from source content.
Expand or collapse full text
A NOVEL AUTOMATIC FRAMEWORK FOR SPEAKER DRIFT DETECTION IN SYNTHESIZED SPEECH 1 Jia-Hong Huang, 2 Seulgi Kim, 2 Yi Chieh Liu, 1 Yixian Shen, 1 Hongyi Zhu, 3 Prayag Tiwari, 1 Stevan Rudinac, 1 Evangelos Kanoulas 1 University of Amsterdam, 2 Georgia Institute of Technology, 3 Halmstad University ABSTRACT Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phe- nomenon undermines the coherence of synthetic speech, es- pecially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by for- mulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similar- ity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured rep- resentations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high- quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the viability of this embedding-to-reasoning pipeline. Our work establishes speaker drift as a standalone research problem and bridges geometric signal analysis with LLM-based perceptual reasoning in modern TTS. Index Terms— Text-to-speech (TTS), Diffusion Model, Speaker Drift Detection 1. INTRODUCTION Recent progress in text-to-speech (TTS) synthesis, particu- larly with diffusion-based models, has significantly enhanced the naturalness, expressiveness, and controllability of gen- erated speech [1–19]. These models can synthesize long- form utterances with high perceptual fidelity, supporting ap- plications such as personalized virtual assistants, audiobook narration, multi-turn dialog systems, and multimedia systems [20–32]. However, an underexplored yet critical challenge persists: speaker drift. This phenomenon refers to a subtle, gradual change in the perceived speaker identity within a sin- gle utterance, even when the synthesis is conditioned on a sin- gle, fixed speaker embedding or prompt. Such intra-utterance inconsistencies can undermine the effectiveness of applica- tions like those above, where maintaining a coherent and sta- ble speaker identity is crucial for a seamless user experience. Speaker drift fundamentally differs from conventional speaker changes addressed in diarization or speaker change detection, which typically assume abrupt and discrete tran- sitions [33–35]. In contrast, speaker drift involves gradual, often imperceptible shifts in vocal characteristics that ac- cumulate throughout an utterance. This subtle degradation presents significant challenges for detection, quantification, and evaluation. A comparison of related tasks is summarized in Table 1. Currently, there is no standardized evaluation protocol, scalable automated method, or standardized dataset tailored specifically to this problem, leaving a critical gap in quality assurance, model validation, and deployment re- liability for both academic research and production-level real-world TTS systems. To address this gap, we propose a novel, LLM-driven framework for automatic speaker drift detection in synthetic speech. We formulate this as a binary classification task at the utterance level. Specifically, we extract speaker embed- dings from short, overlapping segments of each utterance and compute pairwise cosine similarity scores, a compact, interpretable proxy for vocal identity consistency over time. These structured similarity score sequences with specially designed prompts are then fed into state-of-the-art LLMs, e.g., [36–40], to assess whether speaker drift is present based solely on the numerical input. This design bypasses the token limitations of modern LLMs, which cannot directly process high-dimensional embeddings.It offers a reference-free, scalable approach that bridges speaker embedding analysis and the reasoning capabilities of LLMs. To support this architecture, we provide theoretical justi- fication for using cosine similarity as an indicator of drift. Under mild distributional assumptions, we prove that a threshold-based classifier using segment-level cosine sim- ilarity scores can detect speaker drift with exponentially decreasing error as the similarity gap between same-speaker and different-speaker segments increases. This result for- mally grounds the use of cosine similarity as a statistically meaningful signal for vocal identity transitions. To validate our proposed approach and enable systematic evaluation, we address the lack of real-world data by con- structing a controlled benchmark dataset using a high-fidelity, diffusion-based TTS model. Real instances of intra-utterance identity drift are rare, ambiguous, and costly to annotate, arXiv:2604.06327v1 [cs.SD] 7 Apr 2026 making them unsuitable for empirical and systematic study. Instead, we generate utterance samples that either maintain consistent speaker identity or introduce identity shifts within an utterance, then verify them through human annotations. These synthetic samples provide a reliable and reproducible testbed for studying the drift phenomenon and evaluating the proposed framework in a well-defined setting. Together, our contributions offer a principled pipeline for detecting speaker drift and establish a foundation for future work at the intersection of embedding-based speech analysis and LLM-based perceptual evaluation in modern TTS sys- tems. To our knowledge, this is the first work to (1) define and formalize speaker drift detection as a standalone task, (2) construct a dataset tailored for intra-utterance identity vari- ation, and (3) explore LLM-based reasoning as a diagnostic tool for speaker consistency in TTS pipelines. 2. DATASET CONSTRUCTION 2.1 Overview To systematically study the speaker drift phenomenon in synthetic speech, we construct a benchmark dataset designed for binary classification, determining whether a speaker’s identity remains consistent or shifts within a given utterance. Reflecting real-world scenarios where speaker consistency is crucial, the dataset features two subtypes for each class: non-drift and hard negative samples for the “no drift” class, and abrupt drift and smooth morphing samples for the “drift” class. Each sample is created by concatenating consecutive speech segments with a 500 ms silence in between. The dataset contains 32 samples per subtype. 2.2 Controlled Synthetic Construction We begin with a curated set of N distinct speaker samples. Let S = x 1 ,..., x N denote the set of base speech clips, where each x i is associated with a unique speaker embed- ding. We synthesize utterances by concatenating three speech segments s 1 , s 2 , and s 3 : • Non-drift (label = 0): All segments are selected from the same speaker, e.g., [s 1 , s 1 , s 1 ], producing consistent speaker identity throughout the utterance. • Abrupt drift (label = 1): At least one segment orig- inates from a different speaker, e.g., [s 1 , s 2 , s 2 ] or [s 1 , s 1 , s 2 ], introducing a discrete speaker shift at a known boundary. 2.3 Hard Negative Construction (label = 0) To evaluate the effectiveness of our proposed LLM-based automatic speaker drift detection framework, we construct hard negative samples, utterances from the same speaker recorded under different conditions that do not involve actual identity drift. These conditions include variations in speaking rate and pitch (e.g., speedup rate = 1.05) and the addition of background noise (e.g., noise level = –30 dB). While these augmentations introduce noticeable acoustic and prosodic changes, they maintain the speaker’s identity. As such, they serve as challenging counterexamples, allowing us to test whether the framework can reliably distinguish true speaker drift from superficial style or environmental variations. 2.4 Modeling Gradual or Subtle Drift (label = 1) To simulate gradual speaker identity transitions in a more realistic setting, we introduce smooth speaker morphing, where speaker drift is performed directly at the audio level. Instead of generating discrete segments from different speak- ers, we synthesize overlapping speech regions and apply time-domain blending to produce perceptually smooth iden- tity changes.Let x A (t) and x B (t) denote two different speech waveforms. A morphing region t ∈ [T 1 ,T 2 ] ⊂ [0,T ] is defined over which the audio is blended using a linear cross-fade: x morph (t) = (1− α(t))· x A (t) + α(t)· x B (t), where α(t) = t−T 1 T 2 −T 1 . Outside the morphing region, the waveform is taken entirely from one speaker: x(t) = x A (t), t < T 1 , x morph (t), T 1 ≤ t≤ T 2 , x B (t), t > T 2 . This results in a continuous utterance where the speaker identity shifts gradually from A to B over a defined time window (e.g., from 3s to 6s in Figure 2), mimicking real- istic speaker drift at the acoustic level. Compared to hard cuts, this approach produces more subtle transitions and chal- lenges models to detect non-abrupt identity shifts that lack clear boundaries. It serves as an essential component for evaluating model sensitivity to fine-grained speaker variation. Although synthetic, our proposed dataset exhibits natu- ral speech-like quality and variability, closely resembling real human voices. Each sample has been manually inspected to ensure clarity, coherence, and intended speaker characteris- tics, guaranteeing high perceptual quality. The dataset allows precise control over the timing and location of speaker drift within utterances, enabling fine-grained analysis. It is label- balanced, containing equal numbers of drifted and non-drifted samples to avoid classification bias. Furthermore, utterances are structured in fixed-length segments, facilitating segment- level inspection and simplifying downstream detection. 3. METHOD 3.1 Problem Formulation Given a speech utterance x(t), divided into three contigu- ous segments (s 1 ,s 2 ,s 3 ) of equal duration, the goal is to de- tect whether the speaker identity remains consistent through- out or experiences a drift at one or more boundaries. We for- malize the task as a binary classification problem. Let the label y ∈ 0, 1, where y = 0 indicates a consistent speaker and y = 1 indicates at least one speaker shift. Our proposed (Synthetic speech) Speech Segmentation & Encoding Pairwise Cosine Similarity Calculation LLM Reasoning with a Specially Designed Prompt Drift or not Fig. 1: Our proposed LLM-based framework for detecting speaker drift in synthesized speech. Further details are provided in Section IV. TaskGoalKey Characteristics Speaker Change DetectionIdentify speaker boundaries in natural multi-speaker speech streams Assumes abrupt, discrete speaker transitions;aims at accurate change-point detection Speaker Verification or Identifi- cation Confirm or classify speaker identity using labeled reference data Requires ground-truth speaker la- bels; evaluates similarity across separate utterances Speaker DiarizationSegment and cluster speech by speaker in multi-speaker audio Operates on known or unknown speakers; assumes clear speaker turns; insensitive to subtle intra- speaker drift VoiceCloningConsistency Evaluation Evaluatepreservationof speaker identity in synthetic speech Primarily relies on subjective hu- man assessments or limited ABX testing; lacks automated, fine- grained detection methods Out-of-DistributionDetection (Speaker Embeddings) Detect anomalous speaker em- beddings outside known distri- bution Dependsonlargereference datasets;unsuitable for fine- grainedtemporalconsistency within utterances Voice Style Transfer Consis- tency Assess preservation of non- identityattributessuchas prosody, emotion, or accent Focuses on style or affective fea- tures; does not explicitly target speaker identity stability SpeakerDriftDetection (Ours) Detect subtle, gradual vari- ations in speaker identity within a single utterance, par- ticularly in synthetic speech from TTS systems Reference-free; focuses on tem- poral embedding consistency within utterances;addresses identitystabilitychallenges unique to TTS Table 1: Comparison of speaker drift detection with related tasks in speaker and TTS research. Fig. 2: Smooth speaker morphing across an utterance. The morphing region (3s–6s) involves audio-level cross-fading from Speaker A to Speaker B. method aims to predict y from the acoustic features or em- beddings derived from x(t). 3.2 Speaker Embedding Extraction and Cosine Similarity To enable automatic detection of speaker drift, each ut- terance is divided into three consecutive audio segments (s 1 , s 2 , s 3 ), from which we extract fixed-dimensional speaker embeddings e i = f embed (s i ) ∈ R d using a pre-trained model (e.g., Wav2Vec2).We then compute cosine simi- larities between adjacent segments to quantify inter-segment identity consistency: sim i,j = cos(e i , e j ) = e ⊤ i e j ∥e i ∥·∥e j ∥ , where (i,j) ∈ (1, 2), (2, 3). The resulting similarity pair (sim 1,2 , sim 2,3 ) serves as a compact representation of speaker consistency across the utterance, where lower values indicate potential identity shifts and provide an interpretable signal for LLM-based inference. 3.3 Prompt Design We design a standardized prompt template to ensure con- sistent and interpretable LLM-based evaluation of speaker drift.Each prompt concisely conveys the task objective, detecting speaker identity shifts, alongside the relevant pair- wise cosine similarity scores. By emphasizing clarity and minimizing ambiguity, the prompt guides the LLM toward accurate and reproducible decisions. The structure is as fol- lows: Instruction template: You are given a list of N pairwise similar- ity scores derived from three consecutive seg- ments of each utterance.Each score reflects the similarity between adjacent segments, com- puted using your similarity metric, with embeddings obtained fromyour audio encoder. Your task is to assess whether each utterance exhibits speaker drift based on the pro- vided similarity scores. For instance, consider the example: (0.9963, 0.9872).Based on this, determine whether a speaker identity shift has likely occurred. Provide a binary de- cision, same (no drift) or different (drift), and briefly explain your reasoning. Evaluate each case independently and report your decisions with corresponding justifications. 3.4 Theoretical Justification for Drift Detection via Em- bedding Similarity Detecting speaker drift hinges on a key geometric intu- ition: embeddings from the same speaker tend to cluster to- gether, while those from different speakers exhibit separation, particularly when measured via cosine similarity. This section formalizes that intuition through two complementary results: one intuitive and qualitative, and one rigorous and quantita- tive. 3.4.1 Motivating Insight of Drift Separability under Em- bedding Smoothness We begin with an intuitive proposition that connects smooth embedding behavior with the ability to detect drift using cosine similarity. Proposition 1 (Embedding Smoothness & Drift Sepa- rability). Let e 1 , e 2 , e 3 ∈ R d be embeddings from three contiguous speech segments. Define pairwise cosine similar- ities sim i,j = cos(e i , e j ). Suppose: max(sim 1,2 , sim 2,3 ) < τ for some threshold τ ∈ (0, 1). Then any classifier f that is Lipschitz-continuous over the similarity space can sep- arate drift from non-drift samples with bounded error: P (f (sim 1,2 , sim 2,3 )̸= y)≤ ε(τ ), where ε(τ )→ 0 as τ → 0, assuming embeddings vary smoothly for the same speaker and show sufficient separation across speakers. This proposition motivates the use of cosine similarity as a natural proxy for speaker identity consistency. It suggests that if embeddings change gradually within the same speaker and shift abruptly across speakers, then even simple decision boundaries (e.g., thresholding) can detect drift reliably. 3.4.2 Core Theoretical Result: Error Bound under Distri- butional Assumptions We now present a formal result under distributional as- sumptions on cosine similarity scores for drift vs. non-drift segments. Before stating the theorem, note that we assume the embeddings lie on the unit sphere S d−1 ⊂ R d , where S d−1 = x ∈ R d : ∥x∥ = 1. This reflects a common nor- malization step in modern speaker embedding systems (e.g., x-vectors, Wav2Vec2), where embeddings are constrained to have unit norm to make cosine similarity equivalent to the dot product. Since the unit sphere is a (d− 1)-dimensional manifold, each embedding lies on a curved surface, not a flat space, with exactly one degree of freedom removed due to the unit-norm constraint. Theorem 1 (Embedding Separation Bound for Speaker Drift Detection). Let e 1 , e 2 , e 3 ∈ S d−1 denote unit-norm embeddings corresponding to three contiguous speech seg- ments. We assume that for non-drift (i.e., same-speaker) samples, the expected pairwise cosine similarity satisfies E[cos(e i , e j )] ≥ μ 0 with variance bounded by σ 2 . In con- trast, for drift (i.e., different-speaker) samples, the expected similarity is lower, satisfying E[cos(e i , e j )] ≤ μ ′ < μ 0 , with the same variance σ 2 . Define the classifier: f (e 1 , e 2 , e 3 ) = ( 1 if min (cos(e 1 , e 2 ), cos(e 2 , e 3 )) < τ, 0 otherwise, for any threshold τ ∈ (μ ′ ,μ 0 ). Then, the misclassification probability is bounded by: P (f (e 1 , e 2 , e 3 )̸= y)≤ 4 exp − ∆ 2 2σ 2 ,where ∆ = min(μ 0 − τ,τ − μ ′ ). This result provides a concrete error bound, showing that the classifier’s performance improves exponentially as the separation margin ∆ grows. It justifies cosine threshold- ing as a statistically grounded and computationally simple strategy for detecting speaker drift, provided embeddings are well-separated across speakers and stable within speakers. <Proof>. We aim to bound the misclassification probabil- ity of the classifier f (e 1 , e 2 , e 3 ). Let y = 1 denote a drift case (i.e., a speaker change occurs), and y = 0 denote a non-drift case (i.e., all segments are from the same speaker). Let sim i,j := cos(e i , e j ) denote the cosine similarity be- tween embeddings e i and e j . In the non-drift case, we as- sume E[sim 1,2 ] ≥ μ 0 and E[sim 2,3 ] ≥ μ 0 , with variance Var(sim i,j ) ≤ σ 2 . In the drift case, the expected similari- ties are lower: E[sim 1,2 ] ≤ μ ′ , E[sim 2,3 ] ≤ μ ′ < μ 0 , with the same variance bound σ 2 . Let the classifier threshold τ satisfy μ ′ < τ < μ 0 , and define the decision margin as ∆ = min(μ 0 − τ,τ − μ ′ ). Step 1: Bounding False Positives (Type I Error) Suppose the input is non-drift. A false positive occurs when: f (e 1 , e 2 , e 3 ) = 1 (i.e., min(sim 1,2 , sim 2,3 ) < τ ). This implies at least one of the similarities is below τ , so: P(f ̸= y | y = 0)≤ P(sim 1,2 < τ ) + P(sim 2,3 < τ ). Since E[sim i,j ] ≥ μ 0 , and τ < μ 0 , we apply Hoeffding’s inequality (for bounded variables, e.g., cosine similarity in Embedding TypeGPT-4o [36]Gemini-Pro-2.5 [37]Claude-4 [38]Qwen-3 [40]DeepSeek-R1 [39]PCA-based BaselineFixed-threshold Baseline Wav2Vec2 [41]90.70%82.90%88.20%72.70%80.00%71.30%61.70% MFCC [42]39.00%38.64%76.00%71.40%80.00%65.60%57.40% Whisper [43]89.41%80.00%84.38%72.70%90.60%67.30%61.30% Thresholds(0.960, 0.950, 0.995)(0.970, 0.950, 0.998)(0.950, 0.950, 0.995)(0.910, 0.950, 0.990)(0.970, 0.990, 0.995)(0.950, 0.950, 0.950)(0.900, 0.900, 0.900) Table 2: Performance comparison with baselines and ablation studies using different audio embedding methods, evaluated using F1 score and pairwise cosine similarity. [−1, 1]): P(sim i,j < τ ) ≤ exp − (μ 0 −τ ) 2 2σ 2 . Thus, P(f ̸= y | y = 0)≤ 2 exp − (μ 0 −τ ) 2 2σ 2 . Step 2: Bounding False Negatives (Type I Error) Now suppose the input is drift. A false negative occurs when: f (e 1 , e 2 , e 3 ) = 0 (i.e., min(sim 1,2 , sim 2,3 ) ≥ τ ). That is, both similarities are above τ : P(f ̸= y | y = 1) ≤ P(sim 1,2 ≥ τ ) + P(sim 2,3 ≥ τ ). Since E[sim i,j ] ≤ μ ′ , and τ > μ ′ , again apply Hoeffding’s inequality: P(sim i,j ≥ τ )≤ exp − (τ−μ ′ ) 2 2σ 2 . So, P(f ̸= y | y = 1)≤ 2 exp − (τ−μ ′ ) 2 2σ 2 . Final Bound: Total Classification Error Combining both cases: P(f ̸= y)≤ 2 exp − (μ 0 −τ ) 2 2σ 2 + 2 exp − (τ−μ ′ ) 2 2σ 2 . By definition of ∆ = min(μ 0 −τ,τ−μ ′ ), we have: P(f ̸= y)≤ 4 exp − ∆ 2 2σ 2 . 4. EXPERIMENTS 4.1 Experimental Setup Dataset and Evaluation. We use the dataset described in Section I, consisting of 128 samples, 64 with speaker drift and 64 without, synthesized from 384 high-quality utterances by different speakers and verified by human annotators. Each sample is 9 to 40 seconds long. For evaluation, we report ac- curacy and F1 score. Each sample is represented by (1) cosine similarity scores between adjacent segments and (2) princi- pal component analysis (PCA)-reduced speaker embeddings, where each segment is compressed to 8 or 16 dimensions, yielding 24- or 48-dimensional inputs. LLMs, including GPT- 4o [36], Gemini-Pro 2.5 [37], Claude-4 [38], DeepSeek-R1 [39], and Qwen-3 [40], are tested under both zero-shot and few-shot settings on the full dataset. Baselines. For the fixed-threshold baseline, we classify a sample as drift if either cos(s 1 , s 2 ) or cos(s 2 , s 3 ) falls below 0.90; otherwise, it is labeled non-drift. This threshold, se- lected from the empirical distribution of minimum similarity scores, balances overlap between classes; non-drift samples occasionally drop to 0.77, while drift samples can reach as low as 0.80. Despite this compromise, the method achieves a modest F1 score of 0.62, highlighting the limitations of such simple heuristics. To build a strong baseline, we project segment-level speaker embeddings into a lower-dimensional space via PCA to preserve broader variation patterns. These reduced embeddings are then used as input to LLMs for bi- nary classification. MethodInput TypeAccuracyF1 Score GPT-4oPCA Embeddings (8)50.3%66.7% GPT-4oPCA Embeddings (16)73.4%73.6% GPT-4oCosine Scores89.5%90.7% Gemini-Pro-2.5PCA Embeddings (8)50.8%58.3% Gemini-Pro-2.5PCA Embeddings (16)52.3%59.8% Gemini-Pro-2.5Cosine Scores79.7%82.9% Claude-4PCA Embeddings (8)63.7%69.3% Claude-4PCA Embeddings (16)67.5%73.6% Claude-4Cosine Scores83.4%88.2% Qwen-3PCA Embeddings (8)65.3%68.0% Qwen-3PCA Embeddings (16)69.1%72.4% Qwen-3Cosine Scores69.5%72.7% DeepSeek-R1PCA Embeddings (8)60.9%71.2% DeepSeek-R1PCA Embeddings (16)63.4%74.4% DeepSeek-R1Cosine Scores78.8%80.0% Table 3: Ablation study of the proposed LLM-driven speaker drift detection framework, evaluating the impact of different LLM backbones and input feature formats. 4.2 Experimental Results Performance Analysis. We evaluate our LLM-based method against the fixed-threshold and PCA-based baselines de- scribed above. As reported in Table 2, our approach yields a substantially higher F1 score than the fixed-threshold base- line and consistently outperforms the PCA-based baseline, indicating superior performance in detecting speaker drift. These results validate the effectiveness of leveraging LLMs with pairwise cosine similarity scores for the speaker drift detection task. Ablation Studies. We conduct several ablation studies to assess the impact of different design choices: (1) Audio Em- bedding Type: We compare Wav2Vec2, MFCC, and Whis- per embeddings as input features. As shown in Table 2, Wav2Vec2 embeddings achieve the highest performance, indicating their superior ability to capture speaker-relevant characteristics. (2) PCA Dimensionality: We assess the effect of dimensionality reduction by compressing embeddings to 8 dimensions (preserving ≈ 75% of the variance) and 16 dimensions (≈ 87% variance preserved) per segment. Re- sults in Table 3 show that the 16-dimensional setting yields better performance, suggesting that retaining more dimen- sions helps preserve discriminative speaker information. (3) Input Representation: We compare models using pairwise cosine similarity scores against those using PCA-reduced embeddings as input. As shown in Table 3, the former consis- tently outperforms the latter, indicating that explicit relational features between segments more effectively capture speaker identity drift than compressed embedding representations. 5. CONCLUSION AND FUTURE WORK In this work, we introduced the first automated framework for detecting speaker drift in diffusion-based TTS, leveraging cosine similarity as a theoretically grounded proxy for vocal identity consistency and prompting LLMs for perceptual rea- soning. Our method bridges low-level acoustic embeddings with high-level evaluation and is supported by a new bench- mark dataset with human-verified annotations. Looking for- ward, we plan to extend this framework to multilingual and cross-lingual settings and explore fine-tuning LLMs for even greater sensitivity to subtle prosodic and identity cues in gen- erated speech. 6. REFERENCES [1] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al., “Tacotron: Towards end-to-end speech synthesis,” arXiv preprint arXiv:1703.10135, 2017. [2] Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, et al., “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, vol. 12, 2016. [3] Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al., “Autoregressive speech syn- thesis without vector quantization,”arXiv preprint arXiv:2407.08551, 2024. [4] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Kar- rer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al., “Voicebox: Text-guided multilingual universal speech generation at scale,” Advances in neural information processing sys- tems, vol. 36, p. 14005–14034, 2023. [5] Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” arXiv preprint arXiv:2304.09116, 2023. [6] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al., “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, p. 4779–4783. [7] Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, Alessio M. Pacces, and Evangelos Kanoulas, “A novel evaluation framework for image2text gener- ation,”in International ACM SIGIR Conference on Research and Development in Information Retrieval, LLM4Eval Workshop, 2024. [8] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu, “Fastspeech: Fast, robust and controllable text to speech,” Advances in neural information processing systems, vol. 32, 2019. [9] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim,“Diff-tts: A denoising diffusion model for text-to-speech,” arXiv preprint arXiv:2104.01409, 2021. [10] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov, “Grad-tts: A diffusion probabilistic model for text-to-speech,” in ICML, 2021, p. 8599–8608. [11] Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, and Jun Zhu,“Schrodinger bridges beat diffusion models on text-to-speech synthesis,”arXiv preprint arXiv:2312.03491, 2023. [12] Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, and Evangelos Kanoulas, “Image2text2image: A novel framework for label-free evaluation of image- to-text generation with text-to-image diffusion models,” in International Conference on Multimedia Modeling. Springer, 2025, p. 413–427. [13] Weijia Zhang, Mohammad Aliannejadi, Yifei Yuan, Ji- ahuan Pei, Jia-Hong Huang, and Evangelos Kanoulas, “Towards fine-grained citation evaluation in generated text: A comparative analysis of faithfulness metrics,” in Proceedings of the 17th International Natural Language Generation Conference, 2024, p. 427–439. [14] Justin Lovelace, Soham Ray, Kwangyoun Kim, Kil- ian Q Weinberger, and Felix Wu,“Sample-efficient diffusion for text-to-speech synthesis,” arXiv preprint arXiv:2409.03717, 2024. [15] Mateusz Lajszczak, Guillermo C ́ ambara, Yang Li, Fatih Beyhan, Arent Van Korlaar, Fan Yang, Arnaud Joly, ́ Alvaro Mart ́ ın-Cortinas, Ammar Abbas, Adam Michal- ski, et al., “Base tts: Lessons from building a billion- parameter text-to-speech model on 100k hours of data,” arXiv preprint arXiv:2402.08093, 2024. [16] Jia-Hong Huang, Yixian Shen, Hongyi Zhu, Stevan Rudinac, and Evangelos Kanoulas, “Gradient weight- normalized low-rank projection for efficient llm train- ing,” in Proceedings of the AAAI Conference on Artifi- cial Intelligence, 2025, vol. 39, p. 24123–24131. [17] Shivam Mehta, Ruibo Tu, Jonas Beskow, ́ Eva Sz ́ ekely, and Gustav Eje Henter, “Matcha-tts: A fast tts archi- tecture with conditional flow matching,”in ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2024, p. 11341–11345. [18] Zhijun Liu, Shuai Wang, Pengcheng Zhu, Mengxiao Bi, and Haizhou Li,“E1 tts: Simple and fast non- autoregressive tts,” in ICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, p. 1–5. [19] Xinlu He, Swayambhu Nath Ray, Harish Mallidi, Jia-Hong Huang, Ashwin Bellur, Chander Chandak, M Maruf, and Venkatesh Ravichandran, “Continuous- token diffusion for speaker-referenced tts in multimodal llms,”The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems (NeurIPS) Work- shop on Structured Probabilistic Inference & Genera- tive Modeling (SPIGM), 2025. [20] Jia-Hong Huang, Luka Murn, Marta Mrak, and Mar- cel Worring, “Query-based video summarization with pseudo label supervision,” in 2023 IEEE International Conference on Image Processing (ICIP). IEEE, 2023, p. 1430–1434. [21] Hongyi Zhu, Jia-Hong Huang, Stevan Rudinac, and Evangelos Kanoulas, “Enhancing interactive image re- trieval with query rewriting using large language mod- els and vision language models,”in Proceedings of the 2024 International Conference on Multimedia Re- trieval, 2024, p. 978–987. [22] Hongyi Zhu, Jia-Hong Huang, Yixian Shen, Stevan Rudinac, and Evangelos Kanoulas, “Interactive image retrieval meets query rewriting with large language and vision language models,” ACM Transactions on Mul- timedia Computing, Communications and Applications, vol. 21, no. 10, p. 1–23, 2025. [23] Jia-Hong Huang, “Multi-modal video summarization,” in Proceedings of the 2024 International Conference on Multimedia Retrieval, 2024, p. 1214–1218. [24] Changshuo Wang, Shuting He, Xiang Fang, Zhijian Hu, Jia-Hong Huang, Yixian Shen, and Prayag Tiwari, “Reasoning beyond points: A visual introspective ap- proach for few-shot 3d segmentation,” in The Thirty- ninth Annual Conference on Neural Information Pro- cessing Systems, 2025. [25] Yixian Shen, Qi Bi, Jia-Hong Huang, Hongyi Zhu, Andy D Pimentel, and Anuj Pathania, “Macp: Mini- mal yet mighty adaptation via hierarchical cosine pro- jection,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, p. 20602–20618. [26] Yi-Chieh Liu, Hao-Hsiang Yang, C-H Huck Yang, Jia- Hong Huang, Meng Tian, Hiromasa Morikawa, Yi- Chang James Tsai, and Jesper Tegner,“Synthesiz- ing new retinal symptom images by multiple genera- tive models,” in Asian Conference on Computer Vision. Springer, 2018, p. 235–250. [27] Jia-Hong Huang and Marcel Worring,“Query- controllable video summarization,” in Proceedings of the International Conference on Multimedia Retrieval, 2020, p. 242–250. [28] Yixian Shen, Qi Bi, Jia-Hong Huang, Hongyi Zhu, Andy D Pimentel, and Anuj Pathania, “Ssh: Sparse spectrum adaptation via discrete hartley transforma- tion,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), 2025, p. 10400– 10415. [29] Jia-Hong Huang, Luka Murn, Marta Mrak, and Mar- cel Worring,“Gpt2mvs:Generative pre-trained transformer-2 for multi-modal video summarization,” in Proceedings of the International Conference on Multi- media Retrieval, 2021, p. 580–589. [30] Riccardo Di Sipio, Jia-Hong Huang, Samuel Yen-Chi Chen, Stefano Mangini, and Marcel Worring,“The dawn of quantum natural language processing,”in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, p. 8612–8616. [31] Weijia Zhang, Jia-Hong Huang, Svitlana Vakulenko, Yumo Xu, Thilina Rajapakse, and Evangelos Kanoulas, “Beyond relevant documents: A knowledge-intensive approach for query-focused summarization using large language models,” in International Conference on Pat- tern Recognition. Springer, 2024, p. 89–104. [32] Jia-Hong Huang, Chao-Han Huck Yang, Pin-Yu Chen, Andrew Brown, and Marcel Worring, “Causal video summarizer for video exploration,” in 2022 IEEE Inter- national Conference on Multimedia and Expo (ICME). IEEE, 2022, p. 1–6. [33] Jitendra Ajmera, Iain McCowan, and Herv ́ e Bourlard, “Robust speaker change detection,” IEEE signal pro- cessing letters, vol. 11, no. 8, p. 649–651, 2004. [34] Aonan Zhang, Quan Wang, Zhenyao Zhu, John Pais- ley, and Chong Wang, “Fully supervised speaker di- arization,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, p. 6301–6305. [35] Wei Xia, Han Lu, Quan Wang, Anshuman Tripathi, Yil- ing Huang, Ignacio Lopez Moreno, and Hasim Sak, “Turn-to-diarize: Online speaker diarization constrained by transformer transducer speaker turn detection,” in ICASSP, 2022, p. 8077–8081. [36] OpenAI, “Gpt-4 technical report,” 2024. [37] Gemini Team, “Gemini: A family of highly capable multimodal models,” 2024. [38] Anthropic,“Claude (language model),” https: //w.anthropic.com/claude/sonnet, 2025, Accessed: 2025-05-15. [39] DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [40] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xi- aohuan Zhou, and Tianhang Zhu, “Qwen technical re- port,” 2023. [41] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,“wav2vec 2.0:A framework for self-supervised learning of speech representations,” NeurIPS, vol. 33, p. 12449–12460, 2020. [42] Steven Davis and Paul Mermelstein,“Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” IEEE Transactions on Acoustics, Speech, and Signal Process- ing, vol. 28, no. 4, p. 357–366, 1980. [43] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in ICML, 2023, p. 28492–28518.