Paper deep dive
Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech
Hyoeun Kim, Yujun Lee, Kyuhong Shim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 9/24/2026, 5:19:04 AM
Summary
The paper introduces GUARD, a lightweight framework for speaker identity unlearning in zero-shot text-to-speech (ZS-TTS) systems. GUARD prevents the re-identification of speakers who opt out by combining a learned speaker gate (GateNet) with speaker-agnostic activation steering on a frozen TTS backbone. It uses group-relative reward optimization to shift forget-speaker similarity toward population-level impostor similarity, significantly reducing re-identification accuracy while preserving speech quality and retain-speaker reproduction.
Entities (12)
Relation Signals (11)
GUARD → evaluatedon → CosyVoice2
confidence 95% · On CosyVoice2, GUARD reduces forget-speaker similarity from 0.541 to 0.103
GUARD → optimizesusing → group-relative reward optimization
confidence 95% · The steering vectors are optimized using group-relative reward optimization
GUARD → usescomponent → GateNet
confidence 95% · GUARD combines GateNet with speaker-agnostic layer-wise steering vectors.
GateNet → determines → steering application
confidence 90% · GateNet predicts the steering coefficient that weights the layer-wise steering vectors
GUARD → targets → Zero-Shot Text-to-Speech
confidence 90% · we propose GUARD, a lightweight speaker identity unlearning framework... on a frozen TTS backbone.
GUARD → evaluatedondataset → LibriTTS
confidence 85% · We evaluate GUARD on LibriTTS clean-460
GUARD → generalizesto → FireRedTTS
confidence 85% · Table 4: Backbone generalization of GUARD across F5-TTS and FireRedTTS.
GUARD → generalizesto → F5-TTS
confidence 85% · Table 4: Backbone generalization of GUARD across F5-TTS and FireRedTTS.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent zero-shot text-to-speech (ZS-TTS) systems can reproduce a speaker's voice with high fidelity from only a few seconds of reference speech, raising concerns over unauthorized voice cloning and impersonation. Speaker identity unlearning has recently emerged as an approach to selectively suppress this capability for speakers who opt out while preserving synthesis capability for other speakers. Although existing approaches reduce speaker similarity, preventing re-identification often faces severe degradation of speech quality. Motivated by this observation, we propose GUARD, a lightweight speaker identity unlearning framework that combines a learned speaker gate with speaker-agnostic activation steering on a frozen TTS backbone. The steering vectors are optimized using group-relative reward optimization to shift outputs from forget speakers toward population-level impostor similarity while preserving intelligibility and speech naturalness. On CosyVoice2, GUARD reduces forget-speaker similarity from 0.541 to 0.103 and re-identification accuracy in a 150-speaker gallery from 73.5% to 0.5%, while preserving retain-speaker reproduction. The results demonstrate that similarity reduction alone may not fully characterize successful speaker identity unlearning and highlight re-identification as a complementary criterion for its evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2609.27399v1
- Canonical: https://arxiv.org/abs/2609.27399v1
Trouble viewing inline? Open PDF directly →
Full Text
27,455 characters extracted from source content.
Expand or collapse full text
Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech Hyoeun Kim Yujun Lee Kyuhong Shim Abstract Recent zero-shot text-to-speech (ZS-TTS) systems can reproduce a speaker’s voice with high fidelity from only a few seconds of reference speech, raising concerns over unauthorized voice cloning and impersonation. Speaker identity unlearning has recently emerged as an approach to selectively suppress this capability for speakers who opt out while preserving synthesis capability for other speakers. Although existing approaches reduce speaker similarity, preventing re-identification often faces severe degradation of speech quality. Motivated by this observation, we propose GUARD, a lightweight speaker identity unlearning framework that combines a learned speaker gate with speaker-agnostic activation steering on a frozen TTS backbone. The steering vectors are optimized using group-relative reward optimization to shift outputs from forget speakers toward population-level impostor similarity while preserving intelligibility and speech naturalness. On CosyVoice2, GUARD reduces forget-speaker similarity from 0.541 to 0.103 and re-identification accuracy in a 150-speaker gallery from 73.5% to 0.5%, while preserving retain-speaker reproduction. The results demonstrate that similarity reduction alone may not fully characterize successful speaker identity unlearning and highlight re-identification as a complementary criterion for its evaluation. Index Terms: speaker unlearning, zero-shot text-to-speech, speaker privacy, speaker re-identification, activation steering †address: Sungkyunkwan University, Republic of Korea hyoeunkim, yj090744, khshim@skku.edu 1 Introduction Zero-shot text-to-speech (ZS-TTS) systems reproduce the voices of previously unseen speakers from short reference recordings. Models such as VALL-E [19], NaturalSpeech [16], Voicebox [11], F5-TTS [3], and CosyVoice [4] synthesize natural speech that matches the input text while preserving speaker characteristics from the reference. However, this capability raises the risks of unauthorized voice cloning, impersonation, voice phishing, and deepfake misuse. Voice anonymization transforms speaker identity while preserving linguistic content [13, 18], whereas anti-cloning methods introduce adversarial perturbations into speech to disrupt unauthorized voice cloning [22, 6]. These approaches protect individual speech samples but do not address speaker opt-out requests in deployed TTS systems. Speaker identity unlearning instead suppresses the reproduction of speakers who opt out after deployment. Accordingly, we consider a provider-controlled deployment setting in which users access the TTS system through the standard synthesis interface. Prior approaches follow two directions: TGU and SGU [9] fine-tune the flow decoder using alternative speech targets; TruS [12] removes speaker-identity-related directions from internal activations at inference time. Recent work considers continual speaker identity unlearning under sequential opt-out requests [8]. Despite these advances, existing methods do not reduce speaker similarity (SIM) between generated and reference speech enough to suppress gallery-based re-identification. We emphasize that this re-identification risk has received limited attention in prior speaker-unlearning work, and argue that speaker identity unlearning should suppress reliable re-identification while preserving linguistic content, speech naturalness, and retain-speaker reproduction. Figure 1: Motivation for re-identification-aware speaker unlearning: GUARD shifts forget-speaker similarity toward population-level impostor similarity. Figure 2: GUARD architecture. (a) Layer-wise steering vectors modify the residual activations of the decoder transformer blocks. (b) GateNet determines whether steering is applied to the input speaker. (c) Group-relative reward optimization learns the steering vectors from post-generation rewards. To this end, we propose GUARD (Group-relative Unlearning with Activation Steering for Re-identification Defense), a re-identification-aware framework that targets the population-level impostor similarity. We evaluate residual speaker identity through gallery-based re-identification. GUARD combines GateNet with speaker-agnostic layer-wise steering vectors. GateNet predicts the steering coefficient that weights the layer-wise steering vectors added to decoder activations. We optimize these steering vectors using group-relative reward optimization with post-generation rewards that drive forget-speaker SIM toward population-level impostor similarity while preserving linguistic content and speech naturalness. On CosyVoice2, GUARD reduces forget-speaker SIM from 0.541 to 0.103 and top-1 re-identification accuracy in a 150-speaker gallery from 73.5% to 0.5%, without degradation in speech quality and retain-speaker reproduction. Our contributions are threefold: • We introduce a re-identification-aware formulation of speaker identity unlearning that targets population-level impostor similarity and evaluates residual identity through gallery-based re-identification. • We propose GUARD, which combines a learned speaker gate with layer-wise activation steering and group-relative reward optimization on a frozen TTS backbone. • GUARD substantially reduces gallery-based re-identification while preserving speech quality and retain-speaker reproduction, with lightweight roster updates and consistent improvements across other ZS-TTS backbones. 2 Related Work 2.1 Speaker Identity Unlearning for Zero-Shot TTS Recent studies have investigated speaker identity unlearning for selectively suppressing specific speakers in zero-shot TTS. TGU and SGU [9] optimize the flow decoder using alternative speech targets for forget speakers while preserving synthesis capability for retain speakers. However, both methods require fine-tuning the full flow decoder and can affect retain-speaker reproduction. TruS [12] instead suppresses speaker identity by steering internal activations along identity-related directions at inference time without additional training. Increasing its steering strength reduces forget-speaker similarity but can substantially degrade speech naturalness and intelligibility, whereas weaker steering leaves substantial gallery-based re-identification. Recent work also considers continual speaker identity unlearning under sequential opt-out requests [8]. Unlike prior work, we target population-level impostor similarity and explicitly evaluate residual speaker identity through gallery-based re-identification. 2.2 Activation Steering for TTS Activation steering directly modifies internal representations of pretrained TTS models to control speech attributes. EmoSteer-TTS [20] derives activation-difference vectors for training-free emotion control. EmoShift [24] learns trainable steering vectors for emotion-aware speech synthesis. Activation steering has also been applied to accent modification while preserving speaker characteristics [21]. These studies demonstrate that internal activations provide a lightweight intervention space for controlling TTS generation. GUARD instead optimizes speaker-agnostic layer-wise steering vectors for speaker identity unlearning using post-generation anonymity, intelligibility, and naturalness rewards. 2.3 Reward-Based Optimization for TTS Reward-based optimization incorporates post-generation evaluation signals directly into TTS optimization. Seed-TTS [1] uses reinforcement learning, and F5R-TTS [17] applies GRPO; both incorporate WER and speaker-similarity rewards. GLASS [7] optimizes lightweight LoRA-based acoustic style controls on a frozen TTS backbone using GRPO and post-generation acoustic metrics. In contrast to these weight-space adaptations, GUARD performs group-relative reward optimization over layer-wise activation steering vectors while keeping the TTS backbone frozen. Its anonymity reward drives forget-speaker similarity toward population-level impostor similarity, together with WER and UTMOS rewards for intelligibility and speech naturalness. Table 1: Quantitative results on LibriTTS clean-460 using CosyVoice2-0.5B. Gallery Re-ID reports top-1 re-identification accuracy over galleries of different sizes. For forget speakers, μ¯imp=0.129 μ_imp=0.129 denotes the mean population-level impostor similarity across forget speakers and serves as the reference for SIM; the optimal AUROC is 0.5, which indicates chance-level ranking for the evaluated detectability score. Method Forget Speakers (F) Retain Speakers (R) Gallery Re-ID (%)↓ Imp. Det. SIM →μ¯imp→ μ_imp WER ↓ UTMOS ↑ spk-ZRF ↑ SIM ↑ WER ↓ UTMOS ↑ spk-ZRF ↑ id@50 id@100 id@150 AUROC → 0.5 CosyVoice2 0.541 5.5 4.37 0.851 0.607 4.9 4.40 0.832 81.0 77.5 73.5 0.009 SGU 0.481 5.6 4.40 0.856 0.556 5.2 4.41 0.838 67.5 63.0 56.5 0.021 TGU 0.481 5.7 4.38 0.861 0.561 5.7 4.39 0.840 64.5 62.5 56.0 0.021 TruS (α=2.0α=2.0) 0.341 5.3 3.64 0.837 0.602 4.6 4.39 0.832 34.5 30.0 25.0 0.117 TruS (α=2.5α=2.5) 0.078 122.0 1.35 0.807 0.614 4.1 4.38 0.832 3.5 2.5 2.0 0.657 GUARD (Ours) 0.103 5.9 4.37 0.824 0.601 4.7 4.40 0.831 2.0 1.0 0.5 0.579 3 Method 3.1 Problem Formulation and Overview A pretrained zero-shot TTS model ℳM synthesizes speech from a reference recording a and target text y. Given a set of forget speakers ℱF, GUARD suppresses their re-identification while preserving linguistic content, speech naturalness, and retain-speaker reproduction. We use CosyVoice2 [4] as the primary backbone and freeze all backbone parameters. Rather than minimizing speaker similarity, GUARD drives it toward population-level impostor similarity, which characterizes the similarity between unrelated speakers. GUARD modifies residual activations in the flow decoder using layer-wise steering vectors. For decoder block ℓ , we apply ~ℓ=ℓ+g~()ℓ,ℓ=1,…,L, h_ =h_ + g(e)s_ , =1,…,L, (1) at each CFM step and temporal position. ℓ∈ℝds_ ^d is a trainable steering vector shared across forget speakers, and g~() g(e) controls its contribution for the input speaker. CosyVoice2 has L=56L=56 flow-decoder blocks with hidden dimension d=256d=256, for a total of 14.3K trainable steering parameters. 3.2 GateNet for Selective Steering GateNet predicts the steering coefficient from the speaker embedding using a lightweight MLP with SiLU activation. The network uses a 128-dimensional hidden layer and contains 24.8K parameters. We train GateNet with class-weighted binary cross-entropy using forget and retain speaker labels. At inference time, we apply an asymmetric hard threshold: g~()=0,g()<τ,g(),otherwise,τ=0.3. g(e)= cases0,&g(e)<τ,\\ g(e),&otherwise, cases τ=0.3. (2) When g()<τg(e)<τ, the hard gate removes steering entirely, preserving the original computation path of the frozen backbone. GateNet is trained separately and remains fixed during steering-vector optimization. 3.3 Group-Relative Reward Optimization We optimize the layer-wise steering vectors using post-generation rewards for anonymity, intelligibility, and speech naturalness. Let s denote the complete set of layer-wise steering vectors. For each input, we generate G candidate sets of steering vectors as (k)=+σϵks^(k)=s+σ ε_k, where ϵk∼(,) ε_k (0,I). Each candidate produces a synthesized output x(k)x^(k), which is scored using the following reward: R(x(k))= R(x^(k))= −wa|cos(Esv(x(k)),f)−μimp(f)| -w_a | \! (E_sv(x^(k)),p_f )- _imp(f) | (3) −wcWER(x(k),y)+wnmin(0,U(x(k))−U0), -w_c\,WER(x^(k),y)+w_n (0,U(x^(k))-U_0 ), where EsvE_sv denotes the speaker-verification encoder, fp_f denotes the enrollment prototype of forget speaker f, and μimp(f) _imp(f) denotes its population-level impostor similarity, computed as the mean cosine similarity between fp_f and non-matching speaker prototypes. The anonymity term penalizes deviation of forget-speaker SIM from μimp(f) _imp(f). The intelligibility term penalizes transcription errors using WER, while the speech naturalness term penalizes outputs whose UTMOS score falls below U0U_0, the baseline’s UTMOS score. We normalize rewards within each candidate group and estimate the reward gradient as Ak=Rk−R¯std(R)+δ,∇J^=1Gσ∑k=1GAkϵk.A_k= R_k- Rstd(R)+δ, _s J= 1Gσ _k=1^GA_k ε_k. (4) We update the steering vectors with Adam [10] using forget-speaker candidates only, while GateNet remains fixed. At inference, GUARD applies the learned steering vectors with the gated coefficient g~() g(e), without candidate sampling or reward computation. 4 Experiments 4.1 Experimental Settings Dataset and implementation. We evaluate GUARD on LibriTTS clean-460 [23], designating 50 speakers (25 male and 25 female) as forget speakers and the remaining 995 as retain speakers. Evaluation uses 200 utterances from the 50 forget speakers and 80 utterances from 40 retain speakers. All evaluation utterances are held out from GateNet training, and the 200 forget-speaker evaluation utterances are also held out from steering-vector optimization. Each synthesis example uses a 3-second speaker prompt from one utterance and target text from a different utterance of the same speaker. We use the frozen CosyVoice2-0.5B [4] as the primary backbone. For backbone generalization, we additionally evaluate F5-TTS [3] and FireRedTTS [5]. For both models, we use an external speaker-verification encoder to obtain GateNet inputs and optimize the steering vectors separately for each backbone. GateNet is trained separately on cached speaker embeddings for 2.5K steps. We optimize the layer-wise steering vectors over the 50 forget speakers while keeping GateNet fixed. Steering-vector optimization uses G=16G=16 candidates with σ=0.06σ=0.06, reward weights (wa,wc,wn)=(1,24,4)(w_a,w_c,w_n)=(1,24,4), and U0=4.45U_0=4.45. We use Whisper [14] for the WER reward computation and performance evaluation. All experiments are conducted on a single V100-16GB GPU. Baselines and metrics. We compare GUARD with the original CosyVoice2 and three speaker identity unlearning methods: SGU, TGU, and TruS. SGU and TGU fine-tune the full 71.3M-parameter flow decoder using their original objectives with λ=0.2λ=0.2, 60 utterances per forget speaker, and 12K optimization steps. We evaluate TruS at steering strengths α=2.0α=2.0 and 2.52.5 with oracle gating for retain speakers. We report speaker similarity (SIM) [2], WER [14], UTMOS [15], and spk-ZRF [9] for both forget and retain speakers. We additionally measure residual speaker identity using gallery-based top-1 re-identification accuracy, id@Nid@N, for N∈50,100,150N∈\50,100,150\. For each synthesized forget-speaker utterance, the gallery contains the corresponding forget speaker and N−1N-1 distractors, each represented by an enrollment prototype. We also report detectability AUROC between protected outputs and genuine impostor recordings, where values closer to 0.50.5 indicate lower detectability. Averaging μimp(f) _imp(f) over the 50 forget speakers gives μ¯imp=0.129 μ_imp=0.129. 4.2 Results Table 2: Reward ablation of GUARD with all variants trained from scratch under an identical budget. Each row removes one reward term to isolate its effect on unlearning performance. The optimal AUROC score is 0.5. Variant Forget Speakers (F) Gallery Re-ID (%) ↓ Imp. Det. SIM →μ¯imp→ μ_imp WER ↓ UTMOS ↑ id@50 id@100 id@150 AUROC GUARD 0.103 5.9 4.37 2.0 1.0 0.5 0.579 w/o μimp _imp 0.056 6.2 4.39 4.5 2.0 2.0 0.706 w/o RUTMOSR_UTMOS 0.154 11.4 1.25 3.5 2.5 1.0 0.430 w/o RWERR_WER 0.092 6.7 4.44 6.5 3.0 2.5 0.608 Table 3: Cost of updating a single newly opted-out speaker. Method Updated params Update time Memory Update size TGU 71.3M 6.3 h 8–12 GB 286 MB SGU 71.3M 7.8 h 8–12 GB 286 MB GUARD 24.8K 9.9 min <<1 GB 0.1 MB Table 4: Backbone generalization of GUARD across F5-TTS and FireRedTTS. For forget-speaker SIM, μ¯imp=0.129 μ_imp=0.129 denotes the mean population-level impostor similarity across forget speakers. Backbone Method Forget Speakers (F) Gallery Re-ID (%) ↓ SIM →μ¯imp→ μ_imp WER ↓ UTMOS ↑ id@50 id@100 id@150 F5-TTS Baseline 0.651 3.5 4.13 96.5 95.5 95.5 GUARD 0.199 4.2 4.25 14.5 6.5 6.0 FireRedTTS Baseline 0.636 4.2 4.17 94.5 93.0 91.5 GUARD 0.210 4.8 4.15 26.0 20.0 18.0 Speaker unlearning and re-identification. Table 1 shows that GUARD shifts forget-speaker SIM toward the population-level impostor regime while preserving retain-speaker reproduction. SGU and TGU update the full flow decoder; their re-identification rates at id@150id@150 remain at 56.5%56.5\% and 56.0%56.0\%, respectively. TruS reduces forget-speaker SIM as the steering strength α increases; at α=2.0α=2.0, its re-identification rate at id@150id@150 remains at 25.0%25.0\%. At α=2.5α=2.5, the re-identification rate of TruS decreases to 2.0%2.0\%, but WER increases to 122.0%122.0\% and UTMOS decreases to 1.351.35, which indicates a severe performance degradation. GUARD reduces forget-speaker SIM from 0.5410.541 to 0.1030.103 and id@150id@150 from 73.5%73.5\% to 0.5%0.5\%, while maintaining WER, UTMOS, and retain-speaker metrics close to the original backbone. Its detectability AUROC of 0.5790.579 is closest to 0.50.5 among the evaluated methods, indicating that protected outputs are less distinguishable from genuine impostor recordings. Reward ablation. Table 2 examines the effect of each reward term on forget-speaker identity suppression, intelligibility, and speech naturalness. Removing the impostor target (i.e., μimp=0 _imp=0) lowers forget-speaker SIM from 0.1030.103 to 0.0560.056, but does not reduce gallery-based re-identification and increases detectability AUROC from 0.5790.579 to 0.7060.706. This result indicates that reducing forget-speaker SIM below the population-level impostor regime does not further reduce gallery-based re-identification and makes protected outputs more distinguishable from genuine impostor recordings. Removing the UTMOS reward substantially degrades both speech naturalness and intelligibility, with UTMOS decreasing to 1.251.25 and WER increasing to 11.4%11.4\%. Without the WER reward, WER increases to 6.7%6.7\%, while id@150id@150 increases to 2.5%2.5\%. Update cost and backbone generalization. Table 3 compares the cost of adding a newly opted-out speaker to the forget roster. GUARD keeps the speaker-agnostic layer-wise steering vectors fixed as the forget roster changes and updates only the 24.8K-parameter GateNet, without requiring TTS-backbone retraining. This lightweight roster update requires 9.9 minutes, compared with 6.3–7.8 hours for SGU and TGU, which fine-tune the 71.3M-parameter flow decoder. The lightweight roster update also requires substantially less GPU memory and checkpoint storage. Table 4 evaluates GUARD on F5-TTS [3] and FireRedTTS [5]. GUARD reduces id@150id@150 from 95.5%95.5\% to 6.0%6.0\% on F5-TTS and from 91.5%91.5\% to 18.0%18.0\% on FireRedTTS while maintaining intelligibility and speech naturalness. 4.3 Analysis Population-level impostor similarity as the target. Reducing forget-speaker SIM alone does not fully characterize speaker identity unlearning. SIM above the population-level impostor regime can retain speaker-specific information that supports gallery-based re-identification. Reducing SIM below this regime does not further reduce gallery-based re-identification and makes protected outputs more distinguishable from genuine impostor recordings. These observations motivate population-level impostor similarity as a reference target rather than unrestricted SIM minimization. Selective intervention and roster updates. GUARD separates speaker selection from the identity intervention. GateNet determines whether steering is applied, while the speaker-agnostic layer-wise steering vectors define the shared activation intervention for forget speakers. This separation also localizes roster updates to GateNet, so newly opted-out speakers can be incorporated without requiring TTS-backbone retraining. 5 Conclusion We introduced GUARD, a speaker identity unlearning framework that combines GateNet with speaker-agnostic layer-wise steering vectors optimized using group-relative reward optimization on a frozen ZS-TTS backbone. By driving forget-speaker SIM toward the population-level impostor regime, GUARD substantially reduced gallery-based re-identification while preserving speech quality and retain-speaker reproduction. The speaker-agnostic layer-wise steering vectors remained fixed during roster updates, requiring only GateNet retraining for newly opted-out speakers. Future work will investigate speaker-adaptive steering vectors for individualized speaker identity suppression while preserving GUARD’s lightweight update mechanism. 6 Compliance with Ethical Standards This study used existing publicly available datasets, and no new data were collected. No ethical approval was required. References [1] P. Anastassiou, J. Chen, J. Chen, Y. Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, et al. (2024) Seed-tts: a family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430. Cited by: §2.3. [2] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al. (2022) Wavlm: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), p. 1505–1518. Cited by: §4.1. [3] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen (2024) F5-TTS: a fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885. Cited by: §1, §4.1, §4.2. [4] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, et al. (2024) Cosyvoice 2: scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117. Cited by: §1, §3.1, §4.1. [5] H. Guo, Y. Hu, K. Liu, F. Shen, X. Tang, Y. Wu, F. Xie, K. Xie, and K. Xu (2024) Fireredtts: a foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283. Cited by: §4.1, §4.2. [6] Q. Hu, J. Wu, W. Lu, and X. Luo (2026) VoiceCloak: a multi-dimensional defense framework against unauthorized diffusion-based voice cloning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 399–407. Cited by: §1. [7] J. Kang, Y. Lee, and K. Shim (2026) GLASS: grpo-trained lora for acoustic style steering in zero-shot text-to-speech. arXiv preprint arXiv:2606.05889. Cited by: §2.3. [8] J. Kim, Y. Kang, G. Park, and J. H. Ko (2026) Continual speaker identity unlearning with minimal interference. arXiv preprint arXiv:2605.25962. Cited by: §1, §2.1. [9] T. Kim, J. Kim, D. Kim, J. H. Ko, and G. Park (2025) Do not mimic my voice: speaker identity unlearning for zero-shot text-to-speech. arXiv preprint arXiv:2507.20140. Cited by: §1, §2.1, §4.1. [10] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.3. [11] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, and W. Hsu (2023) Voicebox: text-guided multilingual universal speech generation at scale. In Advances in Neural Information Processing Systems, Vol. 36, p. 14005–14034. Cited by: §1. [12] M. Lee, E. Shin, and J. Lee (2026) Erasing your voice before it’s heard: training-free speaker unlearning for zero-shot text-to-speech. arXiv preprint arXiv:2601.20481. Cited by: §1, §2.1. [13] M. Panariello, N. Tomashenko, X. Wang, X. Miao, P. Champion, H. Nourtel, M. Todisco, N. Evans, E. Vincent, and J. Yamagishi (2024) The voiceprivacy 2022 challenge: progress and perspectives in voice anonymisation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, p. 3477–3491. Cited by: §1. [14] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In International conference on machine learning, p. 28492–28518. Cited by: §4.1, §4.1. [15] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari (2022) Utmos: utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152. Cited by: §4.1. [16] K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian (2024) NaturalSpeech 2: latent diffusion models are natural and zero-shot speech and singing synthesizers. In International Conference on Learning Representations (ICLR), Cited by: §1. [17] X. Sun, R. Xiao, J. Mo, B. Wu, Q. Yu, and B. Wang (2025) F5r-tts: improving flow-matching based text-to-speech with group relative policy optimization. arXiv preprint arXiv:2504.02407. Cited by: §2.3. [18] N. Tomashenko, X. Miao, P. Champion, S. Meyer, M. Panariello, X. Wang, N. Evans, E. Vincent, J. Yamagishi, and M. Todisco (2026) The third voiceprivacy challenge: preserving emotional expressiveness and linguistic content in voice anonymization. Computer Speech & Language, p. 101988. Cited by: §1. [19] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei (2023) Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111. Cited by: §1. [20] T. Xie, S. Yang, C. Li, D. Yu, and L. Liu (2025) Emosteer-tts: fine-grained and training-free emotion-controllable text-to-speech via activation steering. arXiv preprint arXiv:2508.03543. Cited by: §2.2. [21] M. Yang and J. H. Hansen (2026) Activation steering for accent-neutralized zero-shot text-to-speech. arXiv preprint arXiv:2603.05977. Cited by: §2.2. [22] Z. Yu, S. Zhai, and N. Zhang (2023) AntiFake: using adversarial audio to prevent unauthorized speech synthesis. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, p. 460–474. External Links: Document Cited by: §1. [23] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu (2019) Libritts: a corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882. Cited by: §4.1. [24] L. Zhou, H. Jiang, J. Li, T. Wang, and H. Li (2026) Emoshift: lightweight activation steering for enhanced emotion-aware speech synthesis. In 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 17262–17266. Cited by: §2.2.