Paper deep dive
Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models
Ye Lu, Yihan Yan, Zhaoyang Zhang, Zhitao Ou, Runze Liu, Li Liu, Shen Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/21/2026, 4:11:09 AM
Summary
This paper investigates the privacy risks of end-to-end speech language models by demonstrating that exposed speech tokens can leak voiceprints. The authors propose a Speaker Inversion Attack (SIA) and a two-stage inversion method called SpInv, which utilizes a trainable model named Audio BERT (AuB) to reconstruct speaker embeddings from discrete speech tokens. Experiments on models like Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni show that SpInv can recover speaker embeddings with high cosine similarity (>0.70) using only three seconds of token data.
Entities (9)
Relation Signals (8)
SpInv → uses → Audio BERT
confidence 95% · SpInv, a two-stage inversion method built on AuB
SpInv → usesdataset → VoxCeleb
confidence 95% · We evaluate... using speaker-disjoint protocols on the VoxCeleb dataset.
SpInv → targets → Speech Tokens
confidence 93% · SpInv... recover embeddings... from exposed speech tokens
Speaker Inversion Attack → formulatedby → SpInv
confidence 92% · We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack... We propose SpInv
SpInv → evaluatedon → Kimi-Audio
confidence 90% · We evaluate... Kimi-Audio... using speaker-disjoint protocols
SpInv → evaluatedon → Qwen3-Omni
confidence 90% · We evaluate... Qwen3-Omni... using speaker-disjoint protocols
SpInv → evaluatedon → Higgs3
confidence 90% · We evaluate... Higgs3... using speaker-disjoint protocols
SpInv → evaluatedon → Moshi
confidence 90% · We evaluate Moshi... using speaker-disjoint protocols
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space.
Tags
Links
- Source: https://arxiv.org/abs/2607.16870v1
- Canonical: https://arxiv.org/abs/2607.16870v1
Trouble viewing inline? Open PDF directly →
Full Text
44,554 characters extracted from source content.
Expand or collapse full text
Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models Ye Lu, Yihan Yan, Zhaoyang Zhang, Zhitao Ou, Runze Liu, Li Liu, and Shen Wang Abstract End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR–LLM–TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space. I Introduction End-to-end speech language models are emerging as a natural interface for interaction with large models. Unlike cascaded ASR–LLM–TTS systems that convert speech into text for language-model processing, these models directly process speech-modality tokens throughout inference. Such representations preserve prosody, emotion, speaking style, and other acoustic cues, enabling emotion-aware and context-sensitive interaction. They also facilitate bidirectional and full-duplex speech modeling while reducing modality-conversion overhead and response latency. Directly transmitting user speech to cloud-side models can expose voiceprints and other sensitive acoustic attributes. A split-inference deployment may instead run a tokenizer or speech frontend locally and transmit only its output to the cloud-side model, leaving the downstream provider or an intermediate service with speech tokens rather than the waveform. Speech tokens are more compact and task-oriented than raw audio, but it remains unclear whether they suppress speaker identity or merely encode it in a different form. This motivates our central question: Do speech tokens in end-to-end speech language models leak voiceprints? Figure 1: Attack scenario. User speech is converted into speech tokens before large-model inference. An eavesdropper or curious provider observing the exposed speech interface uses SpInv to recover an embedding in the space of an attacker-specified speaker encoder. We formulate this privacy question as a Speaker Inversion Attack (SIA) against speech tokens, which extends the concept of a Model Inversion Attack (MIA) [39] to speaker privacy in the speech modality. As illustrated in Fig. 1, the attack arises when an eavesdropper intercepts speech tokens transmitted between the frontend and the downstream model, or when a curious service provider can access them during inference. The adversary then uses the exposed token segment to recover a speaker embedding in the space of a speaker encoder that the adversary specifies. To instantiate this attack, we propose SpInv, a two-stage inversion method that employs Audio BERT (AuB) to aggregate speaker information from speech tokens. AuB maps discrete codebook indices to trainable token embeddings and encodes the resulting sequence with a BERT backbone, while SpInv combines distillation pretraining with discriminative fine-tuning. The trained method recovers a speaker embedding in the attacker-specified speaker-encoder space. We evaluate four representative speech-model frontends: Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni. The evaluation first tests whether their exposed interfaces retain speaker-discriminative information. It then assesses speaker embedding recovery on speaker-disjoint VoxCeleb datasets followed by analyses of input duration and target speaker space. Our results show that SpInv can recover an embedding in the specified speaker-encoder space from only three seconds of exposed frontend outputs, achieving cosine similarities generally above 0.70 with the target embedding. The main contributions of this work are as follows: • To the best of our knowledge, we present the first systematic study of privacy leakage from speech tokens and extend model inversion in the speech modality to the problem of speaker-identity privacy. • We propose SpInv, a two-stage speaker inversion method that employs an AuB model to construct and aggregate token representations and recover embeddings in the space of an attacker-specified speaker encoder. • We conduct extensive experiments on four representative target speech frontends, including leakage validation, speaker embedding recovery, and ablations on key factors. I Related Work I-A End-to-End Speech Language Models and Speech Interfaces Early spoken interaction systems built around large language models were commonly modular or text-mediated. Representative systems such as SpeechGPT [38], Spectron [27], and PSLM [25] connect speech encoders, language models, and speech-generation modules, retaining structural similarities to cascaded ASR–LLM–TTS pipelines. Recent systems increasingly retain speech-specific representations throughout a unified inference pipeline. By avoiding repeated conversion between speech and text, speech-token modeling preserves paralinguistic information and supports lower-latency, full-duplex spoken dialogue, which better matches the interactive requirements of speech language models. Moshi [11] is a real-time, full-duplex speech–text model whose Mimi codec maps 24-kHz audio to one semantic and seven residual acoustic token streams at 12.5 frames/s. Higgs3 [4] is a 4B autoregressive conversational TTS model using eight codebooks at 25 frames/s; its tokenizer is also adopted by the diffusion-based OmniVoice [40]. Kimi-Audio [22] combines a 12.5-Hz semantic-token stream with time-aligned continuous Whisper features. Qwen3-Omni [37] instead uses an Audio Transformer to emit continuous speech representations at 12.5 Hz. Together, these models cover residual-codebook, hybrid discrete–continuous, and continuous interfaces; AuB handles all three within one voiceprint-leakage attack pipeline. I-B Speaker Representation Learning Speaker recognition has evolved from statistical pipelines to deep speaker embeddings. Early methods used GMM-UBM models, joint factor analysis, and i-vectors [30, 21, 12]. Subsequent speaker-verification systems learned discriminative embeddings from large speech corpora [33, 18, 34, 31, 10], with recent work also exploring self-supervised speaker representation learning [8, 6]. ECAPA-TDNN improves TDNN-based verification through channel attention and feature aggregation [14]; CAM++ uses context-aware masking for efficient speaker modeling [35]; and ERes2Net combines local and global multi-scale features [7]. In our threat model, the adversary selects a pretrained speaker encoder to define the target space into which SpInv recovers embeddings. I-C Model Inversion and Audio Privacy Attacks Model Inversion Attacks (MIAs) infer sensitive information about model inputs or training data from outputs and intermediate representations. They have been extensively studied in vision, where an adversary may reconstruct representative inputs or identity-related features [17, 39, 24, 23]. Related studies in natural language processing have shown that continuous text embeddings can retain enough information to recover substantial properties of their source text [26, 9]. These findings demonstrate that replacing raw inputs with learned representations does not necessarily remove private information. Speech and audio systems expose several distinct privacy surfaces. Prior work has introduced inversion attacks against automatic speaker-recognition systems [29] and examined whether speaker identity remains inferable from representations produced by speaker-anonymization mechanisms [3]. Membership-inference studies further investigate whether speech representations reveal the participation of particular users or sensitive attributes in model training [32]. Other attacks exploit released speaker scores to support impersonation [20], while recent evaluations of audio language models consider whether their outputs disclose sensitive information contained in the audio they process [36]. Our work instead targets speech tokens exposed between the speech frontend and the downstream backbone, using a partial token sequence to recover an embedding in the space of an attacker-specified speaker encoder. Unlike waveform-oriented inversion or attacks against speaker-recognition systems, SpInv does not reconstruct the original speech signal or attack the speaker encoder; instead, it recovers an embedding in an attacker-specified speaker space. I Method I-A Threat Model We formalize the attack surface, the adversary’s objective, and the knowledge available to the adversary. Target model Let x denote an arbitrary user speech waveform. The target system is an end-to-end speech language model whose speech frontend converts user audio into speech tokens before downstream inference. Let T denote the tokenizer in this frontend, which converts x into a time-indexed speech-token sequence S=T(x) S=T(x) =[1,…,N], =[s_1,…,s_N], (1) n _n =(sn(1),…,sn(Q)). =(s_n^(1),…,s_n^(Q)). Here, N is the sequence length and Q is the number of codebooks. The q-th codebook has size MqM_q, and sn(q)∈1,…,Mqs_n^(q)∈\1,…,M_q\ is its token ID at frame n. Some frontends additionally expose time-aligned continuous features A=[1,…,N]A=[a_1,…,a_N] alongside the token stream or in place of it. We denote the complete exposed speech representation by R(x)=(S,A)R(x)=(S,A), with any unavailable component omitted. We consider deployments in which these frontend outputs are accessible to an intermediate service or transmitted to the downstream model. Adversary goal The adversary is an eavesdropper observing the exposed speech interface or a curious service provider with access to it during inference, consistent with speech-privacy threat models that assume access to intermediate representations [32, 36]. Let F be a frozen speaker encoder selected by the attacker that maps a waveform to a target speaker embedding =F(x)∈ℝdz,z=F(x) ^d_z, (2) where dzd_z is the embedding dimension. By selecting F, the attacker specifies the target speaker-embedding space to be recovered. Rather than reconstructing the original waveform or recovering the target model’s parameters, the adversary observes a randomly cropped contiguous segment R~ R of R(x)R(x) and seeks to recover z without access to the victim’s waveform at attack time. A recovery model GθG_θ with parameters θ produces ^=Gθ(R~)∈ℝdz z=G_θ( R) ^d_z. We formulate the Speaker Inversion Attack as θ∗=argminθℒ(Gθ(R~),),θ^*= _θ\;L (G_θ( R),z ), (3) where ℒL measures the discrepancy between the recovered and target embeddings in the speaker space defined by F, and θ∗θ^* denotes the optimized recovery-model parameters. Adversary knowledge and capabilities The adversary is assumed to have three capabilities. First, it has query access to the target speech frontend T: it can submit an utterance and observe the corresponding frontend output R(x)R(x), but it does not access the internal parameters of the target model. Second, for a token-based interface, it can observe the number of codebooks Q and their vocabulary sizes Mqq=1Q\M_q\_q=1^Q; the learned codebook embedding values are neither known nor required. Third, it can obtain a public auxiliary speech dataset D and use the queried frontend outputs to train the attack model. The adversary selects and has access to the frozen speaker encoder F; this choice determines the desired speaker-embedding space into which GθG_θ must recover embeddings. In our experiments, F is instantiated by a publicly available pretrained speaker encoder. I-B Overview of SpInv As illustrated in Fig. 2, SpInv encompasses both a training phase and an attack phase. The training phase learns a recovery model GθG_θ from the outputs of the target speech frontend through a two-stage strategy that combines distillation pretraining and discriminative fine-tuning. In the attack phase, the adversary intercepts only a slice of exposed speech tokens from the victim’s frontend stream; the trained AuB model and projection head then recover a speaker embedding z in the selected speaker space directly from this token slice, without requiring the original waveform. Figure 2: Overview of SpInv. The training phase learns a recovery model from frontend outputs via two-stage optimization; the attack phase recovers a speaker embedding from an intercepted token slice. I-C AuB Model Structure AuB is the trainable speaker information extractor used by SpInv. It comprises a codebook embedding layer, a feature fusion layer, and a BERT backbone. For codebook q with vocabulary size MqM_q, AuB maintains a trainable lookup table (q)∈ℝMq×dE^(q) ^M_q× d. The token IDs at frame n are mapped to the shared embedding space and added across codebooks: n=∑q=1Q(q)[sn(q)],n∈ℝd.e_n= _q=1^QE^(q)[s_n^(q)], _n ^d. (4) The lookup tables belong to AuB and are learned independently of the unknown codebook embeddings used by the target model. Since certain target frontends additionally expose the time-aligned continuous feature na_n at each frame as supplementary acoustic information, AuB incorporates a feature fusion layer for compatibility. An MLP projects this feature to the shared embedding space as n=MLP(n)∈ℝdu_n=MLP(a_n) ^d. The fused representation at frame n is obtained by element-wise addition: n=n+n.r_n=e_n+u_n. (5) This design accommodates the native exposed outputs of all target frontends within a single architecture. Similar to ViT [16], AuB prepends a learnable CLS token to [1,…,L][r_1,…,r_L] and processes the sequence with a BERT backbone [15], where L is the length of the input segment. The CLS output h is converted into a fixed 512-dimensional AuB output ∈ℝdao ^d_a, where da=512d_a=512. The projection head is a two-layer MLP with LayerNorm [1] and GELU [19] that maps o to the recovered embedding ^∈ℝdz z ^d_z. Accordingly, the recovery model GθG_θ in (3) comprises AuB and the projection head, with θ=(θA,θP)θ=( _A, _P) denoting their respective parameters. During training, GθG_θ also exposes h and o for auxiliary objectives. A separate DINO head DϕD_ _D uses a two-layer MLP projector with GELU, followed by bottleneck LayerNorm, ℓ2 _2 normalization, and a bias-free prototype layer to produce logits ℓ=DϕD() =D_ _D(o). The DINO head is optimized only as an auxiliary training component and is discarded after training; it is therefore not part of GθG_θ or the final recovery parameters θ∗θ^*. AuB, the projection head, and the auxiliary DINO head are randomly initialized and trained jointly. I-D SpInv Training and Attack Training phase As shown in the training phase of Fig. 2, SpInv organizes three parallel branches. All five views are constructed at the waveform level for each utterance: two local views s=l1,l2V_s=\l_1,l_2\, two global views g=g1,g2V_g=\g_1,g_2\, and one teacher view xitx_i^t. The student model receives only the two local views through the frozen target frontend R and GθG_θ, producing recovered embeddings z, CLS representations h, AuB outputs o, and student logits from DϕD_ _D. The EMA teacher model, comprising EMA copies of AuB and the DINO head, receives only the two global views and generates self-distillation targets. The frozen teacher model F receives only the teacher view and produces the target embedding i=F(xit)z_i=F(x_i^t). The loss block in Fig. 2 shows that LalignL_align and LrelL_rel operate on embeddings recovered from the local views, LdinoL_dino matches student distributions from the local views against EMA teacher targets from the global views, and LvarL_var and LcovL_cov regularize representations from the local views. In Stage B, an ArcMargin classifier adds speaker-discriminative supervision on the CLS representations from the local views. Stage A: Distillation pretraining. Stage A combines alignment to the target speaker space, relational matching of batch geometry, and EMA-based self-distillation across views. Variance and covariance regularization following VICReg [2] prevent representational collapse. Its objective is LA=λ1Lalign+λ2Lrel+λ3Ldino+λ4Lvar+λ5Lcov,L_A= _1L_align+ _2L_rel+ _3L_dino+ _4L_var+ _5L_cov, (6) where λj≥0 _j≥ 0 are loss weights. For a mini-batch of B utterances, let Vs=|s|=2V_s=|V_s|=2, let ^i,v z_i,v be the recovered embedding of utterance i under local view v∈sv _s, and let iz_i be the target embedding obtained from its teacher view. The alignment loss is Lalign=1BVs∑i=1B∑v∈s(1−cos(^i,v,i)).L_align= 1BV_s _i=1^B _v _s (1- ( z_i,v,z_i) ). (7) The relational loss aligns pairwise speaker geometry within the batch: Lrel=1VsB(B−1)∑v∈s∑i≠j(cos(^i,v,^j,v)−cos(i,j))2.L_rel= 1V_sB(B-1) _v _s _i≠ j ( ( z_i,v, z_j,v)- (z_i,z_j) )^2. (8) For self-distillation, the valid teacher–student pairs form the Cartesian product =g×sP=V_g×V_s. The EMA teacher produces centered and sharpened prototype distributions qi,uq_i,u from global view u∈gu _g, while the student produces prototype distributions pi,vp_i,v from local view v∈sv _s. The DINO loss [5] is Ldino=−1B||∑(u,v)∈∑i=1B∑k=1Ksg(qi,u,k)logpi,v,k,L_dino=- 1B|P| _(u,v) _i=1^B _k=1^Ksg(q_i,u,k) p_i,v,k, (9) where K is the number of prototypes and sg(⋅)sg(·) denotes stop-gradient. Let H(v)∈ℝB×dhH^(v) ^B× d_h contain the student representations used for regularization under view v. With γ>0γ>0 denoting the target standard deviation and ϵ>0ε>0 ensuring numerical stability, the view-averaged anti-collapse terms are Lvar=1Vsdh∑v∈s∑j=1dhmax(0,γ−Var(H:,j(v))+ϵ),L_var= 1V_sd_h _v _s _j=1^d_h (0,γ- Var(H^(v)_:,j)+ε ), (10) Lcov=1Vsdh∑v∈s∑j≠j′(Cov(H(v))j,j′)2.L_cov= 1V_sd_h _v _s _j≠ j (Cov(H^(v))_j,j )^2. (11) Stage B: Discriminative fine-tuning. Stage B initializes GθG_θ and the auxiliary DINO head from the selected Stage A checkpoint and adds speaker-discriminative supervision. An ArcMargin classifier [13] is applied to h, while lower-weight versions of the Stage A losses preserve the learned target-space structure: LB=μ1Lspk+μ2Lalign+μ3Lrel+μ4Ldino+μ5Lvar+μ6Lcov,L_B= _1L_spk+ _2L_align+ _3L_rel+ _4L_dino+ _5L_var+ _6L_cov, (12) where μj≥0 _j≥ 0 are loss weights. Let CspkC_spk be the number of speakers in the auxiliary training set, with c∈1,…,Cspkc∈\1,…,C_spk\. For normalized ¯i,v h_i,v and classifier weight ¯c w_c, define αi,v,c=arccos(¯i,v⊤¯c) _i,v,c= ( h_i,v w_c) and the ArcMargin logit βi,v,c=scos(αi,v,c+m[c=yi]) _i,v,c=s ( _i,v,c+mI[c=y_i]). The speaker-classification loss over the local views is Lspk=−1BVs∑i=1B∑v∈slogeβi,v,yi∑c=1Cspkeβi,v,c,L_spk=- 1BV_s _i=1^B _v _s e _i,v,y_i _c=1^C_spke _i,v,c, (13) where yiy_i is the speaker label, m is the angular margin, and s is the logit scale. Stage B retains the same assignment as Stage A: two local views to the student model, two global views to the EMA teacher model, and one teacher view to the teacher model. Attack phase As illustrated in the attack phase of Fig. 2, the adversary does not access the victim’s waveform but only intercepts a contiguous slice of exposed speech tokens (and any accompanying interface features) from the victim’s frontend output stream. This token slice is fed directly to the trained recovery model GθG_θ, comprising AuB and the projection head, to yield z in the speaker space specified by F. The auxiliary DINO head and ArcMargin classifier are not used during the attack. As demonstrated experimentally, the recovered speaker embedding z has high cosine similarity to the raw speaker embedding =F(x)z=F(x) extracted by F when the waveform is available for evaluation. IV Experiments IV-A Experimental Setup Target models We evaluate SpInv against four representative speech-model frontends with their native exposed outputs. Moshi [11] employs the Mimi neural audio codec, which exposes one semantic and seven residual acoustic token streams at 12.5 frames/s. Higgs3 [4] uses an eight-codebook tokenizer operating at 25 frames/s; this tokenizer is also used as the audio frontend in OmniVoice [40]. Kimi-Audio [22] outputs a semantic-token stream together with time-aligned continuous Whisper features, which provide complementary acoustic information. Qwen3-Omni [37] uses a continuous Audio Transformer frontend that emits frame-level feature vectors at 12.5 Hz. The first three models expose discrete speech-token interfaces and constitute the core focus of this study; Qwen3-Omni is additionally evaluated to demonstrate that SpInv also applies to purely continuous frontend features, broadening the evaluation coverage. SpInv uses the exposed output of each frontend directly. For each frontend, input waveforms are resampled to its native sampling rate; all frontend parameters remain frozen throughout attack training. Datasets We use the VoxCeleb corpora, which are widely adopted benchmarks for speaker recognition research. VoxCeleb1 [28] contains over 150,000 utterances from 1,251 speakers, totaling approximately 352 hours of YouTube speech. Under its standard verification partition, the development set contains 1,211 speakers and the test set contains 40 speakers, with no speaker identity overlap. VoxCeleb2 [10] is substantially larger, comprising over 1.1 million utterances, totaling roughly 2,442 hours, from 6,112 speakers. Its standard partition contains 5,994 development speakers and 118 test speakers, which are also identity-disjoint. Both corpora cover diverse acoustic conditions, speaking styles, and demographic backgrounds. We follow the standard VoxCeleb1 and VoxCeleb2 evaluation protocols. Privacy leakage validation uses the official VoxCeleb1 verification split [28]. For speaker embedding recovery, SpInv is trained on VoxCeleb2-dev and evaluated on the speaker-disjoint VoxCeleb2-test and VoxCeleb1 sets. The speakers in VoxCeleb1 and VoxCeleb2-dev do not overlap. Unless stated otherwise, the subsequent ablations use the Moshi tokenizer with VoxCeleb2-dev → VoxCeleb2-test. Evaluation method The frozen teacher model extracts target speaker embeddings from the teacher view, while the student model receives only target-frontend outputs derived from the local views. Let iz_i be the target embedding extracted by the frozen speaker encoder F and ^i z_i be the embedding recovered by SpInv. CosSim measures their average cosine similarity: CosSim=1N∑i=1N^i⊤i‖^i‖2‖i‖2.CosSim= 1N _i=1^N z_i z_i\| z_i\|_2\|z_i\|_2. (14) We also report KL after converting the two embeddings into distributions over dimensions, Pi=softmax(i)P_i=softmax(z_i) and P^i=softmax(^i) P_i=softmax( z_i): KL=1N∑i=1N∑kPi,klogPi,k+ϵP^i,k+ϵ,KL= 1N _i=1^N _kP_i,k P_i,k+ε P_i,k+ε, (15) where ϵε is a small constant. Accuracy (Acc.), equal error rate (EER), and normalized minDCF measure speaker-verification performance. For minDCF, we use Ptarget=0.01P_target=0.01, Cmiss=1C_miss=1, and Cfa=1C_fa=1. The reported threshold is the corresponding optimal cosine-score decision boundary used to compute Acc. Target speaker space The target speaker space is defined by the frozen encoder F selected by the attacker, as formalized in (2). In the main experiments, we adopt ECAPA-TDNN [14] with embedding dimensionality dz=192d_z=192 as the default target space. To assess the robustness of SpInv across different speaker representations, we also evaluate three alternative target spaces: ERes2Net [7] (dz=192d_z=192), which captures multi-scale speaker features, and CAM++ [35] at two embedding dimensionalities (dz=192d_z=192 and dz=512d_z=512), which employs context-aware masking for efficient speaker modeling. All target encoders are publicly available, pretrained, and kept frozen during attack training. Default hyperparameter configuration The AuB codebook embedding layer uses the same number of codebooks and vocabulary sizes as observed in the target frontend. The feature fusion layer uses a one-layer MLP. The projection and DINO heads follow the structures described in Section I-C. AuB uses a BERT hidden dimension of dh=384d_h=384 and produces a fixed-dimensional output with da=512d_a=512. The projection head maps this 512-dimensional output to dzd_z, the dimensionality of the selected target speaker encoder, such as dz=192d_z=192 for the default ECAPA-TDNN. The complete AuB model is trained from random initialization using Adam with a batch size of 96 (which occupies approximately 20 GB of GPU memory), while the target frontend and speaker encoder remain frozen. Stage A runs for 60 epochs with a learning rate of 4×10−44× 10^-4 and weights λ1=1.0 _1=1.0, λ2=0.5 _2=0.5, λ3=1.0 _3=1.0, λ4=25.0 _4=25.0, and λ5=1.0 _5=1.0 for LalignL_align, LrelL_rel, LdinoL_dino, LvarL_var, and LcovL_cov, respectively. Stage B runs for 40 epochs with a learning rate of 1×10−41× 10^-4 and weights μ1=0.2 _1=0.2, μ2=0.8 _2=0.8, μ3=0.4 _3=0.4, μ4=0.4 _4=0.4, μ5=10.0 _5=10.0, and μ6=0.5 _6=0.5 for LspkL_spk, LalignL_align, LrelL_rel, LdinoL_dino, LvarL_var, and LcovL_cov. The ArcMargin margin is linearly increased from 0 to 0.2 between epochs 10 and 20 of Stage B. The DINO center momentum is 0.9, and the EMA momentum follows a cosine schedule from 0.996 to 0.9995. IV-B Privacy Leakage Validation Before evaluating inversion, we determine whether the exposed speech interfaces retain speaker-discriminative information. We train AuB with speaker labels on the VoxCeleb1 development partition and evaluate its representations on the speaker-disjoint VoxCeleb1 verification test partition. Random three-second cropping is the only data augmentation. Table I reports this diagnostic experiment for all four target models. TABLE I: Speaker-information leakage on the speaker-disjoint VoxCeleb1 verification split. Target model EER ↓ minDCF ↓ Moshi 0.0160 0.0512 Higgs3 0.0066 0.0132 Kimi-Audio 0.0034 0.0171 Qwen3-Omni 0.0236 0.0836 All four exposed interfaces retain strong speaker-discriminative information under the speaker-disjoint protocol. Kimi-Audio achieves the lowest EER of 0.0034, followed by Higgs3 at 0.0066 and Moshi at 0.0160, while Qwen3-Omni also exhibits substantial leakage with an EER of 0.0236. These results confirm that the frontend representations preserve speaker information that generalizes to unseen identities. IV-C Speaker Embedding Recovery We next evaluate whether SpInv can recover embeddings in the frozen ECAPA-TDNN speaker space for previously unseen speakers. By default, the attack observes three seconds of exposed frontend output. Table I reports results on VoxCeleb2-test, and Table I reports results on VoxCeleb1; both use VoxCeleb2-dev for attack training. TABLE I: Speaker embedding recovery from VoxCeleb2-dev to VoxCeleb2-test. Target model CosSim ↑ KL ↓ Acc ↑ EER ↓ minDCF ↓ Thres. Moshi 0.7136 0.3346 0.9701 0.0305 0.0885 0.3287 Higgs3 0.7527 0.2858 0.9828 0.0174 0.0627 0.3870 Kimi-Audio 0.7536 0.2919 0.9825 0.0178 0.0723 0.4215 Qwen3-Omni 0.6683 0.3257 0.9624 0.0378 0.1124 0.3269 TABLE I: Speaker embedding recovery from VoxCeleb2-dev to VoxCeleb1. Target model CosSim ↑ KL ↓ Acc ↑ EER ↓ minDCF ↓ Thres. Moshi 0.7037 0.4257 0.9395 0.0606 0.1245 0.2833 Higgs3 0.7060 0.3540 0.9513 0.0489 0.1016 0.3208 Kimi-Audio 0.7359 0.3395 0.9449 0.0554 0.0991 0.3422 Qwen3-Omni 0.6594 0.3991 0.9219 0.0783 0.1335 0.2240 Across both speaker-disjoint evaluation sets, Higgs3 and Kimi-Audio provide the strongest overall recovery, followed by Moshi, while Qwen3-Omni is the most difficult frontend to attack. All four frontends perform worse on VoxCeleb1 than on VoxCeleb2-test. Because SpInv is trained on VoxCeleb2-dev, this consistent decline may result from a slight distribution mismatch between the two evaluation datasets, while the relative performance trend remains stable. IV-D Ablation Study We examine the effects of attack-time input duration and the target speaker space using Moshi with VoxCeleb2-dev → VoxCeleb2-test. Input duration We vary the observed segment duration from 1 to 10 seconds while using ECAPA-TDNN as the target speaker encoder. Table IV reports how the amount of exposed speech affects embedding recovery. TABLE IV: Effect of attack-time input duration using Moshi (VoxCeleb2-dev → VoxCeleb2-test). Duration CosSim ↑ KL ↓ Acc ↑ EER ↓ minDCF ↓ Thres. 1 s 0.4887 0.4353 0.8656 0.1346 0.6670 0.1868 2 s 0.6677 0.3537 0.9559 0.0444 0.4050 0.2697 3 s 0.7136 0.3346 0.9701 0.0305 0.0885 0.3287 5 s 0.7642 0.3176 0.9849 0.0153 0.0878 0.3694 8 s 0.7949 0.3131 0.9892 0.0110 0.0707 0.3723 10 s 0.8065 0.3116 0.9878 0.0124 0.0757 0.4132 Shorter observations reduce speaker-embedding recovery performance, with the largest degradation occurring at one second. Starting from three seconds, the attack achieves substantially stronger recovery. Longer observations generally improve CosSim, although the verification gains become smaller and mildly non-monotonic beyond eight seconds, indicating that the attack approaches saturation. Target speaker space We evaluate recovery into several attacker-specified speaker-encoder spaces instantiated with publicly available models, including ECAPA-TDNN, ERes2Net [7], and CAM++ [35]. Table V reports the results for each target space. TABLE V: Effect of the target speaker space using Moshi (VoxCeleb2-dev → VoxCeleb2-test). Speaker encoder Dim CosSim ↑ Acc ↑ EER ↓ minDCF ↓ Thres. ECAPA-TDNN 192 0.7136 0.9701 0.0305 0.0885 0.3287 ERes2Net 192 0.7562 0.9559 0.0443 0.0882 0.3389 CAM++ 192 0.7658 0.9811 0.0191 0.0957 0.3074 CAM++ 512 0.7167 0.9867 0.0135 0.1044 0.3040 SpInv achieves effective recovery across all four target speaker spaces. CAM++ with 192-dimensional embeddings obtains the highest CosSim of 0.7658, whereas its 512-dimensional variant achieves the lowest EER of 0.0135 and the highest Acc. of 0.9867. These results indicate that the attack is not restricted to a specific speaker encoder or embedding dimensionality. V Conclusion This paper investigates whether speech tokens exposed by end-to-end speech language models leak sufficient voiceprint information to recover speaker embeddings. We formulate this risk as a speaker inversion attack and propose SpInv, a two-stage method that uses AuB to construct and aggregate discrete or continuous frontend representations and recover embeddings in an attacker-specified speaker space. We evaluate SpInv against Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on VoxCeleb2-test and VoxCeleb1. The results show that exposed frontend representations retain substantial speaker information: with only three seconds of observation, SpInv achieves strong recovery on unseen speakers, and the three token-based frontends generally reach cosine similarities above 0.70. This study has two main limitations. First, it focuses on recovering speaker embeddings rather than reconstructing waveforms that reproduce a victim’s acoustic characteristics. Second, the experiments are limited to the VoxCeleb corpora rather than a broader range of unconstrained in-the-wild speech datasets. Future work will investigate privacy-preserving tokenizers that better suppress voiceprint information while retaining the content needed by downstream speech-language tasks. We also plan to study stronger attacks that recover waveforms from exposed speech representations. VI Ethical Statement This work is intended to identify and quantify voiceprint-privacy risks in exposed speech representations so that safer speech interfaces can be developed. All experiments use publicly available research datasets and pretrained models, and results are reported only in aggregate. We do not attempt to identify, contact, or impersonate individuals represented in the evaluation data, nor do we reconstruct their speech waveforms. Nevertheless, speaker inversion could be misused for unauthorized profiling or identity inference. Any release or application of such techniques should follow applicable dataset licenses, consent requirements, and data-protection regulations, and should incorporate access controls that prevent attacks on non-consenting users. Our findings are presented to motivate privacy-preserving tokenizers and stronger safeguards for deployed speech-language systems. References [1] J. L. Ba, J. R. Kiros, and G. E. Hinton (2016) Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §I-C. [2] A. Bardes, J. Ponce, and Y. LeCun (2022) VICReg: variance-invariance-covariance regularization for self-supervised learning. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-D. [3] L. A. Bauer, W. Bao, M. Jadhav, and V. Bindschaedler (2025) Inference attacks for x-vector speaker anonymization. In Proc. IEEE Secur. Priv. Workshops (SPW), p. 152–159. Cited by: §I-C. [4] Boson AI (2026) Higgs TTS 3: conversational speech for voice AI from Boson AI. Note: https://huggingface.co/bosonai/higgs-tts-3-4b Cited by: §I-A, §IV-A. [5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), p. 9650–9660. Cited by: §I-D. [6] Y. Chen, S. Zheng, H. Wang, L. Cheng, Q. Chen, C. Deng, S. Zhang, and W. Wang (2025) Self-distillation prototypes network: learning robust speaker representations without supervision. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 1–5. Cited by: §I-B. [7] Y. Chen, S. Zheng, H. Wang, L. Cheng, Q. Chen, and J. Qi (2023) An enhanced Res2Net with local and global feature fusion for speaker verification. In Proc. Annu. Conf. Int. Speech Commun. Assoc. (Interspeech), p. 2228–2232. Cited by: §I-B, §IV-A, §IV-D. [8] Y. Chen, S. Zheng, H. Wang, L. Cheng, and Q. Chen (2023) Pushing the limits of self-supervised speaker verification using regularized distillation framework. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 1–5. Cited by: §I-B. [9] Y. Chen, H. Lent, and J. Bjerva (2024) Text embedding inversion security for multilingual language models. In Proc. Annu. Meet. Assoc. Comput. Linguist. (ACL), p. 7808–7827. Cited by: §I-C. [10] J. S. Chung, A. Nagrani, and A. Zisserman (2018) VoxCeleb2: deep speaker recognition. In Proc. Annu. Conf. Int. Speech Commun. Assoc. (Interspeech), p. 1086–1090. Cited by: §I-B, §IV-A. [11] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour (2024) Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. Cited by: §I-A, §IV-A. [12] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet (2011) Front-end factor analysis for speaker verification. IEEE Trans. Audio Speech Lang. Process. 19 (4), p. 788–798. Cited by: §I-B. [13] J. Deng, J. Guo, N. Xue, and S. Zafeiriou (2019) ArcFace: additive angular margin loss for deep face recognition. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 4690–4699. Cited by: §I-D. [14] B. Desplanques, J. Thienpondt, and K. Demuynck (2020) ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Proc. Annu. Conf. Int. Speech Commun. Assoc. (Interspeech), p. 3830–3834. Cited by: §I-B, §IV-A. [15] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol. (NAACL-HLT), p. 4171–4186. Cited by: §I-C. [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-C. [17] M. Fredrikson, S. Jha, and T. Ristenpart (2015) Model inversion attacks that exploit confidence information and basic countermeasures. In Proc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), p. 1322–1333. Cited by: §I-C. [18] G. Heigold, I. Lopez-Moreno, S. Bengio, and N. Shazeer (2016) End-to-end text-dependent speaker verification. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 5115–5119. Cited by: §I-B. [19] D. Hendrycks and K. Gimpel (2016) Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415. Cited by: §I-C. [20] C. Hwang, S. Kim, Y. K. Tan, T. Liu, S. Paik, D. Kim, M. Soumik, K. M. M. Aung, and J. H. Seo (2026) Scores know Bob’s voice: speaker impersonation attack. arXiv preprint arXiv:2603.02781. Cited by: §I-C. [21] P. Kenny, G. Boulianne, P. Ouellet, and P. Dumouchel (2007) Joint factor analysis versus eigenchannels in speaker recognition. IEEE Trans. Audio Speech Lang. Process. 15 (4), p. 1435–1447. Cited by: §I-B. [22] Kimi Team, D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, et al. (2025) Kimi-Audio technical report. arXiv preprint arXiv:2504.18425. Cited by: §I-A, §IV-A. [23] O. Li, Y. Hao, Z. Wang, B. Zhu, S. Wang, Z. Zhang, and F. Feng (2024) Model inversion attacks through target-specific conditional diffusion models. arXiv preprint arXiv:2407.11424. Cited by: §I-C. [24] Y. Lu, S. Wang, G. Zhu, Z. Zhang, and J. Huang (2025) FGMIA: feature-guided model inversion attacks against face recognition models. IEEE Trans. Inf. Forensics Secur. 20, p. 8465–8480. Cited by: §I-C. [25] K. Mitsui, K. Mitsuda, T. Wakatsuki, Y. Hono, and K. Sawada (2024) PSLM: parallel generation of text and speech with LLMs for low-latency spoken dialogue systems. In Findings Assoc. Comput. Linguist. (EMNLP), p. 2692–2700. Cited by: §I-A. [26] J. X. Morris, V. Kuleshov, V. Shmatikov, and A. M. Rush (2023) Text embeddings reveal (almost) as much as text. In Proc. Conf. Empir. Methods Nat. Lang. Process. (EMNLP), p. 12448–12460. Cited by: §I-C. [27] E. Nachmani, A. Levkovitch, R. Hirsch, J. Salazar, C. Asawaroengchai, S. Mariooryad, E. Rivlin, R. J. Skerry-Ryan, and M. Tadmor Ramanovich (2024) Spoken question answering and speech continuation using spectrogram-powered LLM. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: §I-A. [28] A. Nagrani, J. S. Chung, and A. Zisserman (2017) VoxCeleb: a large-scale speaker identification dataset. In Proc. Annu. Conf. Int. Speech Commun. Assoc. (Interspeech), p. 2616–2620. Cited by: §IV-A, §IV-A. [29] K. Pizzi, F. Boenisch, U. Sahin, and K. Böttinger (2022) Introducing model inversion attacks on automatic speaker recognition. In Proc. Symp. Secur. Priv. Speech Commun. (SPSC), p. 11–16. Cited by: §I-C. [30] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn (2000) Speaker verification using adapted gaussian mixture models. Digit. Signal Process. 10 (1–3), p. 19–41. Cited by: §I-B. [31] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur (2018) X-vectors: robust DNN embeddings for speaker recognition. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 5329–5333. Cited by: §I-B. [32] E. Tsaprazlis, T. Lertpetchpun, T. Feng, S. P. Karimireddy, and S. Narayanan (2026) VoxGuard: evaluating user and attribute privacy in speech via membership inference attacks. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 19042–19046. Cited by: §I-C, §I-A. [33] E. Variani, X. Lei, E. McDermott, I. Lopez Moreno, and J. Gonzalez-Dominguez (2014) Deep neural networks for small footprint text-dependent speaker verification. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 4052–4056. Cited by: §I-B. [34] L. Wan, Q. Wang, A. Papir, and I. Lopez Moreno (2018) Generalized end-to-end loss for speaker verification. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), p. 4879–4883. Cited by: §I-B. [35] H. Wang, S. Zheng, Y. Chen, L. Cheng, and Q. Chen (2023) CAM++: a fast and efficient network for speaker verification using context-aware masking. In Proc. Annu. Conf. Int. Speech Commun. Assoc. (Interspeech), p. 5301–5305. Cited by: §I-B, §IV-A, §IV-D. [36] J. Wang, K. Luo, L. Lin, W. Wang, Y. Chen, M. Aloqaily, X. Tang, Z. Zhou, K. Wang, L. Sun, and Q. Wen (2026) HearSay benchmark: do audio LLMs leak what they hear?. In Findings Assoc. Comput. Linguist. (ACL), p. 19312–19331. Cited by: §I-C, §I-A. [37] J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, et al. (2025) Qwen3-Omni technical report. arXiv preprint arXiv:2509.17765. Cited by: §I-A, §IV-A. [38] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu (2023) SpeechGPT: empowering large language models with intrinsic cross-modal conversational abilities. In Findings Assoc. Comput. Linguist. (EMNLP), p. 15757–15773. Cited by: §I-A. [39] Y. Zhang, R. Jia, H. Pei, W. Wang, B. Li, and D. Song (2020) The secret revealer: generative model-inversion attacks against deep neural networks. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 253–261. Cited by: §I, §I-C. [40] H. Zhu, L. Ye, W. Kang, Z. Yao, L. Guo, F. Kuang, Z. Han, W. Zhuang, L. Lin, and D. Povey (2026) OmniVoice: towards omnilingual zero-shot text-to-speech with diffusion language models. arXiv preprint arXiv:2604.00688. Cited by: §I-A, §IV-A.