Paper deep dive
Controllable Accent Normalization via Discrete Diffusion
Qibing Bai, Yuhan Du, Tom Ko, Shuai Wang, Yannan Wang, Haizhou Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:07:24 AM
Summary
DLM-AN is a controllable accent normalization system that utilizes masked discrete diffusion over self-supervised speech tokens. It introduces a Common Token Predictor (CTP) to identify and selectively reuse native-like source tokens for accent strength control, and a Duration Ratio Predictor (DP) to adjust speech rhythm, achieving superior word error rates and interpretable control over accent retention.
Entities (5)
Relation Signals (3)
DLM-AN → incorporates → Common Token Predictor
confidence 100% · DLM-AN further incorporates a flow-matching Duration Ratio Predictor... A Common Token Predictor identifies source tokens
DLM-AN → extends → LLaDA
confidence 95% · DLM-AN extends the LLaDA diffusion language model to speech
DLM-AN → uses → WavLM
confidence 90% · We leverage WavLM to extract discrete tokens for conversion and synthesis
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing accent normalization methods do not typically offer control over accent strength, yet many applications-such as language learning and dubbing-require tunable accent retention. We propose DLM-AN, a controllable accent normalization system built on masked discrete diffusion over self-supervised speech tokens. A Common Token Predictor identifies source tokens that likely encode native pronunciation; these tokens are selectively reused to initialize the reverse diffusion process. This provides a simple yet effective mechanism for controlling accent strength: reusing more tokens preserves more of the original accent. DLM-AN further incorporates a flow-matching Duration Ratio Predictor that automatically adjusts the total duration to better match the native rhythm. Experiments on multi-accent English data show that DLM-AN achieves the lowest word error rate among all compared systems while delivering competitive accent reduction and smooth, interpretable accent strength control.
Tags
Links
- Source: https://arxiv.org/abs/2603.14275v1
- Canonical: https://arxiv.org/abs/2603.14275v1
Trouble viewing inline? Open PDF directly →
Full Text
57,498 characters extracted from source content.
Expand or collapse full text
Controllable Accent Normalization via Discrete Diffusion Qibing Bai 1,5 , Yuhan Du 4 , Tom Ko 6 , Shuai Wang 4,6,∗ , Yannan Wang 5 , Haizhou Li 2,3,6 1 SDS, 2 SAI, and 3 SRIBD, The Chinese University of Hong Kong, Shenzhen, China 4 School of Intelligence Science and Technology, Nanjing University, Suzhou, China 5 Tencent Ethereal Audio Lab, Tencent, Shenzhen, China 6 Shenzhen Loop Area Institute, Shenzhen, China qibingbai@link.cuhk.edu.cn, shuaiwang@nju.edu.cn Abstract Existing accent normalization methods do not typically offer control over accent strength, yet many applications—such as language learning and dubbing—require tunable accent reten- tion. We propose DLM-AN, a controllable accent normalization system built on masked discrete diffusion over self-supervised speech tokens. A Common Token Predictor identifies source tokens that likely encode native pronunciation; these tokens are selectively reused to initialize the reverse diffusion process. This provides a simple yet effective mechanism for control- ling accent strength: reusing more tokens preserves more of the original accent. DLM-AN further incorporates a flow-matching Duration Ratio Predictor that automatically adjusts the total duration to better match the native rhythm. Experiments on multi-accent English data show that DLM-AN achieves the lowest word error rate among all compared systems while de- livering competitive accent reduction and smooth, interpretable accent strength control 1 . Index Terms: accent conversion, diffusion language model, speech synthesis, voice conversion, controllability 1. Introduction Accent conversion (AC) seeks to alter speech from one accent to another while preserving the speaker’s characteristics. A spe- cial case, accent normalization (AN) 2 , converts non-native (L2) accented speech into a native (L1) accented form. AN tech- nology enables a wide range of applications, including pronun- ciation training for language learners [1], authentic dubbing in multimedia [2], and personalized text-to-speech systems [3]. Early deep learning approaches for AN are reference- based [4–7], relying on native accent speech samples to gen- erate accent-neutral representations via PPG features [4, 5, 7] or native TTS [6]. Reference-free methods [8–10] eliminate this requirement by directly mapping between accented and native representations, though they still rely on parallel data. Subsequent work removes the parallel data constraint through ASR–TTS pipelines [11], accent feature disentanglement [12], or TTS-guided representations [13, 14], further augmented with flow matching [15] and normalizing flow [16]. However, these approaches depend on TTS-synthesized targets, whose quality can be limited by voice cloning and duration modeling errors. Recent token-based methods [17–20] offer a promising al- ternative. TokAN [19] quantizes speech into self-supervised discrete tokens, performs autoregressive token-to-token con- version, and recovers waveforms via a flow-matching synthe- ** indicates the corresponding author. 1 Samples: https://P1ping.github.io/dlman-demo/ 2 Also referred to as foreign accent conversion (FAC). sizer. CosyAccent [21] adopts a non-autoregressive direct flow- matching approach and introduces a “source-synthesis” data strategy. Both systems support total-duration control. How- ever, none of the above methods provides control over accent strength—a desirable capability for gradual accent reduction or accent preservation as part of speaker identity. The only attempt at accent strength control [22] manipulates the starting timestep of a continuous diffusion process, but it operates under a frame- to-frame framework with fixed duration, lacking fine-grained rhythm adjustability and duration control. In this paper, we propose DLM-AN, a controllable accent normalization system based on masked discrete diffusion over self-supervised speech tokens. DLM-AN extends the LLaDA diffusion language model [23] to speech: a bidirectional Trans- former iteratively predicts masked tokens conditioned on con- tent representations from a CTC-guided [24] token encoder. The key insight enabling controllability is that, under a phonet- ically rich tokenizer, native and accented renditions of the same utterance share many tokens in similarly pronounced regions but differ in accent-affected regions. We introduce a Common Token Predictor (CTP) that identifies tokens likely shared with the native target. By reusing high-confidence source tokens to initialize the masked sequence, users can smoothly control accent strength—from full normalization (all tokens generated from scratch) to near-resynthesis (all source tokens preserved). A flow-matching Duration Ratio Predictor further provides ex- plicit control over the total output duration. Our contributions are as follows: • We propose the first accent normalization system based on discrete diffusion, enabling iterative token generation condi- tioned on phonemically guided content representations. • We introduce a common token predictor that provides smooth, interpretable accent strength control through threshold-based source token reuse. • We demonstrate that DLM-AN achieves the best content preservation (lowest WER) among all compared systems, while offering competitive naturalness and accent reduction, along with robust duration scaling. 2. Related Work 2.1. Controllability in Accent Conversion Most accent conversion/normalization systems perform a one- shot “full” accent shift without a user-controllable knob [5, 20]. Recently, controllability has attracted increasing attention, mo- tivated by applications such as gradual accent reduction in lan- guage learning and adjustable accent retention in dubbing. One line of work studies duration control (speech-rate con- trol) during accent normalization. TokAN [19] performs token- arXiv:2603.14275v1 [eess.AS] 15 Mar 2026 to-token conversion with self-supervised discrete units and sup- ports duration preservation. CosyAccent [21] further enables explicit duration scaling via a duration-ratio predictor and pro- poses a source-synthesis data strategy to reduce reliance on TTS-synthesized supervision artifacts. In contrast, controllable accent strength (i.e., smoothly trading off normalization vs. retaining the original L2 accent) remains less explored. FAC-FACodec [22] introduces an inten- sity control mechanism in a diffusion-based, factorized-codec framework by using the diffusion starting timestep as a user knob. Related directions include fine-grained controllable ac- cent transfer models [25], controllable accented TTS that ren- ders accent intensity at coarse/fine levels [26], and scalable ac- cented TTS with automated accent label discovery [27]. Our work leverages masked discrete diffusion over speech tokens to support both total-duration control and an inter- pretable accent-strength knob via source-token reuse; the iter- ative masking/unmasking procedure also naturally supports lo- calized correction (speech infilling). 2.2. Self-Supervised Speech Tokens Discrete tokens derived from self-supervised learning (SSL) representations [28, 29] correlate strongly with phonetic con- tent [30], which makes them attractive units for speech genera- tion and conversion. Such tokens have been adopted in token- based voice conversion [31–33] and high-fidelity speech gener- ation/TTS frameworks [34]. More broadly, discreteness allows importing “text-like” modeling techniques into speech, includ- ing spoken language modeling [35], direct speech-to-speech translation [36], and speech-centric LLM systems [37]. Recent work also explores SSL tokens for ASR, in- cluding multilingual ASR [38], contextual ASR with Zip- former [39], and accent-robust discrete-token ASR model- ing [40]. For accent conversion/normalization, discrete tokens have been studied under different supervision regimes, includ- ing zero-shot/minimally-supervised conversion [18], pseudo- parallel mapping [17, 19], and prompt-based imitation [20]. We leverage WavLM [29] to extract discrete tokens for conversion and synthesis, capitalizing on its strong phonetic encoding and noise robustness. 2.3. Discrete Diffusion Discrete diffusion has recently been investigated for text gen- eration (i.e., diffusion language models), adapting continuous diffusion principles to categorical data like tokens. Two com- monly used formulations are: (i) multinomial/categorical dif- fusion, where tokens transition probabilistically across the vo- cabulary [41, 42], and (i) masked/absorbing diffusion, where tokens are progressively mapped to a special absorbing state (e.g., [MASK]) and then iteratively recovered [23, 42]. For the absorbing diffusion process, consider a sequence of discrete tokens x 0 ∈ 1,...,V n , where V is the vocabulary size and n is the sequence length. The forward process adds noise independently per token over T timesteps: q(x t | x t−1 ) = n Y i=1 h (1− β t )δ x i t =x i t−1 + β t δ x i t =[MASK] i (1) where β t ∈ (0, 1) is the time-dependent masking probability, and [MASK] is an additional absorbing state (V + 1). The cumulative forward process is: q(x t | x 0 ) = Cat (x t ; ̄α t x 0 + (1− ̄α t )e [MASK] )(2) with ̄α t = Q t s=1 (1− β s ) and e [MASK] the one-hot vector for the mask token. At t = T , x T is fully masked with high proba- bility. The reverse process is parameterized by a neural network p θ (x t−1 | x t ), typically a Transformer predicting a categorical distribution over tokens. Masked diffusion is widely adopted for diffusion language models (e.g., LLaDA [23]), where the forward process ran- domly masks tokens at varying ratios, and the reverse pro- cess predicts masked positions in parallel using a bidirec- tional Transformer.This provides a principled variational lower bound on likelihood, contrasting autoregressive mod- els. Compared with masked generation (e.g., MaskGIT [43], MaskGCT [44], Metis [45]), masked diffusion has a better the- oretical foundation, as it optimizes an evidence lower bound (ELBO) and supports iterative refinement. The log-likelihood admits the evidence lower bound (ELBO): logp(x 0 )≥E q(x 1:T |x 0 ) log p θ (x 0:T ) q(x 1:T | x 0 ) (3) Maximizing the ELBO is equivalent to minimizing the follow- ing loss, which decomposes into a reconstruction term at t=0 and KL divergences for t > 0: L(θ) =E[− logp θ (x 0 | x 1 ) + T X t=2 D KL (q(x t−1 | x t , x 0 )∥p θ (x t−1 | x t )) + D KL (q(x T | x 0 )∥p(x T ))](4) In practice, a simplified loss is used, often reweighted cross- entropy on predicting the original tokens from noisy inputs. For instance, LLaDA derives a continuous-time upper bound on the negative log-likelihood. For t∼U [0, 1], the loss is L(θ) =−E t,x 0 ,x t 1 t X i:x i t =[MASK] logp θ (x i 0 | x t ) (5) where the expectation is approximated via Monte Carlo sam- pling over uniform t (masking rate), data x 0 , and corrupted x t . The 1/t reweighting arises from the continuous-time ELBO derivation, ensuring each masking rate contributes proportion- ally to the bound. Surveys on discrete diffusion highlight its growing role in LLMs [46]. Notable directions include SEDD [47], which ex- tends score matching to discrete spaces through score entropy, and ReMDM [48], which enables iterative refinement via re- masking of previously generated tokens. Entropy-bounded un- masking [49] further accelerates sampling from masked dif- fusion models. More recently, corrective diffusion language models [50–53] explicitly supervise visible incorrect tokens, en- abling discriminative confidence and targeted correction in par- allel decoding scenarios. 3. Methodology We propose to use a diffusion language model (DLM) for con- trollable accent normalization. The method is shortened “DLM- AN”. Figure 1 shows the pipeline of DLM-AN. The SSL tok- enizer extracts SSL representations from the L2-accented wave- form and quantize the features into discrete tokens. A Trans- former token encoder takes further processs these tokens, pro- ducing continuous content representations. To make the con- tent representations phonemic enough, a CTC-based phonemic Token Encoder DLM Decoder SSL Tokenizer m aɪ _ n ˈeɪ m z _ f ˈɜː ɡ ə s ə n Common Token Predictor Duration Ratio Predictor 0.0 0.3 0.8 0.1 0.9 0.83 Target Init. M M M Speech Synthesizer duration scaling (optional) Source speech Target MelL2-accented token L1-accented tokenContent feature Speaker embedding Masked tokenMm aɪ _ ...auxiliary text0.83duration scaling ratio 0.0 ... 0.9 L1 confidence Figure 1: Overview of the DLM-AN pipeline. The SSL tokenizer extracts discrete tokens from L2-accented speech. A Transformer token encoder with CTC-based phonemic guidance produces content representations, which are fed into the Common Token Predictor (CTP), Duration Ratio Predictor (DP), and the DLM decoder. The DLM decoder iteratively generates the target token sequence, optionally initialized with high-CTP-confidence source tokens. A flow-matching synthesizer and vocoder produce the final waveform. guidance is imposed upon them, shown as the auxiliary text in Figure 1. The content representations are further taken as the input for three modules: Common Token Predictor (CTP), Du- ration Ratio Predictor (DP), and the DLM decoder. Note that the content features are concatenated with the source tokens (embeddings) when feeding to CTP and DP, in order to provide detailed pronunciation patterns (i.e., phonetic information). This concatenation is not displayed in Figure 1 for brevity. CTP predicts whether each token is common in both the source and target sequences. The higher the score for a token, the more probable this token is associated with a native pronun- ciation, as will be demonstrated in the subsequent sections. To- kens with high CTP scores could be optionally “reused” for the decoding/generation process later. DP predicts the ratio of the total duration: dur tgt /dur src . This ratio can be optionally used to determine the total duration/length of the target sequence. Given the target duration ratio, either predicted, arbitrarily specified, or kept 1.0 to maintain the source duration, the length of the initial target sequence is determined. By default, the ini- tial target sequence is purely filled with [MASK] for generation from scratch. Optionally, based on the CTP scores and a given threshold or proportion, certain soruce tokens can be reused to initialize the target sequence. After the initial target sequence is determined, the DLM decoder iteratively generates the entire target sequence, conditioned on the content features from the token encoder. The more tokens reused in the target sequence, the more source accent is expected to be preserved. The speech syn- thesizer further generates the corresponding Mel-spectrogram given the target tokens, conditioned on the speaker embedding extracted from the input source speech. The Mel-spectrogram can be converted to a waveform using the HiFT vocoder [54]. 3.1. Discrete Diffusion for Speech Tokens We extend the LLaDA masked diffusion language model [23] to discrete speech tokens for controllable accent normalization. Our forward corruption process uses pure absorbing masking, parameterized by a timestep t∼U [0, 1] per sequence. Let y 0 ∈ 1,...,V L be a clean speech token sequence, where V is the speech vocabulary size and L is the sequence length. We define the absorbing masking rate as λ = (1−ε)t + ε, with ε > 0. Each position i is masked independently: q λ (z i | y i 0 ) = λδ z i =[MASK] + (1− λ)δ z i =y i 0 (6) which induces a masked index set M and a visible set ̄ M = [L]\ M . The corrupted sequence z has corruption setC = M . The model p θ (y 0 | z, c) is a bidirectional Transformer that predicts the original tokens from z, conditioned on the content representations c from the token encoder. The training objective follows LLaDA: L(θ) =−E t,y 0 ,z " 1 λ X i∈M logp θ (y i 0 | z, c) # (7) approximated by Monte Carlo sampling with global per-token normalization for stability. This objective optimizes a prin- cipled upper bound on the negative log-likelihood, enabling strong bidirectional reasoning and iterative refinement. The DLM decoder has a decoder-only Transformer struc- ture without causal masking, allowing parallel prediction over masked positions.The content representations c are inte- grated as conditional inputs via self-attention to guide accent- normalized generation while preserving content. During inference, we employ a greedy sampling algo- rithm with optional initialization: start from a fully or par- tially masked sequence (reusing high-CPT-confidence tokens), and iteratively predict and unmask positions based on confi- dence scores, conditioned on c. For enhanced quality, we use classifier-free guidance (CFG) [55]: compute both conditional logitsp θ (y 0 | z, c) and unconditional logitsp θ (y 0 | z,∅), then combine them as p cfg = (1 + w DLM )p θ (· | z, c)− w DLM p θ (· | z,∅), where w DLM is the guidance strength. This amplifies the conditioning signal for better content preservation. The structure of DLM decoder is shown in Figure 2. It features a self-attention-only structure (i.e., using self-attention to fuse the conditional information). The input is basically two sequences: the content features and the noised target to- kens. Three special tokens, [START], [TASK], and [END], are added to wrap and separate the two sequences, guiding the DLM behavior. To make the condition reusable across time steps (i.e., to- ken sequences with different numbers of masked tokens), the DLM Decoder STMME Speech tokenContent feature ETSStart tokenTask tokenEnd token MMasked token Condition attention regionToken attention region STE Figure 2: Structure of the DLM decoder. The input consists of the content features and the noised target tokens, separated by special tokens [START], [TASK], and [END]. The content features are mutually attentive but do not attend to the token sequence (pink region), while the token sequence attends to the entire input (green region). content features are mutually attentive but with no attention to the token sequence. In contrast, the token sequence can attend to the entire input sequence. The attention regions of the two parts are depicted in Figure 2 with the colors pink and green, respectively. 3.2. Common Token Prediction With a sufficiently phonetic tokenizer, utterances spoken with different accents tend to share many tokens in similarly pro- nounced regions, while differing mainly in accent-affected re- gions. Motivated by this property, we introduce a Common To- ken Predictor (CTP) that assigns each source token a confidence score indicating how likely it is to be shared with the (native) target. Tokens with high CTP confidence can be reused to ini- tialize the target sequence, providing a simple and interpretable control of accent strength: reusing more tokens preserves more of the source accent. We formulate CTP as a sequence-tagging Target tokens Source tokens Common token labels 503 902 622 955 61 1 94 94 415 415 1018 503 902 902 902 577 955 94 456 932 375 1018 1 0 1 0 0 1 1 0 0 0 1 Figure 3: Extraction of common token labels via the longest common subsequence (LCS) between source and target token sequences. For consecutive identical tokens with differing du- rations, center-mode alignment is applied (dashed rectangle). problem. Given paired source and target token sequences, we derive binary labels by computing the longest common subse- quence (LCS) between them. LCS has been used previously to evaluate accent conversion models [18]; here we use it to iden- tify which source tokens are shared with the target. The label extraction procedure is illustrated in Figure 3. We obtain the LCS via dynamic programming and backtracking. For consec- utive identical tokens that have different durations in the source and target, we apply a center alignment: only the centered to- kens are marked positive (dashed rectangle in Figure 3). The training objective for CTP is binary cross-entropy loss. Let S be the source sequence length, and let l∈0, 1 S be the binary labels derived from the LCS searching. Given the source content features, the CTP module (parameterized by φ) outputs predicted probabilities ˆ l∈ [0, 1] S . The loss is: L CTP (φ) =− 1 S S X i=1 h l i log ˆ l i + (1− l i ) log(1− ˆ l i ) i .(8) 3.3. Duration Ratio Prediction Because L2-accented speech often exhibits different rhythm and speaking rate, directly inheriting the source total duration can lead to sub-optimal naturalness. DLM-AN therefore includes a duration ratio predictor (DP) that estimates the global dura- tion ratio r = dur tgt /dur src . DP uses a diffusion Transformer (DiT) [56] backbone with an attentive pooling layer, and is trained with conditional flow matching. The training objective for DP is conditional flow matching loss [57]. Let r > 0 be the target duration ratio (ground truth dur tgt /dur src ), and let c be the input content representations from the token encoder. The flow matching model v ψ (u t ,t, c) (pa- rameterized by ψ) predicts the velocity field: u t = (1− t)u 0 + tr,(9) where t∼U [0, 1] and u 0 ∼N (0, 1) is a standard normal prior. The loss is: L DP (ψ) =E t,u 0 [∥v ψ (u t ,t, c)− (r− u 0 )∥ 2 ](10) This objective trains the model to generate duration ratios con- ditioned on c, enabling global rhythm adjustment. In practice, we also condition DP on the source token embeddings to cap- ture fine-grained pronunciation details; we omit this from the formulation for brevity. 3.4. Phoneme Guidance and Joint Training The token encoder is a Transformer with relative positional em- beddings, producing content representations c. The CTP and DP modules also use Transformer backbones with relative po- sitional embeddings. The DLM decoder is a self-attention-only Transformer with two-block masking, using rotary positional encoding (RoPE) [58] for the entire sequence. To encourage c to be phonemically informative, we impose a CTC-based phonemic guidance on the token encoder outputs. Concretely, we attach a linear projection head on top of the to- ken encoder to predict phoneme logits, and compute a CTC loss against phoneme label sequences derived from the correspond- ing transcripts. Given phoneme sequence p, the loss is L CTC = CTC(Linear(c), p)(11) We train the token-prediction modules jointly, including the token encoder, CTP, DP, and DLM decoder, following a two- stage schedule of pretraining and fine-tuning as described in the experimental setup. Let L DLM denote the masked discrete diffusion loss for token generation (Eq. (7)). The overall joint- training objective is: L token =L DLM + β 1 L DP + β 2 L CTP + β 3 L CTC (12) The token-prediction modules are pre-trained on native- only data and fine-tuned on semi-synthesized parallel data. abushchiefhaddiedanaturaldeath SILahSILbuhuwshchiyfhhfhhSILb ay dahnaechterdehs Frequency (Mel) 0.0 0.2 0.4 0.6 0.8 1.0 CTP Probability CTP Prob. Figure 4: Visualization of common token prediction for a Chinese-accented sample. CTP confidence values are overlaid on the Mel- spectrogram. PPG-predicted phonemes are shown below and their boundaries (white dashed lines) are overlaid on the spectrogram. Aligned words are shown at the bottom. Regions with prominent L2 accent (e.g., prolonged “a”, unclear “had”, /S/-like ending of “death”) receive low CTP confidence. 3.5. Sampling Algorithm for Token Conversion After processing the source L2-accented tokens with the token encoder, CTP, and DP, we can then initialize the target token sequence and use the DLM decoder to complete it (i.e., “Target Init.” and “DLM Decoder” in Figure 1). By default, we use the greedy sampler with a threshold-based strategy for reusing source tokens, as demonstrated in Sec. 5.1.2 and Algorithm 1. Algorithm 1 Greedy sampling with CTP-based initialization. Require: Source tokens y src (length N src ) and duration ratio r CTP scoresl i N src i=1 , reuse threshold τ , sampling steps T Content representations c and CFG strength w DLM Ensure: Target tokens y tgt 1: N tgt ← round(N src · r) 2: K ←⌈N tgt /T⌉▷ tokens to unmask per step 3: I ←i| l i > τ▷ reused source-token indices 4: Initialize z (0) by nearest interpolation from source to target length: z (0) ∈1,...,V,[MASK] N tgt 5: for j = 1 to N tgt do 6: i ⋆ ← round (j− 1 2 ) N src N tgt + 1 2 ▷ source index 7: if i ⋆ ∈I then 8:z (0) j ← y src i ⋆ 9: else 10:z (0) j ← [MASK] 11: end if 12: end for 13: N mask ←|j | z (0) j = [MASK]| 14: T eff ←⌈N mask /K⌉, s 0 ← max(1, T − T eff + 1) ▷ start step from reuse proportion 15: for s = s 0 to T do 16:Compute logits at masked positions: ℓ cond (·| z (s−1) , c) and ℓ uncond (·| z (s−1) ) 17:Apply CFG: ℓ cfg ← (1 + w DLM )ℓ cond − w DLM ℓ uncond 18:For each masked position j, set ˆy j ← arg maxℓ cfg (j) and confidence γ j ← max softmax(ℓ cfg (j)) 19:Select J as the top-min(K, |masked|) masked posi- tions by γ j 20:Unmask: set z (s) j ← ˆy j for j ∈ J ; keep all other positions unchanged 21: end for 22: return y tgt ← z (T) 3.6. Token-to-Speech Synthesis We use a flow-matching speech synthesizer with a vocoder [54] to generate waveforms. The input token sequence is encoded by a relative-positional Transformer encoder, and the encoded features are concatenated with a speaker embedding before be- ing fed into a DiT decoder. The speech synthesizer is trained separately on native-only speech data. Similar to CosyAccent, DLM-AN deploys a two-way CFG strategy for generation: ̄v η (x t ,t, y, s) = v η (x t ,t, y, s) + w 1 (v η (x t ,t, y, s)− v η (x t ,t,∅, s)) + w 2 (v η (x t ,t, y, s)− v η (x t ,t, y,∅)) (13) where v η is the synthesizer, t∈ [0, 1] is the time variable, x t is the Mel-spectrogram at time t, y is the input tokens, and s is the speaker embedding. The two CFG factors w 1 and w 2 control the emphasis on the content and timbre conditions, respectively. 4. Experimental Setup 4.1. Datasets The experiments are conducted on English. Training uses the English subset of Emilia [59] (Emilia-EN) and the LibriTTS-R corpus [60] with synthesized L2-accented counterparts 3 [21]. We also use the L2-ARCTIC corpus [61] together with four American speakers from ARCTIC [62]. We further synthe- size pseudo native targets for this extended L2-ARCTIC set, which are used for supervised fine-tuning and evaluation. The pseudo native targets are generated using a native-only zero- shot Matcha-TTS [63] model trained on LibriTTS-R. Emilia-EN is solely used for pretraining. LibriTTS-R is used to train the SSL tokenizer and the flow-matching speech synthesizer. For fine-tuning, both augmented LibriTTS-R (with synthesized source utterances) and extended L2-ARCTIC (with synthesized targets) are utilized. Since the L2-accented counterparts [21] of LibriTTS-R were synthesized using prompts drawn from L2-ARCTIC, we adopt the same train-valid-test partition of L2-ARCTIC to pre- vent text leakage. 4.2. Tokenizer The proposed method relies on the phonetic richness of the speech tokens. We use WavLM large and extract layer-22 repre- sentations for tokenization. We train an online K-Means model with 1024 clusters (codebook size 1024) on LibriTTS-R. 3 https://huggingface.co/datasets/Piping/L2-LibriTTSR/ 0.00.20.40.60.81.0 Proportion (a) 11 12 13 14 15 WER (%) WER CTP-based Random 0.00.20.40.60.81.0 Proportion (b) 0.275 0.300 0.325 0.350 0.375 0.400 0.425 0.450 PPG PPG w/ Target CTP-based Random 0.00.20.40.60.81.0 Proportion (c) 0.30 0.32 0.34 0.36 0.38 0.40 0.42 0.44 PPG PPG w/ Source CTP-based Random Figure 5: CTP-based vs. random token selection at varying reuse proportions. Three metrics are compared: (a) WER, (b) ∆PPG with the L1-accented target, and (c) ∆PPG with the L2-accented source. CTP-based selection achieves generally lower WER and consistently better accent separation than random selection at the same proportion. 4.3. Compared Systems We evaluate our model against two strong baselines: • TokAN [19]: An autoregressive model operating on dedupli- cated tokens. It recovers token-wise durations in the speech synthesis stage. This model features a flow matching dura- tion predictor with two conditions: 1) the token sequence and 2) the average token duration. When provided with the aver- age token duration, TokAN is able to preserve the total du- ration. We test two modes: TokAN-1, which predicts token durations directly, and TokAN-2, which predicts with total- duration awareness and preserves the total duration. • CosyAccent [21]: A non-autoregressive direct flow-matching model. It features a total-duration ratio predictor similar to our proposed model. It can be regarded as a continuous- diffusion counterpart of the proposed model, but without accent-strength control. We test two modes: CosyAccent- 1, which predicts the total duration ratio, and CosyAccent-2, which inherits the source total duration. For the token-reuse setting of DLM-AN, we deploy a threshold-based selection strategy: tokens with CTP confidence higher than a threshold (τ ) are reused. For simplicity, we do not apply token reuse and duration scaling simultaneously. We evaluate four configurations of DLM-AN: • DLM-AN-1: Uses the predicted total duration ratio. • DLM-AN-2 (τ = 1.0): Inherits the source total duration while predicting the target sequence from scratch. • DLM-AN-2 (τ = 0.3): Inherits the source total duration, with a threshold for common token prediction (i.e., τ = 0.3). • DLM-AN-2 (τ = 0.0): Inherits all the source tokens (i.e., direct resynthesis). For TokAN and DLM-AN, the two token-based systems, we pretrain them on Emilia-EN. BART-style [18, 64] token cor- ruption is applied to the source token sequence. This helps avoid teaching the model to trivially copy the source sequence. Different from the original TokAN, no source accent embed- ding is utilized. Given the corrupted source sequence, TokAN performs autoregressive generation while DLM-AN performs masked generation. TokAN and DLM-AN share a similar ar- chitecture: a speech token encoder with CTC-based phonemic guidance, a self-attention-only decoder, and a flow-matching speech synthesizer. TokAN predicts token-wise durations in the synthesizer module, whereas DLM-AN predicts the total dura- tion ratio during token-level conversion. For DLM-AN, we set the loss weights in Eq. (12) asβ 1 =1.0, β 2 =1.0, andβ 3 =0.2 for joint training of the token-prediction modules. For CTP train- ing, we use a positive weight of 2 for better label balance. For TokAN generation, we use beam search with a beam size of 10. For DLM-AN, we use the greedy sampler with 32 steps; the CFG guidance strength w DLM is set to 1.0. For the speech synthesizers in TokAN and DLM-AN, we use the Euler sampler with 32 steps. The CFG strengths w 1 and w 2 are set to 1.0 and 1.0, respectively. For CosyAccent, we use the official Whisper medium model [65] as the frozen speech frontend. CosyAccent is also pre-trained on Emilia-EN, with the encoder supervised only by the CTC loss. During inference, we use the same CFG weights and number of sampling steps as in the original paper. Resemblyzer 4 is deployed to extract speaker embeddings for the speech synthesizer modules in all the compared mod- els.The final waveform is generated using the HiFTNet vocoder [54] from CosyVoice2 [66]. 4.4. Evaluation Data & Metrics Evaluation Set. Our test set involves the extended L2-ARCTIC dataset, as described in Sec. 4.1. The test set covers seven ac- cents: Arabic, Chinese, Hindi, Korean, Spanish, Vietnamese, and native American English. The set contains 80 sentences. The partitioning is consistent with the source-synthesis training data [21], with no text leakage in the training data. Subjective Evaluation. We conducted listening tests with 25 raters to assess three qualities: Naturalness (NAT) and Accent- edness (ACT) were measured via MUSHRA tests. The native accent was excluded from the ACT evaluation. Speaker Simi- larity (SIM) was measured via Best-Worst Scaling (BWS), with scores aggregated using a standard counting algorithm [67]: (N best − N worst )/N occurrence . Objective Evaluation. We use four objective metrics to as- sess conversion quality automatically. Intelligibility: Word Er- ror Rate (WER) from a native-only ASR model 5 to simulate listener perception. Naturalness: The UTMOSv2 score 6 from a neural naturalness predictor. Timbre Preservation: Speaker En- coding Cosine Similarity (SECS) using the accent-robust Re- semblyzer. Accentedness Reduction: The phonetic posterior- gram distance (∆PPG) 7 [68]. By default, ∆PPG is computed between generated utterances and the synthesized native targets, but some analyses also compute it against the source to measure how much accent is removed. 4 https://github.com/resemble-ai/Resemblyzer 5 https://huggingface.co/facebook/s2t-medium-librispeech-asr 6 https://github.com/sarulab-speech/UTMOSv2 7 https://github.com/interactiveaudiolab/ppgs Table 1: Evaluation results of accent normalization systems. Source-length indicates whether the source total duration is preserved. τ denotes the CTP threshold for token reuse (τ =1.0: generation from scratch; τ =0.0: full token reuse / resynthesis). Best and second- best objective results are in bold and underlined . SystemSource-length SubjectiveObjective NAT (↑)ACT (↓)SIM (↑)WER (%↓)UTMOS (↑)SECS (↑)∆PPG (↓) Source✓58.36±2.8448.46±2.61-15.862.80±0.41-0.5097 TokAN-1 [19]×63.85±2.4923.58±1.85-0.08213.823.07±0.360.84950.2884 TokAN-2 [19]✓59.20±2.6028.01±2.21-0.05114.002.97±0.380.85300.2980 CosyAccent-1 [21]×61.12±2.4225.75±1.89-0.07112.402.99±0.340.82940.2736 CosyAccent-2 [21]✓56.35±2.6129.51±2.120.05113.842.89±0.360.83580.3029 DLM-AN-1×62.20±2.46 22.94±1.85-0.13311.193.05±0.340.83850.2811 DLM-AN-2 (τ =1.0)✓59.50±2.6127.90±2.16-0.02010.642.93±0.370.85210.2773 DLM-AN-2 (τ =0.3)✓57.35±2.5931.34±2.340.09812.522.90±0.390.85900.3580 DLM-AN-2 (τ =0.0)✓56.41±2.7038.37±2.57 0.20814.942.86±0.390.86460.4479 5. Results 5.1. Effectiveness of Common Token Prediction For effective common token prediction, higher confidence should be assigned to native-accented regions, whereas low confidence scores should be assigned to highly-L2-accented re- gions. Figure 4 is a visualization of common token predic- tion for a Chinese-accented sample. The common token con- fidence values are displayed over the spectrogram. Phonemes predicted by ppgs [68] are displayed below the spectrogram, with their boundaries being the white dashed lines on the Mel- spectrogram. Aligned words, obtained via MFA [69], are dis- played at the bottom. Accent can be determined from the com- parison between the words and phonemes. Some prominent L2- accented patterns can be found: 1) the initial word “a” is heav- ily lengthened; correspondingly, the CTP confidence becomes low in the prolonged part. 2) The PPG-predicted phonemes for the word “had” is messy, corresponding with general low confi- dence scores. 3) The ending phoneme of “death” is detected as highly similar to /S/, receiving low confidence scores. 5.1.1. Proportion-based Reuse To progressively preserve or remove the source accent, we can directly control the proportion of reused source tokens based on the CTP confidence scores. However, random reuse may also yield a coarse accent-retention effect. To verify the benefit of CTP-based reuse, we vary the reuse proportion under two strategies—CTP-based selection and random selection. Fig- ure 5 reports three metrics: 1) WER, 2) ∆PPG to the L1- accented target, and 3) ∆PPG to the L2-accented source. Ide- ally, WER should increase as the reuse proportion increases. In contrast, ∆PPG to the target should decrease (more native-like), while ∆PPG to the source should increase (less similar to the accented input). Both strategies follow these general trends, but their out- comes differ. CTP-based selection generally yields lower WER than random selection at the same reuse proportion. Moreover, it consistently achieves higher ∆PPG to the source and lower ∆PPG to the target, indicating better accent removal while remaining closer to the native reference. This suggests that CTP indeed prioritizes more native-accented regions for reuse, whereas random selection more frequently preserves highly ac- cented tokens. Overall, CTP-based token reuse provides a more reliable control knob for progressive accent normalization. 5.1.2. Threshold-based Reuse Input utterances exhibit different strengths of L2 accent. There- fore, enforcing the same reuse proportion for all sources can be suboptimal. We instead adopt threshold-based reuse, where to- kens are reused if their CTP confidence exceeds a threshold τ (Algorithm 1). This makes the effective reuse proportion adap- tive to the input accent strength. 0.00.20.40.60.81.0 Threshold ( ) 0.275 0.300 0.325 0.350 0.375 0.400 0.425 0.450 PPG PPG w/ target PPG w/ source Figure 6: ∆PPG with the L2-accented source and L1-accented target at varying CTP thresholds τ . Lower τ retains more source tokens. As τ increases, ∆PPG with the source increases (more accent removed) while ∆PPG with the target decreases (closer to native). Figure 6 shows ∆PPG to the source and to the L1-accented target under different thresholds. When τ =0.0, all tokens are reused; as τ increases, fewer tokens are kept; and when τ =1.0, generation from scratch is performed. As expected, ∆PPG to the source increases monotonically with τ , while ∆PPG to the target decreases, confirming that τ provides an interpretable and effective knob for accent reduction. 5.2. Comparison with Baselines 5.2.1. Duration-free Conversion The main results are shown in Table 1. We first compare the systems under the free duration setting (“-1” variants), where the model predicts its own target duration. DLM-AN- 1 achieves the lowest ACT score (22.94) among all systems, indicating the strongest accent reduction, while attaining the second-highest NAT score (62.20), close to TokAN-1 (63.85) and above CosyAccent-1 (61.12). In terms of objective content preservation, DLM-AN-1 obtains a WER of 11.19%, notably lower than TokAN-1 (13.82%) and CosyAccent-1 (12.40%). The UTMOS score (3.05) is also competitive with TokAN-1 (3.07), both outperforming CosyAccent-1 (2.99). These results suggest that DLM-AN achieves the best overall balance of ac- cent reduction and content preservation under free duration. 5.2.2. Source-duration-preserved Conversion Under the source-duration-preserved setting (“-2” variants), DLM-AN-2 (τ =1.0) again achieves the best WER (10.64%) and a competitive ∆PPG (0.2773) among all systems. As ex- pected, preserving the source duration generally leads to higher ACT scores and lower NAT scores compared with free-duration counterparts across all models, as the L2-accented rhythm is retained. Notably, DLM-AN-2 (τ =1.0) attains a similar ∆PPG with DLM-AN-1 (0.2773 vs. 0.2811) yet a higher ACT (27.90 vs. 22.94). This suggests that ∆PPG primarily captures segmental (more phonetic) similarity, whereas human raters perceive accentedness more holistically—rhythm and prosody, which are largely determined by duration, play a substantial role in subjective accent judgments. 5.2.3. Controllable Accent Normalization The effect of the CTP-based token reuse is clearly visible across the DLM-AN-2 variants. As the threshold τ decreases from 1.0 to 0.0, more source tokens are preserved, leading to a pro- gressive increase in ACT (27.90→ 31.34→ 38.37), reflecting stronger retention of the source accent. Correspondingly, the SIM score rises monotonically (−0.020 → 0.098 → 0.208), confirming that preserving more source tokens improves per- ceived speaker similarity. The SECS scores follow the same trend (0.8521 → 0.8590 → 0.8646), with the full-reuse vari- ant achieving the highest timbre preservation. This correlation between the source accent and speaker identity aligns with the finding in [22]. Meanwhile, WER degrades mildly (10.64 → 12.52 → 14.94) and ∆PPG increases (0.2773 → 0.3580 → 0.4479), as reusing accented tokens inevitably retains some non-native pro- nunciation patterns. At τ =0.0 (complete resynthesis), the out- put closely mirrors the source accent, as indicated by the ACT score (38.37) approaching the source (48.46), demonstrating smooth and interpretable accent strength control. 5.3. Arbitrary Duration Scaling All three systems support total-duration specification, but via different mechanisms: TokAN predicts token-wise durations after accent normalization, CosyAccent directly generates a target-length spectrogram, and DLM-AN generates a target- length token sequence. To compare robustness under differ- ent target lengths, we vary the total-duration ratio and compare WER across systems. Figure 7 shows the results. DLM-AN achieves the low- est WER (i.e., best content preservation) when the source dura- tion is preserved, and its advantage is more pronounced when the specified ratio is smaller than 1.0. When the target dura- tion is set to half of the source, TokAN degrades substantially because the generated token sequence (after deduplication) is often longer than the desired duration, forcing tokens to be dis- carded in the synthesis stage. DLM-AN maintains an advantage until the ratio reaches 1.5, likely because such extreme stretch- ing is rare in the training data. 0.50.751.01.251.5 Duration Ratio 10 15 20 25 30 35 40 45 WER (%) TokAN CosyAccent DLM Figure 7: WER (%) at varying duration scaling ratios for TokAN, CosyAccent, and DLM-AN. DLM-AN achieves the low- est WER across most ratios, with a notable advantage under compression (ratio < 1.0). 5.4. Ablation Study The ablation results are shown in Table 2, which is based on DLM-AN-2 that generates target tokens from scratch. We ab- late two factors: 1) CFG in token generation, 2) pretraining on Emilia-EN. The results show the effectiveness of these train- ing/inference components in the proposed DLM-AN. Table 2: Ablation results of DLM-AN-2 (τ =1.0). SystemWER (%↓) SECS (↑) ∆PPG (↓) Source15.86-0.5097 DLM-AN10.640.85210.2773 w/o CFG11.230.84880.2907 w/o pretraining16.610.85130.3306 6. Conclusion & Future Work We presented DLM-AN, a controllable accent normalization system based on masked discrete diffusion over self-supervised speech tokens. By introducing a Common Token Predictor (CTP) that identifies source tokens likely shared with the na- tive target, DLM-AN provides a simple yet effective accent- strength knob: reusing more high-confidence tokens preserves more of the original accent, while generating all tokens from scratch yields full normalization. A duration ratio predictor fur- ther enables total-duration adjustment. Experiments on multi- accent English data show that DLM-AN achieves the lowest WER among all compared systems, competitive naturalness and accent reduction, and smooth, interpretable accent strength con- trol across a continuous range. Several directions remain for future work. First, the cur- rent pipeline relies on a recognition-based token encoder for phoneme supervision, whose errors can propagate and degrade conversion quality for heavily accented inputs. Second, re- peated pronunciations can arise during unmasking; incorporat- ing corrective mechanisms [50, 53] may mitigate such artifacts. Third, the SSL tokenizer and synthesizer are trained on native- only data, potentially limiting reconstruction of highly accented speech; incorporating L2-accented data could improve robust- ness. Forth, replacing the K-Means tokenizer with learned dis- crete codebooks (e.g., vector quantization [70]) may yield better phonetic discriminability and further enhance controllability. 7. Generative AI Use Disclosure Generative AI tools were used solely for editing and polishing the manuscript text. No part of the scientific content, including the ideas, methodology, experiments, or analysis, was generated by AI. All authors have reviewed and take full responsibility for the content of this paper. 8. References [1] D. Felps, H. Bortfeld, and R. Gutierrez-Osuna, “Foreign accent conversion in computer assisted pronunciation training,” Speech communication, vol. 51, no. 10, p. 920–932, 2009. [2] O. T ̈ urk and L. M. Arslan, “Subband based voice conversion.” in Proc. Interspeech, 2002, p. 289–292. [3] L. Sun, H. Wang, S. Kang, K. Li, and H. M. Meng, “Personalized, cross-lingual tts using phonetic posteriorgrams.” in Proc. Inter- speech, 2016, p. 322–326. [4] Z. Guanlong, S. Sinem, L. John, C.-H. Evgeny, and G.-O. Ri- cardo, “Accent conversion using phonetic posteriorgrams,” in Proc. ICASSP, 2018, p. 5314–5318. [5] G. Zhao, S. Ding, and R. Gutierrez-Osuna, “Foreign accent con- version by synthesizing speech from phonetic posteriorgrams.” in Proc. Interspeech, 2019, p. 2843–2847. [6] W. Li, B. Tang, X. Yin, Y. Zhao, W. Li, K. Wang, H. Huang, Y. Wang, and Z. Ma, “Improving accent conversion with ref- erence encoder and end-to-end text-to-speech,” arXiv preprint arXiv:2005.09271, 2020. [7] S. Ding, G. Zhao, and R. Gutierrez-Osuna, “Accentron: Foreign accent conversion to arbitrary non-native speakers using zero-shot learning,” Computer Speech & Language, vol. 72, p. 101302, 2022. [8] G. Zhao, S. Ding, and R. Gutierrez-Osuna, “Converting foreign accent speech without a reference,” TASLP, vol. 29, p. 2367– 2381, 2021. [9] T.-N. Nguyen, N.-Q. Pham, and A. Waibel, “Accent conversion using pre-trained model and synthesized data from voice conver- sion.” in Proc. Interspeech, 2022, p. 2583–2587. [10] W. Quamer, A. Das, J. Levis, E. Chukharev-Hudilainen, and R. Gutierrez-Osuna, “Zero-shot foreign accent conversion without a native reference,” in Proc. Interspeech, 2022, p. 4920–4924. [11] S. Liu, D. Wang, Y. Cao, L. Sun, X. Wu, S. Kang, Z. Wu, X. Liu, D. Su, D. Yu et al., “End-to-end accent conversion without using native utterances,” in Proc. ICASSP, 2020, p. 6289–6293. [12] M. Jin, P. Serai, J. Wu, A. Tjandra, V. Manohar, and Q. He, “Voice-preserving zero-shot multiple accent conversion,” in Proc. ICASSP, 2023. [13] Y. Zhou, Z. Wu, M. Zhang, X. Tian, and H. Li, “Tts-guided train- ing for accent conversion without parallel data,” Signal Process- ing Letters, vol. 30, p. 533–537, 2023. [14] X. Chen, J. Pei, L. Xue, and M. Zhang, “Transfer the linguis- tic representations from tts to accent conversion with non-parallel data,” in Proc. ICASSP, 2024. [15] Q. Bai, S. Wang, Z. Liu, M. Zhang, W. Rao, Y. Wang, and H. Li, “Diffusion-based method with tts guidance for foreign accent con- version,” in Proc. ISCSLP, 2024, p. 284–288. [16] T. N. Nguyen, S. Akti, N. Q. Pham, and A. Waibel, “Improving pronunciation and accent conversion through knowledge distilla- tion and synthetic ground-truth from native tts,” in ICASSP, 2025. [17] T.-N. Nguyen, Q. Pham, and A. Waibel, “Accent conversion us- ing discrete units with parallel data synthesized from controllable accented tts,” in Synthetic Data’s Transformative Role in Founda- tional Speech Models, 2024, p. 51–55. [18] Z. Jia, H. Xue, X. Peng, and Y. Lu, “Convert and speak: Zero- shot accent conversion with minimum supervision,” in Multime- dia, 2024. [19] Q. Bai, S. Inoue, S. Wang, Z. Jiang, Y. Wang, and H. Li, “Ac- cent normalization using self-supervised discrete tokens with non- parallel data,” in Interspeech 2025, 2025, p. 1618–1622. [20] X. Zhang, X. Zhang, K. Peng, Z. Tang, V. Manohar, Y. Liu, J. Hwang, D. Li, Y. Wang, J. Chan, Y. Huang, Z. Wu, and M. Ma, “Vevo: Controllable zero-shot voice imitation with self- supervised disentanglement,” in ICLR, 2025. [21] Q. Bai, S. Shi, S. Wang, Y. Ju, Y. Wang, and H. Li, “Cosyac- cent: Duration-controllable accent normalization using source- synthesis training data,” Proc. ICASSP 2026, 2026. [22] Y. Halychanskyi, C. Churchwell, Y. Wen, and V. Kindratenko, “Fac-facodec: Controllable zero-shot foreign accent conversion with factorized speech codec,” Proc. ICASSP 2026, 2026. [23] S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J.-R. Wen, and C. Li, “Large language diffusion models,” arXiv preprint arXiv:2502.09992, 2025. [24] A. Graves, S. Fern ́ andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” in ICML, 2006. [25] L. Wang, Z. Yu, Y. Yang, S. Gao, C. Mao, and Y. Huang, “Non- parallel accent transfer based on fine-grained controllable accent modelling,” in EMNLP 2023.Association for Computational Linguistics, 2023, p. 9288–9298. [26] R. Liu, B. Sisman, G. Gao, and H. Li, “Controllable accented text- to-speech synthesis with fine and coarse-grained intensity render- ing,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, p. 2188–2201, Apr. 2024. [27] H. L. Xinyuan, Z. Cai, A. Garg, K. Duh, L. P. Garc ́ ıa-Perera, S. Khudanpur, N. Andrews, and M. Wiesner, “Scalable control- lable accented tts,” in Proc. ASRU 2025, 2025. [28] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech repre- sentation learning by masked prediction of hidden units,” TASLP, vol. 29, p. 3451–3460, 2021. [29] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” J-STSP, vol. 16, no. 6, p. 1505–1518, 2022. [30] K. Choi, A. Pasad, T. Nakamura, S. Fukayama, K. Livescu, and S. Watanabe, “Self-supervised speech representations are more phonetic than semantic,” in Proc. Interspeech, 2024. [31] W.-C. Huang, Y.-C. Wu, and T. Hayashi, “Any-to-one sequence- to-sequence voice conversion using self-supervised discrete speech representations,” in Proc. ICASSP, 2021. [32] F. Kreuk, A. Polyak, J. Copet, E. Kharitonov, T.-A. Nguyen, M. Rivi ` ere, W.-N. Hsu, A. Mohamed, E. Dupoux, and Y. Adi, “Textless speech emotion conversion using discrete & decom- posed representations,” in Proc. EMNLP, 2022. [33] H.-S. Oh, S.-H. Lee, D.-H. Cho, and S.-W. Lee, “Durflex-evc: Duration-flexible emotional voice conversion leveraging discrete representations without text alignment,” IEEE Transactions on Af- fective Computing, 2025. [34] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with mini- mal supervision,” Trans. ACL, vol. 11, p. 1703–1718, 2023. [35] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed et al., “On generative spoken language modeling from raw audio,” Trans. ACL, vol. 9, p. 1336–1354, 2021. [36] A. Lee, P.-J. Chen, C. Wang, J. Gu, S. Popuri, X. Ma, A. Polyak, Y. Adi, Q. He, Y. Tang, J. Pino, and W.-N. Hsu, “Direct speech- to-speech translation with discrete units,” in Proc. ACL, 2022. [37] Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng, “Llama- omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024. [38] M. Cui, D. Tan, Y. Yang, D. Wang, H. Wang, X. Chen, X. Chen, and X. Liu, “Exploring ssl discrete tokens for multilingual asr,” in Proc. ICASSP 2025, 2025. [39] M. Cui, Y. Yang, J. Deng, J. Kang, S. Hu, T. Wang, Z. Li, S. Zhang, X. Chen, and X. Liu, “Exploring ssl discrete speech features for zipformer-based contextual asr,” in Proc. Interspeech 2025, 2025. [40] K. Onda, S. Fukayama, D. Saito, and N. Minematsu, “Advanced modeling of interlanguage speech intelligibility benefit with l1-l2 multi-task learning using differentiable k-means for accent-robust discrete token-based asr,” in Proc. ICASSP 2026, 2026. [41] E. Hoogeboom, D. Nielsen, P. Jaini, P. Forr ́ e, and M. Welling, “Argmax flows and multinomial diffusion: Learning categorical distributions,” in Advances in neural information processing sys- tems, vol. 34, 2021, p. 12 454–12 465. [42] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg, “Structured denoising diffusion models in discrete state-spaces,” in Advances in Neural Information Processing Systems, 2021. [43] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “Maskgit: Masked generative image transformer,” in Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, p. 11 315–11 325. [44] Y.Wang,H.Zhan,L.Liu,R.Zeng,H.Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “MaskGCT: Zero-shot text-to-speech with masked generative codec transformer,” in The Thirteenth International Confer- ence on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=ExuBFYtCQU [45] Y. Wang, J. Zheng, J. Zhang, X. Zhang, H. Liao, and Z. Wu, “Metis: A foundation speech generation model with masked generative pre-training,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=RTjr4DnS79 [46] R. Yu, Q. Li, and X. Wang, “Discrete diffusion in large language and multimodal models: A survey,” 2025. [Online]. Available: https://arxiv.org/abs/2506.13759 [47] A. Lou, C. Meng, and S. Ermon, “Discrete diffusion modeling by estimating the ratios of the data distribution,” in ICML 2024, 2024. [48] G. Wang, Y. Schiff, S. S. Sahoo, and V. Kuleshov, “Re- masking discrete diffusion models with inference-time scal- ing,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=IJryQAOy0p [49] H. Ben-Hamu, I. Gat, D. Severo, N. Nolte, and B. Karrer, “Accelerated sampling from masked diffusion models via entropy bounded unmasking,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=WBcBhT1NKO [50] Z. Huang, Y. Wang, Z. Chen, and G.-J. Qi, “Don’t settle too early: Self-reflective remasking for diffusion language models,” arXiv preprint arXiv:2509.23653, 2025. [51] S. Zhang, F. Z. Peng, Y. Zhang, J. Pan, and G. G. Chrysos, “Corrective diffusion language models,” 2026. [Online]. Available: https://arxiv.org/abs/2512.15596 [52] Y. Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y. Yang, H. Yu, X. Qu et al., “Seed diffusion: A large-scale dif- fusion language model with high-speed inference,” arXiv preprint arXiv:2508.02193, 2025. [53] T. Bie, M. Cao, X. Cao, B. Chen, F. Chen, K. Chen, L. Du, D. Feng, H. Feng, M. Gong et al., “Llada2. 1: Speeding up text diffusion via token editing,” arXiv preprint arXiv:2602.08676, 2026. [54] Y. A. Li, C. Han, X. Jiang, and N. Mesgarani, “Hiftnet: A fast high-quality neural vocoder with harmonic-plus-noise fil- ter and inverse short time fourier transform,” arXiv preprint arXiv:2309.09493, 2023. [55] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” in NeurIPS 2021 Workshop on Deep Generative Models and Down- stream Applications, 2021. [56] W. Peebles and S. Xie, “Scalable diffusion models with transform- ers,” in Proc. ICCV, 2023, p. 4195–4205. [57] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in ICLR, 2023. [58] J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neuro- computing, vol. 568, p. 127063, 2024. [59] H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in 2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, p. 885–890. [60] Y. Koizumi, H. Zen, S. Karita, Y. Ding, K. Yatabe, N. Morioka, M. Bacchiani, Y. Zhang, W. Han, and A. Bapna, “LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus,” in Proc. Inter- speech, 2023. [61] G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev- Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-ARCTIC: A Non-native English Speech Corpus,” in Proc. Interspeech, 2018. [62] J. Kominek and A. W. Black, “The cmu arctic speech databases,” in Fifth ISCA workshop on speech synthesis, 2004. [63] S. Mehta, R. Tu, J. Beskow, ́ E. Sz ́ ekely, and G. E. Henter, “Matcha-tts: A fast tts architecture with conditional flow match- ing,” in Proc. ICASSP, 2024, p. 11 341–11 345. [64] M. Lewis, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint arXiv:1910.13461, 2019. [65] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in ICML, 2023, p. 28 492–28 518. [66] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024. [67] A. M. V. Ravillion, “A comparison of best-worst scaling and rat- ing scale for timbre characterisation,” 2020. [68] C. Churchwell, M. Morrison, and B. Pardo, “High-fidelity neural phonetic posteriorgrams,” in ICASSP 2024 Workshop on Explain- able Machine Learning for Speech and Audio, 2024. [69] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Son- deregger, “Montreal forced aligner: Trainable text-speech align- ment using kaldi.” in Proc. Interspeech, 2017, p. 498–502. [70] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural dis- crete representation learning,” in Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, 2017, p. 6309–6318.