Paper deep dive
Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech
Yue Heng Yeo, Haoyang Li, Yizhou Peng, Shreyas Gopal, Hexin Liu, Leibny Paola Garcia-Perera, Hardik B. Sailor, Jeremy H. M. Wong, Eng Siong Chng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/20/2026, 6:37:17 AM
Summary
This paper proposes a code-mixing guided preference-learning framework to improve Code-Switch (CS) Automatic Speech Recognition (ASR) through synthetic data augmentation. The authors introduce 'CMI speech', an acoustic-level extension of the Code Mixing Index (CMI), to quantify language-mixing fidelity in synthetic speech. By using Direct Preference Optimization (DPO) with a multi-critic approach—incorporating Mixed Error Rate (MER), UTMOS for perceptual quality, and the proposed ∆CMI—the framework steers Text-to-Speech (TTS) models (specifically CosyVoice2) to generate more realistic language-boundary transitions. Experiments on the SEAME Mandarin-English corpus show that this method significantly reduces the Mixed Error Rate (MER) for downstream ASR models like Whisper Large and CTC-based Conformer compared to standard augmentation techniques.
Entities (8)
Relation Signals (4)
CosyVoice2 → generatesdatafor → Whisper Large
confidence 100% · The resulting speech is then incorporated into downstream ASR training, improving recognition robustness
SEAME → usedtoevaluate → Code-Switch (CS) ASR
confidence 100% · Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning.
CMI speech → guides → Direct Preference Optimization (DPO)
confidence 90% · Leveraging this metric, we design a TTS-based augmentation framework for CS ASR.
Direct Preference Optimization (DPO) → optimizes → CosyVoice2
confidence 90% · Specifically, we first optimize a TTS model through preference learning using Direct Preference Optimization (DPO)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code-switch (CS) Automatic Speech Recognition (ASR) remains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data augmentation via Text-to-speech (TTS) has been explored, existing CS TTS approaches primarily optimise reconstruction fidelity and do not explicitly enforce language-boundary consistency, thereby limiting their effectiveness for CS ASR augmentation. This paper proposes a code-mixing guided preference-learning framework that steers synthetic speech generation toward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed approach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively.
Tags
Links
- Source: https://arxiv.org/abs/2606.19381v1
- Canonical: https://arxiv.org/abs/2606.19381v1
Trouble viewing inline? Open PDF directly →
Full Text
30,184 characters extracted from source content.
Expand or collapse full text
Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speech Yue Heng Yeo ID 1,2 , Haoyang Li 1 , Yizhou Peng ID 1 , Shreyas Gopal ID 1 , Hexin Liu ID 1,∗ , Leibny Paola Garcia-Perera ID 3 , Hardik B. Sailor ID 2 , Jeremy H. M. Wong ID 4 , Eng Siong Chng ID 1 1 College of Computing and Data Science, Nanyang Technological University, Singapore 2 Institute for Infocomm Research (I 2 R), A ⋆ STAR, Singapore 3 HLT-COE & CLSP, Johns Hopkins University, USA 4 Google DeepMind, Singapore yuehengyeo001@e.ntu.edu.sg; hexin.liu@ntu.edu.sg Abstract Code-switch (CS) Automatic Speech Recognition (ASR) re- mains challenging due to limited availability of high quality CS text-speech pairs for training. Although synthetic data aug- mentation via Text-to-speech (TTS) has been explored, exist- ing CS TTS approaches primarily optimise reconstruction fi- delity and do not explicitly enforce language-boundary consis- tency, thereby limiting their effectiveness for CS ASR augmen- tation. This paper proposes a code-mixing guided preference- learning framework that steers synthetic speech generation to- ward improved code-switching fidelity using the Code Mixing Index (CMI). Experiments on the SEAME Mandarin-English conversational corpus demonstrate that the proposed method enhances the utility of synthetic data for ASR fine-tuning. Specifically, when fine-tuning Whisper Large, the proposed ap- proach reduces Mixed Error Rate (MER) from 12.1%/17.8% to 8.9%/14.2% on the DevMAN and DevSGE sets, respectively. Index Terms: speech recognition, code-switching, speech gen- eration, data augmentation, reinforcement learning 1. Introduction Code-Switching (CS), the alternation of multiple languages within a single utterance, is a phenomenon in multilingual com- munities [1, 2, 3]. Despite substantial advances in Automatic Speech Recognition (ASR), recognising conversational CS speech remains challenging due to language alternation, cross- lingual phonetic interference, and informal speaking styles [4, 5]. These challenges result from the scarcity of large-scale, high-quality transcribed CS corpora such as SEAME [6], lim- iting the effectiveness of modern data-intensive ASR systems. As a result, strong end-to-end architectures, including large pre- trained models such as Whisper, struggle to generalise robustly to spontaneous conversational CS [7, 8, 9]. To address data scarcity, data augmentation via text-to- speech (TTS) has become a practical strategy for improving ASR performance in low-resource and domain-mismatched set- tings [10, 11, 12, 13]. Multilingual and multi-speaker TTS sys- tems enable scalable generation of labelled speech while intro- ducing speaker and acoustic variability, which has been shown to improve ASR robustness and generalisation [14, 15, 16]. In code-switching contexts, prior studies have explored generat- ing synthetic bilingual speech or phrase-mixed code-switching data to compensate for limited code-switching conversational recordings [17, 18, 19]. These approaches primarily empha- sise acoustic diversity or text coverage, demonstrating that syn- ** indicates the corresponding author. thetic data can reduce recognition errors when combined with real speech. However, existing augmentation strategies gener- ally assume that all synthesised utterances are equally informa- tive, without explicitly assessing whether the synthesised data’s language alternation patterns in conversational code-switching structure is accurate. Code-Mixing Index (CMI) is a text-based metric that pro- vides a simple and interpretable measure of language mix- ing balance within a tokenized multilingual sequence [20]. While CMI has been widely used to analyze code-switching text [21, 22], its potential as a guiding signal for acoustic model- ing remains largely unexplored, primarily due to a fundamental representational mismatch: CMI is defined over discrete sym- bolic tokens with explicit language labels, whereas speech is a continuous acoustic signal without inherent linguistic segmen- tation. Therefore, it is not trivial to reliably assign frame-level or segment-level language identities directly from the wave- form. We address this challenge by proposing CMI speech , a novel metric that quantifies language mixing within a speech ut- terance using pseudo frame-level language labels. Leverag- ing this metric, we design a TTS-based augmentation frame- work for CS ASR. Specifically, we first optimize a TTS model through preference learning using Direct Preference Optimiza- tion (DPO) [23], incorporating CMI speech into preference pair construction to explicitly enforce code-switching consistency. Unlike prior DPO-based TTS methods that focus on percep- tual quality or intelligibility without directly constraining cross- lingual boundaries, our approach favors samples exhibiting more realistic language mixing as measured by CMI speech . Sub- sequently, we fine-tune the downstream ASR using synthetic data generated by the optimized TTS model. By allowing CS linguistic signals to guide acoustic model optimisation, our framework produces synthetic CS speech that better preserves language boundaries and mixing patterns, improving ASR per- formance on the CS problem. 2. Preference Learning Preference learning has recently emerged as a scalable prefer- ence alignment paradigm for generative models, offering an al- ternative to explicit reward modeling and reinforcement learn- ing [24, 25, 26, 27]. DPO is a methodology derived from preference learning that reformulates alignment as supervised learning over preferred and dis-preferred sample pairs, enabling stable optimisation. Unlike traditional reinforcement-learning- based human feedback methods, which first train a separate reward model and then optimise a policy with reinforcement arXiv:2606.19381v1 [cs.SD] 14 Jun 2026 learning, DPO solves the alignment objective with a simple classification loss. Experimental results in recent TTS works showed that DPO not only simplifies training but also matches the performances in controlling sentiment and improving re- sponse quality [28]. Given a transcript A, and ˆ X + denotes a preferred synthetic speech candidate and ˆ X − denote a dis-preferred candidate un- der the same conditioning input. DPO maximizes the relative preference margin via: L DPO =−E " logσ βlog π θ ( ˆ X + |A) π ref ( ˆ X + |A) −βlog π θ ( ˆ X − |A) π ref ( ˆ X − |A) # (1) where π θ is the trainable TTS model, π ref is a frozen reference TTS model that stabilises optimisation, β > 0 is a tempera- ture parameter controlling the strength of the preference mar- gin, σ(·) denotes the sigmoid function, andE represents expec- tation over the preference dataset. The target TTS model π θ , initialised from π ref , receives the same transcript A and com- putes sequence probabilities. The DPO formulation enables flexible integration of task- specific preference signals without requiring explicit reward modeling, making it particularly suitable for extending pref- erence learning beyond traditional natural language generation settings. While preference-based methods have achieved strong results in natural language generation and large language model alignment, their application to code-switching speech synthesis remains relatively limited. Existing work on preference-guided TTS primarily focuses on perceptual quality, naturalness, or in- telligibility metrics, rather than fine-grained linguistic proper- ties. In particular, linguistically grounded measures of code- switching, computed from synthesised speech, have not been in- vestigated as preference objectives for acoustic-level TTS align- ment, nor has their impact on downstream code-switching ASR been evaluated. 3. Proposed Methodology 3.1. Acoustic-level CMI Speech In this paper, our proposed framework follows the language alignment strategy proposed in [29], generating pseudo frame- level language labels directly from the decoder cross-attention without requiring forced alignment or manual annotations. Specifically, averaged cross-attention from the last decoder layer provides a frame-to-token alignment, allowing token-level language identities to be predicted onto encoder frames as pseudo labels. These labels enable acoustic frame-level Lan- guage Identification (LID) and and is done through training with the Language Alignment Loss (LAL) proposed by the paper. With the capability to pseudo-label acoustic frames of syn- thetic speech, this paper extends CMI to the speech domain to better measure preservation of code-switching patterns in syn- thetic speech. CMI measures the proportion of tokens that do not belong to the dominant language in a text sequence. In- spired by this definition, we extend CMI to the speech domain by treating each acoustic frame as a language-labelled unit, as CMI speech (u) = T (u)− max k∈L T k (u) T (u) ,(2) where T (u) denotes the total number of acoustic frames in ut- terance u, and T k (u) represents the number of frames in u as- signed to language. k ∈L, such that T (u) = P k∈L T k (u). L denotes the set of language labels considered in the utterance. A higher CMI speech (u) indicates higher distribution of languages across frames, reflecting stronger acoustic code-mixing within the utterance. Unlike the traditional token-level CMI computed over discrete textual units, CMI speech is defined at the acoustic frame level, enabling direct measurement of acoustic language synthesis and fine-grained frame-level language transitions in speech. After which, we measure the ∆ CMI to quantify the preserva- tion of code-switching characteristics. We compute this metric for between ground-truth and synthetic speech. The ∆ CMI is defined as ∆ CMI = CMI speech ( ˆ X)− CMI speech (y) .(3) ˆ X denotes the synthetic speech signal generated from the tran- script, andy denotes the corresponding ground-truth speech sig- nal. ∆ CMI compares their frame-level code-mixing structure via CMI speech . Lower values indicate better preservation of overall code-switching proportion between the synthesised speech and ground truth speech. 3.2. Pipeline After deriving ∆ CMI , this paper proposes a DPO framework to explicitly align synthetic code-switching speech with linguis- tic structure and perceptual quality. A multi-critic DPO frame- work is introduced during TTS training. As illustrated in Fig. 1, the optimised TTS model is ultimately used to generate syn- thetic code-switching speech for data augmentation. The result- ing speech is then incorporated into downstream ASR training, improving recognition robustness and reducing cross-language confusion. To obtain this optimised TTS, the paper proposes the pipeline to consist of 3 stages, firstly, basic fine-tuning to adapt the reference TTS model to the CS task. Secondly, DPO alignment, which will be further described in the further sec- tions. Third, ASR training where the resultant optimised TTS model’s synthetic speech will then be combined with original ground truth training data to fine-tune the CS ASR system. The proposed CS DPO framework consists of the following: 3.2.1. Candidate Generation For each transcript A, the fine-tuned TTS model π ref generates N synthetic speech candidates ˆ X (n) N n=1 via stochastic sam- pling. Candidates share the same transcript and prompt speech, but differ in acoustic realization and prosody due to sampling randomness. The temperature modulates the sampling distribu- tion and controls the degree of diversity. 3.2.2. Critic Scoring After generation, each candidate ˆ X (n) is evaluated using au- tomatic critics to capture intelligibility, perceptual quality, and code-switching fidelity. First, the framework computes the mixed error rate (MER) using an ASR model by comparing the transcription of ˆ X (n) with the reference transcript, where lower MER indicates downstream ASR compatibility. Second, per- ceptual naturalness is estimated using a pretrained UTMOS [30] predictor, which provides a non-intrusive speech quality score without requiring human annotation, with higher values reflect- ing better naturalness. Finally, this paper proposes the CMI speech as an explicit critic for code-switching fidelity.Specifically, we measure the discrepancy ∆ CMI between the synthetic and ground-truth DPO Framework TTS TTS TTS ASR SEAME Training Data DPO LOSS Stage 2: DPO Alignment Stage 1: Basic Finetuning Stage 3: ASR Training X (1) X (n) Y (1) Y (n) Candidate Generation Ground Truth Audio Preference Selection X + X (1) X (2) Whisper ASR (MER) UTMOS (MOS) WhisperLAL (CMI speech ) Critic Scoring X - DPO Framework SEAME Training Data Figure 1: Overview of the proposed DPO-based TTS alignment framework. utterances, and prefer candidates with smaller ∆ CMI .This method integrates an acoustic, frame-level code-mixing metric into preference learning for code-switched TTS, directly guid- ing generation toward more realistic language-boundary syn- thetic speech. 3.2.3. Preference Selection To combine heterogeneous critic signals, all metrics are first converted to a unified format through normalization with scores ranging from 0 to 1 for each criteria, the final ranking score is defined as R( ˆ X) = λ ̃ S UTMOS ( ˆ X)− γ ̃ S MER ( ˆ X)− ν ̃ S ∆CMI ( ˆ X)(4) where ̃ S UTMOS ( ˆ X), ̃ S MER ( ˆ X), and ̃ S ∆CMI ( ˆ X) are the nor- malised critic scores, and λ,γ,ν ≥ 0 are their corresponding weights. Preference pairs are constructed by ranking candidates of each transcript according to the normalized critic score. To provide a stable supervision signal, we pair the highest-ranked candidate with the lowest-ranked candidate for each transcript, forming a maximally contrasted pair for DPO optimization. To further improve stability, we apply threshold-based filtering to remove unreliable candidates. Specifically, after forming the pairs, pairs with the preferred candidate with MER greater than 20%, UTMOS lower than 2.5, or CMI speech difference exceed- ing 20% is discarded as they provide an unstable alignment to the training step. 4. Experiment Setup The experiments in this paper are conducted on the SEAME corpus, a benchmark for conversational Mandarin-English code-switching speech recognition [31]. SEAME contains ap- proximately 192 hours of spontaneous conversational speech recorded from over 150 bilingual speakers in Singapore and Malaysia, where Mandarin and English are frequently mixed within and across utterances. The corpus exhibits both intra- sentential and inter-sentential code-switching under informal speaking conditions, making it particularly challenging for ASR systems. TTS training text and prompts are drawn only from SEAME, with no external text data. For speech synthesis, we adopt CosyVoice2[32], a multi- lingual LLM-based auto-regressive TTS model to incorporate DPO training. In our experiments, CosyVoice2 is first fine- tuned on the SEAME conversational Mandarin-English code- switching speech to adapt it to mixed-language conversational acoustics. Fine-tuning is performed using the AdamW opti- mizer [33] with a learning rate of 2× 10 −4 and a linear warm- up schedule. Training is conducted for approximately 50k steps with a batch size of 4 utterances, and early stopping is applied based on validation loss to prevent overfitting. The fine-tuned model is then used to synthesise CS speech for ASR data aug- mentation. For ASR, we use Whisper ASR model as the down- stream evaluation system.Whisper takes log-Mel spectro- grams as input and employs a Transformer-based architec- ture to jointly model acoustic and linguistic information. We fine-tune the Whisper-large v3 model on SEAME conversa- tional Mandarin–English code-switching speech, augmented with synthetic speech generated by the TTS model. Fine-tuning is carried out using the Adam optimizer [34] with a learning rate of 1× 10 −5 . Models are trained till convergence with a batch size of 1 utterance per A40 GPU. All ASR experiments use identical training and decoding configurations across differ- ent augmentation strategies to ensure fair comparison. In par- ticular, it is ensured that summed duration of real and synthetic speech is consistent. 4.1. Scoring models For MER, a SEAME fine-tuned Whisper-largev3 ASR model is used to provide intelligibility-based preference signals for DPO. For each pair of synthesised speech samples generated from the same transcript, we decode the audio using Whisper and compute MER. Preference labels are assigned by favour- ing the sample with lower MER, encouraging the TTS model to generate speech that is more accurately recognized by a strong downstream ASR system. For UTMOS score, UTMOS as an automatic speech qual- ity predictor is used to provide perceptual preference signals for DPO. For each pair of synthesised samples, UTMOS assigns a predicted MOS score, and the sample with higher predicted quality is preferred. These preferences guide the TTS model to- wards generating more natural and perceptually pleasing speech without requiring human listening tests. For ∆ CMI calculation, to encourage realistic code-switching behavior during training, a SEAME based fine-tuned Whisper model with LAL loss is used to generate pseudo frame-level language labels from synthesised speech. Specifically, Whisper provides language identification predictions that are aligned to acoustic frames, enabling automatic identification of contiguous language regions within the waveform. These pseudo labels are then used to compute CMI speech , which quantifies the realised degree of language mixing directly from the synthesised audio. 5. Results Table 1 shows how progressively adding critic signals to DPO better aligns CS TTS with our objective of generating better quality synthetic speech that effectively improves downstream CS-ASR. The table shows the results after DPO fine-tuning CosyVoice TTS on SEAME training set and reproducing Dev- Table 1: Cosyvoice TTS Performance Comparison Across DPO Optimization Metrics (DevMAN and DevSGE test set) ModelRewardUTMOS↑MER↓∆ CMI ↓ CosyVoice–3.116.228.1 DPOMER3.214.925.7 + UTMOS3.813.221.9 + ∆ CMI 3.810.316.1 MAN and DevSGE test sets with the various different reward critics. Optimising with MER alone improves intelligibility (16.2%→ 14.9%) but only modestly reduces ∆ CMI (28.1%→ 25.7%). Adding UTMOS further improves perceptual quality (3.1→ 3.8) and reduces MER to 13.2%, indicating that DPO alignment enhances overall synthesis quality beyond ASR com- patibility. Importantly, incorporating the proposed acoustic-level ∆ CMI critic yields the largest reduction in code-switch dif- ference (28.1%→ 16.1%) while simultaneously achieving the lowest MER (10.3%) without degrading perceptual qual- ity. This demonstrates that explicitly aligning acoustically- perceived language proportion improves structural faithfulness of code-switching speech. Since our objective is to assess how well the TTS system preserves intelligibility, naturalness, and code-switch structure, the consistent improvements across all three metrics confirm the effectiveness of the proposed DPO framework in improving the CS TTS quality. Table 2: Mixed Error Rate (MER, %) on SEAME DevMAN and DevSGE test sets. Real: Ground truth SEAME training data. For Cosyvoice and Cosyvoice with DPO: 200 hours (100 hours of Real + 100 hours of Synthetic Audio Data) Train ConfigurationDevMANDevSGE Whisper ASR Real (100h)12.117.8 + CosyVoice10.116.0 + DPO (UTMOS, MER)9.615.1 + DPO (UTMOS, MER, ∆ CMI )8.914.2 CTC-Based Conformer Real (100h)16.823.6 + CosyVoice16.122.8 + DPO (UTMOS, MER)15.822.3 + DPO (UTMOS, MER, ∆ CMI )15.421.9 Table 2 reports MER on SEAME DevMAN and De- vSGE for both Whisper ASR and a ESPNet’s CTC-based Con- former [35] under different data augmentation strategies to show effects of downstream ASR training. Adding an equal proportion of 100 hours of synthetic speech generated from CosyVoice fine-tuning consistently reduces MER compared to the 100-hour real-data baseline (12.1%/17.8%→ 10.1%/16.0% for Whisper; 16.8%/23.6%→ 16.1%/22.8% for Conformer), demonstrating that high-quality synthetic data can improve ASR robustness for conversational code-switching speech. Applying DPO yields consistent further improvements across both evaluation sets and architectures, suggesting that preference-optimised synthetic speech is more effective than simple generated augmentation. Incorporating the proposed Table 3: Qualitative comparison of code-switching synthesis under different training strategies by Whisper ASR. MethodOutput Ground Truth他 spend too long spend too long去做 o hall olympiad是另外一个也是 什么 science的 but他有一点 CosyVoice2 FT他 spend two龙 spend two龙去做 oh hall奥林匹克是另外一个也是 什 math science the but他有一点 CosyVoice2 FT + DPO (MER, UTMOS) 他 spend too龙 spend too龙去做 o hall奥林匹克是另外一个也是 什 math science的 but他有一点 CosyVoice2 FT + DPO (MER, UTMOS, ∆ CMI ) 他 spend too long spend too long去做 o hall olympiad是另外一个也是 什么 science的 but他有一点 ∆ CMI critic produces the largest gains compared to traditional DPO methods of only using MER and UTMOS, reducing MER to 8.9%/14.2% for Whisper ASR and 15.4%/21.9% for the Con- former on DevMAN/DevSGE, respectively. The consistent im- provements observed across models demonstrate that explic- itly preserving acoustic-level code-switching structure during TTS generation substantially enhances downstream ASR per- formance. 5.1. Ablation Analysis To futher evaluate on the effects of ∆ CMI , Table 3 shows the progressive improvement based on the different methods that has been used to improve the TTS. Basic fine-tuning still has CS errors, where English words are similar shifted in Man- darin and language boundaries become unstable. While DPO with MER and MOS improves overall intelligibility, cross- language substitutions remain, indicating that recognition- and perceptual-based rewards alone do not sufficiently constrain code-switching structure. By incorporating ∆ CMI , the model guides the language distribution in synthesised audio, preserv- ing intended language boundaries and restoring correct pronun- ciation of English or Chinese CS segments. This demonstrates that CMI speech guided preference optimization effectively pro- duces more accurate CS speech. 6. Conclusion In this work, we presented a CS metric guided DPO framework for improving CS TTS. Our approach explicitly aligns syn- thetic speech with intelligibility, perceptual quality, and realis- tic language-mixing structure. By integrating ∆ CMI for measur- ing synthesised CS complexity, we construct contrastive pref- erence pairs through normalized scoring and threshold-based filtering. Experimental results on SEAME demonstrate consis- tent improvements across both TTS and downstream ASR eval- uation, including substantial reductions in MER while main- taining higher perceptual quality and better preservation of lan- guage mixing. These findings highlight the effectiveness of multi-critic preference learning in enforcing structured CS be- havior and suggest that CS alignment-driven optimisation pro- vides a promising direction for CS TTS generation to improve CS ASR. 7. Generative AI Use Disclosure Generative AI tools were used for limited assistance with manuscript editing and presentation (e.g., grammatical valida- tion, removal of redundant sentences and phrases, preparing La- TeX equations and LaTeX formatting suggestions). The litera- ture review and all scientific contributions, including but not limited to problem formulation, methodology, experiments, re- sults, and conclusions, were performed by the authors. All au- thors reviewed the manuscript and are responsible for the final submission. 8. References [1] A. S. Do ̆ gru ̈ oz, S. Sitaram, B. E. Bullock, and A. J. Toribio, “A survey of code-switching: Linguistic and social perspectives for language technologies,” in Proc. ACL, 2021, p. 1654–1666. [2] H. Liu, L. P. Garcia, X. Zhang, J. Dauwels, A. W. H. Khong, S. Khudanpur, and S. J. Styles, “End-to-end language diarization for bilingual code-switching speech,” in Proc. Interspeech, 2021, p. 1489–1493. [3] H. Liu, H. Xu, L. P. Garcia, A. W. H. Khong, Y. He, and S. Khu- danpur, “Reducing language confusion for code-switching speech recognition with token-level language diarization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2023, p. 1–5. [4] E. Y. Ugan, N.-Q. Pham, and A. Waibel, “Decm: Evaluating bilin- gual asr performance on a code-switched speech dataset,” in Proc. LREC-COLING, 2024. [5] H. Liu, L. P. Garcia, X. Zhang, A. W. Khong, and S. Khudan- pur, “Enhancing code-switching speech recognition with interac- tive language biases,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. IEEE, 2024, p. 10 886–10 890. [6] D.-C. Lyu, T.-P. Tan, E. S. Chng, and H. Li, “Seame: a mandarin- english code-switching speech corpus in south-east asia,” in Proc. Interspeech, 2010, p. 1986–1989. [7] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu et al., “Conformer:Convolution-augmented transformer for speech recognition,” in Proc. Interspeech, 2020, p. 5036–5040. [8] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” in Proc. ICML, 2023. [9] H. Yang, Y. Peng, H. Huang, and S. Li, “Adapting whisper for parameter-efficient code-switching speech recognition via soft prompt tuning,” in Proc. Interspeech, 2025. [10] G. Yang, F. Yu, Z. Ma, Z. Du, Z. Gao, S. Zhang, and X. Chen, “Enhancing low-resource asr through versatile tts: Bridging the data gap,” arXiv preprint arXiv:2410.16726, 2024. [11] R. Zevallos, “Text-to-speech data augmentation for low resource speech recognition,” arXiv preprint arXiv:2204.00291, 2022. [12] S. Ueno, K. Kawakami, H. Inaguma, and S. Nakamura, “Data augmentation for asr using tts via a discrete representation,” in Proc. IEEE Autom. Speech Recognit. Understand. Workshop, 2021. [13] K. C. Yuen, L. Haoyang, and C. E. Siong, “Asr model adaptation for rare words using synthetic data generated by multiple text-to- speech systems,” in Proc. APSIPA ASC, 2023, p. 1771–1778. [14] E. Casanova,C. Shulby,A. Korolev,A. C. Junior, A. da Silva Soares, S. Alu ́ ısio, and M. A. Ponti, “Asr data augmentation in low-resource settings using cross-lingual multi-speaker tts and cross-lingual voice conversion,” in Proc. Interspeech, 2023, p. 1244–1248. [15] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, et al., “Neu- ral codec language models are zero-shot text to speech synthesiz- ers,” arXiv preprint arXiv:2301.02111, 2023. [16] B. Lin, B. Zhao, B. Wu, C. Yan, C. Wu, C. Yi, C. Yao, D. Liu, F. Tian, F. Tian et al., “Stepaudio 2.5 technical report,” arXiv preprint arXiv:2605.23463, 2026. [17] Y. H. Yeo, Y. Hu, S. Gopal, Y. Peng, H. Liu, and E. S. Chng, “Improving code-switching speech recognition with tts data aug- mentation,” in Proc. APSIPA ASC, 2025. [18] T. Nguyen and H.-D. Tran, “Can we train asr systems on code- switch without real code-switch data? case study for singapore’s languages,” arXiv preprint arXiv:2506.14177, 2025. [19] B. Yan, I. Hamed, S. Shimizu, V. Lodagala, W. Chen, O. Iakovenko et al., “Cs-fleurs: A massively multilingual and code-switched speech dataset,” in Proc. Interspeech, 2025, p. 743–747. [20] B. Gamb ̈ ack and A. Das, “On measuring the complexity of code- mixing,” in Proc. ICON, Goa, India, 2014, p. 1–7. [21] V. Srivastava, M. Singh, M. Shrivastava, and D. M. Sharma, “Challenges and limitations with the metrics measuring the com- plexity of code-mixed text,” in Proc. Workshop Comput. Ap- proaches Linguist. Code-Switching, 2021. [22] S. K. Rallabandi, S. Sitaram, and A. W. Black, “Automatic detec- tion of code-switching style from acoustics,” in Proc. Interspeech, 2018. [23] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Man- ning, and C. Finn, “Direct preference optimization:Your language model is secretly a reward model,” arXiv preprint arXiv:2305.18290, 2023. [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, et al., “Training language models to follow instruc- tions with human feedback,” in Proc. NeurIPS, 2022. [25] H. Li, X. Zhuang, A. Adnan, Y. Ni, W. Rao, S. Gopal, and E. S. Chng, “Gentse: Enhancing target speaker extraction via a coarse-to-fine generative language model,” arXiv preprint arXiv:2512.20978, 2025. [26] H. Li, N. Hou, Y. Hu, J. Yao, S. M. Siniscalchi, and E. S. Chng, “Aligning generative speech enhancement with human preferences via direct preference optimization,” arXiv preprint arXiv:2507.09929, 2025. [27] D. Wu, H. Zhang, J. Chen, H. Liu, E. S. Chng, F. Tian, X. Yang, X. Zhang, D. Jiang, G. Yu et al., “Mind-paced speaking: A dual- brain approach to real-time reasoning in spoken language mod- els,” arXiv preprint arXiv:2510.09592, 2025. [28] J. Tian, C. Zhang, J. Shi, H. Zhang, J. Yu, S. Watanabe, and D. Yu, “Preference alignment improves language model-based tts,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2025, p. 1–5. [29] H. Liu, X. Zhang, H. Zhang, L. P. Garcia-Perera, A. W. H. Khong, E. S. Chng, and S. Watanabe, “Aligning speech to languages to enhance code-switching speech recognition,” IEEE Trans. Audio, Speech, Lang. Process., vol. 33, p. 4712–4725, 2025. [30] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” in Proc. Interspeech, 2022. [31] H. Liu, H. Zhang, Q. Zhang, X. Zhang, D. Shi, E. S. Chng, and H. Li, “Code-switching speech recognition under the lens: Model- and data-centric perspectives,” IEEE Trans. Audio, Speech, Lang. Process., vol. 34, p. 1853–1865, 2026. [32] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, et al., “Cosyvoice 2: Scalable streaming speech synthesis with large lan- guage models,” arXiv preprint arXiv:2412.10117, 2024. [33] I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in Proc. ICLR, 2019. [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” in Proc. ICLR, 2015. [35] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno et al., “ESPnet: End-to-End Speech Processing Toolkit,” in Proc. Interspeech, 2018, p. 2207–2211.