Paper deep dive
Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/8/2026, 2:24:45 AM
Summary
The paper introduces MSRT, a speech-to-text translation (S2TT) framework that addresses the 'curse of multilinguality' in multimodal large language models. It employs a Mixture of Speech Encoders (MoSE) with a resource-aware router that directs high-resource languages to a frozen expert and medium/low-resource languages to a trainable expert. This architecture, combined with a five-stage curriculum learning strategy, allows the 4B-parameter model to achieve state-of-the-art performance across 45 languages with minimal paired data (10 hours per language), significantly improving low-resource language performance without degrading high-resource results.
Entities (10)
Relation Signals (8)
MSRT → uses → MoSE
confidence 98% · MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE).
MSRT → addressesproblem → Curse of Multilinguality
confidence 97% · Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation
MoSE → routesto → Frozen Expert
confidence 96% · A frozen expert preserves high-resource language capabilities
MoSE → routesto → Trainable Expert
confidence 96% · a trainable expert adapts to and specializes in medium- and low-resource languages
MoSE → consistsof → Whisper
confidence 95% · Two homogeneous Whisper encoders (Radford et al. 2023) assume asymmetric roles
MSRT → trainedon → Fleurs
confidence 94% · We evaluate MSRT on FLEURS across all 45×44 non-identity translation directions
MSRT → usesmethod → Curriculum Learning
confidence 93% · We further introduce a five-stage curriculum learning strategy
MSRT → outperforms → MCAT-27B
confidence 90% · outperforming substantially larger baselines... MCAT-27B
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all $45 \times 44$ translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.
Tags
Links
- Source: https://arxiv.org/abs/2608.04586v2
- Canonical: https://arxiv.org/abs/2608.04586v2
Trouble viewing inline? Open PDF directly →
Full Text
36,772 characters extracted from source content.
Expand or collapse full text
Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders Yexing Du1,2, Kaiyuan Liu1,2, Youcheng Pan2, Bo Yang2, Chengpeng Fu1,2, Yu Wang2, Ming Liu1,2 Abstract Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2T). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2T data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all 45×4445× 44 translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2T research, we release our code and models. Code — https://github.com/yxduir/MSRT Introduction Speech-to-text translation (S2T) aims to translate speech from a source language into text in a target language. Traditionally, these systems have utilized cascaded pipelines: an Automatic Speech Recognition (ASR) component first extracts a transcript from the speech (Radford et al. 2023), which a subsequent Machine Translation (MT) model then converts into the desired language (Shang et al. 2026). Recently, there has been a growing focus on integrating speech encoders with large language models (LLMs) to endow them with multilingual speech capabilities, thereby advancing ASR (Ma et al. 2026) and S2T tasks (Du et al. 2026). Figure 1: Multilingual consistency by source language and resource level. Consistency is the ratio of a source language’s average translation score (across 44 target directions) to the English average. Higher is better, and dashed vertical lines indicate each model’s average within a resource group. Despite these advances, current MLLMs remain strongly English-centric in S2T. They perform well on English-to-X directions but degrade in many-to-many translation from non-English speech. As shown in Figure 1, their consistency scores for low-resource speech inputs fall substantially below those for English, revealing a pronounced cross-lingual performance gap. This disparity is not merely a target-generation issue: changing the source speech language causes substantial quality variation even when the target set and decoder remain unchanged. Consequently, English-centric results can overstate multilingual coverage and obscure failures on multilingual speech inputs. One important cause of this inconsistency is that most multilingual MLLMs (Xu et al. 2025) rely on a single shared speech encoder for all input languages. As language coverage expands, this architecture suffers from the “curse of multilinguality” (Conneau et al. 2020): limited encoder capacity forces languages with different levels of available resources to compete for the same representation space. High-resource languages require preserving pretrained acoustic knowledge, whereas medium- and low-resource languages require greater adaptation. Joint optimization can therefore overwrite established representations or underfit scarce-language features. Inspired by the Mixture-of-Experts architectures in LLMs (Blevins et al. 2024), we propose MSRT, an S2T MLLM centered on a resource-aware Mixture of Speech Encoders (MoSE). Two homogeneous Whisper encoders (Radford et al. 2023) assume asymmetric roles: a frozen expert handles high-resource languages, while a trainable expert adapts to medium- and low-resource speech. A source-language token routes each utterance to one expert, and a target-language token controls generation. This design provides specialization at the inference cost of a single encoder. We further extend the curriculum strategy (Du et al. 2025) into five stages of increasing difficulty, progressively establishing speech–text alignment and translation. We evaluate MSRT on FLEURS across all 45×4445× 44 non-identity translation directions, with detailed comparisons on representative languages from 11 language families. MSRT-4B achieves the highest average COMET score of 83.3, outperforming both the Gemini-3.5-Flash-Lite API and the substantially larger MCAT-27B (Du et al. 2026). Across the complete direction grid, MSRT obtains COMET scores of at least 80 on 1,552 directions (78.4%), compared with 928 for Gemini. Ablation studies show that MoSE improves performance across high-, medium-, and low-resource languages, with the largest gains on low-resource languages, while further strengthening high-resource performance. These consistent gains across all resource groups demonstrate that MoSE breaks the curse of multilinguality rather than trading high-resource performance for low-resource improvements. Experiments on CoVoST-2 further demonstrate robust cross-dataset generalization. Our contributions are summarized as follows: • We propose MoSE, a resource-aware mixture-of-speech-encoder architecture that breaks the curse of multilinguality by preserving high-resource knowledge while enabling targeted adaptation for medium- and low-resource languages. • We introduce MSRT-4B, a compact 4B-parameter model that supports all translation directions among 45 languages using only 10 hours of paired S2T data per language. We open-source its code and model to facilitate reproducible research. • We comprehensively evaluate all 1,980 translation directions across 45 languages, demonstrating broad and reliable many-to-many translation across diverse language families and resource levels, especially for low-resource speech in realistic multilingual settings. Related Work Speech-to-Text Translation. Conventional speech-to-text translation (S2T) cascades automatic speech recognition and machine translation. Although this design can reuse strong task-specific models, recognition errors propagate downstream and two-stage decoding increases latency. End-to-end S2T instead learns a direct mapping from source speech to target text. SeamlessM4T integrates multilingual speech recognition and translation with text translation in a single model, demonstrating that one architecture can support diverse speech–text tasks and languages (Barrault et al. 2023). Its success, however, still depends heavily on paired supervision. ZeroSwot reduces this dependence by connecting an ASR-trained speech encoder to a massively multilingual translation model through CTC compression and optimal-transport alignment, enabling direct S2T without paired translation data (Tsiamas et al. 2024). These studies establish speech–text alignment as a central challenge in end-to-end translation. Many-to-Many Speech-to-Text Translation. Many-to-many S2T expands both source- and target-language coverage, requiring one model to preserve language-specific acoustic cues while controlling multilingual generation. SeamlessM4T demonstrates broad multilingual coverage through shared recognition and translation components (Barrault et al. 2023). General-purpose multimodal LLMs such as Qwen3-Omni further show that pretrained language backbones can support multilingual speech understanding and translation (Xu et al. 2025). Specialized MLLMs more directly exploit the translation knowledge of LLMs: LLM-SRT transfers it to many-to-many S2T through curriculum learning (Du et al. 2025), while MCAT scales MLLM-based translation to 70 languages (Du et al. 2026). Nevertheless, increasing language coverage does not eliminate resource imbalance or interference between languages. This limitation motivates speech-side mechanisms that allocate adaptation capacity according to the data resources and acoustic variation of different languages. Mixture of Experts. MoE has been explored in multimodal and speech models from several perspectives. Uni-MoE activates modality-specific experts to reduce cross-modal interference in a unified MLLM (Li et al. 2025), whereas SimulMega uses expert routers as policies for balancing latency and quality in simultaneous translation (Le et al. 2025). PaM selects heterogeneous audio encoders from task prompts to serve different audio tasks (Shan et al. 2025), while DialectMoE learns dynamic expert routing for multi-dialect ASR (Zhou et al. 2024). In contrast, this work explores MoE specifically for multilingual S2T and focuses on interference induced by imbalanced language resources. MSRT routes source languages by resource level between two homogeneous Whisper encoders with asymmetric roles: a frozen expert preserves high-resource representations, and a trainable expert specializes in medium- and low-resource speech. Thus, its explicit language-level routing investigates multilingual expert specialization without token-level load balancing or task-dependent expert selection. Figure 2: Overview of the MSRT framework. Based on the source-language token in the prompt, MoSE routes high-resource speech to a frozen encoder and medium- and low-resource speech to a trainable encoder. Methodology Model Architecture As shown in Figure 2, MSRT connects a resource-aware Mixture of Speech Encoders (MoSE), a speech adapter, and a pretrained LLM. Mixture of Speech Encoders. MoSE comprises two encoders and an explicit language router. Instead of sharing one encoder across languages with unequal supervision, it separates high-resource knowledge preservation from medium- and low-resource adaptation while activating one expert per utterance. This design breaks the curse of multilinguality by improving underrepresented languages without sacrificing high-resource performance. • Frozen Expert. The frozen expert EfE_f retains the original Whisper encoder with fixed parameters. It preserves general speech representations for high-resource languages, prevents catastrophic forgetting, and anchors the shared speech interface during adaptation. • Trainable Expert. The trainable expert EtE_t shares the Whisper initialization but adapts to medium- and low-resource speech, capturing underrepresented accents and phonological patterns. Its architecture matches EfE_f, so both experts produce compatible representations and share one adapter without expert-specific projections. • Explicit Language Router. The router maps each source-language token to its resource-group expert: high-resource languages use EfE_f, whereas medium- and low-resource languages use EtE_t. Table 3 lists all assignments. Utterance-level routing keeps all frames on one acoustic path and requires neither a learned gate nor a balancing loss. Overall, let ℛ(ℓs)R( _s) denote the expert selected for source language ℓs _s. The MoSE output is written as HMoSE(x,ℓs)=Eℛ(ℓs)(x),ℛ(ℓs)∈f,t.H_MoSE(x, _s)=E_R( _s)(x), ( _s)∈\f,t\. (1) Here, ℛ(ℓs)=fR( _s)=f for ℓs∈ℒhigh _s _high and ℛ(ℓs)=tR( _s)=t otherwise, covering the medium- and low-resource groups. This formulation retains the inference cost of a single encoder. Speech Adapter. The selected encoder produces variable-length features. A Q-Former compresses them into a translation-aware representation, while an MLP projects it into the LLM hidden space. Together, these modules remove acoustic redundancy while retaining information required for translation. LLM. The pretrained LLM provides multilingual translation knowledge and remains frozen during speech-side alignment to prevent catastrophic forgetting. It is unlocked only in the final training stage, where it adapts to the aligned acoustic representation. Tokenizer Expansion. We add dedicated language tokens for the FLEURS languages (Conneau et al. 2023). The source-language token controls MoSE encoder selection, while the target-language token specifies the LLM translation direction. Together, they provide explicit language control and reduce unintended switching. Text Prompt. As shown in Figure 2 and Table 1, the prompt explicitly instructs the LLM to translate from the source language into the target language. The source-language token also controls encoder selection through the explicit language router, while the target-language token specifies the desired output language. Table 1: Prompt design for the five-stage curriculum. Stage Task Speech Text Prompt Prediction I ASR ✓ <|eng|> Text I ASR ✓ <|eng|> Text I SMT ✓ Text<|eng|><|cmn|> MT IV SRT ✓ <|eng|><|cmn|> Text<|eng|><|cmn|> MT V SRT ✓ <|eng|><|cmn|> Text<|eng|><|cmn|> MT Five-Stage Curriculum Learning Following Du et al. (2025), we organize ASR, speech-guided machine translation (SMT), and joint speech recognition and translation (SRT) into the five stages in Table 1. Table 2 summarizes which modules are optimized at each stage; every stage starts from the preceding checkpoint. • Stage I: ASR alignment. The Q-Former and MLP adapter are trained successively on English-only, high-resource multilingual, and fully multilingual ASR data, establishing a stable multilingual speech–text interface while both speech experts and the LLM remain frozen. • Stage I: Expert specialization. MoSE routing is activated for multilingual ASR. The frozen expert EfE_f preserves high-resource representations, whereas EtE_t is optimized for medium- and low-resource speech; the shared adapter continues training. • Stage I: Translation activation. SMT supplies the ground-truth transcription together with source- and target-language tokens. The adapter is optimized to connect routed speech features to the translation knowledge of the frozen LLM. • Stage IV: End-to-end SRT. The ground-truth transcription is removed from the input. With the experts and LLM fixed, the adapter learns to generate both the transcription and translation directly from routed speech. • Stage V: Joint adaptation. LLM LoRA is enabled and jointly optimized with the adapter for SRT, adapting multilingual generation to acoustic inputs without updating the pretrained LLM weights. Table 2: MLLM settings for MSRT. Modules Stage Details Speech Encoder (MoSE) Frozen Expert (EfE_f) – For high-resource Trainable Expert (EtE_t) I For medium-/low-resource Language Router – Explicit expert routing Speech Adapter Q-Former All Feature compression MLP All Dimension alignment LLM ++ Vocabulary – Language tokens ++ LoRA V r=16r=16, α=32α=32 Experimental Settings Datasets. We use Common Voice 24 (Ardila et al. 2020) and FLEURS for multilingual ASR pre-training, combining broad language coverage with curated speech data. For S2T training, we use paired speech–text data from FLEURS, comprising approximately 10 hours of speech per language. We evaluate the S2T models on both FLEURS and the CoVoST-2 (Wang et al. 2021) benchmark. Training Details. As shown in Table 2, the MLLM comprises a MiLMMT-4B LLM (Shang et al. 2026), MoSE, and a speech adapter containing a Q-Former and an MLP. The Q-Former uses 80 learnable queries to compress variable-length speech into a fixed-length interface for the LLM. We train in BF16 with DeepSpeed ZeRO-0 and AdamW, using a peak learning rate of 1×10−41× 10^-4 and 1,000 warmup steps. Training takes less than three days on eight Ascend 910C NPUs. The same implementation can run on eight NVIDIA A100 GPUs with comparable accuracy. Baselines. We compare three representative deployment paradigms: • API-based: Gemini-3.5-Flash-Lite (Team et al. 2023) serves as the API baseline. • Cascade: Whisper-Large-v3 (Radford et al. 2023) first transcribes speech, after which NLLB-200-3.3B (NLLB Team et al. 2024) translates the transcription. • End-to-end: SeamlessM4T-V2-Large (Barrault et al. 2023), Qwen3-Omni-30B (Xu et al. 2025), and MCAT-27B (Du et al. 2026) directly generate translations from speech and cover both specialized and general-purpose multilingual architectures. Evaluation Metrics. We employ COMET (Rei et al. 2022) and spBLEU (Post 2018) as our evaluation metrics. Specifically, spBLEU uses the FLORES-200 tokenizer. All results use a beam size of 1 unless stated otherwise. Table 3: Language support. Resource levels are defined following SeamlessM4T (Barrault et al. 2023). Resource Languages (ISO Code) High Arabic (ara), Bengali (ben), Catalan (cat) (EfE_f) Czech (ces), Chinese (cmn), German (deu) English (eng), Finnish (fin), French (fra) Italian (ita), Japanese (jpn), Dutch (nld) Polish (pol), Romanian (ron), Spanish (spa) Medium Danish (dan), Greek (ell), Hindi (hin) (EtE_t) Croatian (hrv), Hungarian (hun), Indonesian (ind) Kazakh (kaz), Korean (kor), Portuguese (por) Russian (rus), Slovak (slk), Tamil (tam) Tagalog (tgl), Thai (tha), Turkish (tur) Urdu (urd), Uzbek (uzb), Vietnamese (vie) Low Azerbaijani (azj), Bulgarian (bul), Persian (fas) (EtE_t) Hebrew (heb), Khmer (khm), Lao (lao) Malay (msa), Burmese (mya), Norwegian (nob) Slovenian (slv), Swedish (swe), Cantonese (yue) Total 45 languages (15 high, 18 medium, and 12 low) EfE_f denotes the frozen expert, and EtE_t denotes the trainable expert. Table 4: spBLEU / COMET results for 11×4411× 44 and 44×1144× 11 directions on FLEURS. Direction Gemini-3.5 Whisper+ SeamlessM4T Qwen3-Omni MCAT MSRT-4B -Flash-Lite NLLB-3.3B -V2-Large -30B-Instruct -27B (ours) ara→ 44 22.2 / 80.2 21.1 / 78.7 15.7 / 70.1 20.5 / 78.8 20.0 / 79.7 23.7 / 82.0 cmn→ 44 17.2 / 79.5 18.5 / 80.5 13.4 / 74.0 20.9 / 82.8 17.7 / 81.3 20.8 / 83.2 eng→ 44 34.2 / 86.6 30.5 / 84.3 31.8 / 85.3 31.9 / 85.7 31.9 / 87.1 34.2 / 87.9 hun→ 44 14.8 / 70.6 21.2 / 79.9 15.3 / 68.9 11.0 / 65.0 19.6 / 79.9 23.8 / 83.4 ind→ 44 27.4 / 84.6 23.7 / 82.2 15.2 / 71.5 24.7 / 83.4 24.2 / 84.2 27.3 / 85.4 jpn→ 44 18.4 / 80.8 18.9 / 80.5 11.9 / 69.5 19.5 / 81.8 18.0 / 80.8 20.4 / 82.3 kor→ 44 20.8 / 83.1 19.5 / 81.8 14.2 / 74.1 20.4 / 82.7 20.9 / 83.4 22.4 / 84.2 tam→ 44 17.3 / 78.9 12.9 / 71.4 12.6 / 69.8 2.5 / 50.4 13.0 / 74.0 17.9 / 79.4 tha→ 44 20.2 / 82.0 17.1 / 78.6 11.5 / 69.0 18.8 / 81.3 15.6 / 78.9 20.8 / 82.9 tur→ 44 25.6 / 83.7 23.4 / 83.0 15.8 / 72.1 23.2 / 82.5 24.0 / 84.1 25.7 / 85.2 vie→ 44 19.7 / 80.1 18.7 / 78.9 14.1 / 72.6 20.0 / 81.3 18.0 / 80.5 21.4 / 82.7 44→ 21.1 / 78.3 14.3 / 71.2 11.3 / 65.4 21.3 / 78.4 21.5 / 80.2 25.2 / 83.3 44→ 24.9 / 79.2 31.7 / 80.5 31.1 / 83.3 28.1 / 79.7 27.5 / 80.6 34.0 / 84.1 44→ 20.6 / 79.7 18.4 / 75.7 13.8 / 67.9 15.9 / 74.4 17.8 / 79.7 21.4 / 82.4 44→ 21.7 / 80.9 22.9 / 80.5 16.1 / 76.7 21.0 / 79.0 21.9 / 82.1 25.3 / 85.0 44→ 18.2 / 81.8 9.3 / 76.1 7.3 / 71.9 17.2 / 81.0 18.2 / 83.1 19.9 / 85.4 44→ 16.0 / 79.9 13.9 / 77.5 10.2 / 71.8 15.2 / 78.7 15.5 / 80.7 16.8 / 83.2 44→ 20.0 / 80.0 19.8 / 78.5 17.6 / 75.9 15.1 / 74.7 19.0 / 81.0 21.2 / 83.1 44→ 27.6 / 78.1 20.2 / 74.0 20.1 / 70.3 26.7 / 78.1 26.2 / 79.8 29.9 / 82.5 44→ 19.6 / 78.4 18.3 / 76.0 13.6 / 70.0 15.8 / 73.8 18.6 / 78.9 21.1 / 81.6 44→ 23.5 / 79.4 24.0 / 78.7 18.4 / 73.8 22.2 / 77.3 23.2 / 81.1 27.1 / 83.9 11→4411→44 21.6 / 80.9 20.5 / 80.0 15.6 / 72.4 19.4 / 77.8 20.3 / 81.3 23.5 / 83.5 44→1144→11 21.1 / 79.1 19.4 / 76.8 16.0 / 72.6 19.8 / 77.2 21.0 / 80.5 24.1 / 83.2 Avg. 21.3 / 80.0 20.0 / 78.4 15.8 / 72.5 19.6 / 77.5 20.6 / 80.9 23.8 / 83.3 Underlining denotes the second-best results, while bold blue values indicate the best performance. Experiments Main Results Language Selection. As shown in Table 4, the 11 languages cover 11 language families: Arabic (Afro-Asiatic), Mandarin Chinese (Sino-Tibetan), English (Indo-European), Hungarian (Uralic), Indonesian (Austronesian), Japanese (Japonic), Korean (Koreanic), Tamil (Dravidian), Thai (Kra–Dai), Turkish (Turkic), and Vietnamese (Austroasiatic). Overall Analysis. According to the overall average in Table 4, MSRT-4B achieves the highest COMET score of 83.3, outperforming MCAT-27B (80.9), Gemini-3.5-Flash-Lite (80.0), Whisper+NLLB-200-3.3B (78.4), Qwen3-Omni (77.5), and SeamlessM4T-V2-Large (72.5). Moreover, MSRT-4B ranks first in both the 11×4411× 44 and 44×1144× 11 settings, showing that its overall advantage is sustained across translation directions rather than driven by performance in a single setting. Translation Direction Analysis. Performance differs notably between 11×4411× 44 and 44×1144× 11 directions. The cascaded system achieves a competitive COMET score of 80.0 in the former setting but drops to 76.8 in the latter, indicating that translation from a broader set of source speech languages is more challenging, partly due to accumulated multilingual recognition errors. In contrast, MSRT-4B achieves COMET scores of 83.5 and 83.2, respectively, demonstrating the strongest performance and robust consistency across translation directions. Eng→ 44 S2T Overall Analysis of Eng→ 44 Directions. Table 5 reports COMET performance across all 44 English-source translation directions, including the newly evaluated Gemini-3.5-Flash-Lite baseline. MSRT-4B achieves the highest average score of 87.9, outperforming MCAT-27B, Gemini-3.5-Flash-Lite, Qwen3-Omni, and SeamlessM4T-V2-Large by 0.8, 1.3, 2.2, and 2.6 points, respectively. These results demonstrate the strong and consistent English-to-many translation performance of MSRT-4B. English-centric vs. Many-to-Many Translation. Although optimized for many-to-many translation, MSRT-4B also performs strongly on English-source directions, achieving an average COMET score of 87.9 on eng→ 44 and an overall score of 83.3 in Table 4. These results suggest that future speech translation models should emphasize comprehensive multilingual optimization rather than focusing predominantly on English-centric directions. High-Resource vs. Low-Resource Languages. Table 5 shows that Qwen3-Omni performs strongly on high-resource languages such as Chinese (cmn) and Japanese (jpn), but degrades markedly on low-resource languages such as Burmese (mya) and Khmer (khm). In comparison, MSRT-4B maintains consistently strong performance across both resource regimes. This advantage stems from MoSE, whose trainable expert specializes in medium- and low-resource languages, while its frozen expert preserves high-resource performance. Table 5: COMET results on English →44→ 44 directions. ISO Gemini-3.5 Seamless Qwen3 MCAT MSRT -Flash-Lite M4T -Omni -27B -4B ara 83.8 84.5 86.6 86.1 86.0 azj 85.8 83.8 83.1 84.8 87.1 ben 85.1 84.6 83.3 85.2 86.2 bul 88.9 88.8 89.1 90.0 90.3 cat 85.3 85.0 86.2 85.9 86.9 ces 89.1 88.0 88.9 90.1 90.4 cmn 86.4 79.7 88.3 87.2 87.8 dan 88.7 88.7 89.1 89.5 90.1 deu 86.2 84.9 86.8 86.5 86.9 ell 87.1 87.5 87.1 88.7 89.1 fas 86.5 84.6 85.6 86.9 87.1 fin 90.9 88.5 88.7 91.3 91.5 fra 85.8 85.3 87.4 86.3 86.8 heb 85.2 84.5 73.8 87.2 87.8 hin 79.0 78.1 77.7 78.9 79.3 hrv 89.0 87.8 87.8 89.2 90.0 hun 88.1 86.0 86.8 87.6 88.4 ind 89.8 89.0 91.0 90.2 90.7 ita 86.3 85.1 87.3 86.8 87.3 jpn 89.7 84.7 90.9 90.4 90.6 kaz 88.6 87.9 85.5 88.2 89.9 khm 81.4 79.9 77.4 79.7 83.9 kor 88.4 85.1 89.7 88.5 89.0 lao 81.4 81.4 80.3 83.1 84.5 msa 87.6 86.6 88.2 87.5 88.5 mya 87.1 85.7 71.9 85.0 88.3 nld 85.5 85.1 86.0 86.6 87.0 nob 87.9 86.8 88.4 88.7 89.2 pol 87.1 85.7 87.3 88.1 88.3 por 87.3 86.6 88.5 87.9 88.2 ron 88.5 87.8 88.8 89.1 89.8 rus 87.8 86.3 88.9 88.8 88.8 slk 88.4 87.3 87.1 88.8 89.6 slv 87.8 86.8 85.5 88.5 89.0 spa 84.2 83.2 85.4 85.2 85.5 swe 88.7 88.4 88.7 89.3 89.9 tam 87.6 87.3 85.3 88.3 88.5 tha 87.1 82.1 80.1 83.3 88.3 tgl 82.9 83.3 88.9 87.7 83.9 tur 88.3 86.7 88.1 88.2 88.7 urd 81.2 79.4 78.3 80.9 82.0 uzb 88.2 87.7 79.7 88.2 89.6 vie 87.1 85.6 88.6 87.8 88.3 yue 81.7 79.8 88.6 88.0 88.3 Avg. 86.6 85.3 85.7 87.1 87.9 Table 6: COMET results across the 45×4445× 44 directions. Model ≥80≥ 80 [70,80)[70,80) <70<70 Whisper+NLLB-3.3B 1038 680 262 Gemini-3.5-Flash-Lite 928 815 237 SeamlessM4T-V2-Large 117 1109 754 Qwen3-Omni-30B-Instruct 881 557 542 MCAT-27B 1232 554 194 MSRT-4B (ours) 1552 359 69 Figure 3: COMET results for all directions. Shaded regions highlight scores falling below 80 and 70. Scores for identical source and target languages, such as eng→ along the diagonal, are smoothed for visualization. Systematic Analysis on 45×4445× 44 Directions Main Results. Table 6 shows that MSRT-4B achieves COMET scores of at least 80 on 1,552 of the 1,980 directions. MCAT-27B, Whisper+NLLB-3.3B, Gemini-3.5-Flash-Lite, Qwen3-Omni-30B-Instruct, and SeamlessM4T-V2-Large achieve scores of at least 80 on 1,232, 1,038, 928, 881, and 117 directions, respectively. These results demonstrate the broader and more reliable many-to-many translation capabilities of MSRT-4B. API Comparison. MSRT-4B outperforms the Gemini-3.5-Flash-Lite API, achieving COMET scores of at least 80 on 1,552 directions, compared with 928 for Gemini, and scores below 70 on only 69 directions, compared with 237 for Gemini. This result demonstrates broader language coverage and stronger overall performance than the API. English-Centric Bias. SeamlessM4T-V2-Large is centered on English, with its strongest performance concentrated on directions involving English. Its quality drops on non-English-to-non-English translation, yielding only 117 directions with COMET scores of at least 80 and 754 with scores below 70. In contrast, MSRT-4B remains consistent across the translation directions. Parameter Efficiency. With only 4B parameters, MSRT-4B surpasses the much larger Qwen3-Omni-30B-Instruct and MCAT-27B, achieving COMET scores of at least 80 on 1,552 directions versus 881 and 1,232, respectively. It also produces fewer low-scoring directions, demonstrating efficient use of model capacity and broader multilingual coverage without relying on model scale. Figure 4: MoSE ablation and S2T–MT comparison across resource levels. Consistent gains, largest for low-resource languages, show that MoSE breaks the curse of multilinguality. Table 7: Ablation results of MSRT-4B on X→44X\!→\!44 translation. Configuration ara ben khm mya tam uzb Avg. MSRT-4B (2 experts) 82.0 80.6 71.5 60.5 79.4 76.5 75.1 3 experts 81.9 80.7 71.5 62.5 78.9 76.6 75.3 1 expert 80.0 76.8 65.6 53.7 74.2 69.6 70.0 w/o LoRA 81.2 78.9 68.7 56.6 77.7 75.5 73.1 Ablation Study Component Ablations. Table 7 evaluates MSRT-4B on six relatively low-performing source languages, with two selected from each resource group, translated into the other 44 languages. • Number of experts. Three-way routing, with one expert for each resource group, performs similarly to two-way routing between high- and medium/low-resource languages (75.3 vs. 75.1), whereas using a single expert substantially reduces the average COMET score to 70.0. These results show that two experts provide sufficient resource-aware specialization for the current 45 languages, while larger language inventories may benefit from finer-grained expert partitioning. • LoRA. Removing LoRA lowers average COMET from 75.1 to 73.1, showing that lightweight decoder adaptation complements MoSE by mapping speech representations into the LLM’s text space. Resource-Level Analysis. Figure 4 compares MoSE variants and text MT across 45 languages by resource level; each score averages the other 44 target directions. • MoSE gains across 45 languages. MoSE improves COMET by 3.0, 2.2, and 1.4 points for low-, medium-, and high-resource languages, respectively. Gains across all groups show that MoSE breaks the curse of multilinguality by substantially improving low-resource translation while also strengthening high-resource performance. • Text MT vs. S2T. Text MT with source transcriptions provides an oracle upper bound for S2T. Its gap to MSRT-4B narrows as speech resources increase, identifying speech representation and cross-modal alignment as the main bottlenecks. Figure 5: Data scaling on 11 CoVoST-2 English-to-X directions. MSRT-4B and MSRT-4B-SFT use 7.5 hours of FLEURS and 429.6 hours of CoVoST-2 speech, respectively; dashed lines denote average COMET. Data Scaling. Figure 5 shows strong zero-shot transfer: with only 7.5 h of FLEURS English speech, MSRT achieves scores of at least 80 on all 11 English-to-X directions, averaging 83.3. Scaling to 429.6 h of CoVoST-2 speech (57.3×57.3×) raises the average to 85.3, with consistent gains of 1.3–2.6 points across directions. Conclusion We introduced MSRT, a 4B many-to-many S2T framework combining resource-aware MoSE with five-stage curriculum learning. By separating high-resource preservation from medium- and low-resource adaptation, MoSE improves all resource groups, with the largest gains on low-resource speech. Evaluation of all 1,980 directions among 45 languages demonstrates strong multilingual consistency, parameter efficiency, and cross-dataset generalization, showing that MoSE effectively breaks the curse of multilinguality. Future work will extend MoSE to broader language coverage. Limitations MSRT derives its S2T capability from the machine translation knowledge of the pretrained LLM. Consequently, its translation quality is bounded by the LLM’s underlying MT performance, particularly for languages and directions that are weakly represented during pretraining. References R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber (2020) Common voice: a massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis (Eds.), Marseille, France, p. 4218–4222 (eng). External Links: Link, ISBN 979-10-95546-34-4 Cited by: Datasets.. L. Barrault, Y. Chung, M. C. Meglioli, D. Dale, N. Dong, P. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman, et al. (2023) SeamlessM4T: massively multilingual and multimodal machine translation. arXiv preprint arXiv:2308.11596. Cited by: Speech-to-Text Translation., Many-to-Many Speech-to-Text Translation., 3rd item, Table 3. T. Blevins, T. Limisiewicz, S. Gururangan, M. Li, H. Gonen, N. A. Smith, and L. Zettlemoyer (2024) Breaking the curse of multilinguality with cross-lingual expert language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 10822–10837. External Links: Link, Document Cited by: Introduction. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020) Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 8440–8451. Cited by: Introduction. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna (2023) FLEURS: few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop, p. 798–805. Cited by: Tokenizer Expansion.. Y. Du, K. Liu, Y. Pan, B. Yang, K. Deng, X. Chen, Y. Xiang, M. Liu, B. Qin, and Y. Wang (2026) MCAT: scaling many-to-many speech-to-text translation with MLLMs to 70 languages. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: Introduction, Introduction, Many-to-Many Speech-to-Text Translation., 3rd item. Y. Du, Y. Pan, Z. Ma, B. Yang, Y. Yang, K. Deng, X. Chen, Y. Xiang, M. Liu, and B. Qin (2025) Making llms better many-to-many speech-to-text translators with curriculum learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12466–12478. Cited by: Introduction, Many-to-Many Speech-to-Text Translation., Five-Stage Curriculum Learning. C. Le, B. Han, J. Li, S. Chen, and Y. Qian (2025) Simulmega: moe routers are advanced policy makers for simultaneous speech translation. Advances in Neural Information Processing Systems 38, p. 42739–42761. Cited by: Mixture of Experts.. Y. Li, S. Jiang, B. Hu, L. Wang, W. Zhong, W. Luo, L. Ma, and M. Zhang (2025) Uni-moe: scaling unified multimodal llms with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (5), p. 3424–3439. External Links: Document Cited by: Mixture of Experts.. Z. Ma, G. Yang, W. Chen, Z. Gao, Y. Du, X. Li, Z. Zheng, H. Zhu, J. Zhuo, Z. Song, R. Xu, T. Wang, Y. Yang, Y. Zhu, Z. Niu, L. Xue, Y. Ma, R. Yuan, S. Zhang, K. Yu, E. S. Chng, and X. Chen (2026) SLAM-llm: a modular, open-source multimodal large language model framework and best practice for speech, language, audio and music processing. IEEE Journal of Selected Topics in Signal Processing 20 (1), p. 63–76. External Links: Document Cited by: Introduction. NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, et al. (2024) Scaling neural machine translation to 200 languages. Nature 630 (8018), p. 841–846. Cited by: 2nd item. M. Post (2018) A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation, p. 186–191. Cited by: Evaluation Metrics.. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, p. 28492–28518. Cited by: Introduction, Introduction, 2nd item. R. Rei, J. G. C. De Souza, D. Alves, C. Zerva, A. C. Farinha, T. Glushkova, A. Lavie, L. Coheur, and A. F. T. Martins (2022) COMET-22: unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation, p. 578–585. Cited by: Evaluation Metrics.. W. Shan, Y. Li, Y. Zhang, Y. Luo, C. Xu, X. Zhao, L. Meng, Y. Lu, M. Zhang, H. Yang, T. Xiao, and J. Zhu (2025) Enhancing speech large language models with prompt-aware mixture of audio encoders. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 19305–19320. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: Mixture of Experts.. Y. Shang, P. Gao, W. Liu, J. Luan, and J. Su (2026) Scaling model and data for multilingual machine translation with open large language models. External Links: 2602.11961 Cited by: Introduction, Training Details.. G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: 1st item. I. Tsiamas, G. I. Gállego, J. A. Fonollosa, and M. R. Costa-jussà (2024) Pushing the limits of zero-shot end-to-end speech translation. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14245–14267. Cited by: Speech-to-Text Translation.. C. Wang, A. Wu, J. Gu, and J. Pino (2021) CoVoST 2 and massively multilingual speech-to-text translation. p. 2247–2251. Cited by: Datasets.. J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, et al. (2025) Qwen3-Omni technical report. arXiv preprint arXiv:2509.17765. Cited by: Introduction, Many-to-Many Speech-to-Text Translation., 3rd item. J. Zhou, S. Gao, Z. Yu, L. Dong, and W. Wang (2024) DialectMoE: an end-to-end multi-dialect speech recognition model with mixture-of-experts. In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), S. Maosong, L. Jiye, H. Xianpei, L. Zhiyuan, and H. Yulan (Eds.), Taiyuan, China, p. 1148–1159 (eng). External Links: Link Cited by: Mixture of Experts..