Paper deep dive
UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Shengbo Cai, Zhisheng Zhang, Zichao Nie, Jing Peng, Jingran Xie, Zhiyong Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/6/2026, 10:07:23 AM
Summary
The paper introduces UniAE-MoE, a unified audio encoder leveraging a Mixture-of-Experts (MoE) architecture to integrate Qwen2-Audio and Audio-Flamingo 3 backbones. It employs SwiGLU-based experts, a two-stage instruction-tuning strategy, and Task-Specific Data Scaling (TSDS) to achieve state-of-the-art performance on the XARES-LLM benchmark (score 0.802) and the Interspeech 2026 Audio Encoder Capability Challenge.
Entities (8)
Relation Signals (6)
UniAE-MoE → achievesscoreon → XARES-LLM
confidence 95% · On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802
UniAE-MoE → usesarchitecture → Mixture-of-Experts
confidence 95% · achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture.
UniAE-MoE → usesbackbone → Qwen2-Audio
confidence 95% · integrate those from Qwen2-Audio and Audio-Flamingo 3... We therefore combine these encoders into a dual-model backbone
UniAE-MoE → usesbackbone → Audio-Flamingo-3
confidence 95% · integrate those from Qwen2-Audio and Audio-Flamingo 3... We therefore combine these encoders into a dual-model backbone
UniAE-MoE → usestechnique → SwiGLU
confidence 90% · we improve our encoder using SwiGLU with shared experts to decouple encoder networks
UniAE-MoE → usestechnique → Task-Specific Data Scaling
confidence 90% · we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE’s understanding capabilities.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE's understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of UniAE-MoE for unified audio understanding across speech, music, and general audio domains.
Tags
Links
- Source: https://arxiv.org/abs/2609.39199v2
- Canonical: https://arxiv.org/abs/2609.39199v2
Trouble viewing inline? Open PDF directly →
Full Text
35,129 characters extracted from source content.
Expand or collapse full text
UniAE-MoE: A Unified Audio Encoder via Mixture of Experts Shengbo Cai* Zhisheng Zhang* Zichao Nie Jing Peng Jingran Xie Zhiyong Wu✉ Affiliation: Tsinghua University, China E-mail csb25@mails.tsinghua.edu.cn, zywu@sz.tsinghua.edu.cn Affiliation: Affiliation: Equal contribution. Abstract Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE’s understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of UniAE-MoE for unified audio understanding across speech, music, and general audio domains. Keywords: Large Audio Language Models Unified Audio Encoder Mixture of Experts 1 Introduction Large Audio Language Models are shifting from simple audio-to-text conversion toward “General Audio Intelligence”. By integrating powerful audio encoders with Large Language Models (LLMs) [15], recent models, e.g., AudioPaLM [21] and Kimi-Audio [9], unify speech, general audio, and music perception. More recently, Qwen2-Audio [7] leverages natural language prompts to enhance instruction-following across diverse benchmarks. These advancements underscore the necessity of high-quality audio encoders that provide generalized representations for a broad spectrum of audio tasks. Despite these advancements, developing a unified audio encoder remains challenging. Traditional pipelines typically separate audio classification from comprehension and generation, often relying on task-specific heads that limit architectural flexibility and impede knowledge transfer across domains. Meanwhile, current LALMs commonly employ monolithic encoders or dense shared adapters [9, 12, 20], forcing a single parameter space to capture heterogeneous acoustic characteristics ranging from fine-grained temporal cues to high-level semantics. General-purpose representation models such as AudioMAE [14] and BEATs [6], although effective for perception and classification, are not inherently aligned with generative audio-to-language inference and instruction-following objectives. Integrating multiple encoders, as explored by SALMONN [24] and WavLLM [13], provides access to complementary representations, but straightforward fusion can introduce feature redundancy and performance trade-offs, particularly in speech recognition. Recent advances in MoE-based adapters [16, 22] indicate that sparsely routing inputs to specialized experts can decouple audio attributes or sub-domains while retaining shared knowledge, offering a promising direction for robust multi-task audio modeling. In this work, we introduce UniAE-MoE, a novel unified audio encoder designed to explicitly decouple the often conflicting demands of cross-domain audio using Mixture of Experts. By leveraging instructionalized unified training objectives, we converge multi-task learning into a cohesive generative optimization framework. Concretely, we employ two encoders with top-performance, Qwen2-Audio and Audio-Flamingo 3, and exploit the MoE architecture to decouple the parameter space, allowing experts to specialize in acoustic modalities. To facilitate the extraction of high-quality, task-tailored semantic features from raw audio, we unify 20 downstream tasks into an ‘instruction-response’ text sequence generation paradigm and devise a two-stage training strategy that advances from semantic alignment to multi-task instruction fine-tuning. Considering the unbalance of each task, we propose the task-specific data scaling method for better understanding performance. Our methodology achieves robust generalization across speech, music, and ambient sounds, delivering excellent performance in both classification and generation tasks simultaneously. We release our code11 1 https://github.com/Syclus123/UniAE-MoE for future comparison. Our contributions are as follows: • We propose UniAE-MoE, a unified audio encoder that integrates dual high-performance backbones (Qwen2-Audio and Audio-Flamingo 3) via a SwiGLU-based MoE fusion module with a shared expert, enabling the effective decoupling and fusion of cross-domain audio representations. • We establish a unified generative paradigm that reformulates 20 diverse downstream tasks into an instruction-following framework, complemented by a two-stage instruction-tuning strategy that facilitates a seamless transition from modality alignment to robust multi-task generalization. • We introduce Task-Specific Data Scaling to mitigate cross-task data imbalance. UniAE-MoE achieves state-of-the-art performance on the XARES-LLM benchmark with a high score of 0.802, and further demonstrates strong generalization in Interspeech 2026 Audio Encoder Capability Challenge, attaining top-tier performance across both classification and understanding tasks. Figure 1: The UniAE-MoE framework comprises the overall architecture (top), the SwiGLU-based expert network (bottom left), and the two-stage instruction-tuning strategy (bottom right). 2 Methodology 2.1 High-Performance Dual-Model Backbone Using Dasheng-base as a baseline, we benchmark representative encoders, including the Qwen series, Audio-Flamingo, and Whisper. As shown in Table 3, Audio-Flamingo 3 and Qwen2-Audio achieve the two highest overall scores of 0.767 and 0.737, respectively. Despite its strong performance, Qwen2-Audio’s pre-training distribution places greater emphasis on speech and music, with comparatively limited coverage of general acoustic events. Audio-Flamingo 3, in contrast, exhibits more balanced capabilities for long-form audio and environmental sounds, complementing Qwen2-Audio in general audio modeling. We therefore combine these encoders into a dual-model backbone and investigate their fusion for robust, fine-grained unified audio modeling, with the corresponding analyses presented in Section 3.2. Table 1: Comparison of different audio encoders on the XARES-LLM benchmark Num # Task Baseline Model Fusion (Concat) UniAE-MoE (Ours) Dasheng-Base Whisper-Base (Q2A, DS) (Q2A, KA) (Q2A, AF3) (Q2.5O, DS) (AF3, KA) (Q2A, AF3, DS) Task 1 Performance 1 ASVSpoof 2015 0.937 0.943 0.980 0.978 0.985 0.986 0.995 0.994 0.982 2 CREMA-D 0.621 0.516 0.732 0.795 0.855 0.626 0.858 0.808 0.847 3 ESC-50 0.755 0.635 0.900 0.898 0.932 0.865 0.922 0.915 0.882 4 Fluent Speech Commands 0.984 0.817 0.983 0.992 0.994 0.993 0.982 0.993 0.994 5 Free Music Archive 0.429 0.579 0.718 0.720 0.887 0.632 0.723 0.904 0.979 6 FSD50K 0.063 0.092 0.201 0.221 0.239 0.150 0.236 0.239 0.251 7 FSDKaggle2018 0.415 0.552 0.770 0.798 0.852 0.727 0.832 0.851 0.872 8 GTZAN 0.323 0.697 0.939 0.909 0.869 0.859 0.899 0.889 0.899 9 LibriCount 0.386 0.409 0.108 0.150 0.190 0.115 0.116 0.161 0.381 10 NSynth 0.675 0.638 0.753 0.681 0.784 0.710 0.784 0.788 0.782 11 Speech Commands V1 0.655 0.694 0.929 0.927 0.936 0.959 0.928 0.931 0.968 12 UrbanSound8K 0.829 0.737 0.846 0.828 0.898 0.864 0.897 0.883 0.888 13 VocalSound 0.855 0.867 0.941 0.938 0.948 0.935 0.947 0.947 0.941 14 VoxCeleb1 0.974 0.762 0.970 0.973 0.977 0.627 0.980 0.982 0.974 15 VoxLingua33 0.311 0.835 0.876 0.960 0.897 0.886 0.952 0.890 0.940 Task 1 0.614 0.652 0.777 0.785 0.816 0.729 0.803 0.812 0.839 Task 2 Performance 16 AISHELL-1 0.018 0.361 0.700 0.780 0.796 0.733 0.388 0.718 0.882 17 Clotho 0.207 0.358 0.455 0.458 0.469 0.418 0.460 0.465 0.449 18 LibriSpeech 0.103 0.385 0.824 0.882 0.881 0.843 0.396 0.832 0.914 19 MECAT 0.600 0.624 0.616 0.617 0.660 0.620 0.646 0.647 0.700 20 SongDescriber 0.410 0.448 0.494 0.502 0.503 0.471 0.525 0.494 0.512 Task 2 0.270 0.400 0.618 0.648 0.662 0.617 0.483 0.613 0.691 Overall Performance (Task 1 + Task 2) Overall 0.528 0.597 0.737 0.750 0.778 0.708 0.723 0.766 0.802 • Model Abbreviation. (1) Q2A: Qwen2-Audio. (2) DS: Dasheng-1.2B. (3) KA: Kimi-Audio. (4) AF3: Audio-Flamingo 3. (5) Q2.5O: Qwen2.5-Omni. 2.2 Universal Audio Representation with MoE Fusion Across heterogeneous tasks such as speech recognition, audio classification, and audio captioning, a single encoder often struggles to simultaneously preserve fine-grained acoustic details and high-level semantics. Although Audio-Flamingo 3 and Qwen2-Audio share a Whisper-based backbone, their distinct pre-training objectives and instruction-tuning strategies encourage them to encode complementary acoustic and semantic information. To effectively reconcile these heterogeneous features, we introduce a Mixture-of-Experts (MoE) framework to derive universal audio representations. As illustrated in Fig. 1, the routing mechanism dynamically activates specialized experts according to the input characteristics, allowing different audio attributes to be modeled in separate parameter spaces while retaining task adaptability and parameter efficiency. Specifically, we project the feature sequences FAF_AF from Audio-Flamingo 3 and FQWF_QW from Qwen2-Audio into a unified embedding space. Both audio sequences undergo temporal and feature-wise alignment, followed by concatenation and normalization along the feature dimension to obtain the composite feature vector x. Because the concatenated dual-encoder features remain heterogeneous and audio signals contain substantial time–frequency redundancy, we instantiate each expert with SwiGLU [23]. Its feature-wise gating and multiplicative interactions selectively preserve informative dimensions while reconciling acoustic and semantic features from the two encoders: SwiGLU(x)=[ SiLU(xWgate)⊙(xWup)]Wdown,SwiGLU(x)= [SiLU(xW_gate) (xW_up) ]W_down, (1) where WgateW_gate, WupW_up, and WdownW_down denote the learnable parameter matrices within the expert layer. The element-wise multiplication between the two projection branches introduces multiplicative feature interactions, enabling each expert to more effectively integrate heterogeneous representations from the two audio encoders. Unlike a dense feed-forward layer that applies the same set of parameters to every token, our MoE module performs token-level conditional expert aggregation. Specifically, an MLP-based router computes routing scores for each token xtx_t over N experts. The top-K experts are then selected, and their outputs are combined using normalized routing weights: yrouted=∑i=1K Softmax(G(xt))i⋅Ei(xt),y_routed= _i=1^KSoftmax(G(x_t))_i· E_i(x_t), (2) where EiE_i denotes the i-th selected SwiGLU expert. Activating only a subset of experts encourages domain- and task-specific specialization while improving the computation-to-capacity ratio for efficient inference. To further enhance modeling robustness in complex audio environments, we incorporate a Shared Expert mechanism. Distinct from the experts dynamically activated via routing, the shared expert EsharedE_shared processes every token, aiming to capture invariant acoustic features across diverse tasks. The final output representation is a dynamic reconciliation of the routed result and the shared expert’s output, modulated by a learnable gating parameter gsharedg_shared: alignedα=σ(g shared),Y=RMSNorm(αy routed+(1−α)E shared(x)). α&=σ(g_shared),\\ Y=RMSNorm (α\,y_routed+(1-α\,)E_shared(x) ). (3) By doing so, we not only preserve the complementarity of features across different models but also mitigate the issue of expert collapse through the shared expert, yielding more robust representations for multi-task audio understanding. 2.3 Multi-task Instruction Tuning To support diverse audio tasks within a single architecture, we reformulate 20 downstream tasks under a unified instruction-following paradigm. Specifically, both discriminative tasks, whose labels are serialized as textual responses, and open-ended generation tasks are cast as autoregressive conditional sequence modeling problems. This formulation eliminates task-specific prediction heads and enables joint optimization using the standard Next Token Prediction (NTP) objective of the LLM. Under this shared objective, end-to-end training encourages the MoE router to retain semantically informative features across task domains. Formally, given an audio sequence A and a corresponding instruction I, the model generates a target response Y=y1,…,yLY=\y_1,…,y_L\ by maximizing its conditional likelihood: P(Y∣A,I;θ)=∏i=1LP(yi∣y<i,moe,I;θ),P(Y A,I;θ)= _i=1^LP(y_i y_<i,F_moe,I;θ), (4) where moeF_moe denotes the universal audio representation produced by MoE fusion. To bridge the modality gap between raw acoustic features and the LLM’s embedding space, we employ a two-stage training strategy that provides a stable transition from low-level acoustics to high-level semantic understanding. Table 2: Statistics of our training data Task Dataset Train # New Task 1 for Stage-2 Training Spoofing Detection ASVSpoof 2015 15,720 - Emotion Recognition CREMA-D 6,000 - IEMOCAP/TESS/RAVDESS 3,552 ✔ Environment Class ESC-50 1,600 - Intent Classification Fluent Speech Commands 23,132 - Music Genre Class Free Music Archive 6,400 - Free Music Archive (medium) 25,000 ✔ Sound Event Detection FSD50K 4,009 - Sound Event Detection FSDKaggle2018 9,473 - Genre Classification GTZAN 900 - Speaker Counting LibriCount 4,576 - Instruments Class NSynth 289,205 - Keyword Spotting SpeechCommand_v1 51,088 - Urban Sound Class. UrbanSound8K 7,895 - Non-speech Sounds VocalSound 15,531 - Binary Speaker ID VoxCeleb1 22,576 - Language ID VoxLingua33 66,000 - Task 2 for Stage-1 and Stage-2 Training Speech Recognition AISHELL-1 120,098 - Sound Caption Clotho 3,839 - Speech Recognition LibriSpeech 28,539 - Audio Caption MECAT 20,039 - MACS 3,930 ✔ AudioSetCaps 1,729,996 ✔ Music Caption SongDescriber 359 - MusicBench 52,768 ✔ MusicCaps 5,380 ✔ In the first stage, we leverage large-scale sequence-level supervision from Automatic Speech Recognition (ASR) and audio captioning to map MoE-fused audio features into the LLM’s embedding space, establishing preliminary audio–language alignment. We freeze both the audio encoders and the LLM while optimizing the MoE module and projector, thereby protecting the LLM’s pre-trained linguistic priors from noise introduced by initially unaligned audio features. In the second stage, we keep the audio encoders frozen and jointly fine-tune the remaining components using diverse instruction-following data covering all 20 tasks. This allows the instruction I to guide the extraction of task-relevant information from the aligned MoE representations. Despite employing a lightweight 135M-parameter LLM, the resulting model achieves strong cross-task generalization, highlighting the effectiveness of the fused audio representations and staged optimization. 2.4 Task-Specific Data Scaling for Training As shown in Table 2, the training datasets vary substantially in scale, ranging from only a few hundred samples in SongDescriber to hundreds of thousands in NSynth. During unified multi-task optimization, this disparity can bias learning toward high-resource tasks, while low-resource tasks, particularly audio captioning, remain prone to overfitting and limited output diversity. To mitigate this cross-task imbalance, we propose Task-Specific Data Scaling (TSDS), which selectively augments underrepresented tasks with task-relevant public datasets rather than uniformly scaling all tasks. The newly introduced datasets are highlighted in Table 2. For captioning, we incorporate MusicBench [18], MusicCaps [1], MACS [19], and AudioSetCaps [4], with AudioSetCaps alone providing approximately 1.73M audio–text pairs. These additions expand both acoustic and linguistic diversity, promoting robust cross-modal alignment and more diverse caption generation. For classification tasks affected by class imbalance, we supplement genre and emotion recognition with Free Music Archive [8] and IEMOCAP [5], respectively, broadening category coverage and reducing bias toward dominant classes. 3 Experimental Results and Analyses 3.1 Experimental Setups Preprocess. To adhere to the 24GB VRAM competition limit, we employ memory-efficient preprocessing: audio clips are uniformly truncated to a maximum duration of 30 seconds to prevent out-of-memory (OOM) errors, and individual samples are processed iteratively to further minimize peak memory usage during feature extraction. The resulting modality-specific inputs are then fed into the Qwen2-Audio and Audio-Flamingo 3 branches as Log-Mel spectrograms and raw waveforms, respectively. Training Data. We train on diverse public datasets, including only the official training split of XARES-LLM and supplementary datasets for scaling. To improve multi-task learning and mitigate cross-task data imbalance, we apply Task-Specific Data Scaling (TSDS) tailored to each task. Details of TSDS and training data statistics are provided in Section 2.4 and Table 2, respectively. Training Details. We train UniAE-MoE for 30,000 and 80,000 steps in Stages 1 and 2, respectively, using three NVIDIA L20 GPUs. The MoE comprises four routed experts and one shared expert with top-2 routing (K=2K=2), and each expert network has a hidden dimension of 3,413. As expert utilization remains balanced, we do not apply an auxiliary load-balancing loss. Evaluation. We evaluate all models on the 20 public tasks of XARES-LLM [27] under a unified generative protocol that replaces task-specific heads with text prediction. Task 1 (public Track A) comprises 15 speech, sound, and music classification datasets, evaluated by accuracy for single-label tasks and mean average precision (mAP) for multi-label tasks. Task 2 (public Track B) comprises five ASR and audio-captioning datasets, evaluated by inverted character or word error rate (iCER/iWER) and FENSE or DATE, respectively. The overall XARES-LLM score is macro-averaged across all datasets. The official Interspeech 2026 Audio Encoder Capability Challenge [11] further includes six hidden classification datasets and four hidden understanding datasets to assess generalization to unseen data. For each submission, the organizers freeze the encoder and train a lightweight projector and LoRA-adapted SmolLM2-135M decoder [2]. Track A and A (Hidden) use separate decoders because of their different label spaces, whereas B (Hidden) reuses the public Track B decoder for out-of-domain evaluation. Table 3: Benchmarking results of various audio encoders Encoder Domain Task 1 Task 2 Overall Audio Encoders for Specific Tasks. Whisper-large [20] Speech 0.698 0.441 0.634 FunASR-nano [3] Speech 0.552 0.516 0.543 MERT-330M [17] Music 0.475 0.236 0.415 Dasheng-base [10] Unified 0.540 0.233 0.463 Dasheng-1.2B [10] Unified 0.587 0.236 0.499 Audio Encoders from Large Audio Language Models. Kimi-Audio [9] Unified 0.720 0.435 0.648 Qwen3-Omni [26] Unified 0.693 0.559 0.659 Qwen2.5-Omni [25] Unified 0.732 0.597 0.698 Qwen2-Audio [7] Unified 0.770 0.638 0.737 Audio-Flamingo 3 [12] Unified 0.810 0.635 0.767 3.2 Benchmarking Audio Encoder We evaluate representative encoders from speech (Whisper-large), music (MERT-330M), and general audio (Dasheng-1.2B) domains on the XARES-LLM benchmark (Table 3). Notably, the audio encoders from pre-trained LALMs, specifically Qwen2-Audio and Audio-Flamingo 3, yield the highest average scores of 0.737 and 0.767, respectively (Table 3). Our analysis suggests that single-encoder models often exhibit domain-specific biases or coverage deficiencies. Motivated by this, we propose UniAE-MoE to leverage the complementary strengths of these leading LALM backbones for more robust unified audio modeling. 3.3 MoE Makes Better Fusion We compare our MoE with three fusion baselines—Concatenation (Concat), Addition (Add), and Gating—across two encoder pairs (Table 4). Here, Gating denotes a dense encoder-level fusion strategy that combines both encoder representations using a learnable coefficient; unlike the sparse MoE router, it performs neither conditional expert selection nor token-level routing. Results show that Concat consistently outperforms Add and Gating. We attribute this to Concat’s ability to preserve feature integrity and avoid the information blurring inherent in summation, thereby providing richer representations for the downstream LLM. Table 4: Comparison of model fusion strategies (Q2A, AF3) (Q2A, KA) Concat Gating MoE (Ours) Concat Add MoE (Ours) Task 1 0.816 0.800 0.839 0.785 0.778 0.816 Task 2 0.662 0.660 0.691 0.648 0.604 0.682 Overall 0.778 0.764 0.802 0.750 0.735 0.782 Table 5: Ablation study of the UniAE-MoE Params (M) Task 1 Task 2 Overall UniAE-MoE 115.8 0.839 0.691 0.802 w/o SwiGLU 85.2 0.827 0.693 0.794 UniAE-MoE-Light 88.5 0.832 0.693 0.797 Further exploration of encoder combinations (Table 1) reveals that the dual-model setup (Qwen2-Audio and Audio-Flamingo 3) achieves the best baseline (0.778 on XARES-LLM). Interestingly, a triple-model configuration (adding Dasheng-1.2B) degrades performance, suggesting that indiscriminate encoder stacking introduces redundancy rather than complementarity. Leveraging the optimal dual-model backbone, our UniAE-MoE further improves the score to 0.802. This significant gain over the best Concat baseline demonstrates that the MoE framework more effectively captures and exploits the synergies between complementary audio encoders. 3.4 Ablation Study We first evaluate the impact of the SwiGLU activation function on the MoE experts. As shown in Table 5, removing SwiGLU degrades the overall score from 0.802 to 0.794, confirming its importance in enhancing the non-linear expressivity of the MoE architecture. To assess parameter efficiency, we introduce UniAE-MoE-Light, a compressed variant with 88.5 M parameters. Despite its reduced footprint, UniAE-MoE-Light maintains competitive performance (0.797), demonstrating the robustness of our architecture. These results validate the parameter efficiency of UniAE-MoE and suggest that our data scaling strategy effectively compensates for reduced model capacity. 3.5 Data Scaling Strategies To address task imbalance within XARES-LLM, we implement Task-Specific Data Scaling (TSDS) for music, emotion, and audio captioning. As shown in Table 6, TSDS substantially improves the standard MoE from 0.759 to 0.794, whereas the higher-capacity MoE-SwiGLU exhibits a smaller aggregate gain from 0.799 to 0.802. This contrast suggests that the stronger architecture may already capture much of the available overall improvement. Nevertheless, TSDS remains important for low-resource or imbalanced tasks: on Free Music Archive, for example, it improves performance from 0.848 to 0.979, demonstrating that aggregate scores can understate its task-specific benefits. Table 6: The table compares data-scaling performance across tasks; “w/o” denotes training without TSDS, whereas “w” denotes training with TSDS. Model Method Task 1 Task 2 Overall 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 MoE Ours 0.987 0.856 0.905 0.994 0.932 0.253 0.858 0.929 0.207 0.771 0.966 0.886 0.947 0.973 0.939 0.851 0.474 0.904 0.688 0.550 0.794 w/o TSDS 0.981 0.841 0.890 0.993 0.923 0.204 0.808 0.909 0.184 0.765 0.928 0.885 0.944 0.976 0.920 0.714 0.439 0.855 0.633 0.493 0.759 w AudioSet 0.992 0.797 0.875 0.994 0.87 0.247 0.875 0.909 0.191 0.747 0.964 0.886 0.941 0.978 0.943 0.829 0.485 0.897 0.694 0.528 0.782 MoE- SwiGLU Ours 0.982 0.847 0.882 0.994 0.979 0.251 0.872 0.899 0.381 0.782 0.968 0.888 0.941 0.974 0.94 0.882 0.449 0.914 0.7 0.512 0.802 w/o TSDS 0.993 0.847 0.9 0.995 0.848 0.254 0.87 0.9 0.375 0.79 0.974 0.889 0.941 0.978 0.938 0.870 0.470 0.91 0.7 0.533 0.799 w AudioSet 0.984 0.818 0.858 0.995 0.836 0.248 0.875 0.909 0.267 0.791 0.972 0.873 0.942 0.978 0.937 0.868 0.461 0.909 0.695 0.529 0.787 Table 7: Official leaderboard results of UniAE-MoE in the Interspeech 2026 Audio Encoder Capability Challenge, covering Track A (classification) and Track B (audio understanding) Public Tasks Hidden Tasks Aggregate Results Dataset Score Dataset Score Result Score Track A Performance ASVspoof 2015 0.9900 Fingersnap 0.8750 Overall 0.9122 CREMA-D 0.8793 KeyScratch 0.9990 Public test 0.8527 ESC-50 0.9125 King-ASR-457 0.9850 Hidden test 0.9717 Fluent Speech Commands 0.9939 King-ASR-719 0.9810 Free Music Archive 0.9587 King-ASR-876 1.0000 FSD50K 0.3208 King-ASR-955 0.9900 FSDKaggle2018 0.8844 GTZAN 0.9091 LibriCount 0.4790 NSynth 0.7881 Speech Commands V1 0.9352 UrbanSound8K 0.8889 VocalSound 0.9393 VoxCeleb1 0.9720 VoxLingua33 0.9391 Track B Performance AISHELL-1 0.9094 AISHELL-6 0.5830 Overall 0.6299 Clotho 0.4789 LibriHeavy 0.7366 Public test 0.7014 LibriSpeech 0.9247 MusicCaps 0.5155 Hidden test 0.5583 MECAT 0.6966 TACOS 0.3983 SongDescriber 0.4973 Note. All entries are official normalized leaderboard scores in [0,1][0,1]. Task-specific raw metrics are linearly mapped to normalized scores by the challenge organizers. The overall score for each track is the arithmetic mean of its public- and hidden-test aggregate scores. Boldface denotes the best result, including ties, among the 23 participating teams. For comparison, we also evaluate broad data scaling by incorporating AudioSet, AudioSetCaps, and ASR data during Stage 1. This strategy improves the standard MoE from 0.759 to 0.782 but provides no corresponding gain for MoE-SwiGLU, indicating that indiscriminately increasing data volume yields diminishing returns for high-capacity architectures. These results further motivate targeted rather than uniform data scaling. 3.6 Official Challenge Evaluation In addition to the controlled comparisons above, we report the official leaderboard results of UniAE-MoE to the Interspeech 2026 Audio Encoder Capability Challenge. Track A evaluates discriminative audio representations, whereas Track B focuses on audio understanding. Table 7 presents both aggregate and dataset-level results for the two tracks. On Track A, UniAE-MoE achieves top-tier overall performance, with consistently strong results on both the public and hidden tests. At the dataset level, it delivers top-tier performance on eight tasks, including five public tasks and three hidden tasks. These leading results span speech, music, environmental sound, and spoofing detection, demonstrating robust transfer across heterogeneous discriminative domains and highlighting the benefit of complementary expert fusion. Performance on FSD50K and LibriCount nevertheless indicates that multi-label sound-event recognition and speaker counting remain challenging. On Track B, UniAE-MoE also achieves competitive overall performance and strong results on the public test. It delivers top-tier performance on the public AISHELL-1 and MECAT tasks and, notably, on the hidden MusicCaps task, demonstrating strong speech recognition and effective audio–language modeling across general audio and music. Nevertheless, its relatively lower hidden-test aggregate performance, particularly on AISHELL-6 and LibriHeavy, suggests that robust generalization under unseen generative conditions remains an important direction for future work. 4 Conclusion This paper presents UniAE-MoE, a unified audio encoder that integrates complementary encoders from Qwen2-Audio and Audio-Flamingo 3 through MoE-based fusion. We employ SwiGLU and shared experts to facilitate interaction between heterogeneous encoder representations, together with Task-Specific Data Scaling in a two-stage training framework to improve cross-task generalization. On the XARES-LLM benchmark, UniAE-MoE achieves state-of-the-art performance with an overall score of 0.802. In the official Interspeech 2026 Audio Encoder Capability Challenge, UniAE-MoE achieves top-tier overall performance on Track A, including top-tier results on eight datasets, while also demonstrating strong performance on Track B. These results validate the effectiveness of sparse expert architectures for integrating heterogeneous audio encoders and establish a scalable approach to unified, general-purpose audio understanding. Acknowledgments. This work was supported by National Natural Science Foundation of China (62076144). Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References [1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, et al. (2023) MusicLM: generating music from text. arXiv preprint arXiv:2301.11325. Cited by: §2.4. [2] L. B. Allal, A. Lozhkov, E. Bakouch, G. M. Blázquez, G. Penedo, L. Tunstall, et al. (2025) SmolLM2: when smol goes big – data-centric training of a small language model. External Links: 2502.02737, Link Cited by: §3.1. [3] K. An, Y. Chen, Z. Chen, C. Deng, Z. Du, C. Gao, et al. (2025) Fun-ASR technical report. arXiv preprint arXiv:2509.12508. Cited by: Table 3. [4] J. Bai, H. Liu, M. Wang, D. Shi, W. Wang, M. D. Plumbley, et al. (2024) AudioSetCaps: enriched audio captioning dataset generation using large audio language models. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, Cited by: §2.4. [5] C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, et al. (2008) IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, p. 335–359. Cited by: §2.4. [6] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, et al. (2023) BEATs: audio pre-training with acoustic tokenizers. In The 40th International Conference on Machine Learning (ICML), A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Machine Learning Research, p. 5178–5193. Cited by: §1. [7] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, et al. (2024) Qwen2-audio technical report. External Links: 2407.10759, Link Cited by: §1, Table 3. [8] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson (2016) FMA: a dataset for music analysis. arXiv preprint arXiv:1612.01840. Cited by: §2.4. [9] D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, et al. (2025) Kimi-Audio technical report. arXiv preprint arXiv:2504.18425. Cited by: §1, §1, Table 3. [10] H. Dinkel, Z. Yan, Y. Wang, J. Zhang, Y. Wang, and B. Wang (2024) Scaling up masked audio encoder learning for general audio classification. In Annual Conference of the International Speech Communication Association (INTERSPEECH), p. 547–551. Cited by: Table 3, Table 3. [11] H. Dinkel, J. Zhou, G. Wang, Y. Niu, J. Zhang, Y. Hao, et al. (2026) The interspeech 2026 audio encoder capability challenge for large audio language models. In Interspeech 2026, Cited by: §3.1. [12] A. Goel, S. Ghosh, J. Kim, S. Kumar, Z. Kong, S. Lee, et al. (2025) Audio flamingo 3: advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128. Cited by: §1, Table 3. [13] S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, et al. (2024) WavLLM: towards robust and adaptive speech large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), p. 4552–4572. Cited by: §1. [14] P. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, et al. (2022) Masked autoencoders that listen. In Advances in Neural Information Processing Systems (NeurIPS), S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), p. 28708–28720. Cited by: §1. [15] R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, et al. (2024) AudioGPT: understanding and generating speech, music, sound, and talking head. In The AAAI Conference on Artificial Intelligence (AAAI), p. 23802–23804. Cited by: §1. [16] Y. Lei, S. He, J. Hu, D. Zhang, X. Luo, D. Zhu, et al. (2026) MoE adapter for large audio language models: sparsity, disentanglement, and gradient-conflict-free. External Links: 2601.02967, Link Cited by: §1. [17] Y. Li, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, et al. (2023) Mert: acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107. Cited by: Table 3. [18] J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria (2024) Mustango: toward controllable text-to-music generation. In The 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), p. 8286–8309. Cited by: §2.4. [19] I. M. Morato and A. Mesaros (2021) Macs-multi-annotator captioned soundscapes. MACS-Multi-Annotator Captioned Soundscapes. Cited by: §2.4. [20] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In The International Conference on Machine Learning (ICML), p. 28492–28518. Cited by: §1, Table 3. [21] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, et al. (2023) AudioPaLM: a large language model that can speak and listen. External Links: 2306.12925, Link Cited by: §1. [22] W. Shan, Y. Li, Y. Zhang, Y. Luo, C. Xu, X. Zhao, et al. (2025) Enhancing speech large language models with prompt-aware mixture of audio encoders. In 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), p. 19305–19320. External Links: Link Cited by: §1. [23] N. Shazeer (2020) Glu variants improve transformer. arXiv preprint arXiv:2002.05202. Cited by: §2.2. [24] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, et al. (2024) Salmonn: towards generic hearing abilities for large language models. In International Conference on Learning Representations, Vol. 2024, p. 16607–16629. Cited by: §1. [25] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, et al. (2025) Qwen2.5-Omni technical report. arXiv preprint abs:2503.20215. Cited by: Table 3. [26] J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, et al. (2025) Qwen3-Omni technical report. arXiv preprint arXiv:2509.17765. Cited by: Table 3. [27] J. Zhang, H. Dinkel, Y. Niu, C. Liu, S. Cheng, A. Zhao, et al. (2025) X-ARES: a comprehensive framework for assessing audio encoder performance. In Interspeech 2025, Cited by: §3.1.