Paper deep dive
Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs
Jing Peng, Zichao Nie, Zhisheng Zhang, Jingran Xie, Zhiyong Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/23/2026, 3:46:52 AM
Summary
This paper presents a comprehensive study comparing continuous and discrete audio representations within Large Audio Language Models (LALMs) using the UniARC framework. It evaluates various encoders (HuBERT, WavLM, Whisper, DAC, SpeechTokenizer, etc.) across speech, sound, and music domains using SmolLM2 and Llama-3 backbones. The study finds that semantic compatibility, rather than representation type (continuous vs. discrete), is the primary determinant of performance. Discrete tokens with strong semantic constraints (e.g., SpeechTokenizer) can outperform continuous features, while high-fidelity but semantically weak codecs (e.g., DAC) struggle. Additionally, scaling LLM backbones does not compensate for poor audio representation quality, especially in data-limited tasks.
Entities (13)
Relation Signals (7)
UniARC → uses → Llama 3
confidence 95% · frozen-backbone probing pipeline using Llama-3-1B/8B
UniARC → uses → SmolLM2
confidence 95% · Utilizing our UniARC framework... leveraging SmolLM2-135M/360M
Scaling LLM Backbones → failstocompensate → information loss in audio representation
confidence 92% · scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks
Semantic Constraints → improve → Discrete Token Performance
confidence 90% · discrete tokens can surpass continuous features when they incorporate strong semantic constraints
DAC → underperformsin → semantic understanding tasks
confidence 90% · reconstruction-oriented codecs (e.g., DAC and WavTokenizer) struggle with semantic understanding tasks
SpeechTokenizer → outperforms → HuBERT
confidence 85% · SpeechTokenizer consistently surpasses its continuous teacher, HuBERT, in specific tasks such as ER
Whisper → outperforms → HuBERT
confidence 85% · Weakly supervised Whisper aligns features with textual semantics, leads most benchmarks... HuBERT excels in tasks heavily reliant on speech-specific semantics
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC framework with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic relationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic constraints in tokenization for audio understanding and demonstrate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs.
Tags
Links
- Source: https://arxiv.org/abs/2609.22851v2
- Canonical: https://arxiv.org/abs/2609.22851v2
Trouble viewing inline? Open PDF directly →
Full Text
33,622 characters extracted from source content.
Expand or collapse full text
Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs Jing Peng ID 1,∗ , Zichao Nie ID 1,∗ , Zhisheng Zhang ID 1 , Jingran Xie ID 1 , Zhiyong Wu ID 1,∗ 1 Shenzhen International Graduate School, Tsinghua University, Shenzhen, China zywu@se.cuhk.edu.hk Abstract Large Audio Language Models (LALMs) utilize either contin- uous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Ex- isting benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC frame- work with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic re- lationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic con- straints in tokenization for audio understanding and demon- strate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs. Index Terms: large audio language models, audio encoders, benchmarking, unified audio understanding 1. Introduction In recent years, Large Audio Language Models (LALMs) have advanced significantly [1, 2]. The underlying representation schemes of these models have undergone several developments, evolving from continuous features to discrete tokens, and cur- rently towards hybrid approaches where both paradigms coexist or are integrated [3, 4, 5]. Discrete representations, owing to their structural consistency with text tokens, have greatly sim- plified the process of cross-modal alignment [6, 7]. However, the quantization bottleneck inherent in discretization often leads to the loss of crucial acoustic details, prompting researchers to re-examine the value of continuous features. While numerous studies have emerged in this field, they of- ten exhibit specific limitations. Some works focus on compar- ing semantic discrete tokens with continuous features, neglect- ing acoustic-based discrete representations [3, 4]. Others con- duct evaluations outside the context of large language models (LLMs) [8, 9], or concentrate on discrete tokens without a com- prehensive benchmark against continuous features [10, 11]. No- tably, most research focuses narrowly on speech, overlooking general audio understanding [3]. Furthermore, existing evalua- tions are often confined to constrained or “toy” settings, neglect- ing the complex interplay between representation paradigms, model capacity, and data scalability [4]. Therefore, a system- atic comparison addressing both cross-domain capabilities and real-world scaling dynamics is urgently needed. * These authors contributed equally. ** indicates the corresponding author. To bridge these gaps, we propose the Unified Audio Representation Comparison (UniARC) framework, specifi- cally designed for audio understanding.In this study, we unify the terminology by referring to audio representation modules as encoders, encompassing continuous features from Self-Supervised Learning (SSL), and discrete tokens derived from either clustering-based or neural codec-based paradigms. To ensure the robustness of our findings and simulate real- world deployment constraints, UniARC evaluates encoders un- der two distinct strategies: (1) a parameter-efficient fine-tuning setup leveraging SmolLM2-135M/360M [12], and (2) a frozen- backbone probing pipeline using Llama-3-1B/8B [13]. Beyond a direct comparison of encoding paradigms, we leverage the UniARC to investigate the multi-dimensional interplay between data volume, model capacity, and computational efficiency. Our analysis reveals that regardless of the representational form, the efficacy of audio encoders for understanding tasks is primarily determined by the richness of their semantic infor- mation. Specifically, we find that discrete tokens can surpass continuous features when they incorporate strong semantic con- straints, whereas high-fidelity representations focused purely on signal reconstruction often fail in understanding tasks. These findings suggest that the semantic compatibility of audio rep- resentations with the language model’s reasoning space is the primary determinant of performance in LALMs. Furthermore, we demonstrate that for audio understanding, scaling the lan- guage backbone cannot effectively compensate for insufficient representational quality. Our findings reveal that the front-end features effectively set a performance ceiling that remains un- changed or even declines despite increases in model scale. The primary contributions of this work are as follows: • We comprehensively evaluate across three diverse audio do- mains, i.e., speech, general sound, and music, to compare LLM-based continuous and discrete representations specifi- cally for audio understanding tasks. • We analyze how representation paradigms, model scales and data scalability interact, demonstrating that for audio under- standing, aligning audio encoders with textual semantics is more decisive than scaling LLM backbones. • We identify the task-dependent strengths of different paradigms within the understanding context, providing guidelines for balancing semantic density, acoustic fidelity, and training efficiency in future LALMs designs. 2. Evaluation Methodology 2.1. Evaluation Framework Building upon the open-source XARES-LLM project [9], we propose UniARC, a modular framework that systematically arXiv:2609.22851v2 [cs.SD] 22 Sep 2026 MLP LLM Backbone Audio Encoder Continuous Audio Encoder Discrete Audio Encoder Audio Emb Audio Token Audio Emb Projector Prompt (ASR) Identify the text corresponding to the speech: (ER) Identify the emotion corresponding to the speech: ...... Output Text LoRA / 푑 푎푢푑푖표 →푑 푙푚 Audio Emb Prompt Emb ... LLM Decoder Figure 1: The UniARC evaluation framework. The modular design supports flexible configurations of encoders, projectors, and LLM backbones. Fire and snowflake icons represent the training and frozen status of components, respectively. evaluates audio encoders via an instruction-tuning paradigm, as shown in Figure 1. Audio embeddings are appended to task- specific prompts, guiding the LLM within a unified sequence- to-sequence generation framework. To simulate diverse real- world deployment scenarios, we implement two strategies. The first scenario targets domain-specific experts via parameter-efficient fine-tuning. For lightweight applications where backbone updates are computationally feasible, we em- ploy SmolLM2 (135M/360M) with a LoRA-based instruction- tuning strategy [14], which updates only the adapters while mapping audio embeddings via a projector, maximizing perfor- mance on specific tasks within limited parameter budgets. The second scenario evaluates general-purpose LLMs via frozen-backbone probing. To preserve inherent reasoning capabilities and minimize confounding variables from model adaptation, we utilize Llama-3 (1B/8B) with frozen parameters. To bridge the modality gap without updating the LLM, we em- ploy a robust two-layer MLP projector and a 10-frame tempo- ral concatenation to handle long acoustic sequences efficiently. This design isolates the adaptive capacity of LLM, directly ex- posing the intrinsic semantic quality of frontend encoders. 2.2. Processing of Audio representations To ensure fair comparison, UniARC standardizes the conver- sion of diverse audio inputs into a unified format compatible with the LLM projector. For continuous representations, raw audio is processed by encoders into high-dimensional dense embeddings, preserv- ing representational density within a continuous latent space. For discrete representations derived from semantic clustering (K-means) or neural codecs, inputs are first converted to token indices and then mapped to codebook embeddings, effectively transforming discrete units into a continuous vector space. In both paradigms, the resulting embeddings are pro- cessed by a projector to align dimensions with the LLM’s text space, enabling end-to-end differentiable training. Finally, task- specific prompts guide the model across all downstream tasks. 3. Experiments 3.1. Encoder Selection We include a diverse set of encoders to cover different repre- sentation paradigms. For continuous features, we select Hu- BERT [15], WavLM [16], Wav2Vec 2.0 [17], and Whisper [18] to represent SSL and weakly supervised approaches. For discrete representations, we investigate two primary categories of encoders based on discretization mechanisms: 1) Clustering-based tokens, derived via K-means (1,000 centroids) on SSL features (HuBERT, WavLM) to prioritize high-level se- mantic abstraction [3]; and 2) Neural Codec tokens, learned through end-to-end reconstruction. For the latter, we evalu- ate DAC [19] and WavTokenizer [20], alongside SpeechTok- enizer [21], a specialized codec that integrates semantic distil- lation to decouple semantic content from acoustic details. To ensure consistent feature quality, we operate all encoders at 16 kHz, with the exception of WavTokenizer, which utilizes its native 24 kHz configuration. For continuous encoders (Hu- BERT, WavLM), we consistently extract the last hidden state. For clustering-based discrete tokens, we set K-means clusters to 1000 for SmolLm2-135M/360M lightweight decoders with LoRA fine-tuning, balancing speech unit complexity, training stability, inference efficiency and computational cost [3]. 3.2. Tasks and Datasets To align with the two evaluation strategies defined in UniARC, we adopt a stratified task selection protocol covering Speech, Sound, and Music. A comprehensive overview of all evaluated datasets, task types, and metrics is summarized in Table 1. Table 1: Overview of datasets, tasks, and metrics utilized in the UniARC framework. Mode denotes the evaluation setup: Tuned (T) with SmolLM2, Frozen (F) with Llama-3, or Both. DatasetTask Type (Abbr.)MetricMode Speech LibriSpeech (LS) [22]Speech Recognition (ASR)WER/iWERBoth AISHELL-1 [23]Speech Recognition (ASR)iCERT Fluent Speech Commands [24]Intent Classification (IC)AccT SLURP [25]Intent Classification (IC)AccF Speech Commands V1 [26]Keyword Spotting (KS)AccT CREMA-D [27]Emotion Recognition (ER)AccBoth IEMOCAP [28]Emotion Recognition (ER)AccF VoxCeleb1 [29]Speaker Identification (SI)AccT VoxLingua33 [30]Language Identification (LID)AccT LibriCount [31]Speaker Counting (SC)AccT VocalSound [32]Human Sound Classification (HSC)AccT ASVspoof2015 [33]Spoofing Detection (SD)AccT Sound ESC-50 [34]Environment Classification (EnvC)AccT UrbanSound8K [35]Urban Sound Classification (USC)AccBoth FSD50K [36]Sound Event Detection (SED)mAPT FSDKaggle2018 [37]Sound Event Detection (SED)mAPT Clotho [38]Sound Captioning (SCap)FENSEBoth MECAT [39]General Caption (GCap)DATET Music GTZAN [40]Genre Classification (GC)AccBoth Free Music Archive (FMA) [41]Genre Classification (GC)AccT NSynth [42]Instrument Classification (InstC)AccT Song Describer (SDD) [43]Music Captioning (MCap)FENSEBoth For parameter-efficient fine-tuning evaluations, we utilize a broad suite of instruction-tuning tasks adhering to XARES- LLM to establish a fundamental baseline.Conversely, for HuBERT Wav2Vec2 WavLM Whisper HuBERT-KM WavLM-KM DAC SpeechTokenizer WavTokenizer librispeech aishell-1 fluentspeechcommands speechcommandsv1 voxceleb1 cremad asvspoof2015 voxlingua33 libricount esc-50 urbansound8k fsd50k fsdkaggle2018 clotho mecat vocalsound gtzan freemusicarchive nsynth songdescriber 0.960.810.930.940.940.850.010.740.00 0.450.020.570.890.000.000.020.330.01 0.990.940.991.000.970.980.420.980.29 0.970.950.970.980.890.940.890.890.88 0.180.790.920.980.180.720.580.280.25 0.270.210.420.620.250.280.350.400.32 0.800.570.800.990.930.820.900.890.94 0.780.600.910.980.400.740.270.630.22 0.300.280.380.350.250.260.320.200.26 0.240.150.510.760.180.290.270.550.43 0.430.190.570.850.410.500.610.600.56 0.030.030.070.100.030.030.040.070.05 0.170.120.350.640.090.150.310.340.34 0.220.120.310.410.170.240.230.260.26 0.490.460.610.680.460.540.500.450.50 0.750.630.840.880.510.770.710.810.73 0.400.240.670.830.290.590.640.630.54 0.410.410.510.620.330.440.450.500.48 0.580.400.640.730.330.480.720.610.69 0.380.430.420.480.460.440.410.420.43 SmolLM2-135M HuBERT Wav2Vec2 WavLM Whisper HuBERT-KM WavLM-KM DAC SpeechTokenizer WavTokenizer 0.960.930.940.950.950.870.210.770.00 0.150.000.620.930.010.200.000.370.03 1.000.951.000.990.940.980.620.980.29 0.970.950.970.980.900.930.910.870.88 0.270.710.900.560.210.810.440.180.18 0.230.200.350.570.210.250.330.370.34 0.710.610.800.960.810.830.910.890.91 0.820.650.910.980.390.720.310.640.18 0.310.280.100.340.100.190.190.170.29 0.220.150.500.800.120.270.320.390.37 0.400.260.420.440.320.460.640.340.58 0.040.030.060.120.030.040.040.060.06 0.160.110.310.550.090.130.310.310.31 0.230.150.330.420.180.270.240.280.26 0.530.410.620.700.450.560.520.460.50 0.750.550.830.890.610.740.670.790.57 0.340.270.650.730.250.490.080.490.58 0.400.370.530.600.330.420.470.490.43 0.570.390.620.720.360.470.710.610.68 0.410.430.420.490.450.440.410.430.40 SmolLM2-360M 0.0 0.2 0.4 0.6 0.8 1.0 Figure 2: Performance of representation paradigms within the UniARC fine-tuning pipeline (KM stands for K-means). frozen-backbone probing evaluations, we strategically curate a representative subset of core tasks. Notably, SLURP and IEMOCAP are introduced here to rigorously probe encoders’ intrinsic semantic reasoning. We employ metrics tailored to each task type: Accuracy (Acc) for single-label classification, mAP for multi-label event detection, FENSE [44] for audio/music captioning to evalu- ate semantic fluency, and DATE [39] for general captioning. For speech recognition, standard Word/Character Error Rates (WER/CER) are calculated. To facilitate intuitive visualization of fine-tuning evaluations, WER/CER are inverted into Inverse Word/Character Error Rate (iWER/iCER), ensuring that higher scores consistently indicate better performance. 3.3. Experimental Setup Our experiments are conducted on NVIDIA A100 GPUs us- ing PyTorch. For Fine-Tuning Strategy, we apply LoRA (r = 8,α = 32) to LLM backbone on single GPU, maintaining con- sistent default hyperparameters on a single GPU. For Frozen- Probing Strategy, we freeze LLM and optimize only projector using AdamW optimizer [45] with a learning rate of 1× 10 −5 across four GPUs. To ensure fair comparison of representation quality, all task-specific configurations within each strategy are kept strictly identical across different audio encoders. 4. Results and Analysis We comprehensively evaluate continuous features and discrete tokens across diverse tasks under UniARC framework. Results from the parameter-efficient fine-tuning strategy are visualized in Figure 2, where datasets are color-coded by domain(speech in blue, sound in green, music in red) on y-axis, and encoders are color-coded by representation type (continuous in black, discrete in gray). These results are numerically summarized in Table 2 by averaging the scores within each domain. Con- versely, detailed findings from the strict frozen-backbone prob- ing strategy are presented in Table 3. The following subsec- tions systematically analyze these results regarding representa- tion paradigms, scaling effects, and training efficiency. Table 2: Encoder Performance on Different Backbones. Higher scores indicate better performance (↑). EncoderSpeech Sound Music Overall SmolLM2-135M HuBERT0.6340.3330.4450.491 Wav2Vec20.5750.2430.3700.418 WavLM0.7660.4660.5610.620 Whisper0.8580.6180.6660.735 HuBERT+Kmeans0.5340.2630.3560.403 WavLM+Kmeans0.6200.3610.4880.503 DAC0.4190.3810.5540.432 SpeechTokenizer0.5930.4400.5410.529 WavTokenizer0.3530.4080.5350.409 SmolLM2-360M HuBERT0.6030.3310.4320.474 Wav2Vec20.5890.2380.3680.422 WavLM0.7320.4380.5550.594 Whisper0.8070.5610.6350.687 HuBERT+Kmeans0.5020.3040.3660.407 WavLM+Kmeans0.6650.4070.4570.533 DAC0.4350.3910.4170.416 SpeechTokenizer0.5830.3750.5050.494 WavTokenizer0.3470.3780.5230.393 4.1. Analysis of Representation Paradigms Based on the experimental results in Figure 2 and Table 3, we find that whether an encoder is discrete or continuous is not the primary factor governing performance; rather, semantic com- patibility fundamentally dictates its efficacy for LLM-based un- derstanding, far outweighing raw signal fidelity. This principle is evident among continuous encoders. Weakly supervised Whisper aligns features with textual seman- tics, leads across most benchmarks. Conversely, Wav2Vec2’s self-supervised focus on low-level acoustics limits its efficacy in non-speech semantic tasks. This confirms that while SSL mod- els excel at signal modeling, Whisper’s text-aligned features are more effective for high-level LLM reasoning. A similar trend governs discrete representations, explaining Table 3: Experimental results across Speech, Sound, and Music domains. Best results within each model scale (1B/8B) are bolded. Model SpeechSoundMusic ASR (WER %↓)ER (ACC %↑)IC (ACC %↑)USC (ACC %↑)SCap (FENSE↑)GC (ACC %↑)MCap (FENSE↑) LS-960 (clean/other) LS-100 (clean) IEMOCAPCREMA-DSLURP UrbanSound8KClothoGTZANSDD 1B Models HuBERT2.31/4.623.4552.3050.1271.2355.560.19630.690.453 SpeechTokenizer 40.62/70.0880.8257.7051.9741.8260.330.14351.030.476 WavLM4.02/6.556.0064.0664.6268.6769.890.16461.720.545 DAC171.02/177.48208.4647.4636.247.9051.140.21135.170.478 8B Models HuBERT2.24/4.383.1347.6252.2170.1348.030.12945.190.507 SpeechTokenizer18.33/38.7583.4056.4954.3041.4771.920.08351.720.392 WavLM2.83/5.575.4962.2967.8165.3971.450.17759.310.523 DAC147.35/148.41166.5746.0129.857.4248.860.10136.550.501 the divergence among tokenization paradigms. While K-means quantization typically lags behind continuous features due to information loss, encoders that incorporate explicit semantic constraints defy this trend. Notably, SpeechTokenizer consis- tently surpasses its continuous teacher, HuBERT, in specific tasks such as ER. This demonstrates that semantically-aware discretization can act as a regularizer, distilling high-density se- mantic features optimally tailored for LLM comprehension. Conversely, reconstruction-oriented codecs (e.g., DAC and WavTokenizer) struggle with semantic understanding tasks de- spite their superior fidelity. The preservation of low-level acous- tic details, such as phase and microstructural information, may introduce irrelevant noise that complicates the mapping of au- dio features into the LLM’s semantic space. However, this high fidelity remains valuable for timbre-sensitive tasks like music classification (NSynth), where fine-grained frequency details are more critical than high-level semantic abstractions. 4.2. Scaling, Data Diversity, and Efficiency Backbone Scaling and the Inverse Scaling Phenomenon. We analyze scaling behavior separately within each model family: SmolLM2 (135M to 360M) and Llama-3 (1B to 8B). As shown in Tables 2 and 3, increasing capacity does not yield univer- sal improvements. For SmolLM2, scaling from 135M to 360M provides marginal gains or fluctuations, suggesting capacity sat- uration for the evaluated encoders. For Llama-3, while scaling laws [46] hold for the data-abundant ASR task (LS-960), perfor- mance regression occurs in many data-limited scenarios. Over- all, across both frameworks, performance decreased in approxi- mately 53% of the evaluated encoder-task configurations. These results indicate that for audio understanding, scaling backbones cannot compensate for low feature quality or limited training data. While we focus on representations rather than projector design, our simple MLP might act as a bottleneck limiting these scaling benefits, which future context-aware architectures could mitigate by preventing surjective mappings. Impact of Pre-training Data Diversity. Encoder under- standing performance depends on the breadth of pre-training. While semantic compatibility governs overall efficacy, the task- level breakdowns in Tables 2 and 3 further elucidate the criti- cal impact of pre-training data diversity. As observed in Table 2, encoders pre-trained on diverse, multi-domain data (Whis- per and WavLM) show a clear advantage over those trained ex- clusively on speech (HuBERT and SpeechTokenizer). Further- more, results in Table 3 provide a more nuanced perspective: although HuBERT excels in tasks heavily reliant on speech- specific semantics, such as ASR and IC—sometimes even sur- passing WavLM—the latter maintains its dominance across most of other general-audio categories. This gap underscores that multi-domain pre-training is essential for universal audio understanding, since instruction-tuning alone cannot bridge rep- resentation gaps inherited from narrow pre-training data. ASRERICUSCSCapGCMCap 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Relative Duration (DAC = 1.0) 2.5 1.9 2.2 3.8 3.1 2.3 2.7 1.9 1.6 1.8 2.9 1.9 1.3 1.3 1.3 1.0 1.0 1.0 1.0 1.1 1.1 1.01.01.01.01.01.01.0 WavlmHubertSpeechTokenizerDAC Figure 3: Total training time until convergence for encoders. Computational Efficiency and Training Convergence. As illustrated in Figure 3, the computational cost of continu- ous features is a practical challenge. The sequence processing latency for continuous encoders (HuBERT and WavLM) signifi- cantly exceeds that of discrete tokenizers (SpeechTokenizer and DAC) across all benchmarks. Notably, discrete representations converge much faster during modality alignment, with valida- tion performance stabilizing in fewer training steps. Despite the performance edge of continuous features in certain domains, the rapid convergence and lower overhead of discrete tokens make them an attractive paradigm for resource-constrained research and rapid system deployment. 5. Conclusion This paper systematically evaluates continuous and discrete rep- resentations for audio understanding across speech, sound, and music. Results reveal that discrete and continuous encoders exhibit distinct and complementary strengths across task re- quirements. We highlight the necessity of incorporating seman- tic constraints into discrete representations. Furthermore, our study underscores multi-domain pre-training data in achieving universal audio intelligence. Crucially, for audio understand- ing, backbone scaling fails to consistently compensate for an encoder’s low semantic density, often leading to performance regression in data-limited tasks. These insights guide future LALM development, suggesting a focus on balancing seman- tics and fidelity through multi-domain pre-training. 6. Acknowledgments This work is supported by National Natural Science Founda- tion of China (62076144) and Shenzhen Science and Technol- ogy Program (JCYJ20220818101014030). 7. Generative AI Use Disclosure During the preparation of this manuscript, the authors used gen- erative AI tools exclusively for the purpose of language editing and manuscript polishing to improve readability. These tools were not used to generate any core scientific ideas, experimen- tal data, or technical contributions. All authors have thoroughly reviewed and approved the final version of the manuscript, and assume full responsibility for the integrity and entirety of its content. 8. References [1] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023. [2] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” in International Conference on Learning Representations (ICLR). [3] D. Wang, J. Li, M. Cui, D. Yang, X. Chen, and H. Meng, “Speech discrete tokens or continuous features? a comparative analysis for spoken language understanding in speechllms,” in Empirical Methods in Natural Language Processing (EMNLP), 2025, p. 24 924–24 935. [4] Y. Xu, S.-X. Zhang, J. Yu, Z. Wu, and D. Yu, “Comparing dis- crete and continuous space llms for speech recognition,” in An- nual Conference of the International Speech Communication As- sociation (INTERSPEECH). ISCA, 2024, p. 2509–2513. [5] Y. Zhou, G. Zeng, X. Liu, X. Li, R. Yu, Z. Wang, R. Ye, W. Sun, J. Gui, K. Li et al., “Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning,” arXiv preprint arXiv:2509.24650, 2025. [6] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: A language modeling approach to audio gener- ation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), p. 2523–2533, 2023. [7] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,” in Findings of the Associa- tion for Computational Linguistics: EMNLP 2023 (EMNLP Find- ings), 2023, p. 15 757–15 773. [8] P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli, “Dasb - discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294, 2024. [9] J. Zhang, H. Dinkel, Y. Niu, C. Liu, S. Cheng, A. Zhao, and J. Luan, “X-ares: A comprehensive framework for assessing audio encoder performance,” in Annual Conference of the International Speech Communication Association (INTERSPEECH).ISCA, 2025, p. 4868–4872. [10] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer et al., “Discrete audio tokens: More than a survey!” Transactions on Machine Learning Research (TMLR), 2025. [11] Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, “Recent advances in discrete speech tokens: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). [12] L. B. allal, A. Lozhkov, E. Bakouch et al., “SmolLM2: When smol goes big — data-centric training of a fully open small language model,” in Second Conference on Language Modeling (COLM), 2025. [13] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models,” International Conference on Learning Repre- sentations (ICLR), p. 3, 2022. [15] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “Hubert: Self-supervised speech rep- resentation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing (TASLP), p. 3451–3460, 2021. [16] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing (JSTSP), p. 1505, 2022. [17] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” in Advances in Neural Information Processing Systems (NeurIPS), 2020, p. 12 449–12 460. [18] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning (ICML). PMLR, 2023, p. 28 492–28 518. [19] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” 2023, p. 27 980–27 993. [20] S. Ji, Z. Jiang, W. Wang, Y. Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li et al., “Wavtokenizer: An efficient acoustic discrete codec tokenizer for audio language modeling,” in International Conference on Learning Representations (ICLR). [21] X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech language models,” in International Conference on Learning Representations (ICLR), 2024. [22] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, p. 5206–5210. [23] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An open- source mandarin speech corpus and a speech recognition base- line,” in conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O sys- tems and assessment (O-COCOSDA). IEEE, 2017, p. 1–5. [24] M. Ravanelli, V. Tomar, P. Ignoto, Y. Rachmad, and L. Lugosch, “Speech model pre-training for end-to-end spoken language un- derstanding,” in Annual Conference of the International Speech Communication Association (INTERSPEECH). ISCA, 2019, p. 814–818. [25] E. Bastianelli, A. Vanzo, P. Swietojanski, and V. Rieser, “Slurp: A spoken language understanding resource package,” in Empirical Methods in Natural Language Processing (EMNLP), 2020, p. 7252–7262. [26] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209, 2018. [27] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “CREMA-D: Crowd-sourced emotional multi- modal actors dataset,” IEEE Transactions on Affective Computing (TAC), p. 377–390, 2014. [28] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,” Language Resources and Evaluation (LRE), p. 335–359, 2008. [29] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,” Computer Speech & Language (CSL), vol. 60, p. 101027, 2019. [30] J. Valk and T. Alum ̈ ae, “Voxlingua107: A dataset for spoken language recognition,” 2021 IEEE Spoken Language Technology Workshop (SLT), p. 652–658, 2020. [31] F.-R. St ̈ oter, S. Chakrabarty, E. Habets, and B. Edler, “Libricount, a dataset for speaker count estimation,” 2018. [32] Y. Gong, J. Yu, and J. Glass, “Vocalsound: A dataset for improv- ing human vocal sounds recognition,” in IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2022, p. 151–155. [33] Z. Wu, J. Yamagishi, T. Kinnunen, C. Hanilc ̧i, M. Sahidullah, A. Sizov, N. Evans, M. Todisco, and H. Delgado, “Asvspoof: The automatic speaker verification spoofing and countermeasures challenge,” IEEE Journal of Selected Topics in Signal Processing (JSTSP), vol. 11, no. 4, p. 588–604, 2017. [34] K. J. Piczak, “Esc: Dataset for environmental sound classifi- cation,” in ACM international conference on Multimedia (M), 2015, p. 1015–1018. [35] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in ACM international conference on Multimedia (M), 2014, p. 1041–1044. [36] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: an open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), vol. 30, p. 829–852, 2021. [37] E. Fonseca, M. Plakal, F. Font, D. P. W. Ellis, X. Favory, J. Pons, and X. Serra, “General-purpose tagging of freesound audio with audioset labels: Task description, dataset and baseline,” in Detec- tion and Classification of Acoustic Scenes and Events Workshop (DCASE), 2018. [38] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2020, p. 736– 740. [39] Y. Niu, T. Wang, H. Dinkel, X. Sun, J. Zhou, G. Li, J. Liu, X. Liu, J. Zhang, and J. Luan, “Mecat: A multi-experts con- structed benchmark for fine-grained audio understanding tasks,” arXiv preprint arXiv:2507.23511, 2025. [40] G. Tzanetakis, G. Essl, and P. Cook, “Automatic musical genre classification of audio signals,” in International Symposium on Music Information Retrieval (ISMIR), 2001. [41] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,” in International Society for Music Information Retrieval Conference (ISMIR), 2016. [42] J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Si- monyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” in International Conference on Machine Learning (ICML). JMLR. org, 2017, p. 1068–1077. [43] I. Manco, B. Weck, S. Doh, M. Won, Y. Zhang, D. Bogdanov, Y. Wu, K. Chen, P. Tovstogan, E. Benetos et al., “The song describer dataset: A corpus of audio captions for music-and- language evaluation,” in Workshop on Machine Learning for Au- dio, Neural Information Processing Systems (NeurIPS).Neural Information Processing Systems, 2023. [44] Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, “Can audio captions be evaluated with image caption metrics?” in IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), 2022, p. 981–985. [45] I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” in International Conference on Learning Representations (ICLR), 2019. [46] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020.