Paper deep dive
ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models' In-Context Learning Ability
Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien, Toshiki Ogimoto, Shang-Tse Chen, Yun-Nung Chen, Chun-Yi Lee, Shao-Yuan Lo
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/24/2026, 3:10:23 AM
Summary
This paper introduces ALICE (Audio-Language In-Context learning Evaluation), a three-stage evaluation framework for assessing in-context learning (ICL) ability of Large Audio-Language Models (LALMs) under audio conditioning. The framework progressively reduces textual guidance across three stages: Explicit Constraint, Implicit Constraint, and Audio Only. Evaluating six LALMs across four audio understanding tasks (ASR, SER, GR, MMAU) under two output constraint categories (CEQ and CoT), the authors find a consistent asymmetry: in-context demonstrations reliably improve format compliance but fail to improve—and often degrade—core task performance. This suggests LALMs can match surface-level formatting patterns but struggle with cross-modal semantic grounding needed to infer task objectives from audio-conditioned examples.
Entities (34)
Relation Signals (31)
Yen-Ting Piao → affiliatedwith → National Taiwan University
confidence 99% · National Taiwan University, Taiwan {r14922010, r13922210, r13922197, r14922163, stchen}@csie.ntu.edu.tw
Yen-Ting Piao → authored → ALICE
confidence 99% · Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien, Toshiki Ogimoto, Shang-Tse Chen, Yun-Nung Chen, Chun-Yi Lee, Shao-Yuan Lo
ALICE → consistsof → Stage 3 Audio Only
confidence 99% · In Stage 3 (Audio Only), task descriptions are further removed, providing the models with only audio inputs paired with correctly formatted outputs
ALICE → consistsof → Stage 1 Explicit Constraint
confidence 99% · In Stage 1 (Explicit Constraint), each demonstration example and test sample contains both a task description and explicit format constraint instruction
ALICE → consistsof → Stage 2 Implicit Constraint
confidence 99% · In Stage 2 (Implicit Constraint), the explicit format constraint instructions are removed from both the demonstration examples and test samples
Qwen2.5-Omni → evaluatedby → ALICE
confidence 99% · Qwen2.5-Omni (10.7B)
Phi-4-Multimodal → evaluatedby → ALICE
confidence 99% · Phi-4-Multimodal (5.6B)
Qwen2-Audio → evaluatedby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Large Audio-Language Models (LALMs) have been shown to exhibit degraded instruction-following capabilities, their ability to infer task patterns from in-context examples under audio conditioning remains unstudied. To address this gap, we present ALICE, a three-stage framework that progressively reduces textual guidance to systematically evaluate LALMs' in-context learning ability under audio conditioning. Evaluating six LALMs across four audio understanding tasks under two output constraint categories, we uncover a consistent asymmetry across all stages and LALMs: in-context demonstrations reliably improve format compliance but fail to improve, and often degrade, the core task performance. This suggests that LALMs can glean surface-level formatting patterns from demonstrations but may struggle to leverage cross-modal semantic grounding to reliably infer task objectives from audio-conditioned examples, highlighting potential limitations in current cross-modal integration.
Tags
Links
- Source: https://arxiv.org/abs/2603.20433v1
- Canonical: https://arxiv.org/abs/2603.20433v1
Trouble viewing inline? Open PDF directly →
Full Text
33,730 characters extracted from source content.
Expand or collapse full text
ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models’ In-Context Learning Ability Yen-Ting Piao ∗,∗ , Jay Chiehen Liao ∗ , Wei-Tang Chien ∗ , Toshiki Ogimoto ∗ , Shang-Tse Chen, Yun-Nung Chen, Chun-Yi Lee, Shao-Yuan Lo ∗ National Taiwan University, Taiwan r14922010, r13922210, r13922197, r14922163, stchen@csie.ntu.edu.tw, y.v.chen@ieee.org, cylee, sylo@csie.ntu.edu.tw Abstract While Large Audio-Language Models (LALMs) have been shown to exhibit degraded instruction-following capabilities, their ability to infer task patterns from in-context examples un- der audio conditioning remains unstudied. To address this gap, we present ALICE, a three-stage framework that progressively reduces textual guidance to systematically evaluate LALMs’ in- context learning ability under audio conditioning. Evaluating six LALMs across four audio understanding tasks under two output constraint categories, we uncover a consistent asymme- try across all stages and LALMs: in-context demonstrations re- liably improve format compliance but fail to improve, and often degrade, the core task performance. This suggests that LALMs can glean surface-level formatting patterns from demonstrations but may struggle to leverage cross-modal semantic grounding to reliably infer task objectives from audio-conditioned examples, highlighting potential limitations in current cross-modal inte- gration. Index Terms: large audio-language model, in-context learning, evaluation framework 1. Introduction Large Audio-Language Models (LALMs) [1–6] extend Large Language Models (LLMs) to the auditory domain, support- ing applications such as semantic-level speech question an- swering, spoken dialogue generation, paralinguistic analysis (e.g., emotion or speaker traits), and general non-speech au- dio event understanding [7–10].Despite this progress, re- cent studies [11–13] reveal that LALMs often exhibit degraded instruction-following ability compared to their text-only coun- terparts, particularly under explicit output constraints [11]. This gap is hypothesized to stem from catastrophic forgetting during speech-text integration, suggesting that multimodal adaptation could affect not only modality alignment but also higher-level task inference mechanisms inherited from text pretraining. In the text-only domain, in-context learning (ICL) enables LLMs to infer latent task structures and output formats through few-shot demonstrations at inference time [14–17]. Crucially, ICL and instruction-following share a common underlying mechanism: both require the model to infer task objectives and output format constraints from contextual signals, whether those signals are explicit natural language instructions or im- plicit demonstrations [18, 19]. Therefore, evaluating LALMs’ ICL behaviors provides a principled framework for probing whether these higher-level task recognition mechanisms gen- eralize to the audio modality after speech–text integration. * These authors contributed equally. ** indicates the corresponding authors. However, despite extensive studies of ICL in text-only LLMs [14–17], its behavior in audio-conditioned multimodal settings remains underexplored. Prior work has explored ICL for specific tasks such as automatic speech recognition [20] or auditory knowledge editing [21]. To the best of our knowledge, there is no systematic study that characterizes ICL behavior in the audio-conditioned setting of LALMs. Rather than compar- ing LALMs with their text-only counterparts, our goal is to dis- entangle the joint influence of acoustic inputs and in-context demonstrations on task inference. To bridge this gap, we introduce ALICE (Audio-Language In-Context learning Evaluation), a systematic three-stage eval- uation framework for analyzing LALMs’ ICL ability. As il- lustrated in Figure 1, under a fixed audio-conditioning setting, we design three conditions that progressively reduce textual guidance in the in-context demonstrations, which are all task- domain consistent, while keeping the audio inputs and demon- stration outputs unchanged. Such controlled setup isolates the contribution of textual cues and assesses whether LALMs can infer task objectives and generate correctly formatted responses from audio-conditioned examples. Across six LALMs, we observe a strongly asymmetric ICL effect consistently: in-context demonstrations only im- prove format compliance but not core speech task perfor- mance, even when examples with reasoning traces are pro- vided. Notably, the evaluated LALMs with stronger instruction- following ability undergo more degradation in format compli- ance when explicit constraint instructions are removed, indicat- ing that good instruction-following capability does not imply robust ICL deduction ability from demonstrations alone. Overall, our key contributions are threefold: (i) we conduct the first systematic analysis of LALMs’ ICL deduction ability under audio conditioning; (i) we provide empirical evidence that instruction-following ability does not guarantee ICL deduc- tion ability; (i) we show that providing examples with reason- ing traces fails to improve task performance, suggesting that LALMs’ ICL limitations possibly stem from integrating audi- tory information rather than a deficiency in text-based reason- ing. The inference code and related resources are available at: https://github.com/yenting-biao/ALICE. 2. ALICE: ICL Evaluation Framework To investigate whether LALMs exhibit ICL behavior under au- dio conditioning, we adopt a subset of Speech-IFEval [11] as our evaluation testbed, a benchmark of audio understand- ing tasks augmented with verifiable output constraints [11, 22] that enables systematic assessment of format compliance and speech-processing ability under audio-conditioned prompts. arXiv:2603.20433v1 [cs.SD] 20 Mar 2026 Figure 1: ALICE’s three evaluation stages: Textual guidance progressively decreases from Stage 1 (Explicit Constraint) to Stage 3 (Audio Only), with removed elements grayed out. 2.1. Audio Understanding Tasks and Format Constraints We adopt the audio understanding tasks from Speech-IFEval, including Automatic Speech Recognition (ASR), Speech Emotion Recognition (SER), Gender Recognition (GR), and Massive Multi-Task Audio Understanding (MMAU) [23]. We consider two constraint categories: (1) Closed-Ended Ques- tions (CEQ), which enforce strict deterministic formats (e.g., all-uppercase output, JSON formatting, or a required pre- fix/suffix); and (2) Chain-of-Thought (CoT), which requires the model to produce explicit intermediate reasoning steps alongside the final answer. Refer to Figure 1 for an example. 2.2. Construction of ICL Examples Instruction rephrasing. For each task-constraint combination, we use GPT-5 [24] to rephrase task instructions and format constraints respectively while preserving semantics, diversify- ing linguistic expressions across demonstrations and reducing lexical and structural bias. Audio and answer sourcing. For ASR, SER, and GR, we sample audio from CREMA-D [25], a corpus disjoint from Speech-IFEval’s data containing diverse emotional speech with labelled speaker information. For MMAU, we sample audio clips from the MMAU dataset, ensuring that none overlap with Speech-IFEval. To maintain annotation quality, we only in- clude MMAU items with either provided ground-truth answers or confidently human-annotated labels. Table 1 summarizes the audio duration statistics for our evaluation dataset (988 sam- ples) and demonstration dataset (912 samples). Format label generation. CEQ demonstration labels are generated via rule-based transformations on the answers (e.g., uppercasing text), while CoT reasoning traces are generated by Gemini 2.5 Pro [26] and verified by a human expert. The resulting demonstration pool consists of audio- instruction-label triples that are linguistically diverse and dis- joint from the evaluation data. They are label-balanced to avoid exposing the models to skewed label distributions during ICL. 2.3. Three-Stage Evaluation Framework The core evaluation methodology of ALICE is illustrated in Figure 1. We propose a three-stage framework that progres- sively reduces the amount of explicit textual instruction pro- vided, thereby probing the degree to which LALMs can learn from audio-conditioned examples when textual guidance is sys- tematically withheld. In Stage 1 (Explicit Constraint), each demonstration ex- ample and test sample contains both a task description and ex- Table 1: Summary statistics of audio duration (seconds). DatasetMean Std Min Max Evaluation8.087.730.1334.48 Demonstration11.9710.911.6538.75 plicit format constraint instruction alongside the audio input. Demonstrations are drawn from the same task-constraint cate- gory as the test sample and always contain correctly formatted outputs. This stage serves as a strong upper-bound condition: textual instructions fully specify both the task objective and how the output should be formatted, so any observed ICL effect re- flects learning on top of already complete textual guidance. In Stage 2 (Implicit Constraint), the explicit format con- straint instructions are removed from both the demonstration examples and test samples while other parts remain intact. This stage tests if the models can infer formatting conventions from example outputs alone, without being explicitly told the rules. In Stage 3 (Audio Only), task descriptions are further re- moved, providing the models with only audio inputs paired with correctly formatted outputs as demonstrations. This most chal- lenging stage requires the models to jointly infer task objectives and formatting patterns from the relationship between audio in- puts and textual outputs across demonstrations, constituting a strict test of cross-modal ICL. Note that MMAU is excluded at this stage as its textual instruction is necessary to answer the question, making the task ill-defined without it. 3. Experimental Setup We explore the effects of the number of in-context examples (k) by varyingk from 1 to 8, a range sufficient to observe conver- gence trends while remaining computationally feasible. Exam- ples are sampled from the demonstration pool to match the test sample’s task domain and constraint category, then inserted into the dialogue context in random order. To anchor the compari- son baseline against ICL, Stage 1 includes a zero-shot setting in which the LALMs rely solely on natural language instruc- tions without demonstrations. Conversely, we omit zero-shot conditions in Stages 2 and 3: format compliance in Stage 2 is undefined without explicit constraints, and providing only audio inputs in Stage 3 renders the task ill-defined. 3.1. Models Our goal is to investigate whether LALMs can integrate au- ditory information during ICL, but given the heterogeneity of the underlying backbones (e.g., pretrained versus instruction- tuned) across different LALMs, directly comparing them with their text-only counterparts lacks strict experimental control. Therefore, we focus our comparative design exclusively within the audio-conditioned setting, rather than evaluating their re- tention of text-based ICL abilities. We evaluate five open- source LALMs of various sizes: Qwen2-Audio (8.4B) [2], DeSTA2.5-Audio (9.7B) [3], BLSP-Emo (8.5B) [4], Qwen2.5- Omni (10.7B) [5], and Phi-4-Multimodal (5.6B) [6]. In addi- tion, we include one proprietary model, Gemini 2.5 Flash [26], evaluated under two configurations: dynamic thinking enabled (“On”) and thinking entirely disabled (“Off”). 3.2. Evaluation Metrics We evaluate ICL ability along two complementary dimensions: • Format Compliance Rate (FCR): This measures whether a model’s response adheres to the prescribed output constraints Table 2: Format Compliance Rate (FCR, %) reported with 95% confidence intervals. “Few-shot” is averaged overk ≥ 1. Shaded cells indicate the zero-shot baseline within each task group, and bold marks the highest FCR within each model×task-group block. Model Task Group A: ASR/SER/GR/MMAUTask Group B: ASR/SER/GR Zero-shotOne-shotFew-shotZero-shotOne-shotFew-shot Stage 1 Stage 1 Stage 2 Stage 1 Stage 2Stage 1 Stage 1 Stage 2 Stage 3 Stage 1 Stage 2 Stage 3 QWEN2-AUDIO40.08± 3.05 55.67± 3.09 22.57± 2.60 57.06± 3.08 40.02± 3.0141.61± 3.59 56.17± 3.61 13.31± 2.48 27.60± 3.26 57.63± 3.60 32.11± 3.32 44.40± 3.57 DESTA2.5-AUDIO95.45± 1.31 98.48± 0.78 33.81± 2.94 98.94± 0.66 71.38± 2.6295.70± 1.50 98.75± 0.85 32.87± 3.42 34.81± 3.47 98.79± 0.83 69.38± 3.15 67.11± 3.19 BLSP-EMO 66.80± 2.93 97.98± 0.90 72.57± 2.78 97.48± 0.99 88.35± 1.9370.60± 3.32 98.20± 1.00 71.01± 3.30 60.75± 3.56 97.99± 1.05 90.60± 1.97 88.12± 2.13 QWEN2.5-OMNI82.79± 2.35 91.09± 1.78 58.50± 3.07 94.67± 1.40 82.19± 2.2080.17± 2.91 89.32± 2.26 54.23± 3.63 67.96± 3.40 93.57± 1.78 79.66± 2.72 88.59± 2.13 PHI-4-MULTIMODAL60.12± 3.05 98.58± 0.76 85.12± 2.22 98.15± 0.85 89.85± 1.8853.40± 3.63 98.34± 0.97 84.19± 2.66 90.43± 2.15 97.76± 1.10 89.55± 2.22 96.64± 1.28 GEMINI 2.5 FLASH (ON) 98.68± 0.73 99.70± 0.39 69.33± 2.87 99.85± 0.30 78.77± 2.5399.03± 0.76 99.72± 0.46 72.26± 3.26 82.39± 2.78 99.93± 0.32 81.14± 2.84 85.75± 2.55 GEMINI 2.5 FLASH (OFF)91.90± 1.71 99.19± 0.59 72.98± 2.77 99.48± 0.48 80.68± 2.4591.26± 2.07 98.89± 0.81 70.18± 3.33 80.03± 2.91 99.29± 0.65 77.79± 3.02 83.03± 2.74 shown in the instructions or demonstrations. For CEQ con- straints, we employ a rule-based processor [22]; for CoT, we adopt LLM-as-a-judge [27] with GPT-5 mini 1 to determine whether the response exhibits reasoning behavior. • Task Performance: We report standard metrics for each au- dio understanding task: Word Error Rate (WER;↓) for ASR, and accuracy (↑) for SER, GR, and MMAU. Model responses are preprocessed prior to scoring. CEQ predictions are ex- tracted using a rule-based processor [11]. For CoT-based ASR, instances of hallucinated or non-terminating reason- ing [28–30] are detected via regular expressions, and the re- sulting transcriptions are replaced with empty strings; other- wise, GPT-5 mini extracts the transcription from the model’s response. For other CoT-based tasks, GPT-5 mini directly determines prediction correctness. To validate the reliability of the LLM judge, we conduct hu- man verification on 300 randomly selected samples, achieving 99% agreement, indicating strong robustness. 4. Experimental Results Table 2 reports the FCR of zero-shot, one-shot, and few-shot across ALICE’s three stages. Since MMAU is not applicable to Stage 3, as described in Sec. 2.3, we present two task groups: Task Group A covers all audio understanding tasks for Stages 1–2, while Task Group B excludes MMAU and covers Stages 1– 3. Figures 2 and 3 show the trends in task performance across varying numbers of in-context examples for the three stages un- der CEQ and CoT constraint categories, respectively. 4.1. Impact of Demonstrations with Explicit Format Con- straints FCR is near-saturated for instruction-strong models, yet one-shot still yields large compliance jumps for others. As shown in Table 2, Gemini 2.5 Flash (On) is essentially at the ceiling even in Stage 1 zero-shot (Task Group A: 98.68%; Task Group B: 99.03%), and DeSTA2.5-Audio is likewise highly compliant (A: 95.45%; B: 95.70%). In con- trast, several models are only moderately compliant at zero-shot but jump to near-ceiling with a single example (e.g., BLSP- Emo: 66.80%→97.98%, Phi-4-Multimodal: 60.12%→98.58% in Task Group A), suggesting that even with explicit constraints, demonstrations help “instantiate” the intended output schema. Qwen2-Audio is the clear outlier, staying low even at few-shot (A: 57.06%; B: 57.63%), indicating difficulty in consistently adhering to output constraints. Task performance is mostly stable across the number of examples. In the leftmost columns ((a1)–(d1)) of Figures 2 1 gpt-5-mini-2025-08-07, minimal reasoning, random seed = 42 and 3, most curves are relatively flat for CEQ and fluctuate for CoT, consistent with Stage 1 being a near-upper-bound setting where text already specifies the task and format. WER for ASR is generally low and MMAU primarily reflects intrinsic model capabilities and exhibits weakk-dependence. High format compliance is independent of high task per- formance. Table 2 and Figures 2–3 jointly suggest that all LALMs benefit from ICL examples to infer output format, yet they can not leverage such examples to internalize the core concepts of accomplishing speech-processing tasks, even under CoT, where models are provided with reasoning traces gener- ated by a stronger LALM. This phenomenon persists in Stages 2 and 3; a detailed discussion is presented in Sec. 4.4. 4.2. Impact of Removing Explicit Format Constraints Format inference from demonstrations alone is unreliable for LALMs. Comparing Stages 1 and 2 in Table 2, FCR consis- tently decreases across all models, with few-shot drops ranging from 7.39% (BLSP-Emo: 97.99% → 90.60% for Task Group B) to 29.41% (DeSTA2.5-Audio: 98.79%→ 69.38% for Task Group B). This indicates that without explicit constraint in- structions, format inference from demonstrations remains chal- lenging for LALMs. Furthermore, in Task Group A, LALMs achieving the highest Stage 1 zero-shot FCR show substantial FCR drops in Stage 2, namely DeSTA2.5-Audio (95.45% → 71.38%) and two Gemini variants (On: 98.68% → 78.77%; Off: 91.90%→ 80.68%). This reveals that strong instruction- following ability does not guarantee robust format deduction. Task performance remains stable despite constraint changes. Unlike FCR, removing explicit output constraints does not lead to a discernible change in task performance, as the task description is preserved (see the middle-column sub- figures, (a2)–(d2), of Figures 2 and 3). As in Stage 1, increasing the number of ICL examples yields only negligible performance fluctuations. 4.3. Impact of Further Removing Textual Task Descrip- tions Removing task descriptions can lead to higher format com- pliance. Comparing the FCR of Stages 2 and 3 for Task Group B in Table 2, we observe a counterintuitive trend: Most mod- els maintain or even improve FCR in both one-shot and few- shot settings. In few-shot settings, minor degradation occurs only for DeSTA2.5-Audio (69.38%→ 67.11%) and BLSP-Emo (90.60%→88.12%), while Phi-4-Multimodal achieves a high FCR of 96.64%, nearly matching its Stage 1 score (97.76%). This suggests that decreasing textual guidance in demonstra- tions does not always hinder format inference. Since formatting relies on surface-level patterns that can be matched from audio- Figure 2: Task Performance of CEQ. Performance vs. num- ber of in-context examplesk under the three-stage evalua- tion framework for closed-ended questions. Rows correspond to tasks (ASR in WER↓; SER/GR/MMAU in accuracy↑), and columns correspond to Stage 1–3. label pairs without integrating auditory semantics with textual context, the textual task descriptions retained in Stage 2 may in- stead introduce distracting variability, consistent with the find- ings that LALMs are sensitive to instruction phrasing [13, 31]. Removing them in Stage 3 reduces this variability, enabling more reliable pattern extraction. Relying solely on audio inputs and formatted outputs to infer task objective is challenging for LALMs. Unlike for- mat patterns, inferring the task objective requires cross-modal semantic grounding that cannot be achieved through surface- level pattern matching alone. As shown in the rightmost col- umn ((a3)–(c3)) of Figure 2, CEQ task performance in Stage 3 remains substantially below Stage 2, even with multiple shots, with the gap especially pronounced for ASR in DeSTA2.5- Audio (due to its inability to infer the task objective) and GR for all LALMs. In contrast, Figure 3 shows that CoT Stage 3 performance remains close to Stage 2, suggesting that reason- ing traces may provide implicit task guidance by exposing the decision process linking audio to text outputs. Notably, Qwen2- Audio and Qwen2.5-Omni exhibit an overall increasing trend in GR accuracy across Stage 3 under CoT, further highlighting this role. Crucially, however, this gain reflects LALM’s reliance on textual content within reasoning traces rather than true speech- text integration [8, 32], as evidenced by the persistent failure under CEQ where no such textual guidance exists. 4.4. Summary Across Three Stages Our three-stage results suggest that format and task inference rely on fundamentally distinct mechanisms. Format inference requires only surface-level pattern matching, allowing LALMs to maintain or improve FCR in Stage 3 using audio-label pairs alone. Conversely, task inference requires cross-modal seman- tic grounding: the model must integrate auditory information to map it to a correct response, a capability that remains chal- lenging for current LALMs. This distinction also clarifies the Figure 3: Task Performance of CoT. Performance vs. number of in-context examplesk under the three-stage evaluation frame- work for chain-of-thought prompting. Rows correspond to tasks (ASR in WER↓; SER/GR/MMAU in accuracy↑), and columns correspond to Stage 1–3. CoT Stage 3 findings: the performance recovery reflects re- liance on textual clues in reasoning traces to infer task objective rather than genuine speech-text integration, evidenced by the persistent failure under CEQ. The fact that providing reason- ing traces generated by stronger models does not improve task performance in Stages 1 and 2 further confirms that LALMs’ ICL limitations are not attributable to insufficient reasoning ca- pacity, but to their inability to ground auditory perception into task-relevant semantics and integrate it with textual context, consistent with prior findings on speech/audio reason- ing [8]. 5. Conclusion and Limitations We propose ALICE, a three-stage evaluation framework that progressively removes textual guidance to probe ICL in LALMs. Across six LALMs and four audio understanding tasks, we observe an asymmetry: in-context demonstrations consistently boost format compliance yet do not improve—and often degrade—core task performance. Such dissociation sug- gests that format inference is largely driven by surface-level pattern matching, whereas task inference requires cross-modal semantic grounding that appears insufficiently developed in current LALMs. Moreover, strong instruction-following does not translate into reliable demonstration-based deduction, and the apparent CoT gains observed in Stage 3, where only au- dio inputs and formatted outputs are presented as demonstra- tions, indicate reliance on textual cues in reasoning traces rather than genuine audio–text integration. These findings suggest a need for better cross-modal integration training paradigms for LALMs. Nevertheless, our evaluation focuses on format- constrained audio understanding tasks; extending ALICE to open-ended generation, richer semantic settings, more diverse tasks such as speech synthesis, and investigating demonstration selection and ordering [15, 33] remains an important direction for future work. 6. Acknowledgement We acknowledge the computational and storage support pro- vided by the National Center for High-performance Comput- ing (NCHC) of the National Applied Research Laboratories (NARLabs) in Taiwan. 7. Generative AI Use Disclosure In this work, generative AI tools, including LLMs, were used as judge models and as auxiliary support for language editing (e.g., style polishing, grammar refinement, and proofreading). The research conceptualization, experiment execution, and the analysis and interpretation of results were conducted entirely by the authors without LLM involvement. All technical con- tributions and intellectual work are attributable to the authors, with LLMs used only to support evaluation and to enhance the manuscript’s clarity and readability. 8. References [1] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, p. 1–8. [2] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin et al., “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024. [3] K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, S.-F. Huang, C.- K. Yang, C.-E. Yu, C.-W. Chen, W.-C. Chen, C.-y. Huang et al., “Desta2.5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment,” arXiv preprint arXiv:2507.02768, 2025. [4] C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang, “BLSP-emo: Towards empathetic large speech-language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds.Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, p. 19 186–19 199. [Online]. Available: https://aclanthology.org/2024.emnlp-main. 1070/ [5] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin, “Qwen2.5-omni technical report,” 2025. [6] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen et al., “Phi-4-mini technical report:Compact yet powerful multi- modal language models via mixture-of-loras,” arXiv preprint arXiv:2503.01743, 2025. [7] C.-K. Yang, N. S. Ho, and H.-y. Lee, “Towards holistic evalua- tion of large audio-language models: A comprehensive survey,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, p. 10 155–10 181. [8] C.-K. Yang, N. Ho, Y.-T. Piao, and H. yi Lee, “SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information,” in Interspeech 2025, 2025, p. 1788–1792. [9] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,” in Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, 2023, p. 15 757–15 773. [10] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=14rn7HpKVk [11] K.-H. Lu, C.-Y. Kuan, and H.-y. Lee, “Speech-ifeval: Evaluat- ing instruction-following and quantifying catastrophic forgetting in speech-aware language models,” in Interspeech 2025.Inter- national Speech Communication Association, 2025, presented at Interspeech 2025, Rotterdam, The Netherlands. [12] Y. Gao, B. Wang, C. Wei, S. Sun, and A. Aw, “IFEval-audio: Benchmarking instruction-following capability in audio-based large language models,” in Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, K. Inui, S. Sakti, H. Wang, D. F. Wong, P. Bhattacharyya, B. Banerjee, A. Ekbal, T. Chakraborty, and D. P. Singh, Eds.Mumbai, India: The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, Dec. 2025, p. 120–127. [Online]. Available: https://aclanthology.org/2025.ijcnlp-short.11/ [13] B. Li, W. Huang, Y. Qiu, Y. Guo, H. Wang, Z. Li, J. Peng, Z. Ma, X. Chen, and K. Yu, “Isa-bench: Benchmarking instruc- tion sensitivity for large audio language models,” arXiv preprint arXiv:2510.23558, 2025. [14] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhari- wal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Lan- guage models are few-shot learners,” Advances in neural informa- tion processing systems, vol. 33, p. 1877–1901, 2020. [15] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?”in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, p. 11 048–11 064. [Online]. Available: https://aclanthology.org/ 2022.emnlp-main.759/ [16] Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, B. Chang, X. Sun, L. Li, and Z. Sui, “A survey on in-context learning,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds.Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, p. 1107–1128. [Online]. Available: https://aclanthology.org/2024.emnlp-main.64/ [17] W. Chen, Y. Lin, Z. Zhou, H. Huang, Y. Jia, Z. Cao, and J.-R. Wen, “ICLEval:Evaluating in-context learning ability of large language models,” in Proceedings of the 31st International Conference on Computational Linguistics, O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert, Eds. Abu Dhabi, UAE: Association for Computational Linguistics, Jan. 2025, p. 10 398–10 422. [Online]. Available: https://aclanthology.org/2025.coling-main. 693/ [18] J. Pan, T. Gao, H. Chen, and D. Chen, “What in-context learn- ing “learns” in-context: Disentangling task recognition and task learning,” in Findings of the Association for Computational Lin- guistics: ACL 2023, 2023, p. 8298–8319. [19] S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, “An explanation of in-context learning as implicit bayesian inference,” in International Conference on Learning Representations, 2022. [Online]. Available:https://openreview.net/forum?id= RdJVFCHjUMI [20] N. Roll, C. Graham, Y. Tatsumi, K. T. Nguyen, M. Sumner, and D. Jurafsky, “In-context learning boosts speech recognition via human-like adaptation to speakers and language varieties,” in Pro- ceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds.Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 4412–4426. [Online]. Available: https://aclanthology.org/2025.emnlp-main.219/ [21] C.-K. Yang, Y.-T. Piao, T.-W. Hsu, S.-W. Fu, Z. Chen, K.-H. Lu, S.-F. Huang, C.-H. H. Yang, Y.-C. F. Wang, Y.-N. Chen et al., “Sake: Towards editing auditory attribute knowledge of large audio-language models,” arXiv preprint arXiv:2510.16917, 2025. [22] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,” arXiv preprint arXiv:2311.07911, 2023. [23] S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available:https: //openreview.net/forum?id=TeVAZXr3yv [24] OpenAI, “Introducing gpt-5,” 2025. [Online]. Available: https: //openai.com/index/introducing-gpt-5/ [25] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,” IEEE Transactions on Affective Computing, vol. 5, no. 4, p. 377–390, 2014. [26] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil- ities,” arXiv preprint arXiv:2507.06261, 2025. [27] C.-H. Chiang and H.-y. Lee, “Can large language models be an alternative to human evaluations?”in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd- Graber, and N. Okazaki, Eds.Toronto, Canada: Association for Computational Linguistics, Jul. 2023, p. 15 607–15 631. [Online]. Available: https://aclanthology.org/2023.acl-long.870/ [28] R. Frieske and B. E. Shi, “Hallucinations in neural automatic speech recognition: Identifying errors and hallucinatory models,” arXiv preprint arXiv:2401.01572, 2024. [29] C.-Y. Kuan and H.-y. Lee, “Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning,” in ICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, p. 1–5. [30] S. Kumar, I. Thorbecke, S. Burdisso, E. Villatoro-Tello, M. KE, K. Hacio ̆ glu, P. Rangappa, P. Motlicek, A. Ganapathiraju, and A. Stolcke, “Performance evaluation of slam-asr: The good, the bad, the ugly, and the way forward,” in 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing Work- shops (ICASSPW). IEEE, 2025, p. 1–5. [31] J. Peng, Y. Wang, B. Li, Y. Guo, H. Wang, Y. Fang, Y. Xi, H. Li, X. Li, K. Zhang, S. Wang, and K. Yu, “A survey on speech large language models for understanding,” IEEE Journal of Selected Topics in Signal Processing, vol. 20, no. 1, p. 2–31, 2026. [32] C. Wang, G. Deng, X. Yang, H. Qiu, and T. Zhang, “When audio and text disagree: Revealing text bias in large audio-language models,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds.Suzhou, China: Association for Computational Linguistics, Nov. 2025, p. 4878–4888. [Online]. Available: https://aclanthology.org/2025. emnlp-main.246/ [33] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language mod- els use long contexts,” Transactions of the Association for Com- putational Linguistics, vol. 12, p. 157–173, 2024.