Paper deep dive
Investigating Faithfulness in Large Audio Language Models
Yunfei Chu, Jin Xu, Qian Yang
Models: Qwen2-Audio-7B-Instruct, SALMONN-13B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/11/2026, 12:56:58 AM
Summary
This paper investigates the faithfulness of chain-of-thought (CoT) reasoning in Large Audio-Language Models (LALMs) such as Qwen2-Audio and SALMONN. By applying targeted interventions—including filler token injection, paraphrasing, early answering, and adding mistakes—the authors demonstrate that LALMs generally produce CoTs that are faithful to their underlying decision processes, contrasting with findings in text-only LLMs.
Entities (5)
Relation Signals (3)
Qwen2-Audio → evaluatedon → SAKURA
confidence 100% · We evaluate our method on two benchmarks: SAKURA [13] and MMAR [24].
SALMONN → evaluatedon → MMAR
confidence 100% · We evaluate our method on two benchmarks: SAKURA [13] and MMAR [24].
LALMs → exhibits → Faithfulness
confidence 90% · LALMs generally produce CoTs that appear to be faithful to their underlying decision processes.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Faithfulness measures whether chain-of-thought (CoT) representations accurately reflect a model's decision process and can be used as reliable explanations. Prior work has shown that CoTs from text-based LLMs are often unfaithful. This question has not been explored for large audio-language models (LALMs), where faithfulness is critical for safety-sensitive applications. Reasoning in LALMs is also more challenging, as models must first extract relevant clues from audio before reasoning over them. In this paper, we investigate the faithfulness of CoTs produced by several LALMs by applying targeted interventions, including paraphrasing, filler token injection, early answering, and introducing mistakes, on two challenging reasoning datasets: SAKURA and MMAR. After going through the aforementioned interventions across several datasets and tasks, our experiments suggest that, LALMs generally produce CoTs that appear to be faithful to their underlying decision processes.
Tags
Links
- Source: https://arxiv.org/abs/2509.22363
- Canonical: https://arxiv.org/abs/2509.22363
Trouble viewing inline? Open PDF directly →
Full Text
22,236 characters extracted from source content.
Expand or collapse full text
INVESTIGATING FAITHFULNESS IN LARGE AUDIO LANGUAGE MODELS Lovenya Jain ∗,1 , Pooneh Mousavi ∗,2,3 , Mirco Ravanelli 2,3 , Cem Subakan 4,2,3 1 Birla Institute of Technology and Science-Pilani, 2 Concordia University, 3 Mila, Quebec-AI-Institute, 4 Laval University, ∗ Equal contribution ABSTRACT Faithfulness measures whether chain-of-thought (CoT) represen- tations accurately reflect a model’s decision process and can be used as reliable explanations. Prior work has shown that CoTs from text-based LLMs are often unfaithful. This question has not been explored for large audio-language models (LALMs), where faithfulness is critical for safety-sensitive applications. Reasoning in LALMs is also more challenging, as models must first extract relevant clues from audio before reasoning over them. In this paper, we investigate the faithfulness of CoTs produced by several LALMs by applying targeted interventions, including paraphrasing, filler token injection, early answering, and introducing mistakes, on two challenging reasoning datasets: SAKURA and MMAR. After going through the aforementioned interventions across several datasets and tasks, our experiments suggest that, LALMs generally produce CoTs that appear to be faithful to their underlying decision processes. Index Terms— Faithfulness, Large Audio Language Models, Explainable AI. 1. INTRODUCTION Large Language Models (LLMs) have transformed machine learning in recent years. An interesting feature of LLMs is that they can be prompted to provide reasoning for their decisions, potentially help- ing their deployment in decision-critical applications such as health- care or forensics. Prior studies show that generating intermediate reasoning steps, often called chain-of-thought (CoT) or reasoning chains, can improve explainability and trustworthiness [1–3]. CoT decomposes complex tasks into smaller subproblems and allocates more computation to harder questions, which can make predictions more accurate and interpretable. However, this raises a key question for trustworthy AI: How faithful are the chain-of-thought explanations produced by LLMs? In machine learning, faithfulness refers to whether an explanation reflects the model’s actual reasoning process. A faithful explana- tion correctly shows why the model produced a specific answer. An unfaithful one may sound plausible, but it does not match the true decision process. Faithfulness is therefore crucial for building reli- able and safe AI systems. Recent work suggests that for text-only LLMs, CoT representations may not reflect the model’s underlying reasoning [4–7]. Other studies have proposed methods to measure the faithfulness of CoT explanations [6, 8–11]. While LLMs have shown strong reasoning abilities through language, extending these systems to understand audio is essential This research was also enabled in part by support provided by Cal- cul Qu ́ ebec (https://w.calculquebec.ca/) and the Digital Re- search Alliance of Canada (https://alliancecan.ca/en), as well as support provided by Natural Sciences and Engineering Research Council of Canada (NSERC). for building models that can reason using contextual auditory cues. Large Audio-Language Models (LALMs) integrate audio encoders with pre-trained decoder-based LLMs, enabling open-ended Audio Question Answering (AQA) and free-form response generation [12]. Several recent LALMs incorporate chain-of-thought (CoT) reason- ing to improve perception and reasoning over audio [13–18]. Al- though these works report accuracy gains from CoT, it is unclear whether such reasoning can serve as faithful explanations of the model’s decision process. This question is important because rea- soning in audio-language models is inherently more challenging than in text-only models. Despite recent progress, even the most advanced LALMs underperform on expert-level reasoning tasks compared to foundational tasks such as event classification [13]. There are several reasons why CoT may fail as a faithful expla- nation: (i) Post-hoc reasoning: The model may generate reasoning after it has already decided on an answer [19]. Since this reasoning does not influence the decision, it may not reflect the true internal process. (i) Extra test-time computation: The performance gain may come from the extra computation allowed by generating more tokens between the question and the answer [20]. (i) Encoded rea- soning in CoT: The model may encode useful information in ways not understandable to humans. This could involve subtle changes in wording, punctuation, or phrasing. Because Large Audio-Language Models (LALMs) operate on an additional data modality involving sound, it remains unclear whether empirical findings on the faithfulness of CoT representa- tions in text-only LLMs extend to LALMs. In this paper, we present a faithfulness analysis of several LALMs, including Qwen2 and SALMONN. Our evaluation uses modified chain-of-thought (CoT) representations to test whether the semantic content of the CoT influences the model’s final predictions. We analyze faithfulness by systematically modifying the CoTs and observing how these changes affect model accuracy, aiming to identify potential sources of unfaithfulness. We find that LALMs generally produce CoTs that accurately reflect their decision process across tasks and datasets. 2. METHODOLOGY In this section, we describe the interventions used to assess the faith- fulness of CoT representations. The prompting setup is illustrated in Figure 1. An Audio-Language Model is given an input audio sam- ple along with a text prompt containing a question about the audio. The prompt also instructs the model to reason step by step. The model first generates a CoT explaining its reasoning process. Af- ter obtaining the CoT, we modify it for the faithfulness analysis and then prompt the model to produce a final answer based on the mod- ified CoT. In the following subsection, we describe the intervention strategies in detail. arXiv:2509.22363v2 [cs.LG] 14 Oct 2025 What animal is causing this sound? It sounds like a horse, because I hear that typical clip-clap sound that horse hooves make on old timey streets. It ..... ......., because I hear that ........... sound that horse hooves make ......... ......... ........ Final answer? Horse User CoT User Model Output Modified CoT (Filler Tokens) What animal is causing this sound? It sounds like a horse, because I hear that typical clip-clap sound that horse hooves make on old timey streets. It resembles a horse, as I can hear the distinctive clip-clop noise made by horse hooves on vintage streets. Final answer? Horse User CoT User Model Output Modified CoT (Paraphrasing) What animal is causing this sound? It sounds like a horse, because I hear that typical clip-clap sound that horse hooves make on old timey streets. It sounds like a horse. Final answer? Horse User CoT User Model Output Modified CoT (EarlyAnswering) What animal is causing this sound? It sounds like a horse, because I hear that typical clip-clap sound that horse hooves make on old timey streets. It sounds like a dog, because I hear that typical clip-clap sound that dog paws make on old timey streets. Final answer? Horse User CoT User Model Output Modified CoT (AddingMistakes) Fig. 1. (left) Filler token modification of CoT representation. We randomly mask a certain percentage of the CoT to see the effect on the answer that the model gives. (middle-left) Paraphrasing of CoT representation. We use an LLM to paraphrase the CoT. (middle-right) Early answering modification of CoT. We remove the last sentences depending on the rate of early answering. (right) Adding mistakes to the CoT. We add mistakes to the CoT with a certain rate. 2.1. Chain of Thought Interventions We apply the following CoT modifications to assess the faithfulness of the selected LALMs. Filler Tokens To assess whether the model relies on the semantic content of the CoT representations, we replace a given percentage of CoT tokens with filler tokens. If the answer of the model after the replacement with filler tokens does not change compared to the original CoT, (if the results remain consistent), we infer that the model may not be faithful to its generated CoT, likely due to extra test time compute. Specifically, we randomly replace portions of the CoT with sequences of filler tokens, each represented as a space followed by three periods (“ ...”). We test filler token strings ranging from 0 tokens up to the full length of the CoT, with a step size of 5% of a given CoT. Paraphrasing To test whether CoTs contain hidden information encoded in their phrasing, we paraphrase each CoT using a large lan- guage model while preserving the semantic content of the CoT. This removes specific wording choices that could carry implicit signals. If phrasing-encoded information contributes to the accuracy gains from CoT, paraphrasing should reduce the consistency of results be- tween the original CoT and the paraphrased one. This intervention targets potential unfaithfulness due to encoded reasoning in CoTs. We progressively paraphrase each sentence until reaching the end of the CoT. For paraphrasing, we use the same LLM backbone as used within the corresponding LALM. For example, LLAMA 1 [21] for SALMONN and Qwen 2 [22] LLM for Qwen2-Audio. Early Answering To test whether accuracy gains of CoT originate 1 lmsys/vicuna-13b-v1.1 2 Qwen/Qwen-7B from post-hoc reasoning, we truncate the CoT and prompt the model to answer using only the partial reasoning. For each collected CoT, we progressively remove sentences from the end, producing trun- cated versions such as [], [x 1 ], [x 1 , x 2 ], ..., [x 1 , ..., x n ]. Each truncated CoT replaces the original one in the sample, and the model is then prompted to answer as before. If performance remains consistent with the original CoT even with shorter CoTs, this then suggests that reasoning is likely to be post-hoc, since the earlier sentences were generated after the decision was made. Adding Mistakes To assess whether the model relies on the se- mantic content of the CoT, we incrementally introduce mistakes into the reasoning. If the model’s final answer changes after these mod- ifications, it suggests that the CoT is a faithful representation of the model’s decision process. Conversely, if the answer remains un- changed, it indicates that the semantic content of the CoT may not influence the decision. Similar to paraphrasing, we progressively add mistakes to the CoT sentences using the same LLM backbone as used within the corresponding LALM. Unlike paraphrasing, after introducing an incorrect sentence (e.g., at positionx 5 ), we prompt the model to continue reasoning from this incorrect step and use the resulting CoT to evaluate the model’s faithfulness. 3. EXPERIMENTAL SETUP Models We use Qwen2-Audio-7B-Instruct [16], and SALMONN- 13B [23]. We chose these models because they have shown promis- ing results in audio understanding and reasoning tasks. Datasets We evaluate our method on two benchmarks: SAKURA [13] and MMAR [24].SAKURA tests single- and multi-hop rea- soning over 500 multiple-choice questions per track across four audio attributes: gender, language, emotion, and animal sounds. MMARSAKURA AnimalSAKURA EmotionSAKURA GenderSAKURA Language 020406080100 Percentage % of Words Replaced 50 60 70 80 90 100 Consistency (%) Random Partial Filler, QWEN 020406080100 Percentage % of Words Replaced 50 60 70 80 90 100 Consistency (%) Random Partial Filler, SALMONN Fig. 2. Injecting filler tokens inside CoTs (left) for QWEN, (right) for SALMONN. MMARSAKURA AnimalSAKURA EmotionSAKURA GenderSAKURA Language 020406080100 Percentage % of Sentences Paraphrased 80 85 90 95 100 Consistency (%) Paraphrasing, QWEN 020406080100 Percentage % of Sentences Paraphrased 70 75 80 85 90 95 100 Consistency (%) Paraphrasing, SALMONN Fig. 3. Paraphrasing of CoTs (left) for QWEN, (right) for SALMONN. MMAR is a more challenging benchmark with 1,000 curated audio–question–answer triplets from real-world videos, requiring multi-step reasoning and domain knowledge. Unlike prior bench- marks, MMAR spans not only speech, audio, and music, but also their mixtures, making it a diverse and difficult evaluation set. Prompting and Sampling For each example, we generate three CoT chains to improve the stability and robustness of our evaluation. We include only CoTs containing 2–7 sentences to avoid outliers caused by overly short or excessively long responses. Our filtering criteria and prompting templates for the different intervention types are available in our GitHub repository. 4. EXPERIMENTAL RESULTS We evaluate model faithfulness by measuring consistency, defined as the agreement between the predicted answer using the original CoT and the predicted answer using a modified CoT. We analyze how consistency varies with the percentage of modifications applied to the original CoT. 4.1. Filler Tokens Figure 2 shows the effect of progressively replacing CoT words with filler tokens on answer consistency. For both Qwen2-Audio and SALMONN, consistency decreases as a larger proportion of the CoT is replaced, indicating that removing semantic content degrades per- formance. The degree of performance drop varies across datasets. For example, in SALMONN, the SAKURA-Gender shows a smaller decline compared to other tracks, suggesting that gender-related rea- soning may be less dependent on explicit CoT content and more influenced by other factors. Overall, across both models and all datasets, we observe a clear downward trend in consistency as the proportion of filler tokens increases. 4.2. Paraphrasing In Figure 3, we see the effect of paraphrasing the CoTs. We initially observe a drop in consistency, but unlike the other modifications the observation is that both QWEN2-Audio and SALMONN remain more or less consistent with the original answers which points to- MMARSAKURA AnimalSAKURA EmotionSAKURA GenderSAKURA Language 020406080100 Percentage % of Sentences Kept 50 60 70 80 90 100 Consistency (%) Early Answering, QWEN 020406080100 Percentage % of Sentences Kept 50 60 70 80 90 100 Consistency (%) Early Answering, SALMONN Fig. 4. Early Answering modification on CoTs (left) for QWEN, (right) for SALMONN. MMARSAKURA AnimalSAKURA EmotionSAKURA GenderSAKURA Language 020406080100 Percentage % of Chain Without Mistake 50 60 70 80 90 100 Consistency (%) Adding Mistakes, QWEN 020406080100 Percentage % of Chain Without Mistake 50 60 70 80 90 100 Consistency (%) Adding Mistakes, SALMONN Fig. 5. Adding Mistakes modification on CoTs (left) for QWEN, (right) for SALMONN. wards the fact that the models do not rely on a specific hidden encod- ing in CoTs, but rather pay attention to the actual semantic content. 4.3. Early Answering Figure 4 shows the effect of progressively truncating the CoT and prompting the model to answer with only the remaining initial sen- tences. Unlike text-only LLMs, where prior work [8] observed a slower decline or even flat trends, LALMs show a clear dependence on the amount of reasoning retained. When fewer sentences are kept, answer consistency drops sharply, and it increases almost monoton- ically as more of the original CoT is preserved. This suggests that LALMs rely heavily on the full reasoning chain rather than produc- ing post-hoc justifications. 4.4. Adding Mistakes Figure 5 shows the effect of injecting random mistakes into the CoTs. For both Qwen2-Audio and SALMONN, answer consistency decreases when a larger portion of the reasoning is corrupted, then recovers when the original unaltered CoT is fully preserved (100%). This pattern suggests that both models rely on the semantic content of the CoTs: introducing incorrect reasoning disrupts their predic- tions, indicating that the models tend to attend to the meaning of the CoTs rather than treating them as post-hoc explanations. 5. CONCLUSION In this paper, we presented an empirical study on the faithfulness of chain-of-thought reasoning in Large Audio-Language Models. Our results suggest that, unlike text-only LLMs, after going through the list of interventions we described, LALMs generally produce CoTs that seem to be faithful to their underlying decision processes, with no substantial variation across different datasets or tasks. To the best of our knowledge, this is the first work to investigate faithfulness in LALMs, and we view it as an important step toward more compre- hensive studies of reliability and interpretability in multimodal rea- soning models. For future work, we plan to evaluate a broader range of LALMs and explore a more comprehensive list of intervention strategies. 6. REFERENCES [1] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao, “React: Synergizing rea- soning and acting in language models,” in International Con- ference on Learning Representations (ICLR), 2023. [2] Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan, “Explanations from large language models make small reasoners better,” 2022. [3] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou, “Rationale-augmented ensembles in lan- guage models,” 2022. [4] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” 2023. [5] Fazl Barez, Tung-Yu Wu, Iv ́ an Arcuschin, Michael Lan, Vin- cent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Is- abelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Dami- ano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio, “Chain-of-thought is not explainability,” 2025. [6] Iv ́ an Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy, “Chain-of- thought reasoning in the wild is not always faithful,” 2025. [7] Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, and Kathleen McKeown, “Do mod- els explain themselves? counterfactual simulatability of natu- ral language explanations,” 2023. [8] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil ̇ e Luko ˇ si ̄ ut ̇ e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tris- tan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez, “Measuring faithful- ness in chain-of-thought reasoning,” 2023. [9] Katie Matton, Robert Osazuwa Ness, John Guttag, and Emre Kıcıman, “Walk the talk? measuring the faithfulness of large language model explanations,” 2025. [10] Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin,“Can large language models explain themselves? a study of llm-generated self- explanations,” 2023. [11] Andreas Madsen, Sarath Chandar, and Siva Reddy, “Are self- explanations from large language models faithful?,” 2024. [12] Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe, “On the landscape of spoken language models: A comprehensive survey,” arXiv preprint arXiv:2504.08528, 2025. [13] Chih-Kai Yang, Neo Ho, Yen-Ting Piao, and Hung yi Lee, “Sakura: On the multi-hop reasoning of large audio-language models based on speech and audio information,” 2025. [14] Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catan- zaro, “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” 2025. [15] Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao, “Audio-reasoner: Im- proving reasoning capability in large audio language models,” 2025. [16] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhi- fang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou, “Qwen2-audio technical report,” 2024. [17] Zhifeng Kong, Arushi Goel, Joao Felipe Santos, Sreyan Ghosh, Rafael Valle, Wei Ping, and Bryan Catanzaro, “Audio flamingo sound-cot technical report: Improving chain-of-thought rea- soning in sound understanding,” 2025. [18] Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen,“Audio-cot: Exploring chain-of-thought reasoning in large audio language model,” arXiv preprint arXiv:2501.07246, 2025. [19] Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell, “What do we need to build explainable ai systems for the medical domain?,” 2017. [20] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, “Chain-of-thought prompting elicits reasoning in large lan- guage models,” 2023. [21] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [22] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.,“Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023. [23] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, “Salmonn: Towards generic hearing abilities for large language models,” 2024. [24] Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, Tianrui Wang, Yuping Wang, Yuxuan Wang, Yihao Wu, Guan- rou Yang, Jianwei Yu, Ruibin Yuan, Zhisheng Zheng, Ziya Zhou, Haina Zhu, Wei Xue, Emmanouil Benetos, Kai Yu, Eng- Siong Chng, and Xie Chen, “Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix,” 2025.