Paper deep dive
The Sound of Absence: Audio-Language Embedding Models Struggle with Negation
Chun-Yi Kuan, Hung-yi Lee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/15/2026, 5:30:34 AM
Summary
This paper investigates the inability of audio-language embedding models (e.g., CLAP, M2D-CLAP, WAVE) to handle negation, revealing an 'affirmation bias' where affirmative and negated captions map to nearly identical embeddings. The authors introduce NegEval-Audio, a framework that transforms standard datasets (AudioCaps, Clotho) into negation-aware tasks (Retrieval-Neg and MCQ-Neg). Experiments show sharp performance degradation under negation, with a training-free steering method only partially mitigating the issue, indicating a fundamental flaw in representation geometry that requires explicit negation-aware training.
Entities (10)
Relation Signals (14)
NegEval-Audio → introduces → Retrieval-Neg
confidence 98% · we introduce NegEval-Audio, a framework that converts existing datasets into two negation-aware tasks, Retrieval-Neg and Multiple-Choice Negation
NegEval-Audio → introduces → MCQ-Neg
confidence 98% · we introduce NegEval-Audio, a framework that converts existing datasets into two negation-aware tasks, Retrieval-Neg and Multiple-Choice Negation
Audio-language embedding models → exhibits → Affirmation Bias
confidence 96% · these models fail to encode negated sound concepts, mapping affirmative and negated captions to nearly identical representations
M2D-CLAP → evaluatedon → Clotho
confidence 95% · Table I shows M2D-CLAP performance on Clotho evaluation split
WAVE → evaluatedon → AudioCaps
confidence 95% · Table I shows WAVE performance on AudioCaps test split
WAVE → evaluatedon → Clotho
confidence 95% · Table I shows WAVE performance on Clotho evaluation split
CLAP → evaluatedon → AudioCaps
confidence 95% · Table I shows CLAP performance on AudioCaps test split
CLAP → evaluatedon → Clotho
confidence 95% · Table I shows CLAP performance on Clotho evaluation split
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Audio-language embedding models such as CLAP are widely evaluated on matching present sound events, but rarely on negation. We show this affirmation-only evaluation hides a key limitation: these models fail to encode negated sound concepts, mapping affirmative and negated captions to nearly identical representations. To expose this blind spot, we introduce NegEval-Audio, a framework that converts existing datasets into two negation-aware tasks, Retrieval-Neg and Multiple-Choice Negation (MCQ-Neg), to probe whether models distinguish present from absent events. On AudioCaps and Clotho, performance degrades sharply under negation, with negation-type MCQ accuracy falling far below chance, and the failure persists even for a recent multimodal LLM-based embedding model. While a training-free steering method improves MCQ-Neg, it yields marginal gains for Retrieval-Neg. This indicates that affirmation bias is a fundamental flaw in the representation geometry, necessitating explicit negation-aware training objectives.
Tags
Links
- Source: https://arxiv.org/abs/2607.12290v1
- Canonical: https://arxiv.org/abs/2607.12290v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
68,601 characters extracted from source content.
Expand or collapse full text
The Sound of Absence: Audio-Language Embedding Models Struggle with Negation Chun-Yi Kuan♡, Hung-yi Lee♡♣ ♡Graduate Institute of Communication Engineering, National Taiwan University, Taiwan ♣Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan chunyi.kuan.tw@gmail.com Abstract Audio-language embedding models such as CLAP are widely evaluated on matching present sound events, but rarely on negation. We show this affirmation-only evaluation hides a key limitation: these models fail to encode negated sound concepts, mapping affirmative and negated captions to nearly identical representations. To expose this blind spot, we introduce NegEval-Audio, a framework that converts existing datasets into two negation-aware tasks, Retrieval-Neg and Multiple-Choice Negation (MCQ-Neg), to probe whether models distinguish present from absent events. On AudioCaps and Clotho, performance degrades sharply under negation, with negation-type MCQ accuracy falling far below chance, and the failure persists even for a recent multimodal LLM-based embedding model. While a training-free steering method improves MCQ-Neg, it yields marginal gains for Retrieval-Neg. This indicates that affirmation bias is a fundamental flaw in the representation geometry, necessitating explicit negation-aware training objectives. Figure 1: Overview of NegEval-Audio. Existing audio-caption datasets are converted into negation-aware retrieval and multiple-choice tasks by extracting present sound concepts and verifying plausible absent concepts. I Introduction Recent progress in audio-language embedding models, which map audio clips and text into a shared space for retrieval and matching, has enabled strong performance on benchmarks such as AudioCaps [32] and Clotho [15]. This family now spans contrastive bi-encoders such as CLAP [64, 16, 70, 71, 30, 43, 21, 55, 59, 14] and M2D-CLAP [48, 49], as well as recent embedding models built on multimodal LLMs such as WAVE [56]. As a result, modern models have become increasingly effective at detecting and matching acoustic events that are present. However, these evaluations share an implicit assumption: audio semantics is affirmation-only. Models are rewarded for recognizing sounds that occur, but are rarely examined on whether they understand which plausible sounds do not occur. This blind spot matters: in natural language, negation is fundamental, and many practical audio–language applications depend on it. For example, a user may search for queries such as “rain without thunder” or “speech without background music”, which are common, precise, and meaningful in practice. Despite this importance, negation remains relatively underexplored in audio–language embedding models. Current embedding models are naturally trained to associate audio with sounds that are present, but not to account for sounds that a description explicitly negates. To address this gap, we introduce NegEval-Audio, an evaluation framework designed to probe negation understanding in audio–language embedding models. Figure 1 provides an overview. Instead of constructing a new dataset from scratch, our framework transforms existing audio retrieval datasets into negation-aware evaluation sets. We first extract positive (existent) sound concepts from the caption and then propose corresponding negative (non-existent) concepts. These negative concepts are verified using audio-aware large language models (ALLMs) [10, 22, 12, 2, 66, 1, 57, 11, 23, 20, 19, 68, 8, 18, 60, 61, 29, 25, 51, 13, 47, 63, 24, 40, 38, 37, 5, 6], ensuring that they truly do not occur in the audio. This pipeline allows standard retrieval datasets to be directly converted into negation-aware formats without additional data collection. Based on the verified concept pairs, we construct two evaluation tasks. The first is text-to-audio retrieval with negation (Retrieval-Neg), which tests whether models can handle realistic queries that combine affirmative and negative statements, e.g., “Birds are chirping and rain is falling, with no human voices in the background.” This requires the model not only to retrieve audio containing specific events, but also to ensure that other events are absent, unlike traditional retrieval, which considers only presence. The second is multiple-choice questions with negation (MCQ-Neg), whose carefully designed options reveal specific failure types: given several closely related descriptions, the model must select the one that correctly reflects the audio, while distractors differ only by affirming or denying particular events. For instance, “A baby is crying with no adult voices present” and “Adult voices are present with no baby crying” are surface-similar yet imply opposite judgments, requiring attention to subtle but meaningful differences. Through this protocol, we find that audio–language embedding models struggle with negation. In the embedding space, affirmative and negated captions often occupy highly overlapping regions; for example, “a sound of cat meowing” and “no cat meowing sound” receive similar representations. The model thus fails to encode the distinction between the presence and the explicit absence of a sound event, revealing a persistent affirmation bias that holds even for the recent multimodal LLM-based WAVE [56]. Recognizing this limitation, we further explore mitigation strategies. Rather than the obvious data-centric route of augmenting training captions with synthetic negation, we investigate a lightweight, training-free steering technique to adjust the embedding space directly. While this intervention improves performance on MCQ-Neg, it yields only marginal gains for Retrieval-Neg. This contrast reveals that the models’ inability to handle negation is not a mere surface-level artifact, but a fundamental flaw in their representation geometry, underscoring the need for explicit negation-aware training objectives. Our findings indicate that robust audio understanding has to go beyond recognizing present events and incorporate explicit reasoning about negation and sound absence. Our contributions are summarized as follows: (1) We identify negation as a critical and previously overlooked failure mode in audio–language embedding models. (2) We introduce NegEval-Audio, a systematic framework that transforms existing audio-caption datasets into negation-aware retrieval and multiple-choice benchmarks. Across CLAP, M2D-CLAP, and a recent MLLM-based embedding model, we show that current models fail to reliably distinguish present from absent sound events. (3) We provide a controlled text-side diagnosis showing that negated descriptions remain close to both their affirmative counterparts and bag-of-concepts representations, and explore training-free embedding steering as a lightweight mitigation. Steering improves local option ranking in MCQ-Neg but yields only marginal retrieval gains, indicating a deeper limitation in the joint embedding geometry. I Related Work I-A Audio–language embedding models. Contrastive audio–language models [64, 16, 71, 43, 21, 48, 49, 56] learn a shared space between audio clips and natural language descriptions, enabling text-to-audio retrieval, audio–text matching, and zero-shot audio classification. Representative models such as CLAP [64, 16] and its variants [70, 71, 30, 43, 21, 48, 55, 59, 49, 14] have shown strong performance on standard benchmarks such as AudioCaps [32] and Clotho [15]. Recent benchmarks further examine compositional audio understanding, including multiple sound events, attributes, and temporal relations. However, these evaluations are still largely built around affirmative captions that describe sounds present in the audio. Whether audio–language embedding models can also handle descriptions that specify what should be absent remains underexplored. I-B Compositionality and negation in multimodal embeddings. Prior work [72, 58, 74, 45, 26] in vision–language modeling has shown that contrastive embeddings often rely on salient content words while underusing relational structure, word order, and logical operators. This behavior is commonly described as bag-of-words-like matching [42]: models recognize individual concepts but may fail to represent how these concepts are composed. Negation is a representative failure case. Vision–language models [4, 73, 3] can assign similar representations to affirmative and negated descriptions, such as “a dog” and “no dog,” revealing an affirmation bias in the joint embedding space. These findings suggest that strong retrieval performance does not necessarily imply robust logical understanding. Motivated by this observation, we ask whether a similar failure mode appears in audio–language embeddings, where queries such as “speech without background music” and “rain without thunder” are natural and practically meaningful. I NegEval-Audio Benchmark Construction I-A Overview of NegEval-Audio Construction Pipeline Existing audio-language datasets are typically built under an affirmation-only assumption, where captions describe sound events that are present in the audio. Our key idea is to transform these standard datasets into negation-aware evaluation sets without collecting new audio. Instead of creating new annotations from scratch, we derive negation semantics from the audio-caption pair itself, using captions to identify present sound events and audio-side verification to confirm plausible absent ones. Formally, let an audio sample be denoted as a, paired with its caption c. From c, we extract a set of positive sound concepts (a)=p1,p2,…,pmP(a)=\p_1,p_2,…,p_m\, where each pip_i corresponds to a sound event explicitly described as occurring in the audio. We then construct a complementary set of negative sound concepts (a)=n1,n2,…,nkN(a)=\n_1,n_2,…,n_k\, where each njn_j represents a sound event that is plausible in the same acoustic scene but does not occur in the audio. These negative concepts are first proposed by an LLM based on contextual plausibility inferred from the caption, and are then verified using audio-aware LLMs to ensure their absence. Through this process, each audio sample is associated with a pair of concept sets ((a),(a))(P(a),N(a)), representing what the audio contains and what it explicitly does not contain. This concept-level representation enables us to systematically construct two negation-aware evaluation tasks: (1) Retrieval-Neg, where text queries include both positive and negative statements, and (2) MCQ-Neg, where closely related descriptions differ only by affirming or negating specific sound events. Importantly, NegEval-Audio is dataset-agnostic and can convert existing affirmation-only audio-language retrieval benchmarks into negation-aware evaluation sets. In the following subsections, we describe how positive concepts are extracted from captions, how negative concepts are proposed and verified, and how these concept sets are used to construct the two tasks. I-B Positive Concept Extraction from Captions Given an audio sample a and its paired caption c, the first step of our pipeline is to extract the set of sound events that are explicitly described as occurring in the audio. We employ an LLM to perform semantic extraction of sound events from captions. The LLM is prompted to identify and normalize the sound events described in c into concise concept phrases that represent distinct acoustic events. For example, the caption “A dog barking while rain is falling in the background” is mapped to (a)=dog barking,rain falling.P(a)=\dog barking,rain falling\. This step preserves the semantic intent of the caption while converting it into a structured concept-level representation. The extracted positive concepts serve as the semantic anchor of each audio sample and form the basis for constructing negation-aware evaluation tasks in subsequent stages. I-C Negative Concept Proposal and Verification After obtaining the positive concept set (a)P(a) from the caption, the next step is to construct a complementary set of negative sound concepts. We first denote the set of candidate negatives proposed by the LLM as ~(a)=n~1,n~2,…,n~k N(a)=\ n_1, n_2,…, n_k\, where each n~j n_j represents a sound event that is plausible in the same acoustic scene but not mentioned in the caption. A naive approach would be to randomly select sound events as negatives. However, this would lead to unrealistic negation cases that do not reflect meaningful semantic reasoning. Instead, we require candidate negatives to satisfy a contextual plausibility constraint. Given the positive concept set (a)P(a), we prompt the LLM to suggest sound events that could reasonably co-occur with these concepts in the same acoustic scene. For example, given the caption “A dog barking while rain is falling”, the LLM might propose candidates like human speech, car passing, or thunder. However, plausibility inferred from text alone is insufficient. We therefore introduce an audio-side verification step using an audio-aware LLM. For each candidate concept n~j∈~(a) n_j∈ N(a), we query the audio-aware LLM with a sound-existence question of the form: “Does the audio contain the sound of n~j n_j?” Its response is then used to determine whether n~j n_j is present in the audio. To assess the reliability of the audio-aware LLM verification step, we manually validated a subset of the constructed negative concepts. We randomly sampled 200 audio clips from AudioCaps and Clotho, covering 460 automatically verified negative concepts. A human annotator judged whether each negative concept verified by the audio-aware LLM was indeed absent from the corresponding audio. The annotator agreed with the audio-aware LLM verification in 445 out of 460 cases, yielding a 96.7% agreement rate. Only the candidates verified as absent are retained to form the final negative concept set: v(a)=nj∣n~j∈~(a),Audio-aware LLM indicates absence in aN_v(a)=\n_j n_j∈ N(a),\ Audio-aware LLM indicates absence in a\. Note that this verification step does not presuppose the capability under evaluation. The audio-aware LLM is only asked affirmative sound-existence questions (i.e., whether a sound is present), a task on which such models perform reliably; it is never required to interpret negated language. The negation semantics are introduced afterward, at the text-construction stage. Our benchmark then evaluates embedding models, a different model family, on these negated descriptions. The 96.7% human agreement further confirms the reliability of this step. Through this two-stage process, negative concepts are text-proposed and audio-verified. This ensures that v(a)N_v(a) consists of realistic, semantically meaningful, and truly absent sound events, rather than arbitrary negatives. The resulting pair ((a),v(a))(P(a),N_v(a)) provides a structured semantic representation of what the audio contains and what it explicitly does not contain. I-D Construction of Retrieval-Neg Task Using the verified concept sets ((a),v(a))(P(a),N_v(a)), we transform the original standard retrieval setting into a negation-aware task. For each audio sample a, we construct a negation-aware query qnegq_neg that contains both positive and negative statements. Conceptually, the query is designed to satisfy two semantic constraints: the audio must contain (a)P(a) and must not contain v(a)N_v(a). This changes the nature of the retrieval problem from affirmation-only matching to negation-constrained matching, while keeping the retrieval protocol unchanged. In practice, we construct a negation-aware query qnegq_neg using a simple rule-based template to ensure consistency. For each audio sample, we start from the original caption c and append a negation statement using one sampled concept from v(a)N_v(a). The template takes the form: original caption c + “There is no sound of absent sound”, where absent sound is sampled from the verified negative concept set v(a)N_v(a). For example, suppose the original caption is “A dog barking while rain is falling” and a verified negative concept is human speech. The rule-based construction yields “A dog barking while rain is falling. There is no sound of human speech”. Since this template-based sentence may sound slightly unnatural, we further apply the LLM to paraphrase the query into a more fluent form without altering its semantics. The above example becomes “A dog barking while rain is falling, with no human speech”. This two-step process ensures that negation queries are generated in a controlled manner, while maintaining natural linguistic quality for retrieval evaluation. Importantly, this task remains within the standard retrieval framework. The only change lies in the semantic structure of the query. This allows us to directly compare model performance between standard retrieval and Retrieval-Neg. I-E Construction of MCQ-Neg Task While Retrieval-Neg evaluates coarse-grained negation in a retrieval setting, MCQ-Neg provides a sentence-level diagnostic of how models interpret negation. For each audio a, we construct a four-option multiple-choice question derived from the structured pair ((a),v(a))(P(a),N_v(a)), where candidate sentences differ only in the affirmation or negation of specific sound events. Following prior work [41, 4], we categorize MCQs into three types based on the structure of the correct answer: Affirmation, Negation, and Hybrid. Each type probes distinct aspects of negation and exposes different failure modes. Options are generated programmatically using prompt-encoded logical templates with an LLM. Given (a)P(a) and v(a)N_v(a), the model produces one correct option and three distractors according to predefined logical rules (e.g., false affirmation, reversed logic, false exclusion). For each MCQ type, we specify dedicated prompts that enforce exact logical constraints. Affirmation MCQ. The correct option describes only (a)P(a). Distractors introduce incorrect events from v(a)N_v(a) or omit true ones. Negation MCQ. The correct option negates a concept from v(a)N_v(a) while remaining consistent with (a)P(a). Distractors invert this logic. Hybrid MCQ. The correct option simultaneously affirms (a)P(a) and negates v(a)N_v(a), while distractors minimally alter this structure to create semantically opposite statements. For example, if (a)=dog barking,rainP(a)=\dog barking,rain\ and v(a)=human speechN_v(a)=\human speech\, a generated Hybrid MCQ may include: (A) A dog is barking while rain is falling, with no human speech present (correct); (B) A dog is barking while rain is falling, and human speech can be heard (distractor); with additional options constructed by systematically reversing affirmation and negation. All generated options are lightly paraphrased to improve naturalness without changing the underlying sound concepts. A subset of the generated MCQs was manually verified to ensure logical correctness and linguistic validity. IV Training-Free Negation Steering Beyond diagnosis, we further explore whether negation bias can be mitigated. Inspired by prior work [3, 50, 65, 9, 75, 17, 31, 54], we adopt a training-free steering method to mitigate negation bias in audio-language retrieval models. Similar to observations in vision-language embeddings [3], we find an affirmation bias, where captions such as “dog barking” and “no dog barking” are mapped to highly similar embeddings, leading to confusion between presence and absence. To counter this, we edit the caption embedding directly. Given a caption C containing a negated concept CnegC_neg, we compute e∗=eC−λeCnege^*=e_C-λ e_C_neg , where eCe_C is the full caption embedding and eCnege_C_neg is the embedding of the negated sound concepts (e.g., “dog barking”). We set λ=0.2λ=0.2 for all models, datasets, and tasks, rather than tuning it per setting. This provides a fair, non-oracle comparison and keeps the intervention in a small-perturbation regime, where the embedding is steered away from the negated concept while preserving the remaining caption semantics. We report full λ-sweep results in Appendix B. TABLE I: Text-to-audio retrieval performance on standard and negation-aware queries under AudioCaps and Clotho. For each column, the best and second-best scores are highlighted in bold and underline, respectively. AudioCaps test [32] Clotho evaluation [15] Model Standard Retrieval-Neg Standard Retrieval-Neg R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 LAION-CLAP [64] – 630k-audioset-best 35.3 70.1 82.8 26.5 60.3 75.6 14.8 37.3 51.2 14.4 35.4 48.7 – 630k-best 31.4 67.5 80.8 26.3 59.1 75.8 15.3 37.4 51.6 16.1 37.0 50.6 – music-audioset 33.0 71.9 85.0 23.5 57.0 74.8 13.9 35.2 49.2 13.1 33.8 46.1 – music-speech-audioset 35.0 68.6 84.9 21.7 57.9 73.4 14.4 37.7 50.9 12.2 34.2 46.8 – music-speech 30.6 67.0 80.5 24.5 59.4 75.6 14.9 38.2 51.2 14.8 38.0 50.1 M2D-CLAP [49] – M2D-CLAP-2025 40.9 77.9 89.2 29.9 63.2 77.1 18.9 43.9 58.1 16.2 38.0 51.4 – M2D-CLAP-2024 22.6 55.0 70.6 16.9 46.1 62.1 9.6 24.5 32.6 6.5 19.6 27.4 WAVE [56] – WAVE-7B 44.0 77.9 88.7 39.6 75.2 86.1 26.6 52.7 66.3 22.1 50.0 61.5 V Experimental Setup Evaluation data construction. We construct NegEval-Audio on the test split of AudioCaps [32] and the evaluation split of Clotho [15]. For LLM-based components, we use Claude Opus 4.8 for text generation, including caption refinement and MCQ construction. For audio-based sound-existence verification, we use the audio-aware LLM Qwen2.5-Omni-7B [66]. We choose this model due to its strong performance on recent audio-related benchmarks [28, 27, 34, 36, 53, 46, 44, 62, 69, 7, 39, 33, 35]. Evaluated models. We evaluate a broad set of publicly available audio–text embedding models. We first consider LAION-CLAP checkpoints [64], covering general audio, music, speech, and their combinations. We then include M2D-CLAP [49, 48], which departs from the original CLAP design through architectural and training modifications. Finally, we evaluate WAVE [56], a recent multimodal embedding model built upon multimodal LLMs, to reflect the emerging trend of MLLM-based audio–text representation learning. Evaluation protocol. To isolate the effect of negation, we compare standard Retrieval with Retrieval-Neg and evaluate MCQ-Neg. In text-to-audio retrieval, queries rank candidate audio samples by embedding similarity, and performance is measured by Recall@K. We report K equal to 1, 5, and 10. The retrieval protocol is identical across settings; only the query semantics differ. For MCQ-Neg, each question contains four candidate descriptions with one correct answer. The model selects the option whose text embedding has the highest cosine similarity to the audio embedding; no generation or task-specific tuning is involved. We report accuracy for each question type and their overall average to analyze model behavior under affirmation, negation, and hybrid conditions. TABLE I: MCQ-Neg performance (%) across three question types on AudioCaps and Clotho. Best and second-best scores are highlighted. AudioCaps test [32] Clotho evaluation [15] Model Affirmation Negation Hybrid Average Affirmation Negation Hybrid Average LAION-CLAP [64] – 630k-audioset-best 81.1 0.6 47.8 43.2 59.8 1.9 49.9 37.2 – 630k-best 80.9 0.6 44.1 41.9 66.5 3.5 44.2 38.1 – music-audioset 85.4 0.6 43.8 43.3 59.6 7.1 48.3 38.4 – music-speech-audioset 80.8 0.3 42.1 41.1 56.1 3.9 32.8 30.9 – music-speech 74.1 1.1 29.6 34.9 49.0 3.5 31.3 27.9 M2D-CLAP [49] – M2D-CLAP-2025 69.4 0.2 14.0 27.9 42.7 3.9 19.7 22.1 – M2D-CLAP-2024 67.4 0.6 24.2 30.8 41.1 4.9 23.7 23.2 WAVE [56] – WAVE-7B 78.3 0.9 51.3 43.5 75.0 5.3 49.2 42.2 TABLE I: Effect of steering (λ=0.2λ=0.2) on negation-aware tasks: Retrieval-Neg and MCQ-Neg. The best and second-best results in each column are shown in bold and underlined, respectively. Δ denotes the change after steering relative to the corresponding baseline; Δ columns are shaded light blue. AudioCaps test [32] Clotho evaluation [15] Model Retrieval-Neg (R@5) MCQ-Neg (Avg) Retrieval-Neg (R@5) MCQ-Neg (Avg) Before After Δ Before After Δ Before After Δ Before After Δ LAION-CLAP [64] – 630k-audioset-best 60.3 61.8 +1.5 43.2 45.0 +1.8 35.4 36.7 +1.3 37.2 36.6 -0.6 – 630k-best 59.1 58.5 -0.6 41.9 42.8 +0.9 37.0 38.7 +1.7 38.1 39.4 +1.3 – music-audioset 57.0 57.3 +0.3 43.3 44.9 +1.6 33.8 35.0 +1.2 38.4 37.7 -0.7 – music-speech-audioset 57.9 58.6 +0.7 41.1 42.2 +1.1 34.2 34.8 +0.6 30.9 31.1 +0.2 – music-speech 59.4 58.1 -1.3 34.9 38.4 +3.5 38.0 37.5 -0.5 27.9 29.5 +1.6 M2D-CLAP [49] – M2D-CLAP-2025 63.2 65.8 +2.6 27.9 30.2 +2.3 38.0 42.3 +4.3 22.1 22.9 +0.8 – M2D-CLAP-2024 46.1 50.2 +4.1 30.8 30.9 +0.1 19.6 22.5 +2.9 23.2 24.1 +0.9 WAVE [56] – WAVE-7B 75.2 77.1 +1.9 43.5 54.5 +11.0 50.0 52.0 +2.0 42.2 52.4 +10.2 VI Results and Analysis VI-A Standard Retrieval Masks Negation Failure Standard retrieval looks reliable, but negation-aware retrieval exposes a hidden failure. Table I compares standard text-to-audio retrieval with Retrieval-Neg, where the audio candidates and evaluation protocol are unchanged, and only the query is modified to include an explicit absent sound concept. On AudioCaps, all models degrade under Retrieval-Neg. Averaged over the eight checkpoints, R@1 drops from 34.1 to 26.1, R@5 from 69.5 to 59.8, and R@10 from 82.8 to 75.1. The drop is also clear for strong models: M2D-CLAP-2025 falls from 40.9 to 29.9 in R@1, and LAION-CLAP music-speech-audioset from 35.0 to 21.7. On Clotho, the degradation is smaller but consistent on average, with R@1, R@5, and R@10 dropping from 16.1, 38.4, and 51.4 to 14.4, 35.8, and 47.8, respectively. These results show that affirmation-only retrieval can overestimate semantic understanding: retrieving present sounds well does not imply that a model can use negated sound concepts as exclusion constraints. VI-B MCQ-Neg Reveals an Affirmation Bias MCQ-Neg confirms that the failure is specific to negation, not general audio recognition. Table I breaks performance into Affirmation, Negation, and Hybrid questions. For Affirmation questions, where the correct option describes sounds present in the audio, models remain substantially above chance: accuracy ranges from 67.4% to 85.4% on AudioCaps and from 41.1% to 75.0% on Clotho. In contrast, Negation accuracy collapses to 0.2–1.1% on AudioCaps and 1.9–7.1% on Clotho, far below the 25% random baseline for four-way multiple choice. This near-zero performance indicates a systematic affirmation bias: mentioned sound concepts are treated as positive evidence even when explicitly negated. Hybrid questions show an intermediate pattern, with accuracy ranging from 14.0% to 51.3% on AudioCaps and from 19.7% to 49.9% on Clotho. Together, these results show that current audio–language embeddings can recognize present events, but fail to bind negation to the intended sound concept. VI-C Steering Helps Local Decisions but Not Global Retrieval Steering shows that the failure is partially editable, but not fully solved. Table I reports the effect of training-free steering with λ=0.2λ=0.2. For Retrieval-Neg, gains are limited: R@5 changes by +1.2 points on average on AudioCaps, with per-model changes from -1.3 to +4.1, and by +1.7 points on Clotho, with changes from -0.5 to +4.3. This suggests that steering slightly reduces the attraction toward negated concepts, but does not substantially reshape the global retrieval space. The effect is clearer for MCQ-Neg, especially on WAVE-7B, which improves from 43.5% to 54.5% on AudioCaps and from 42.2% to 52.4% on Clotho. For CLAP and M2D-CLAP checkpoints, however, MCQ gains are modest, ranging from +0.1 to +3.5 points on AudioCaps and from -0.7 to +1.6 points on Clotho. This contrast suggests that steering can alter local option rankings in MCQ-Neg, but full-corpus retrieval requires a more globally consistent geometry where audio containing the forbidden sound is pushed away. The limited retrieval gains therefore suggest that negation failure is not a surface artifact of the word “no”, but reflects a deeper weakness in the joint embedding space. Details on the selection of λ are provided in Appendix B. TABLE IV: Controlled diagnosis of CLAP text-side negation handling. Values are averaged over five LAION-CLAP checkpoints. Diagnostic Mean Polarity sensitivity Normalized affinity to affirmative text 1.07 Polarity change / content swap 0.30 Negation scope Scope permutation similarity 0.70 Content swap similarity 0.58 Bag-of-concepts similarity 0.82 Audio retrieval bridge Corr. between negated and affirmative margins 0.76 Negated-query margin +0.018 Affirmative-query margin +0.072 VI-D Text-Side Diagnosis of Negation Failure The retrieval results reveal a recurring failure mode in CLAP-style audio–text retrieval: negated queries are often matched to audio containing the very sound that should be excluded. To understand this mechanism, we conduct a controlled diagnosis on five LAION-CLAP checkpoints. Our goal is not to introduce another benchmark, but to isolate whether a negated description, such as “P, with no Y”, is sufficiently separated from its affirmative counterpart, “P, with Y”, in the embedding space. To eliminate confounding variables in real-world datasets, such as complex free-form captions and noisy acoustic backgrounds, we construct a clean and controlled diagnostic set using ESC-50 [52]. This design allows us to isolate the role of negation more precisely. Each item contains two positive sound events and one designated negated concept Y. We render matched target–distractor audio pairs that share the same base mixture and differ only in whether Y is present: the distractor contains Y, while the target replaces it with a matched filler sound from the same category group and with similar loudness. This gives 1,000 controlled items where the presence or absence of each sound event is known by construction, avoiding the verification noise of caption-derived negatives. Table IV summarizes the diagnostic results averaged over the five checkpoints. Negation barely moves the text embedding. We first compare each negated query with its affirmative counterpart. Since raw cosine similarities in CLAP text space are difficult to interpret due to anisotropy, we calibrate them using within-query references: an unrelated query as the floor and a meaning-preserving paraphrase as the ceiling. We report normalized affinity, where 0 corresponds to the unrelated floor and 1 corresponds to the paraphrase ceiling. Under this calibration, the affirmative counterpart reaches the paraphrase ceiling, with a normalized affinity of 1.07. In other words, “P, with Y” is as close to “P, with no Y” as a meaning-preserving paraphrase is. We further compare this polarity change against a content-swap reference, where one mentioned sound concept is replaced. The polarity change produces only 30% of the embedding distance caused by the content swap, suggesting that the text encoder treats negation as a weak perturbation rather than a distinct semantic condition. Negation is weakly bound to the intended concept. We then test whether the encoder tracks which sound concept is negated. For each query, we construct three variants that mention the same concepts but negate a different one. If negation scope were well represented, these variants should be clearly separated. To make this comparison interpretable, we again use content swap as a reference scale: it measures how much the embedding changes when one mentioned sound concept is replaced. As shown in Table IV, scope variants remain more similar to each other than content-swapped variants, and the query is also highly similar to its bag-of-concepts form, which removes function and polarity words and keeps only the mentioned sound concepts. These results indicate that the encoder preserves which concepts are mentioned more strongly than whether a specific concept is negated. The text-side collapse propagates to retrieval. Finally, we examine whether this text-side behavior is reflected in audio retrieval. For each item, we compute the audio margin between the Y-containing distractor and the Y-absent target: m(q)=s(q,adist)−s(q,atgt)m(q)=s(q,a_dist)-s(q,a_tgt), where adista_dist and atgta_tgt denote the distractor and target audio, respectively. A correctly interpreted negated query should yield a negative margin, preferring the target. Instead, the average margin for negated queries remains positive, meaning that they still prefer audio containing the forbidden sound. The affirmative query yields a stronger positive margin, and the two margins are strongly correlated across items. This means that examples where the affirmative query strongly prefers the Y-containing audio are also examples where the negated query tends to prefer it. Thus, negated queries behave like attenuated affirmative queries rather than reversing the preference toward the Y-absent audio. More details are provided in Appendix A. Together, these diagnostics suggest that CLAP under-encodes negation in its text space. A negated description remains close to a bag of mentioned sound concepts and to its affirmative counterpart, so the joint embedding space has little basis for retrieving audio that excludes the negated sound. VII Conclusion We presented NegEval-Audio, a framework for evaluating negation understanding in audio-language embedding models. Our results reveal a systematic affirmation bias: models can match sounds that are present, but often fail to treat negated sounds as constraints on what should be absent. To address this, future work should explore explicit negation-aware training objectives [67] and compositional alignment. Ultimately, these findings call for audio-language models that go beyond what is heard and account for what should not be heard. VIII Limitations First, our steering method assumes access to the negated concept. While this concept is known by construction in NegEval-Audio, applying the method to free-form queries would require an additional parsing step to identify the negated span. Second, each Retrieval-Neg query contains only a single negated concept; queries involving multiple, coordinated, or nested negations therefore remain unexplored. Third, NegEval-Audio is constructed from AudioCaps and Clotho, both of which primarily cover general environmental sounds. Extending the benchmark to other domains, such as spoken content and musical attributes, is an important direction for future work. Finally, we use a fixed steering strength, λ, across all models to ensure a fair and controlled comparison. Although this avoids model-specific tuning, selecting λ separately for each model may produce different trade-offs between negation sensitivity and overall retrieval performance. Generative AI Use Disclosure The authors used ChatGPT and Claude to assist with grammar checking, language polishing, and improving the readability of the manuscript. Acknowledgments We thank Kai-Wei Chang for insightful discussions and valuable feedback, which helped improve this work. References [1] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al. (2025) Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743. Cited by: §I. [2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §I. [3] B. Aggarwal, A. More, M. Soni, and S. D. Bhat (2026) Seeing what’s not there: negation understanding needs more than training. In The Fourteenth International Conference on Learning Representations, Cited by: §I-B, §IV. [4] K. Alhamoud, S. Alshammari, Y. Tian, G. Li, P. H. Torr, Y. Kim, and M. Ghassemi (2025) Vision-language models do not understand negation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29612–29622. Cited by: §I-B, §I-E. [5] S. Arora, K. Chang, C. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H. Lee, K. Livescu, and S. Watanabe (2025) On the landscape of spoken language models: a comprehensive survey. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §I. [6] K. Chang, W. Chen, E. Hu, H. Lee, and J. Glass (2026) TiCo: time-controllable training for spoken dialogue models. arXiv preprint arXiv:2603.22267. Cited by: §I. [7] K. Chang, E. Hu, C. Kuan, W. Ren, W. Chen, G. Lin, Y. Tsao, S. Sun, H. Lee, and J. Glass (2026) Game-time: evaluating temporal dynamics in spoken language models. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §V. [8] K. Chang, H. Wu, Y. Wang, Y. Wu, H. Shen, W. Tseng, I. Kang, S. Li, and H. Lee (2024) Speechprompt: prompting speech language models for speech processing tasks. IEEE/ACM Transactions on Audio, Speech, and Language Processing. Cited by: §I. [9] R. Chen, A. Arditi, H. Sleight, O. Evans, and J. Lindsey (2025) Persona vectors: monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509. Cited by: §IV. [10] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, et al. (2024) Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. Cited by: §I. [11] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou (2023) Qwen-audio: advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. Cited by: §I. [12] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §I. [13] W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, S. Y. Guo, and I. King (2025) Recent advances in speech language models: a survey. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13943–13970. Cited by: §I. [14] H. Dinkel, Z. Yan, T. Wang, Y. Wang, X. Sun, Y. Niu, J. Liu, G. Li, J. Zhang, and J. Luan (2026) GLAP: general contrastive audio-text pretraining across domains and languages. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 14737–14741. Cited by: §I, §I-A. [15] K. Drossos, S. Lipping, and T. Virtanen (2020) Clotho: an audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 736–740. Cited by: Figure 2, Figure 3, TABLE VII, TABLE IX, §I, §I-A, TABLE I, TABLE I, TABLE I, §V. [16] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang (2023) Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §I, §I-A. [17] S. Facchiano, G. Strano, D. Crisostomi, I. Tallini, T. Mencattini, F. Galasso, and E. Rodolà (2025) Activation patching for interpretable steering in music generation. arXiv preprint arXiv:2504.04479. Cited by: §IV. [18] Y. Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y. Shangguan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer (2024) Audiochatllama: towards general-purpose speech abilities for llms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5522–5532. Cited by: §I. [19] S. Ghosh, A. Goel, J. Kim, S. Kumar, Z. Kong, S. Lee, C. H. Yang, R. Duraiswami, D. Manocha, R. Valle, et al. (2025) Audio flamingo 3: advancing audio intelligence with fully open large audio language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §I. [20] S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro (2025) Audio flamingo 2: an audio-language model with long-audio understanding and expert reasoning abilities. In Forty-second International Conference on Machine Learning, Cited by: §I. [21] S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. R. Evuru, S. Sakshi, O. Nieto, R. Duraiswami, D. Manocha, et al. (2024) CompA: addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference on Learning Representations, Cited by: §I, §I-A. [22] Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass (2023) Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), p. 1–8. Cited by: §I. [23] Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. R. Glass (2024) Listen, think, and understand. In The Twelfth International Conference on Learning Representations, Cited by: §I. [24] Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu (2025) Recent advances in discrete speech tokens: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §I. [25] W. Held, Y. Zhang, M. Li, W. Shi, M. J. Ryan, and D. Yang (2025) Distilling an end-to-end voice assistant without instruction training data. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7876–7891. Cited by: §I. [26] C. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna (2023) Sugarcrepe: fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems 36, p. 31096–31116. Cited by: §I-B. [27] C. Huang, W. Chen, S. Yang, A. T. Liu, C. Li, Y. Lin, W. Tseng, A. Diwan, Y. Shih, J. Shi, et al. (2025) Dynamic-superb phase-2: a collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. In The Thirteenth International Conference on Learning Representations, Cited by: §V. [28] C. Huang, K. Lu, S. Wang, C. Hsiao, C. Kuan, H. Wu, S. Arora, K. Chang, J. Shi, Y. Peng, et al. (2024) Dynamic-superb: towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 12136–12140. Cited by: §V. [29] S. Ji, Y. Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng, et al. (2024) Wavchat: a survey of spoken dialogue models. arXiv preprint arXiv:2411.13577. Cited by: §I. [30] X. Jing, A. Triantafyllopoulos, and B. Schuller (2024) ParaCLAP–towards a general language-audio model for computational paralinguistic tasks. arXiv preprint arXiv:2406.07203. Cited by: §I, §I-A. [31] X. Kang, D. Shi, and L. Chen (2025) Model whisper: steering vectors unlock large language models’ potential in test-time. arXiv preprint arXiv:2512.04748. Cited by: §IV. [32] C. D. Kim, B. Kim, H. Lee, and G. Kim (2019) Audiocaps: generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 119–132. Cited by: Figure 2, Figure 3, TABLE VI, TABLE VIII, §I, §I-A, TABLE I, TABLE I, TABLE I, §V. [33] C. Kuan, K. Chang, and H. Lee (2026) AQAScore: evaluating semantic alignment in text-to-audio generation via audio question answering. arXiv preprint arXiv:2601.14728. Cited by: §V. [34] C. Kuan, W. Huang, and H. Lee (2024) Understanding sounds, missing the questions: the challenge of object hallucination in large audio-language models. In Interspeech 2024, Cited by: §V. [35] C. Kuan, W. Huang, and H. Lee (2026) Walking through uncertainty: an empirical study of uncertainty estimation for audio-aware large language models. arXiv preprint arXiv:2604.25591. Cited by: §V. [36] C. Kuan and H. Lee (2025) Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §V. [37] C. Kuan and H. Lee (2025) From alignment to advancement: bootstrapping audio-language alignment with synthetic data. IEEE Transactions on Audio, Speech and Language Processing 33, p. 4604–4619. External Links: Document Cited by: §I. [38] C. Kuan and H. Lee (2025) Teaching audio-aware large language models what does not hear: mitigating hallucinations through synthesized negative samples. In Proc. Interspeech 2025, p. 2073–2077. Cited by: §I. [39] C. Kuan and H. Lee (2026) AQUA-bench: beyond finding answers to knowing when there are none in audio question answering. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §V. [40] C. Kuan, C. Yang, W. Huang, K. Lu, and H. Lee (2024) Speech-copilot: leveraging large language models for speech processing via task decomposition, modularization, and program generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), p. 1060–1067. Cited by: §I. [41] M. I. Laka Mugarza (1990) Negation in syntax–on the nature of functional categories and projections. Ph.D. Thesis, Massachusetts Institute of Technology. Cited by: §I-E. [42] M. Lewis, N. Nayak, P. Yu, J. Merullo, Q. Yu, S. Bach, and E. Pavlick (2024) Does clip bind concepts? probing compositionality in large image models. In Findings of the Association for Computational Linguistics: EACL 2024, p. 1487–1500. Cited by: §I-B. [43] Y. Li, Z. Guo, X. Wang, and H. Liu (2024) Advancing multi-grained alignment for contrastive language-audio pre-training. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 7356–7365. Cited by: §I, §I-A. [44] K. Lu, C. Kuan, and H. Lee (2025) Speech-ifeval: evaluating instruction-following and quantifying catastrophic forgetting in speech-aware language models. In Proc. Interspeech 2025, p. 2078–2082. Cited by: §V. [45] Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna (2023) Crepe: can vision-language foundation models reason compositionally?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10910–10921. Cited by: §I-B. [46] Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y. Chao, R. Xu, et al. (2025) MMAR: a challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032. Cited by: §V. [47] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer, et al. (2025) Discrete audio tokens: more than a survey!. Transactions on Machine Learning Research. Cited by: §I. [48] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, M. Yasuda, S. Tsubaki, and K. Imoto (2024) M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation. In Interspeech, p. 57–61. External Links: Document Cited by: §I, §I-A, §V. [49] D. Niizumi, D. Takeuchi, M. Yasuda, B. Thien Nguyen, Y. Ohishi, and N. Harada (2025) M2D-clap: exploring general-purpose audio-language representations beyond clap. IEEE Access 13 (), p. 163313–163330. External Links: Document Cited by: §I, §I-A, TABLE I, TABLE I, TABLE I, §V. [50] T. Pai, J. Wang, L. Lu, S. Sun, H. Lee, and K. Chang (2025) Billy: steering large language models via merging persona vectors for creative generation. arXiv preprint arXiv:2510.10157. Cited by: §IV. [51] J. Peng, Y. Wang, B. Li, Y. Guo, H. Wang, Y. Fang, Y. Xi, H. Li, X. Li, K. Zhang, et al. (2025) A survey on speech large language models for understanding. IEEE Journal of Selected Topics in Signal Processing. Cited by: §I. [52] K. J. Piczak (2015) ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, p. 1015–1018. Cited by: §A-A, §VI-D. [53] S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha (2025) MMAU: a massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations, Cited by: §V. [54] F. Sammani, T. Chamiti, P. Gavrikov, and N. Deligiannis (2026) When negation is a geometry problem in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11553–11562. Cited by: §IV. [55] T. Takano, Y. Okamoto, Y. Kanamori, Y. Saito, R. Nagase, and H. Saruwatari (2025) Human-clap: human-perception-based contrastive language-audio pretraining. In 2025 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), p. 131–136. Cited by: §I, §I-A. [56] C. Tang, Q. Xiao, K. Mei, T. Wang, F. Rao, and C. Zhang (2025) WAVE: learning unified & versatile audio-visual embeddings with multimodal llm. arXiv preprint arXiv:2509.21990. Cited by: Appendix A, §I, §I, §I-A, TABLE I, TABLE I, TABLE I, §V. [57] C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang (2024) SALMONN: towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, Cited by: §I. [58] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross (2022) Winoground: probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5238–5248. Cited by: §I-B. [59] W. Tseng, X. Zhou, M. Huo, Y. Shao, H. Zhang, and D. Yu (2025) Revisiting audio-language pretraining for learning general-purpose audio representation. arXiv preprint arXiv:2511.16757. Cited by: §I, §I-A. [60] C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y. Liu, C. Zong, and J. Zhang (2023) Blsp: bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916. Cited by: §I. [61] C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang (2024) BLSP-emo: towards empathetic large speech-language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 19186–19199. Cited by: §I. [62] D. Wang, J. Wu, J. Li, D. Yang, X. Chen, T. Zhang, and H. Meng (2025) Mmsu: a massive multi-task spoken language understanding and reasoning benchmark. arXiv preprint arXiv:2506.04779. Cited by: §V. [63] H. Wu, X. Chen, Y. Lin, K. Chang, H. Chung, A. H. Liu, and H. Lee (2024) Towards audio language modeling–an overview. arXiv preprint arXiv:2402.13236. Cited by: §I. [64] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov (2023) Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: Appendix A, §I, §I-A, TABLE I, TABLE I, TABLE I, §V. [65] T. Xie, S. Yang, C. Li, D. Yu, and L. Liu (2025) EmoSteer-tts: fine-grained and training-free emotion-controllable text-to-speech via activation steering. arXiv preprint arXiv:2508.03543. Cited by: §IV. [66] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, et al. (2025) Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215. Cited by: §I, §V. [67] J. Xu (2026) Omni-negclip: enhancing clip with front-layer contrastive fine-tuning for comprehensive negation understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9392–9401. Cited by: §VII. [68] C. Yang, Y. Fu, C. Li, Y. Lin, Y. Lin, W. Chen, H. L. Chung, C. Kuan, W. Huang, K. Lu, et al. (2024) Building a taiwanese mandarin spoken language model: a first attempt. arXiv preprint arXiv:2411.07111. Cited by: §I. [69] S. Yang, M. Tu, A. T. Liu, X. Qu, H. Lee, L. Lu, Y. Wang, and Y. Wu (2025) ParaS2S: benchmarking and aligning spoken language models for paralinguistic-aware speech-to-speech interaction. arXiv preprint arXiv:2511.08723. Cited by: §V. [70] Z. Ye, R. Huang, Y. Ren, Z. Jiang, J. Liu, J. He, X. Yin, and Z. Zhao (2023) Clapspeech: learning prosody from text context with contrastive language-audio pre-training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9317–9331. Cited by: §I, §I-A. [71] Y. Yuan, Z. Chen, X. Liu, H. Liu, X. Xu, D. Jia, Y. Chen, M. D. Plumbley, and W. Wang (2024) T-clap: temporal-enhanced contrastive language-audio pretraining. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), p. 1–6. Cited by: §I, §I-A. [72] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023) When and why vision-language models behave like bags-of-words, and what to do about it?. In The Eleventh International Conference on Learning Representations, Cited by: §I-B. [73] Y. Zhang, Y. Su, Y. Liu, and S. Yeung-Levy (2025) NegVQA: can vision language models understand negation?. arXiv preprint arXiv:2505.22946. Cited by: §I-B. [74] T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin (2022) Vl-checklist: evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221. Cited by: §I-B. [75] L. Zhou, H. Jiang, J. Li, T. Wang, and H. Li (2026) EmoShift: lightweight activation steering for enhanced emotion-aware speech synthesis. arXiv preprint arXiv:2601.22873. Cited by: §IV. Appendix A Controlled Diagnosis of Text-Side Negation Handling This appendix provides the implementation details and full results for the controlled diagnosis in Section VI-D. The main paper reports the results averaged over five LAION-CLAP [64] checkpoints; here, we additionally include the LLM-based embedding model WAVE-7B [56]. A-A Controlled setup We construct 1,000 controlled target–distractor pairs from ESC-50 [52]. Each item contains two positive sound events, p1p_1 and p2p_2, and one designated negated concept Y. The two positive events are concatenated into a 10 s base clip. We then overlay either Y or a filler event F at the same onset and gain: • target: the base clip with filler F, but without Y; • distractor: the same base clip with Y. The filler F is matched to Y in coarse ESC-50 category and loudness. Therefore, the target and distractor differ only in whether the overlaid event is F or the negated concept Y. Sound presence is known by construction and does not require ALLM verification. For each item, we construct a negated query, “P, with no Y”, its affirmative counterpart, “P, with Y”, and the oracle description “P”, where P contains the two positive concepts. We additionally construct the following deterministic text references: • a meaning-preserving paraphrase, obtained by reversing the positive-concept order and replacing “no” with “without”; • a content-swap reference, obtained by replacing one positive concept; • an unrelated query from another item; • three scope variants, which negate each of p1,p2,Y\p_1,p_2,Y\ in turn; • a bag-of-concepts reference containing only the three concept phrases. For WAVE-7B, an instruction-aware MLLM embedding model, audio embeddings are conditioned on a text instruction. We use “Please describe the audio.” as the instruction. All embeddings are L2L_2-normalized before computing cosine similarity. A-B Metrics Let q be the negated-query embedding, and let tafft_aff, tparat_para, tswapt_swap, and tunrelt_unrel denote the affirmative, paraphrase, content-swap, and unrelated references, respectively. Polarity sensitivity. We report normalized affinity =s(q,taff)−s(q,tunrel)s(q,tpara)−s(q,tunrel), = s(q,t_aff)-s(q,t_unrel)s(q,t_para)-s(q,t_unrel), (1) polarity/content /content =1−s(q,taff)1−s(q,tswap). = 1-s(q,t_aff)1-s(q,t_swap). (2) A normalized affinity near or above 11 means that the truth-opposite affirmative is as close to the query as its meaning-preserving paraphrase. A polarity/content ratio below 11 means that changing polarity moves the embedding less than replacing one sound concept. Negation scope. For the three scope variants v1,v2,v3\v_1,v_2,v_3\, we compute scope-perm=meani<js(vi,vj).scope-perm=mean_i<j\,s(v_i,v_j). We compare this value with the content-swap similarity s(q,tswap)s(q,t_swap) and the bag-of-concepts similarity s(q,tbag)s(q,t_bag). High scope-permutation and bag-of-concepts similarities indicate weak encoding of which concept is negated. Audio-retrieval bridge. For each item, we define m(q)=s(q,adist)−s(q,atgt).m(q)=s(q,a_dist)-s(q,a_tgt). A positive margin indicates that the query prefers the distractor containing the negated concept. We report the mean margins for the negated and affirmative queries, together with their item-level correlation. A-C Full results TABLE V: Controlled text-side negation diagnosis for the mean over five LAION-CLAP checkpoints and WAVE-7B. Diagnostic CLAP WAVE-7B Polarity sensitivity Normalized affinity to affirmative text 1.07 1.05 Polarity change / content swap 0.30 0.12 Negation scope Scope-permutation similarity 0.70 0.90 Content-swap similarity 0.58 0.70 Bag-of-concepts similarity 0.82 0.91 Audio-retrieval bridge Negated–affirmative margin correlation 0.76 0.94 Negated-query margin +0.018 +0.016 Affirmative-query margin +0.072 +0.022 Table V reveals three consistent patterns across both model families. Negation is only a weak perturbation. The affirmative counterpart is at least as close to the negated query as the meaning-preserving paraphrase, with normalized affinities of 1.071.07 for CLAP and 1.051.05 for WAVE-7B. Polarity changes are also substantially smaller than content swaps, particularly for WAVE-7B, whose polarity/content ratio is only 0.120.12. Thus, changing whether a concept is affirmed or negated affects the embedding much less than changing the concept itself. Negation is weakly bound to the intended concept. For CLAP, the scope-permutation similarity (0.700.70) exceeds the content-swap similarity (0.580.58), while the bag-of-concepts similarity remains high (0.820.82). The pattern is even stronger for WAVE-7B: its scope variants have a similarity of 0.900.90, close to its bag-of-concepts similarity of 0.910.91. These results indicate that the representations preserve which sound concepts are mentioned more strongly than which concept is negated. The text-side failure carries over to audio ranking. Negated and affirmative queries induce strongly correlated audio margins (0.760.76 for CLAP and 0.940.94 for WAVE-7B). Moreover, the negated-query margins remain positive (+0.018+0.018 and +0.016+0.016, respectively), indicating continued preference for the audio containing the forbidden concept. Negated queries therefore behave like attenuated affirmative queries rather than reversing the ranking toward the target audio. Overall takeaway. Across both model families, negation is represented as a weak modifier of mentioned concepts rather than an exclusion operator. This behavior persists from text representations to cross-modal audio ranking and is, on several measures, even more pronounced for WAVE-7B. Appendix B Full λ-Sweep for Training-Free Steering TABLE VI: Retrieval-Neg (R@5) versus steering strength λ on AudioCaps test [32]. Per-model best results are shown in bold, the value used in the main paper (λ=0.2λ=0.2) is underlined, and gray cells denote the no-steering baseline (λ=0λ=0). Model 0 0.1 0.2 0.3 0.5 0.7 1 1.5 2 3 5 8 LAION-CLAP – 630k-audioset-best 60.3 61.5 61.8 60.8 55.4 48.6 36.8 20.9 10.8 4.2 1.9 0.9 – 630k-best 59.1 59.5 58.5 57.3 51.1 43.0 27.0 12.8 6.5 2.4 0.4 0.1 – music-audioset 57.0 57.2 57.3 55.9 50.8 44.8 33.4 18.7 10.4 3.8 2.0 0.8 – music-speech-audioset 57.9 58.4 58.6 59.0 54.4 49.1 39.0 21.7 13.5 6.0 2.8 1.2 – music-speech 59.4 59.8 58.1 56.8 53.1 44.9 28.8 12.8 6.2 2.4 0.6 0.1 M2D-CLAP – M2D-CLAP-2025 63.2 64.9 65.8 66.8 67.1 64.7 58.4 44.6 30.8 13.7 4.5 1.6 – M2D-CLAP-2024 46.1 49.0 50.2 50.9 48.8 41.9 29.7 14.8 7.7 3.6 1.6 0.9 WAVE – WAVE-7B 75.2 75.8 77.1 77.2 76.1 73.4 68.4 53.0 39.0 19.4 4.8 2.3 TABLE VII: Retrieval-Neg (R@5) versus steering strength λ on Clotho evaluation [15]. Per-model best results are shown in bold, the value used in the main paper (λ=0.2λ=0.2) is underlined, and gray cells denote the no-steering baseline (λ=0λ=0). Model 0 0.1 0.2 0.3 0.5 0.7 1 1.5 2 3 5 8 LAION-CLAP – 630k-audioset-best 35.4 35.9 36.7 36.3 34.8 30.0 22.2 11.4 7.1 2.7 1.2 0.9 – 630k-best 37.0 37.9 38.7 38.2 36.2 31.8 24.4 13.8 9.3 3.6 1.5 0.9 – music-audioset 33.8 34.2 35.0 34.6 30.2 27.6 21.9 13.0 7.5 3.6 1.8 1.3 – music-speech-audioset 34.2 35.5 34.8 34.6 32.3 27.7 20.8 11.1 6.0 3.1 1.5 0.7 – music-speech 38.0 36.6 37.5 36.2 34.5 32.0 26.7 16.0 8.2 3.6 1.8 1.1 M2D-CLAP – M2D-CLAP-2025 38.0 40.4 42.3 42.9 43.5 41.8 35.4 25.3 17.4 8.4 3.1 1.5 – M2D-CLAP-2024 19.6 20.9 22.5 22.9 22.8 20.6 14.4 8.8 5.4 2.8 0.8 0.5 WAVE – WAVE-7B 50.0 51.6 52.0 51.9 51.1 49.8 46.5 38.3 28.8 14.4 5.4 2.4 TABLE VIII: MCQ-Neg (Avg) vs. steering strength λ on AudioCaps test [32]. Per-model best results are bold, λ=0.2λ=0.2 is underlined, and the no-steering baseline (λ=0λ=0) is shaded gray. Model 0 0.1 0.2 0.3 0.5 0.7 1 1.5 2 3 5 8 LAION-CLAP – 630k-audioset-best 43.2 44.6 45.0 45.4 44.9 42.6 39.1 36.8 36.1 35.0 34.7 34.5 – 630k-best 41.9 42.7 42.8 42.1 39.9 38.3 36.3 34.7 34.4 33.9 33.9 33.8 – music-audioset 43.3 44.9 44.9 44.7 42.5 40.6 39.3 37.5 36.3 35.0 35.0 34.6 – music-speech-audioset 41.1 42.0 42.2 41.1 41.0 41.2 40.8 38.4 36.9 36.0 35.2 34.9 – music-speech 34.9 36.5 38.4 40.0 39.7 38.6 36.8 34.9 34.4 34.0 33.9 33.7 M2D-CLAP – M2D-CLAP-2025 27.9 29.2 30.2 31.6 34.4 35.9 36.2 34.7 33.9 33.6 33.4 33.4 – M2D-CLAP-2024 30.8 31.3 30.9 31.0 31.7 31.9 31.7 31.4 31.7 31.9 32.6 33.0 WAVE – WAVE-7B 43.5 51.8 54.5 50.8 40.1 34.7 33.6 33.3 33.3 33.3 33.3 33.3 TABLE IX: MCQ-Neg (Avg) vs. steering strength λ on Clotho evaluation [15]. Per-model best results are bold, λ=0.2λ=0.2 is underlined, and the no-steering baseline (λ=0λ=0) is shaded gray. Model 0 0.1 0.2 0.3 0.5 0.7 1 1.5 2 3 5 8 LAION-CLAP – 630k-audioset-best 37.2 37.0 36.6 36.8 37.8 37.8 36.6 35.4 34.8 34.3 34.4 34.2 – 630k-best 38.1 39.2 39.4 39.6 38.8 38.1 36.3 35.0 34.4 34.4 34.0 33.9 – music-audioset 38.4 38.9 37.7 37.1 36.8 36.7 35.5 34.6 34.4 33.9 33.7 33.7 – music-speech-audioset 30.9 31.0 31.1 31.3 32.4 33.0 32.9 32.8 32.8 33.2 33.6 33.5 – music-speech 27.9 29.4 29.5 30.0 32.1 33.1 34.0 34.5 34.0 34.4 34.1 33.7 M2D-CLAP – M2D-CLAP-2025 22.1 22.2 22.9 24.1 26.2 27.2 29.4 31.9 32.3 33.2 33.4 33.5 – M2D-CLAP-2024 23.2 23.6 24.1 24.3 25.5 26.8 28.0 29.6 30.4 31.1 31.6 31.9 WAVE – WAVE-7B 42.2 49.3 52.4 51.2 43.5 36.9 34.1 33.4 33.3 33.3 33.3 33.3 Figure 2: Effect of steering strength λ on Retrieval-Neg R@5 for AudioCaps [32] (top) and Clotho [15] (bottom). Performance generally peaks at a small or moderate λ and declines sharply under stronger steering, consistent with excessive subtraction removing useful query semantics. Figure 3: MCQ-Neg performance across steering strengths λ on AudioCaps [32] (top) and Clotho [15] (bottom). Steering improves local option ranking for several models, but the optimal strength varies across models and datasets. This suggests that inference-time steering can partially modify negation-related decisions, although the gains are not consistently robust. To complement the main results, we report the full λ-sweep of training-free negation steering in Tables VI, VII, VIII and IX, with corresponding trends visualized in Figures 2 and 3. The sweep shows that small steering strengths can improve negation-aware matching, but the gains are limited and model-dependent. In Retrieval-Neg, performance usually peaks at a small or moderate λ and then drops sharply as λ increases, indicating that excessive steering removes not only the negated concept but also useful semantic information from the query. In MCQ-Neg, some models benefit more from larger λ values, especially when the task only requires local ranking among a small set of options. These trends support our main conclusion: training-free steering can partially edit local decisions, but it does not provide a robust solution to negation failure in the global retrieval space. Appendix C On the Use of ALLMs for Verification A potential concern is circularity, as we use an audio-aware LLM (ALLM) to construct a benchmark for diagnosing negation failures in audio-language models. We argue that this design does not compromise the validity of NegEval-Audio for three reasons. First, the verification task is purely affirmative. For each candidate concept, the ALLM is asked, “Does the audio contain the sound of X?” This is a presence-detection task that does not require interpreting a negated query. Negation is introduced only after verification, when the resulting concept sets are used to construct retrieval queries and MCQ options. Thus, the capability being tested by NegEval-Audio is not required during benchmark construction. Second, the verifier and the evaluated models differ in both architecture and inference paradigm. The evaluated models are embedding-based systems that rank audio–text pairs according to similarity in a shared representation space, whereas the verifier is a generative ALLM that produces a binary answer. Although this distinction alone does not guarantee independent failure modes, it reduces the concern that the benchmark construction procedure directly inherits the specific negation-related ranking failures being evaluated. Most importantly, the verification results are independently validated by human annotators. Across 200 audio clips and 460 verified negative concepts, human judgments agree with the ALLM verification results in 96.7% of cases. This high agreement provides direct evidence that the resulting negative concepts are reliable.