Paper deep dive

CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation

Insung Lee, Taeyoung Jeong, Haejun Yoo, Du-Seong Chang, Myoung-Wan Koo

Year: 2026Venue: arXiv preprintArea: cs.SDType: PreprintEmbeddings: 53

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/23/2026, 12:07:30 PM

Summary

The paper introduces CAF-Score, a reference-free metric for evaluating audio captioning models. It calibrates the coarse-grained semantic alignment of CLAP models with the fine-grained syntactic and semantic reasoning capabilities of Large Audio-Language Models (LALMs) using a probabilistic FLEUR-based scoring mechanism. Experiments on the BRACE benchmark demonstrate that CAF-Score achieves superior correlation with human judgments compared to existing reference-free and reference-based metrics.

Entities (5)

BRACE · benchmark · 100%CAF-Score · metric · 100%CLAP · model-architecture · 100%LALM · model-architecture · 100%FLEUR · methodology · 95%

Relation Signals (3)

CAF-Score → evaluatedon → BRACE

confidence 100% · Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation

CAF-Score → calibrates → CLAP

confidence 95% · CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment

CAF-Score → integrates → LALM

confidence 95% · CAF-Score... calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference-based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)-based approaches frequently overlook syntactic errors and fine-grained details. We propose CAF-Score, a reference-free metric that calibrates CLAP's coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperforming reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

52,769 characters extracted from source content.

Expand or collapse full text

CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation Insung Lee, Taeyoung Jeong, HaeJun Yoo, Du-Seong Chang, Myoung-Wan Koo ∗ Department of Artificial Intelligence, Sogang University, South Korea dlstjd6474@sogang.ac.kr, mwkoo@sogang.ac.kr Abstract While Large Audio-Language Models (LALMs) have advanced audio captioning, robust evaluation remains difficult. Reference- based metrics are expensive and often fail to assess acoustic fidelity, while Contrastive Language-Audio Pretraining (CLAP)- based approaches frequently overlook syntactic errors and fine- grained details. We propose CAF-Score, a reference-free met- ric that calibrates CLAP’s coarse-grained semantic alignment with the fine-grained comprehension and syntactic awareness of LALMs. By combining contrastive audio-text embeddings with LALM reasoning, CAF-Score effectively detects syntactic inconsistencies and subtle hallucinations. Experiments on the BRACE benchmark demonstrate that our approach achieves the highest correlation with human judgments, even outperform- ing reference-based baselines in challenging scenarios. These results highlight the efficacy of CAF-Score for reference-free audio captioning evaluation. Code and results are available at https://github.com/inseong00/CAF-Score. Index Terms: audio captioning evaluation, reference-free metric, CLAP, LALM, human preference alignment 1. Introduction Recent advances in Large Audio-Language Models (LALMs) have demonstrated strong capabilities in fine-grained audio un- derstanding and reasoning [1,2,3]. These developments have, in turn, spurred the emergence of specialized audio captioning models [3, 4]. As illustrated in Figure 1 (top left), most existing evaluation metrics rely predominantly on text-to-text comparisons against reference captions. These include traditional n-gram–based met- rics [5,6,7,8] and scene graph-based metrics [9,10], and more recent sentence-embedding-based metrics such as FENSE [11] and LLM-based approaches like CLAIR-A [12]. While X- ACE [13] incorporates audio information, it still necessitates reference captions. Consequently, these metrics face limited scalability due to their reliance on expensive, labor-intensive annotated datasets. To address these limitations, reference-free evaluation meth- ods have emerged, drawing inspiration from the vision do- main (Figure 1, bottom left). While approaches such as CLIP- Score [14] and FLEUR [15] have established effective paradigms for image–text alignment, their extension to audio remains rel- atively underexplored. In the audio domain, Contrastive Lan- guage–Audio Pretraining (CLAP) models [16,17,18,19] have enabled reference-free evaluation by narrowing the modality gap between audio and text [20,21]. However, as summarized in Figure 1 (top right), CLAP-based metrics often fail to cap- ** indicates the corresponding author. Figure 1: Overview of audio captioning evaluation metrics. Traditional metrics (top left) depend on ground-truth reference captions. Although CLAPScore enables reference-free evalua- tion, it lacks fine-grained semantic understanding and frequently overlooks syntactic errors. Inspired by FLEUR, a vision-domain approach, the proposed CAF-Score addresses these limitations by integrating the coarse-grained semantic alignment of CLAP with the fine-grained semantic reasoning and syntactic aware- ness of LALMs, resulting in stronger alignment with human preference judgments. ture syntactic errors and fine-grained semantic distinctions due to their predominantly coarse-grained semantic training objec- tives [22]. To overcome this limitation, we propose the CLAP-aligned FLEUR score (CAF-Score). By calibrating CLAPScore us- ing FLEUR, CAF-Score combines the robust coarse-grained semantic alignment of CLAP models with the syntactic sensitiv- ity and fine-grained semantic reasoning capabilities of LALMs. This hybrid approach yields consistent performance improve- ments over standalone CLAP-based metrics across all evaluated combinations of LALMs and CLAP backbones on the BRACE benchmark, achieving substantially higher correlation with hu- man preference judgments. The contributions of this study are threefold: •First, we extend the FLEUR methodology to the audio domain using LALMs, demonstrating the effectiveness of LALMs for evaluating audio-text alignment. •Second, we introduce the CAF-Score, a novel reference-free metric that leverages LALMs to capture syntactic structure and fine-grained semantic nuances often overlooked by CLAP- based approaches, achieving competitive or superior perfor- mance compared to even reference-based baselines. •Third, we perform extensive experiments across multiple CLAP backbones and state-of-the-art open-source LALMs, offering comprehensive insights into reference-free audio-text arXiv:2603.19615v1 [cs.SD] 20 Mar 2026 The outdoor ambient noises includes birds, wind, and insects. Text Encoder Audio Encoder LALM-based Evaluation (FLEUR) Cosine Similarity Calculation Max Pooling CAF-Score LALM Audio Waveform ( ) Caption ( ) Sliding Windows FLEUR Score FLEUR Calculation Digit Token Probability Distribution CLAP Module Input CLAP-based Acoustic Alignment (S-CLAP) Cosine Similarity 0123456789 Figure 2: Overall architecture of CAF-Score. The framework comprises two parallel branches. The CLAP-based coarse-grained semantic alignment branch applies a sliding-window strategy to the input audio and computes cosine similarity with the candidate caption using CLAP encoders; Max pooling is then used to select the most salient segment score (S-CLAPScore). The LALM-based evaluation branch assesses caption fidelity using an LALM. Rather than relying on discrete text generation, it computes a FLEUR score from token probability distributions to capture fine-grained semantic and syntactic information. The final CAF-Score is obtained as a weighted combination of the two metrics. Notably, our framework operates entirely at inference time, utilizing frozen pre-trained backbones without requiring additional training or fine-tuning. alignment evaluation. 2. Related Works 2.1. Contrastive Audio-Text Alignment To address the need for reference-free evaluation, contrastive learning-based metrics have gained prominence. Building on the success of CLIPScore [14] in the vision domain, CLAP- Score [16] computes cosine similarity between audio and text embeddings within a shared latent space. Several CLAP vari- ants have since been proposed to improve audio-text alignment, including LAION-CLAP [17] for variable-length audio process- ing, MGA-CLAP [18] for fine-grained frame-level alignment, and M2D-CLAP [19] for general-purpose representation learn- ing. Despite their overall robustness, results from the BRACE benchmark [22] indicate that these embedding-based metrics frequently fail to capture fine-grained acoustic characteristics and syntactic errors, largely due to the inherently coarse-grained nature of vector-space alignment. 2.2. Generative Evaluation with LALMs In the vision domain, FLEUR [15] has shown that Large Vision-Language Models (LVLMs) can effectively assess vision- language alignment through smoothed token probabilities. Al- though this generative evaluation paradigm is, in principle, ex- tensible to audio, its applicability and effectiveness in the audio- language domain remain largely unexplored. Robust progress in LALMs has further demonstrated strong capabilities in fine- grained audio understanding and syntactic awareness [1,2,3]. However, when used as standalone evaluators, LALMs are still subject to well-documented limitations, including hallucina- tions [23,22], which undermine their reliability for consistent evaluation. To mitigate these issues, we propose CAF-Score, which uses the robust coarse-grained semantic alignment pro- vided by CLAP as a foundation and employs LALMs as a calibra- tion mechanism to capture fine-grained semantic and syntactic nuances. This hybrid design enables more reliable reference-free audio-text evaluation. 2.3. Benchmarks for Audio Captioning Evaluation Several benchmarks have been proposed for evaluating audio cap- tioning metrics. The FENSE benchmark [11] measures the corre- lation between automatic metric scores and human preferences; however, it was constructed using earlier-generation captioning systems and does not explicitly target hallucinations or fine- grained audio-text alignment. More recent benchmarks, such as Comp-A [24] and AHa-Bench [25], focus on hallucination- related errors but remain largely reference-based and do not directly assess alignment in a reference-free setting. Accord- ingly, we adopt BRACE [22], which is designed for modern LALMs and supports reference-free evaluation of both misalign- ment and hallucination via its two subsets, BRACE-Main and BRACE-Hallucination. 3. Methodology As illustrated in Figure 2, the CAF-Score framework processes the input audio and candidate caption through independent CLAP and LALM components, producing an S-CLAPScore and a FLEUR score, respectively. These outputs are then combined to compute the final CAF-Score. The details of each component are described in the following subsections. 3.1. CLAP-based Alignment (S-CLAPScore) CLAP models comprise a text encoderE t and an audio encoder E a . Given a candidate captionx t and an audio inputx a , the standard CLAPScoreS CLAP is computed as the cosine similarity between their L2-normalized embeddings: S CLAP (a,t) = Norm E t (x t ) ⊤ Norm E a (x a ) (1) where Norm(·) denotes L2-normalization. Standard CLAP models are constrained by a fixed maximum audio input duration. As a result, evaluating variable-length au- dio via naive truncation or random clipping can lead to the loss of critical acoustic information [18]. To alleviate this limitation and better preserve temporal structure, we adopt a sliding-window strategy following the evaluation protocol used in the BRACE benchmark [22]. Although the sliding-window strategy enables full coverage of the audio without truncation, simply averaging the window- level scores—as done in prior work [22]—can result in semantic dilution. Audio captions often refer to brief but salient acoustic events that may occupy only a small portion of the total du- ration [18]. In such cases, average pooling attenuates the high alignment score of the relevant segment by combining it with low scores from unrelated background or silent segments. We there- fore adopt a Max-pooling strategy to identify the best-matching segment within the audio. We denote the resulting metric as S-CLAPScore (Sliding-window CLAPScore)S S-CLAP , defined as: S S-CLAP (x a ,x t ) = N max n=1 S CLAP (x w n ,x t ) (2) wherex w n N n=1 denotes the set of sliding-windows ex- tracted from the audiox a . A hop size of1 sis used for all models. The window length is set to7 sfor MS-CLAP and10 s for other CLAP variants to ensure stable score estimation. Com- parative analyses of this aggregation strategy are presented in Section 5.3.1. 3.2. LALM-based Evaluation (FLEUR) Although LALMs can directly produce discrete numerical scores (e.g., “0.85”), the reliability of using such raw outputs as align- ment measures has not been fully established. In practice, we observed that relying on raw scores often results in frequent ties among candidate captions, hindering meaningful analysis of human preferences and obscuring subtle quality differences. To overcome this limitation, we adopt FLEUR [15], which computes continuous scores from the token-level probability distribution rather than from deterministic numeric outputs. To obtain these scores, we prompt the LALM with a grading task adapted to the audio captioning domain by replacing “image” with “audio” in the original prompt from [15]. The exact prompt template is as follows: Evaluation Prompt Your task is to evaluate and rate the caption on a scale of 0.0 to 1.0 based on the given Grading Criteria. (Print Real Number Score ONLY) Grading Criteria: 0.0: The caption does not describe the audio at all. 1.0: The caption accurately and clearly describes the audio. Caption: pred_caption Score(Choose a rating from 0.0 to 1.0): Rather than using the deterministic output, FLEUR com- putes a continuous score from the probability distribution over digit tokens. Specifically, letp(i,j)denote the probability of digit tokeni∈1,..., 9appearing at thej-th decimal place. The FLEUR score is defined as: FLEUR = 2 X j=1 10 −j 9 X i=1 i· p(i,j)(3) For illustrative purposes, consider a simplified example where the model outputs “0.85”. Instead of taking this value di- rectly, FLEUR examines the token-level probabilities at each dec- imal position. Suppose the first decimal place hasP(‘8’) = 0.7 andP(‘9’) = 0.3, and the second decimal place hasP(‘5’) = 0.6 andP(‘4’) = 0.4, with all other digit probabilities near zero. The FLEUR score then becomes: FLEUR = 0.1× (8× 0.7 + 9× 0.3) + 0.01× (5× 0.6 + 4× 0.4) = 0.1× 8.3 + 0.01× 4.6 = 0.876 (4) In this case, while two captions might both produce “0.85” as raw output, their underlying probability distributions can differ, yielding distinct FLEUR scores (e.g., 0.876 vs. 0.851) and thereby resolving ties. A detailed analysis of the tie phenomenon is presented in Section 5.3.3. 3.3. CAF-Score (CLAP-aligned FLEUR score) CLAP models achieve robust coarse-grained semantic alignment performance by projecting audio and text into a shared embed- ding space and computing cosine similarity [16,17,18,19]. However, due to the inherent limitations of vector-based sim- ilarity measures, these embedding-based approaches struggle to capture fine-grained phenomena, including syntactic errors, causal inconsistencies, and specific acoustic hallucinations. In contrast, LALMs exhibit strong language understanding and rea- soning capabilities that can address these limitations [1,2,3]. Nevertheless, when used as standalone evaluators, LALMs re- main vulnerable to model biases and hallucinations [23, 22]. To bridge this gap, we propose the CAF-Score (CLAP- aligned FLEUR score), which synergistically combines the complementary strengths of both approaches. Specifically, CAF- Score uses S-CLAPScore as a foundation for coarse-grained semantic alignment and applies FLEUR as a calibration mecha- nism to assess fine-grained semantic and syntactic accuracy. The final CAF-Score is computed as a linear combination of the two component scores using a weighting parameter α: S CAF = α· S S-CLAP + (1− α)· FLEUR(5) whereαcontrols the trade-off between coarse-grained se- mantic alignment captured by CLAP and fine-grained semantic understanding and syntactic assessment provided by FLEUR. In this study, we setα = 0.8, placing primary emphasis on the sta- bility of S-CLAPScore, while allowing FLEUR to correct subtle semantic and syntactic errors, thereby achieving the most robust and reliable performance. A sensitivity analysis with respect to α is reported in Section 5.3.2. 4. Experiments 4.1. Dataset We conduct our evaluation using BRACE, a reference-free bench- mark for audio captioning assessment. BRACE comprises two subsets: BRACE-Main, which contains 2,496 caption pairs for fine-grained caption comparison, and BRACE-Hallucination, which includes 2,027 audio clips paired with noun-modified cap- tions for hallucination detection. The audio samples are drawn from the evaluation splits of AudioCaps [26] and Clotho [27]. The audio durations typically range from5 s to 30 s, with Au- dioCaps clips specifically limited to a maximum of10 s, and caption lengths are generally under 20 words. The original BRACE benchmark evaluates LALMs using a pairwise comparison protocol, in which two captions are pre- sented simultaneously and the model selects the better one. While effective for eliciting preferences, pairwise prompting can encourage relative judgments based on superficial cues—such as fluency, length, or lexical priors—rather than on explicit au- dio grounding. This makes it difficult to attribute decisions solely to audio-text alignment. To enable a more grounded and analyzable evaluation, we adopt a single-caption evaluation pro- tocol, in which each caption is scored independently against the corresponding audio by both CLAP and the LALM, yielding S-CLAPScore and FLEUR for each audio-caption pair. This approach allows fine-grained analysis at the individual caption level, while still permitting reconstruction of the original BRACE pairwise preferences by comparing the independently obtained scores. We use the original BRACE data without modification, changing only the evaluation protocol. 4.2. Experimental Setup 4.2.1. Baseline Models To ensure a comprehensive evaluation, we selected representative models spanning diverse architectures and training paradigms. CLAP Models. We used four representative variants to assess audio-text alignment: • MS-CLAP 1 : The 2023 released version. • LAION-CLAP 2 : The‘laion/clap-htsat-unfused’ checkpoint from Hugging Face. •MGA-CLAP 3 : Pre-trained weights downloaded via the offi- cial Google Drive link provided in the repository. •M2D-CLAP 4 : Pre-trained weights provided in the repository. LALMs. We evaluated two prominent families, considering both standard and reasoning variants: •AudioFlamingo3 5 : We evaluated both the Base and Think variants. •Qwen3-Omni: We evaluated both the Instruct 6 and Thinking 7 variants. 4.2.2. Implementation Details All experiments were conducted on a single NVIDIA A100 (80GB) GPU. For the Qwen3-Omni family, we employed the vLLM library on the same hardware to efficiently handle the large model size. In this setup, the GPU memory utilization parameter (‘gpu_memory_utilization’) was set to 0.95 to maxi- mize throughput while maintaining stable inference. Under this configuration, the average inference latency was approximately 1 https://github.com/microsoft/CLAP 2 https://huggingface.co/laion/ clap-htsat-unfused 3 https://github.com/Ming-er/MGA-CLAP 4 https://github.com/nttcslab/m2d 5 https://huggingface.co/nvidia/ audio-flamingo-3-hf 6 https://huggingface.co/Qwen/ Qwen3-Omni-30B-A3B-Instruct 7 https://huggingface.co/Qwen/ Qwen3-Omni-30B-A3B-Thinking 0.28 sper sample. Additionally, to ensure fair and deterministic evaluation across all models, the temperature was set to 0.0 for all LALM generations. 4.2.3. Evaluation Protocol Our experiments were designed to validate each component of the CAF-Score: •Sliding-window Analysis (CLAP): We compared the stan- dard CLAPScore with S-CLAPScore. For CLAPScore, to ensure fair comparison and reproducibility, the input audio was truncated to the model’s maximum supported duration: 7 s for MS-CLAP and 10 s for the other CLAP variants. •Scoring Method Analysis (LALM): To evaluate the robust- ness of FLEUR and the alignment capability of LALMs, we compared the probabilistic FLEUR metric against raw discrete generation scores. This analysis demonstrates the necessity of probabilistic smoothing (using token log-probabilities) for reliable evaluation. •CAF-Score Configuration: For the final CAF-Score cal- culation, we exclusively used the non-reasoning variants of LALMs (AudioFlamingo3-Base, Qwen3-Omni-Instruct), as reasoning variants exhibited lower stability in score genera- tion, as discussed in Section 5.1. 5. Results 5.1. BRACE-Main Quantitative results on the BRACE-Main benchmark are summa- rized in Table 1. All reported scores represent accuracy against human preference judgments in the BRACE benchmark, which is constructed from intensive human annotations of subjective caption quality. Thus, higher accuracy directly reflects stronger correlation with human evaluation. Our analysis of sliding- window strategies is limited to the Clotho-Main subset, as the short duration of AudioCaps clips minimizes the impact of win- dowing. In this subset, S-CLAPScore consistently outperforms the truncated baseline across most backbones, demonstrating that applying a sliding-window strategy followed by pooling yields higher correlation with human judgments than naive truncation. In evaluating LALM-based approaches, we find that the probabilistic FLEUR metric substantially outperforms using raw discrete scores generated by the models. The lower effec- tiveness of raw scores is primarily due to the high frequency of ties—identical scores assigned to different captions—which reduces their discriminative power. A detailed analysis of this “tie” phenomenon is provided in Section 5.3.3. We further investigated the performance differences between reasoning and non-reasoning variants. For AudioFlamingo3, the non-reasoning variant consistently outperformed the reasoning variant across both raw score and FLEUR-based evaluations. In contrast, Qwen3-Omni exhibited a different pattern: the rea- soning variant achieved higher performance with raw scores, accompanied by a lower tie rate (see Section 5.3.3). However, when applying FLEUR with logit-based score smoothing, the non-reasoning variant performed better. These observations indi- cate that non-reasoning variants are more suitable for FLEUR- based evaluation. Accordingly, we used non-reasoning variants (AudioFlamingo3-Base and Qwen3-Omni-Instruct) for the final calculation of CAF-Score. The proposed CAF-Score, which integrates both CLAP and LALM components, achieves the highest performance on both AudioCaps-Main and Clotho-Main subsets. Notably, combining Table 1: Performance comparison on the BRACE benchmark, including BRACE-Main and BRACE-Hallucination subsets. The upper section reports reference-free metrics (standalone CLAP, LALM, and the proposed CAF-Score), while the lower section reports reference- based baselines for comparison. The Overall column represents the weighted average across subsets based on sample counts. AF3 denotes AudioFlamingo3. Within the CAF-Score results, AF3 and Qwen3 correspond to AudioFlamingo3-Base and Qwen3-Omni- Instruct, respectively. The configuration combining Qwen3-Omni-Instruct with M2D-CLAP achieves the highest overall performance. Model (Method) BRACE-MainBRACE-Hallucination AudioCaps-MainClotho-Main OverallAudioCaps-HalluClotho-HalluOverall HHHMMMTotalHHHMMMTotal Reference-free CLAPScore (CLAP-ONLY) MS-CLAP64.0348.7655.0853.7167.0770.9265.9367.9561.4272.9985.2380.61 LAION-CLAP63.3177.3065.4269.7859.2875.6465.6368.6269.15 87.0384.3185.34 MGA-CLAP62.5973.4865.2468.1271.2667.5863.4165.9566.9592.1887.2689.12 M2D-CLAP65.4780.2263.9970.4867.6666.2162.8164.6967.3591.0383.9386.61 S-CLAPScore (CLAP-ONLY) MS-CLAP64.0346.0753.6551.9765.8772.8967.2669.2161.3073.7387.0782.04 LAION-CLAP63.3177.3065.4269.7859.8875.8366.9669.4369.59 87.0384.4585.12 MGA-CLAP62.5973.4865.2468.1272.4668.9662.5266.1767.5192.1888.0889.48 M2D-CLAP65.4780.2263.9970.4870.0669.9463.8566.9168.5991.0384.4786.98 Raw Score (LALM-ONLY) AF3-Base33.8128.0935.1232.2313.7722.4023.2621.7626.5625.2860.8347.41 AF3-Think15.8322.4719.0720.0021.5621.2221.3321.3220.7124.8121.9023.00 Qwen3-Omni-Instruct16.5548.5441.0040.9635.9354.6245.0447.5244.5181.7882.4782.21 Qwen3-Omni-Thinking25.9055.7343.1445.9441.9255.4042.2247.1546.5981.8675.8078.09 FLEUR (LALM-ONLY) AF3-Base53.2452.5866.4959.4863.4768.3761.1964.1762.0293.8395.8695.09 AF3-Think41.7352.5856.5153.1959.2854.8155.1155.1154.4585.8090.1488.50 Qwen3-Omni-Instruct53.2478.6566.3169.5267.6674.6661.6367.2868.3198.4898.0098.18 Qwen3-Omni-Thinking51.8071.4658.8262.8858.6868.9656.7461.5862.1889.7086.9988.01 CAF-Score (Ours, CLAP + LALM) AF3+MS-CLAP62.5948.0960.0755.7264.6773.8768.7470.1763.5488.1692.9891.16 AF3+LAION-CLAP63.3176.6367.3870.4858.6877.0167.8570.1770.3193.2089.9891.20 AF3+MGA-CLAP65.4771.0167.9168.8270.0670.9263.2666.9967.8395.8493.2094.20 AF3+M2D-CLAP64.0370.3467.3868.1267.6671.7163.7067.2167.6395.6992.3893.63 Qwen3+MS-CLAP64.0360.6763.9962.7167.6679.1770.0773.2168.3993.5296.6195.44 Qwen3+LAION-CLAP63.3181.5768.0972.7561.6880.1668.0071.8072.2396.2494.9495.43 Qwen3+MGA-CLAP63.3181.1267.7472.4073.6575.4463.8569.4370.7997.5796.8097.09 Qwen3+M2D-CLAP67.6386.9767.9175.2870.6677.8066.5271.2873.1197.9696.5997.11 Reference-based FENSE [11]61.1584.4967.3873.2856.8984.6864.1570.9872.0496.7696.1896.40 CLAIR-A [12]66.1990.7967.7476.5156.2991.9465.6374.3975.3691.4591.5591.51 CLAP with Qwen3-Omni-Instruct outperforms configurations using AudioFlamingo3, likely due to the superior capabilities of the Qwen3-Omni family, which benefits from a larger parameter count and more extensive pretraining data. Specifically, the M2D-CLAP + Qwen3-Omni-Instruct combination achieves the best performance on AudioCaps-Main and overall, while MS- CLAP + Qwen3-Omni-Instruct attains the highest performance on Clotho-Main. To contextualize these results, we additionally compare CAF-Score against two representative reference-based metrics: FENSE [11], which combines Sentence-BERT embeddings with a grammar error detector, and CLAIR-A [12], which leverages GPT-4o for reasoning-based semantic distance calculation. De- spite operating without reference captions, CAF-Score achieves competitive overall performance against these baselines. No- tably, CAF-Score demonstrates a clear advantage in the more challenging H (Human-Human) scenarios, where both can- didate captions are written by humans and exhibit subtle qual- ity differences. On this subset, CAF-Score outperforms both FENSE and CLAIR-A, suggesting that our hybrid integration of LALM-based semantic reasoning captures fine-grained nu- ances of human preference that reference-based metrics often overlook. 5.2. BRACE-Hallucination As shown in Table 1, consistent with the trends observed in BRACE-Main, S-CLAPScore outperforms CLAPScore, and FLEUR outperforms raw LALM scores in the BRACE- Hallucination. However, in this case, the highest performance is achieved by FLEUR rather than CAF-Score. Compared to BRACE-Main, the overall performance scores in BRACE-Hallucination are generally higher, and a pronounced performance gap is observed between CLAP- and LALM-based methods, with LALMs achieving near-perfect performance. This does not contradict our original hypothesis—that cal- ibrating CLAP’s coarse-grained alignment using LALMs im- Table 2: Comparison of pooling strategies for CAF-Score (using Qwen3-Omni-Instruct) evaluated on Clotho-Main across differ- ent CLAP backbones. The pooling methods are: no pooling (X), average pooling (Avg), and maximum pooling (Max). BackbonePool Clotho-Main (CAF-Score) HHHMMMTotal MS-CLAP X68.86 78.00 67.56 71.65 Avg66.47 78.59 69.48 72.54 Max67.66 79.17 70.07 73.21 LAION-CLAP X61.08 78.59 67.85 71.06 Avg61.68 80.35 67.70 71.72 Max61.68 80.16 68.00 71.80 MGA-CLAP X71.26 74.46 64.74 69.21 Avg70.66 75.83 65.04 69.80 Max73.65 75.44 63.85 69.43 M2D-CLAP X73.05 77.01 66.22 71.13 Avg70.06 77.01 66.81 71.06 Max70.66 77.80 66.52 71.28 proves alignment—but rather indicates that, in this context, the fine-grained capabilities of LALMs dominate. BRACE- Hallucination focuses on hallucinations involving specific ob- jects (e.g., changing a subject from “man” to “woman”) rather than broader context-level errors. Consequently, effective eval- uation in this benchmark requires precise fine-grained compre- hension rather than general contextual understanding. Although CLAP models achieved satisfactory performance in the 80-90 range, the raw scores produced by Qwen3-Omni- Instruct alone were comparable to S-CLAPScore and even sur- passed MS-CLAP. When evaluated using FLEUR, performance improved substantially: AudioFlamingo3-Base achieved 95.09, and Qwen3-Omni-Instruct reached 98.18. These results sup- port our fundamental assumption that LALMs excel at capturing fine-grained audio information. Moreover, although FLEUR outperformed CAF-Score, CAF-Score still surpassed standalone CLAP, highlighting the value of calibrating coarse-grained CLAP alignment with LALM capabilities, particularly in the context of hallucination detection. This trend is even more pronounced when compared against reference-based metrics: CAF-Score achieves 97.11, substan- tially outperforming CLAIR-A (91.51) and FENSE (96.40), demonstrating that the combination of coarse-grained CLAP alignment and fine-grained LALM evaluation is particularly ef- fective at detecting specific acoustic hallucinations. These results collectively demonstrate that CAF-Score offers a scalable and robust alternative to reference-based evaluation, eliminating the dependency on costly ground-truth annotations while maintain- ing—and in critical scenarios surpassing—the alignment with human judgment. 5.3. Metric Design Analysis In this section, we present a detailed component analysis to validate the architectural choices underlying CAF-Score. First, we evaluate the effectiveness of the sliding-window and pooling strategies in achieving robust alignment. Next, we examine the impact of the weighting parameterαto identify the optimal balance between CLAP and LALM contributions. Finally, we highlight the necessity of the probabilistic FLEUR method by analyzing the limitations of raw scores produced by LALMs. 0 (FLEUR) 0.20.50.8 1 (S S−CLAP ) 60 65 70 75 Weighting Parameter α Performance Score MS-CLAP LAION-CLAP MGA-CLAP M2D-CLAP Figure 3: Performance variation of CAF-Score across different weighting parameters α on BRACE-Main. Table 3: Comparison of optimal fixedαversus entropy-based adaptiveα on BRACE-Main. MethodMS-CLAPLAION-CLAPMGA-CLAPM2D-CLAP Best Fixedα0.20.80.50.8 Fixedα70.1172.2471.3273.11 Adaptiveα70.0371.9971.1569.99 5.3.1. Impact of Sliding Windows and Pooling Strategy We evaluated the robustness of our sliding-window approach by comparing it with the standard non-pooling baseline (X), and further analyzed the effectiveness of maximum pooling relative to average pooling. As detailed in Section 4.2.3, this analysis focuses on the Clotho dataset, given the variable length of its audio samples. The results are summarized in Table 2. Consistent with our hypothesis, across most CLAP back- bones, applying a sliding-window strategy with pooling consis- tently outperformed the non-pooling baseline (X). This demon- strates that capturing temporal segments of variable length audio is more effective than processing the entire sequence at once. Regarding the pooling strategy, maximum pooling (Max) proved more robust than average pooling (Avg). Specifically, MS-CLAP, LAION-CLAP, and M2D-CLAP achieved their high- est total CAF-Scores using Max pooling—for example, MS- CLAP improved from 71.65 (X) to 73.21 (Max). Although MGA-CLAP showed a slight preference for Avg pooling in the total score, Max pooling still outperformed the baseline. These findings suggest that selecting the most salient acoustic-semantic features (Max) is more effective for alignment than averaging (Avg), which can dilute critical information during audio cap- tioning evaluation. 5.3.2. Effect of Weighting Parameter α To determine the optimal balance between CLAP and LALM contributions, we varied the weighting parameterα, combining Qwen3-Omni-Instruct with different CLAP backbones. The results of this experiment are shown in Figure 3. We observed that for both LAION-CLAP and M2D-CLAP, performance peaked atα = 0.8. In contrast, MS-CLAP and MGA-CLAP favored a higher contribution from the LALM component, with peak performance atα = 0.2andα = 0.5, re- spectively. Overall, the highest scores were achieved by the com- binations using M2D-CLAP + Qwen3-Omni-Instruct (73.11) Figure 4: Tie rates in raw scores for AudioFlamingo3 (AF3) and Qwen3-Omni models. and LAION-CLAP + Qwen3-Omni-Instruct (72.24). Importantly, these results show that the hybrid approach (0 < α < 1) consistently outperforms either CLAP (α = 1) or LALM (α = 0) alone, with M2D-CLAP + Qwen3-Omni- Instruct at α = 0.8 representing the optimal configuration. We further investigated whether a sample-level adaptive weighting could improve upon the fixedα. Specifically, we definedadaptiveαas the entropy of the digit token probabil- ity distribution used in the FLEUR calculation:adaptiveα = − P i p(i) logp(i) . The rationale is that higher entropy indi- cates greater LALM uncertainty, warranting a larger weight on S-CLAPScore to maintain evaluation stability. However, as shown in Table 3, the optimal fixedαconsistently outperformed the adaptive approach across all CLAP backbones, confirming that a fixed weighting provides more robust performance than sample-level adaptation. 5.3.3. Analysis of LALM Raw Score As shown in Table 1, FLEUR substantially outperforms raw LALM scores. To investigate the underlying cause, we ana- lyzed the tie rate, i.e., the frequency with which a model assigns identical scores to a pair of captions. The primary reason for these frequent ties is that raw LALM outputs tend to collapse onto a small set of discrete values. In our experiments, AudioFlamingo3 predicted both captions as “0.0” in 54.5% of tie cases and “1.0” in 29.1%, while Qwen3-Omni predicted “0” for 46.0% and “1” for 43.5% of ties. This lack of discriminative power in raw scores is a documented issue in LLM-based evaluation [12,15], which FLEUR effectively mitigates by leveraging the underlying token-level probability distribution. Figure 4 further confirms that high tie rates are prevalent across all models when using raw scores. For AudioFlamingo3, tie rates exceeded 50% in most cases, with variation between reasoning and non-reasoning variants across subsets. Similarly, the reasoning variant of Qwen3-Omni exhibited a lower tie rate than the instruction variant, but it remained insufficient for fine- grained ranking. These observations confirm that, independent of the relative performance of reasoning versus non-reasoning variants, applying FLEUR is the more reliable approach for score smoothing. We also note that the prompts used were adapted directly from vision tasks; exploring audio-specific prompt engi- neering represents a promising direction for future research. 6. Qualitative Analysis We present qualitative examples of CAF-Score to illustrate its strengths and limitations. Figure 5: LALM-driven Correction on TIKTOK_1.wav. Figure6:CLAP-drivenStabilityonCafeteria Ambience.wav. 6.1. Success Cases CAF-Score effectively combines the strengths of both models to mitigate their individual limitations. We categorize these success cases into two complementary scenarios. LALM-driven Correction: In Figure 5, S-CLAPScore assigns a higher score to an unpreferred caption than to the preferred one, due to superficial keyword overlap between “Clock” and the acoustic event “ticks”. CAF-Score, however, leverages the fine-grained understanding of the LALM to detect this mismatch, adjusting the final score to align with human preference. CLAP-driven Stability: Conversely, in Figure 6, both caption candidates are linguistically valid, causing the LALM to show weak and ambiguous discrimination. In this case, CAF-Score relies on the robust coarse-grained alignment of S-CLAPScore to correctly identify the more accurate caption. Together, these examples highlight the complementary nature of our hybrid ap- proach: the LALM compensates for CLAP’s shallow matching, while CLAP stabilizes the LALM’s uncertain judgments. 6.2. Failure Cases Since the performance of CAF-Score is inherently bounded by the capabilities of the underlying models, we also analyzed in- stances where CAF-Score failed to align with human preference. We identify three failure types, each stemming from a different mode of model dependency. Double Failure: As shown in Figure 7, when both CLAP and the LALM are simultaneously misaligned with human preference, Figure 7: Double Failure on lama2.wav. Figure 8: Insufficient Calibration on 105166.wav. CAF-Score is unable to rectify the prediction. This typically occurs when the audio contains ambiguous sounds that mislead both models at the same time. Insufficient Calibration: In some cases, CLAP favors the in- correct caption while the LALM correctly prefers the human- selected one. However, as illustrated in Figure 8, the LALM assigned extremely high and nearly identical scores to both candi- dates, producing an insufficient margin to overcome the weighted contribution of the incorrect CLAPScore. Over-calibration: Conversely, CLAP may correctly identify the preferred caption, but the LALM makes an erroneous judgment due to hallucination or hypersensitivity to syntax. As shown in Figure 9, FLEUR exhibited an excessive preference for the non- preferred candidate, effectively overriding the correct alignment from CLAP and leading to a wrong prediction. 7. Discussion In this section, we discuss the implications of our findings and the inherent limitations of the proposed framework. Complementarity of CLAP and LALMs: Our results consis- tently show that the hybrid approach (0<α< 1) outperforms either CLAP or LALM alone, confirming that coarse-grained contrastive alignment and fine-grained generative evaluation capture fundamentally different aspects of audio-text correspon- dence. As illustrated in Section 6, CLAP provides stable acoustic grounding when the LALM struggles with linguistically similar captions, while the LALM corrects CLAP’s superficial keyword- Figure 9: Over-calibration on 108925.wav. matching errors. This complementarity suggests that future reference-free metrics may benefit from integrating contrastive and generative paradigms more broadly. Failure Modes and Their Implications:Despite these strengths, CAF-Score inherits the limitations of its underlying components. When both CLAP and the LALM simultaneously misalign with human preference—as observed in the “double failure” cases—the metric cannot recover. Furthermore, the fixed weighting parameterα = 0.8may not be universally optimal across all audio-caption pairs. As discussed in Section 5.3.2, entropy-based adaptive weighting did not outperform the fixed configuration; however, exploring alternative adaptive strategies remains a promising direction for future work. Computational Considerations: Incorporating LALMs in- troduces substantial computational overhead compared to lightweight embedding-based models such as CLAP. Conse- quently, CAF-Score presents a trade-off between evaluation ac- curacy and computational efficiency. However, with the con- tinued advances in efficient inference techniques, this gap is expected to narrow in future iterations. Generalizability: Our analysis relies exclusively on the BRACE benchmark, which, while designed for modern LALMs, may not fully represent the diversity of audio captioning scenarios en- countered in practice. Expanding evaluations to additional bench- marks and diverse comparison methods would further strengthen confidence in CAF-Score. We hope that this work serves as a foundation to encourage the development of more diverse evalu- ation resources for reference-free audio captioning. 8. Conclusion We proposed the CAF-Score, a novel reference-free metric for audio captioning evaluation that combines the coarse-grained semantic alignment of CLAP with the fine-grained semantic comprehension and syntactic awareness of LALMs. Methodologically, we demonstrated that employing a sliding-window strategy with maximum pooling substantially improves alignment compared to standard truncation approaches. Additionally, we successfully adapted the FLEUR metric to the audio domain, confirming its effectiveness in capturing linguis- tic nuances. Through extensive experiments, we showed that CAF-Score achieves superior alignment with human preference compared to existing metrics, without relying on ground-truth references. Furthermore, CAF-Score demonstrates competitive or superior performance against reference-based baselines, par- ticularly in challenging human-human comparison and hallucina- Table 4: Full performance comparison on TTA benchmarks including RELATE and PAM datasets. We report Pearson (LCC), Spearman (SRCC), and Kendall’s Tau (KTAU) correlations with human judgments. Model RELATEPAM LCCSRCCKTAULCCSRCCKTAU RELATE [28]0.3850.3830.265--- RELATE w/ CB [28]0.3770.3740.259--- CLAPScore (CLAP-ONLY) MS-CLAP0.2560.2240.1530.3450.3120.215 LAION-CLAP0.4040.3860.2680.4610.4520.314 MGA-CLAP0.4460.4110.286 0.5320.5220.368 M2D-CLAP0.4920.4570.3220.5660.5480.388 AQA-Score [29] (LALM-ONLY) Qwen-2.5-Omni-3B0.4430.4530.3270.5400.5600.410 Qwen-2.5-Omni-7B0.5440.5560.3960.5180.5890.429 AF3-base0.4750.5080.3570.4960.5380.383 AF3-think0.4350.4740.330 0.5820.5870.419 Raw Score (LALM-ONLY) Qwen-2.5-Omni-3B0.3370.3410.2720.3760.3640.287 Qwen-2.5-Omni-7B0.3280.3160.2620.3050.2990.302 AF3-Base0.1200.1110.092 0.1260.0890.132 AF3-Think0.1490.1450.116-0.070-0.062-0.049 Qwen3-Omni-Instruct0.3170.3480.2820.4200.3840.299 Qwen3-Omni-Thinking0.2890.2790.2180.3690.3620.276 FLEUR (LALM-ONLY) Qwen-2.5-Omni-3B0.3630.4080.2870.3720.4110.288 Qwen-2.5-Omni-7B0.3990.4070.2860.4180.4290.302 AF3-Base0.1510.2160.1480.1520.1940.132 AF3-Think0.1630.1630.111-0.025-0.042-0.029 Qwen3-Omni-Instruct0.3960.4510.3160.4580.4260.297 Qwen3-Omni-Think0.2920.2460.1710.3670.3510.242 Ours (CAF-Score, CLAP + LALM) Qwen2.5-3B + MS0.3820.3510.2430.4850.4510.316 Qwen2.5-3B + LAION0.4690.4550.3200.5380.5260.372 Qwen2.5-3B + MGA0.5040.4770.3340.5590.5450.389 Qwen2.5-3B + M2D0.5040.4820.3400.5560.5430.388 Qwen2.5-7B + MS0.3480.3120.2160.4370.4000.278 Qwen2.5-7B + LAION0.4550.4350.3040.5190.5060.356 Qwen2.5-7B + MGA0.4940.4590.3210.5700.5510.392 Qwen2.5-7B + M2D0.5400.5170.3670.6090.5980.430 AF3 + MS0.2700.2340.1600.3620.3240.223 AF3 + LAION0.4130.3930.2730.4720.4590.319 AF3 + MGA0.4520.4160.2890.5340.5220.368 AF3 + M2D0.4910.4640.3270.5440.5430.383 Qwen3 + MS0.3720.3290.2270.5210.4830.340 Qwen3 + LAION0.4660.4390.3060.5820.5720.407 Qwen3 + MGA0.5000.4600.3210.5960.5790.412 Qwen3 + M2D0.5260.5200.3670.6080.6060.436 tion detection scenarios. These results indicate that CAF-Score provides a robust and scalable evaluation framework for the rapidly evolving field of audio captioning. 9. Generative AI Use Disclosure GitHub Copilot was utilized as an AI-powered code assistant to aid in writing and editing the experimental scripts. Additionally, Gemini was employed for polishing the manuscript to improve linguistic clarity and grammatical accuracy. In accordance with the ISCA policy, the authors have reviewed and edited all AI- generated content and remain fully responsible and accountable for the entire work and content of this paper. 10. Extension to Text-to-Audio Evaluation While the original submission was primarily focused on audio captioning due to space constraints, we extend our analysis in this version to demonstrate the generalizability of CAF-Score to the Text-to-Audio (TTA) generation task. This expansion aims to verify if CAF-Score can effectively evaluate the alignment be- tween user-provided text prompts and generated audio samples. To this end, we conducted experiments using the RELATE [28] and PAM [30] benchmarks, which provide gold-standard hu- man subjective ratings for TTA generation quality. Furthermore, we included a comparative analysis with AQA-Score [29], a recent evaluation framework that also leverages LALMs for TTA assessment through question-based prompting. To ensure a comprehensive and fair comparison across different model architectures, we additionally performed experiments using the Qwen-2.5-Omni [31] family as backbones for our framework. • Experimental Setup: Following the methodology established in Section 3, we utilized the CLAPScore as the foundation for coarse-grained alignment. Given that TTA samples in these benchmarks typically range around 10 seconds, the impact of sliding windows was found to be minimal. To adapt the eval- uation for the TTA task, we modified the LALM evaluation prompt by simply inverting the perspective between the audio and the caption as follows: TTA Evaluation Prompt Your task is to evaluate and rate the audio on a scale of 0.0 to 1.0 based on the given Grading Criteria. (Print Real Number Score ONLY) Grading Criteria: 0.0: The audio does not describe the caption at all. 1.0: The audio accurately and clearly describes the caption. Caption: caption Score (Choose a rating from 0.0 to 1.0): •Analysis of RELATE: On the RELATE dataset, we observed that AQA-Score remains a strong baseline. However, our CAF-Score consistently improves upon standalone LALM or CLAP metrics, reinforcing the hypothesis that LALM-based calibration effectively refines coarse-grained semantic align- ment. •Analysis of PAM: In the PAM benchmark, CAF-Score con- figurations utilizing the Qwen family—including both Qwen3 and Qwen-2.5 [31] variants—combined with M2D-CLAP achieved superior performance. Across various model scales, CAF-Score consistently improves upon standalone LALM or CLAP metrics. These results further reinforce our hypothesis. •Discussion on Generalizability: The experimental results, summarized in Table 4, demonstrate that CAF-Score serves as a robust and versatile evaluation framework. By bridg- ing the gap between coarse-grained acoustic grounding from CLAP and fine-grained semantic reasoning from LALMs, CAF-Score generalizes effectively across diverse audio-text alignment tasks beyond captioning, including TTA genera- tion. These findings suggest that the hybrid integration of contrastive and generative paradigms provides a more reli- able reference-free assessment that aligns closely with human subjective preferences in various audio intelligence scenarios. 11. References [1]G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabili- ties,” arXiv preprint arXiv:2507.06261, 2025. [2]A. Goel, S. Ghosh, J. Kim, S. Kumar, Z. Kong, S.-g. Lee, C.- H. H. Yang, R. Duraiswami, D. Manocha, R. Valle et al., “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” arXiv preprint arXiv:2507.08128, 2025. [3]J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, Y. Lv, Y. Wang, D. Guo, H. Wang, L. Ma, P. Zhang, X. Zhang, H. Hao, Z. Guo, B. Yang, B. Zhang, Z. Ma, X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, and M. Huang, “Qwen3-omni technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2509.17765 [4]H. Dinkel, G. Li, J. Liu, J. Luan, Y. Niu, X. Sun, T. Wang, Q. Xiao, J. Zhang, and J. Zhou, “Midashenglm: Efficient audio understand- ing with general audio captions,” arXiv preprint arXiv:2508.03983, 2025. [5]K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, p. 311–318. [6]C.-Y. Lin, “Rouge: A package for automatic evaluation of sum- maries,” in Text summarization branches out, 2004, p. 74–81. [7]S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic eval- uation measures for machine translation and/or summarization, 2005, p. 65–72. [8]R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, p. 4566–4575. [9]P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in European conference on computer vision. Springer, 2016, p. 382–398. [10]S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in Proceedings of the IEEE international conference on computer vision, 2017, p. 873–881. [11]Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, “Can au- dio captions be evaluated with image caption metrics?” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, p. 981–985. [12]T.-H. Wu, J. E. Gonzalez, T. Darrell, and D. M. Chan, “Clair-a: Leveraging large language models to judge audio captions,” arXiv preprint arXiv:2409.12962, 2024. [13]Q. Wang, J.-C. Gu, and Z.-H. Ling, “X-ace: Explainable and multi-factor audio captioning evaluation,” in Findings of the As- sociation for Computational Linguistics: ACL 2024, 2024, p. 12 273–12 287. [14]J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi, “Clip- score: A reference-free evaluation metric for image captioning,” in Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, p. 7514–7528. [15]Y. Lee, I. Park, and M. Kang, “Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model,” arXiv preprint arXiv:2406.06004, 2024. [16] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, p. 1–5. [17] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dub- nov, “Large-scale contrastive language-audio pretraining with fea- ture fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, p. 1–5. [18]Y. Li, Z. Guo, X. Wang, and H. Liu, “Advancing multi-grained alignment for contrastive language-audio pre-training,” in Proceed- ings of the 32nd ACM International Conference on Multimedia, 2024, p. 7356–7365. [19]D. Niizumi, D. Takeuchi, M. Yasuda, B. T. Nguyen, Y. Ohishi, and N. Harada, “M2d-clap: Exploring general-purpose audio-language representations beyond clap,” IEEE Access, 2025. [20]H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXiv preprint arXiv:2301.12503, 2023. [21] H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley, “Audioldm 2: Learn- ing holistic audio generation with self-supervised pretraining,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 32, p. 2871–2883, 2024. [22] T. Guo, H. Chen, H. Liang, M. Qiang, B. Zeng, L. Sun, B. CUI, and W. Zhang, “BRACE: A benchmark for robust audio caption quality evaluation,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [Online]. Available:https: //openreview.net/forum?id=MdEkxa6PV9 [23] T.-w. Hsu, K.-H. Lu, C.-H. Chiang, and H.-y. Lee, “Reducing object hallucination in large audio-language models via audio- aware decoding,” arXiv preprint arXiv:2506.07233, 2025. [24]S. Ghosh, A. Seth, S. Kumar, U. Tyagi, C. K. Evuru, S. Ra- maneswaran, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “Compa: Addressing the gap in compositional reasoning in audio- language models,” arXiv preprint arXiv:2310.08753, 2023. [25]X. Cheng, D. Fu, C. Wen, S. Yu, Z. Wang, S. Ji, S. Arora, T. Jin, S. Watanabe, and Z. Zhao, “AHa-bench: Benchmarking audio hallucinations in large audio-language models,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [Online]. Available: https://openreview.net/forum?id=vCej5sO61x [26]C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds.Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, p. 119–132. [Online]. Available: https://aclanthology.org/N19-1011/ [27] K. Drossos, S. Lipping, and T. Virtanen, “Clotho:An audio captioning dataset,” 2019. [Online]. Available: https: //arxiv.org/abs/1910.09387 [28]Y. Kanamori, Y. Okamoto, T. Takano, S. Takamichi, Y. Saito, and H. Saruwatari, “Relate: Subjective evaluation dataset for auto- matic evaluation of relevance between text and audio,” in Proc. Interspeech 2025, 2025, p. 3155–3159. [29]C.-Y. Kuan, K.-W. Chang, and H.-y. Lee, “Aqascore: Evaluating semantic alignment in text-to-audio generation via audio question answering,” arXiv preprint arXiv:2601.14728, 2026. [30]S. Deshmukh, D. Alharthi, B. Elizalde, H. Gamper, M. Al Ismail, R. Singh, B. Raj, and H. Wang, “Pam: Prompting audio-language models for audio quality assessment,” in Proc. Interspeech 2024, 2024, p. 3320–3324. [31]X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, and M. Huang, “Qwen2.5-omni technical report,” arXiv preprint arXiv:2503.20215, 2025. [Online]. Available: https://arxiv.org/abs/2503.20215

Instant research discovery