Paper deep dive

Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification

Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh, Ahmed Rashad

Year: 2026Venue: arXiv preprintArea: cs.SDType: PreprintEmbeddings: 63

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We benchmark eleven audio classification methods: five task-aware closed-set LLMs (four Gemini models plus open-weight Kimi-Audio-7B-Instruct), four fixed-vocabulary taggers (YAMNet, PANNs, Whisper-AT, and SSLAM), a zero-shot audio-text model (CLAP), and an audio-grounded LLM (BAT). We evaluate them on a closed-set sound-source identification task over 2,242 clips spanning 23 fine-grained classes and 11 categories. Since these methods differ fundamentally in how they receive the task and how outputs are scored, we group them into four evaluation tiers rather than one leaderboard, reporting macro Precision, Recall, F1, and false-negative rate per tier. The best model, Gemini-3.1-Pro-Preview, reaches 85.6 percent category-level F1 and 56.7 percent fine-grained F1. Kimi-Audio is competitive for its size, reaching 67.5 percent category-level F1 and 32.9 percent fine-grained F1, but fails to answer 1.6 percent of samples. SSLAM and CLAP match or exceed the best closed-set model at the category level without seeing the candidate list, but fall behind at the fine-grained level. Analyzing the Gemini models' chain-of-thought across 8,968 responses, we find that response length does not predict accuracy, an apparent "holistic judgment beats detailed analysis" effect is better explained as a difficulty confound, and wrong answers are stated confidently 92 to 100 percent of the time. We report full per-class confusion matrices and metrics for all eleven methods, identify the structural error modes behind most of the accuracy loss between granularities, and give practical guidance for choosing among these method families.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

61,218 characters extracted from source content.

Expand or collapse full text

August 4, 2026 Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification Sajjad Abdoli 1,*,† 1 ,Ghassan Al-Sumaidaee 1,*,† 2 ,Ahmad ElShiekh 1,* 3 ,Ahmed Rashad 1 1 Perle * Equal contribution; names sorted alphabetically. † Corresponding authors sajjad@perle.aighassan.al-sumaidaee@perle.ai mad.elshiekh@perle.aiahmed@perle.ai We benchmark eleven audio classification methods — five task-aware closed-set LLMs (four Gemini models plus open-weight Kimi-Audio-7B-Instruct), four fixed-vocabulary taggers (YAMNet, PANNs, Whisper-AT, SSLAM), a zero-shot audio-text model (CLAP), and an audio-grounded LLM (BAT) — on a closed-set sound-source identification task over 2,242 clips spanning 23 fine-grained classes (11 categories). Since these methods differ fundamentally in how they receive the task and how outputs are scored, we group them into four evaluation tiers rather than one leaderboard, reporting macro Precision, Recall, F1, and false-negative rate per tier. The best model (Gemini-3.1-Pro-Preview) reaches 85.6% category-level and 56.7% fine-grained F1; Kimi-Audio is competitive for its size (67.5%/32.9%) but fails to answer 1.6% of samples. SSLAM and CLAP match or exceed the best closed-set model at the category level without seeing the candidate list, but fall behind at the fine-grained level. Analyzing the Gemini models’ chain-of-thought across 8,968 responses, we find response length does not predict accuracy, an apparent ”holistic judgment beats detailed analysis” effect is better explained as a diﬀiculty confound, and wrong answers are stated confidently 92–100% of the time. We report full per-class confusion matrices and metrics for all eleven methods, identify the structural error modes behind most of the accuracy loss between granularities, and give practical guidance for choosing among these method families. Keywords:environmental sound classification; audio-language models; large language model reasoning; closed-set audio classification; benchmark evaluation methodology 1.Introduction Identifying the source of an everyday sound — a doorbell, a siren, a running tap — is a task humans perform almost effortlessly, but it has historically required purpose-built audio classifiers [1] trained on fixed label vocabularies drawn from ontologies such as AudioSet [6]. The recent emergence of general-purpose multimodal large language models (LLMs) with native audio input, alongside a new generation of audio- grounded LLMs [14,8] and self-supervised audio encoders [2], raises a practical question for anyone building a product on top of audio understanding: given a fixed set of sounds we care about, which class of method should we use, and how much can we trust its answer? This is harder to answer cleanly than it first appears, because these method families are not interchangeable in how they are evaluated. A multimodal LLM can be handed an explicit list of candidate answers and asked to pick one; a fixed-vocabulary audio tagger has no notion of a candidate list at all and must be scored by mapping its native output vocabulary onto the target taxonomy after the fact; a zero-shot audio-text 1 Corresponding author:sajjad@perle.ai 2 Corresponding author:ghassan.al-sumaidaee@perle.ai; ORCID:0000-0002-5536-0252 3 ORCID:0009-0001-6837-6202; current contact:ahmadelshiekh9@gmail.com © 2026 Perle. All rights reserved.1 arXiv:2608.02397v1 [cs.SD] 3 Aug 2026 August 4, 2026 similarity model is given natural-language descriptions of the candidates and ranks them by embedding similarity; and an audio-grounded LLM fine-tuned primarily for open-ended dialogue may produce free text that has to be graded by a second model. Comparing raw accuracy numbers across these four setups risks conflating ”this method understood the sound better” with ”this method’s task was easier.” We address this by running all eleven methods on the identical 2,242-sample, 23-class balanced dataset, and by treating the four evaluation paradigms as distinct tiers rather than collapsing them into one leaderboard. We report standard per-class detection metrics (macro Precision, Recall/TPR, F1, FNR) rather than raw accuracy, since these are the conventional reporting choice in the environmental sound classification (ESC) literature [13] and remain well-defined even for the one method (BAT) for which a full multiclass confusion matrix cannot be constructed. We further exploit the fact that the four LLMs in our study produce free-text chain-of-thought reasoning, and run a secondary LLM-based classification pass over all 8,968 saved responses to characterizehowthese models reason their way to an answer — not just whether they are right. The main contributions of this paper are: •A four-tier evaluation framework for comparing task-aware closed-set selection, task-agnostic fixed- vocabulary tagging, zero-shot audio-text similarity, and open-vocabulary LLM-judged scoring on the same underlying data, with an explicit accounting of why cross-tier comparisons must be read directionally rather than as a strict ranking. •A full benchmark of eleven methods across four families on a 2,242-sample, 23-class (11-category) closed- set sound-source identification task, with complete per-class Precision/Recall/F1/TPR/FNR metrics and confusion matrices for every method at both granularities. •An analysis of 8,968 LLM chain-of-thought traces showing that (a) response length does not predict fine-grained accuracy, (b) an apparent strategy-accuracy correlation is confounded by which classes elicit which strategy, and (c) none of the four LLMs reliably signals uncertainty when wrong, with 92–100% of incorrect answers phrased with confident language. 2.Background and Related Work Environmental sound classification (ESC) is typically framed as a fixed-vocabulary multiclass or multil- abel problem: a model is trained (or evaluated) against a closed ontology such as the 527-class AudioSet ontology [ 6] or the 50-class ESC-50 dataset [13]. Classical approaches to this problem are convolutional or transformer audio taggers trained end-to-end on labeled audio, of which YAMNet [9,10] and PANNs [12] are widely used pretrained baselines. More recent work adapts speech-recognition backbones for tagging (Whisper-AT [ 7]) or introduces self-supervised pretraining objectives specifically designed for overlapping, polyphonic soundscapes (SSLAM [ 2]). A parallel line of work moves away from fixed vocabularies entirely. Contrastive audio-text models such as CLAP [ 3] embed audio and natural-language descriptions into a shared space, enabling zero-shot classification against arbitrary text prompts rather than a fixed label set. Most recently, general-purpose multimodal LLMs with native audio understanding (e.g., the Gemini family [5]) and audio-grounded instruction-tuned LLMs such as LTU [ 8], BAT [14], and Kimi-Audio [11] reframe audio classification as a language generation or selection task, in principle allowing free-form reasoning before an answer is produced. Kimi-Audio in particular is an open-weight 7B audio-native LLM (an audio tokenizer, a Qwen2.5-based transformer core, and an audio detokenizer for optional speech output), distinguishing it architecturally from both the much larger proprietary Gemini models and BAT’s LoRA-adapted encoder-plus-frozen-LLM design. All three of Kimi-Audio, BAT, and (almost certainly) Gemini share a common architectural commit- ment despite their differences: a separately pretrained audio or speechencoderconverts the raw signal into a compressed representation before the LLM ever sees it. Concurrent work by Fan et al. [ 4] (Mel-LLM) questions whether this encoder is necessary at all, removing it entirely and projecting lightly-processed Mel-spectrogram patches directly into the LLM via a single linear layer — the speech-domain analogue of encoder-free vision-language modeling. Their central finding is a trade-off directly relevant to sound-source identification: the encoder-free model shows the largest gains on tasks where the answer is carried by low-level Sound Source Identification Benchmark2 August 4, 2026 acoustic structure rather than lexical content (including a 50-class environmental sound classification bench- mark, ESC-50 [13], where it improves substantially over an otherwise-comparable encoder-based baseline), while remaining weaker on knowledge-intensive spoken question answering. This suggests that how much an audio-LLM’s front end compresses toward semantic abstraction, versus preserving raw acoustic detail, is itself a variable worth manipulating for tasks like ours — a possibility we return to in Section6.4. These four families — fixed-vocabulary taggers, zero-shot audio-text models, general-purpose multimodal LLMs, and audio-grounded instruction-tuned LLMs — differ in a way that matters for benchmarking: a fixed-vocabulary tagger has no mechanism for accepting an externally-defined candidate list at inference time, while an LLM has no native scoring mechanism over a fixed label set without first being given one as part of the prompt. Comparing across families therefore risks conflating a model’s audio understanding with how much task information it was given. This is why our study deliberately keeps the underlying audio, ground truth, and taxonomy identical across all eleven methods (Section4), but treats the resulting comparison as tiered by task-awareness and scoring mechanism rather than flat, making the source of any given accuracy gap between methods (audio understanding versus task framing) explicit rather than implicit. 2.1.Other General-Purpose Multimodal LLMs Considered Beyond the four Gemini models and Kimi-Audio, Tier A of this study (Section4) excludes several other general-purpose multimodal LLMs we investigated, because we could not confirm they support native audio input at the time of this study: •Anthropic’s Claude modelsdo not accept audio as an input modality at all (text and image only, per Anthropic’s own model overview 4 — “All current Claude models support text and image input”), so no Claude model could be included in Tier A. •OpenAI’s frontier reasoning models(the GPT-5.x family) likewise do not accept audio input: Ope- nAI’s own model documentation lists audio as “Not supported” for GPT-5 5 . OpenAI’s dedicated audio models (gpt-audio,gpt-audio-mini) do accept audio, per OpenAI’s audio-model guide 6 , and were smoke-tested on a small sample of this benchmark, but were not scaled to the full 2,242-sample run reported here and are therefore omitted from Tier A. •Meta’s Muse Spark, released via the Meta Model API, is advertised as a multimodal reasoning model, but Meta’s own developer documentation 7 lists only text, image, video, and PDF as supported input modalities — audio is not listed. We tested this directly by attempting to upload one of our benchmark audio clips to Meta’s consumer-facing assistant atmeta.ai, which rejected the file outright with “Unsup- ported file format. Please upload an image.” With the primary documentation not listing audio support and a direct upload attempt rejecting audio files entirely, we did not include Muse Spark in this study. •Moonshot AI’s Kimi K3(the general-purpose chat model, distinct from Kimi-Audio) likewise does not accept audio: its oﬀicial documentation overview 8 and K3 quickstart guide 9 list only text, image, and video as supported inputs, with no audio modality documented anywhere in the platform’s indexed documentation. We note this explicitly because both models share a developer and a name prefix; only Kimi-Audio, a separate, dedicated open-weight audio foundation model [11], was included in this study (Tier A). We flag this explicitly because it is a real constraint on how far the Tier A comparison generalizes: it spans four Gemini model versions and one open-weight audio-native model, not the full landscape of general-purpose multimodal LLMs, and should be read accordingly. 4 https://platform.claude.com/docs/en/about-claude/models/overview 5 https://developers.openai.com/api/docs/models/gpt-5 6 https://developers.openai.com/api/docs/guides/audio 7 https://ai.developer.meta.com/docs/getting-started/models 8 https://platform.kimi.ai/docs/overview 9 https://platform.kimi.ai/docs/guide/kimi-k3-quickstart Sound Source Identification Benchmark3 August 4, 2026 3.Data 3.1.Collection and Class Structure The benchmark dataset consists of 2,242 short audio clips drawn from a curated, balanced subset of everyday sound recordings. Each clip is labeled with asound_type(broad category) and asub_category (fine-grained sub-type); the unique combination of the two fields defines 23 fine-grained classes, which collapse into 11 broader categories. Table1lists both levels; the full 23-class list with its lettered candidate mapping is given in AppendixA. 3.2.Dataset Summary The dataset is balanced at approximately 100 samples per fine-grained class, with one exception (Siren - Other, 42 samples) reflecting natural data availability rather than a sampling choice. Table 1:Dataset structure. All 2,242 samples were attempted by all eleven methods. PropertyValue Total samples2,242 Category-level classes (sound_type)11 Fine-grained classes (sound_type+sub_category)23 Samples per fine-grained class∼100 (42–104) Methods evaluated11 Samples evaluated per method2,242 (100% coverage; 0 unresolved errors for ten methods, 36/2,242 unresolved for Kimi-Audio — Section4) 4.Methodology 4.1.Study Design Every method was evaluated on the identical 2,242-sample set and scored against the identical ground truth at both the category and fine-grained level. What differs across methods ishoweach one is given the task and how its raw output is converted into a predicted class. We group the eleven methods into four evaluation tiers along these two axes. 4.1.1. Tier A — Task-aware closed-set, exact match.Four Gemini models (gemini-2.5-flash, gemini-2.5-pro,gemini-3.5-flash,gemini-3.1-pro-preview) were each given the audio clip together with a fixed, lettered list of all 23 candidate labels (e.g.,A. Appliance Alarm - Appliance Alarm, ...,W. Yell - Yell), an explicit instruction to reason freely (chain-of-thought), and a requirement to commit to exactly one letter as the final answer (full prompt text in Appendix B.1). Scoring is exact string match between the predicted and true label at each granularity. We additionally testedKimi-Audio-7B-Instruct[ 11], an open-weight audio-native LLM, with the iden- tical prompt format. Unlike BAT (Tier D), Kimi-Audio is instruction-tuned broadly enough to follow the closed-set letter-selection task, making it a genuine Tier A member rather than a forced fallback to open- vocabulary scoring. In practice its compliance with the ”share your reasoning” instruction was inconsistent: the large majority of responses were a bare letter with no reasoning text at all, a minority included full chain-of-thought reasoning ending in the requested JSON answer, and 36/2,242 samples (1.6%) produced no scoreable answer whatsoever — 32 repetition-loop generations that never concluded (the model restates a description in a loop until hitting the generation-length limit) and 4 explicit refusals (e.g., “none of the candidates match this sound”; one instance, “I cannot analyze the sound as there is no audio provided,” Sound Source Identification Benchmark4 August 4, 2026 hallucinates theabsenceof input that was in fact given — the mirror image of the Gemini grounding failure in Section5.5that hallucinates thepresenceof input never given, and then answers anyway rather than abstaining). These 36 samples are excluded from Kimi-Audio’s reported metrics at both granularities, iden- tically to how an audio-file error would be excluded for any other method; unlike BAT’s exclusions, these are not a scoring design choice but genuine non-answers we could not parse a letter from even after two rounds of extraction-logic refinement (JSON, LaTeX-boxed, bare-letter, natural-language, and verbatim-candidate- echo matching) and a same-settings retry that reproduced the identical failures, confirming they are not transient. 4.1.2. Tier B — Task-agnostic fixed-vocabulary, post-hoc mapped.Four models (YAMNet [9], PANNs [12], Whisper-AT [7], SSLAM [2]) were run with no knowledge of our 23-class taxonomy at all. Each produces a ranked list of scores over its native∼521–527-class AudioSet vocabulary. We restrict this ranking to the subset of AudioSet labels with a known mapping into our taxonomy (taggedexactorapproxi- matematch quality by hand) and take the top-ranked mapped label as the prediction. Because these models never see the candidate set, their effective task is harder — ranking over the full AudioSet ontology — than the 23-way selection given to Tier A. 4.1.3. Tier C — Zero-shot audio-text similarity.Microsoft CLAP [3] is given the same 23 candidates as Tier A, but phrased as natural-language descriptions (e.g.,”a police car siren”) rather than lettered options (full candidate phrasings in Appendix B.2), and produces a prediction via cosine similarity between the audio embedding and each candidate’s text embedding, without any generative reasoning step. CLAP is never separately queried against an 11-candidate category-level set: as with Tier A, both granularities are scored from this single 23-way ranking — Fine-Grained accuracy compares the full top-ranked candidate against the true label, and Category-Level accuracy compares only itssound_typecomponent against the true category, rather than from an independently-elicited coarser prediction. 4.1.4. Tier D — Open-vocabulary generation with LLM-judged scoring.BAT [14] (a Spatial-AST audio encoder coupled with a LoRA-adapted LLaMA-2-7B via the SLAM-LLM framework) was run in its native open-vocabulary free-text question-answering mode (exact question text in AppendixB.3), since a direct experiment giving BAT the same lettered candidate list as Tier A failed: BAT’s LoRA fine-tuning produces a strong bias toward binary yes/no answers regardless of instruction, and could not be induced to select a letter. BAT’s free-text answers were instead scored by an independent LLM judge (gemini-2.5- flash-lite), prompted to determine separately whether the answer correctly identifies the category-level and fine-grained sound source, allowing for synonyms and paraphrases (full judge prompt in Appendix B.4). 4.2.Evaluation Metrics For every method and every sample, we treat each of theCclasses (C= 11at the category level,C= 23 at the fine-grained level) as a one-vs-rest detection problem and compute, per classc: true positives TP c , false positives FP c , false negatives FN c , and true negatives TN c . From these we derive Precision c = TP c TP c +FP c ,Recall c (TPR c ) = TP c TP c +FN c ,(1) F1 c = 2·Precision c ·Recall c Precision c +Recall c ,FNR c = 1−Recall c = FN c TP c +FN c .(2) We report the unweighted (macro) average of each quantity across allCclasses as the headline number for a method at a given granularity. We use macro-averaged F1, TPR, and FNR rather than raw accuracy as our primary reported metrics, for two reasons. First, this is the conventional reporting choice in the ESC literature [ 13,12], where per-class detection rate and false-negative rate are more informative than a single pooled accuracy figure when class diﬀiculty is uneven (as it clearly is here — see Section 5). Second, macro Recall/TPR and FNR remain well-defined for Tier D (BAT), where no discrete predicted class exists (the output is free text judged correct/incorrect per sample), so no full confusion matrix — and therefore no Sound Source Identification Benchmark5 August 4, 2026 Precision or F1 — can be constructed; reporting only what is actually measurable for each tier, rather than forcing every method into the same metric, is itself part of the tiered-comparison discipline in Section5. For completeness and comparability with the accompanying project write-up, raw accuracy for every method is also given in AppendixD. 4.2.1. Match-quality tagging for Tier B.Tier B methods never see our 23-class taxonomy; each one tags audio using its own native AudioSet vocabulary (∼521–527 classes), and a hand-built lookup table translates the relevant AudioSet labels into our(sound_type, sub_category)pairs. Each entry in that lookup table is taggedexactorapproximate, describing the quality of thetranslation itself— not the model’s confidence, and not how close a wrong guess was. •exact: AudioSet has a label that specifically corresponds to our sub-category. For example, the true label Siren - Policematched against YAMNet’s raw tag"Police car (siren)", which the lookup table maps to(Siren, Police, exact). AudioSet has a genuinely specific concept here, so a hit reflects the model actually detecting that particular sound. •approximate: AudioSet has no label specific enough to express our sub-category, only a broader, generic one, and we chose to map that generic bucket onto the closest available sub-category when building the lookup table. For example, the true labelAppliance Alarm - Appliance Alarmmatched against YAMNet’s raw tag"Alarm", which the lookup table maps to(Appliance Alarm, Appliance Alarm, approximate). AudioSet’s"Alarm"label is a single bucket that also covers car alarms, smoke alarms, and alarm clocks; the model is not telling us “this is specifically an appliance,” only “this sounds like some kind of alarm,” and our table is the one deciding that generic signal counts as a match for this particular sub-category. Critically, this tag hasno effect on scoring. Whether a mapped prediction isexactorapproximate, it is scored identically to every other prediction in this study: does the translated label string equal the true label string? A match is full credit regardless of tag; a non-match is a miss regardless of tag — there is no partial credit forapproximatematches and no penalty either.match_qualityis recorded purely as a diagnostic column for post-hoc error analysis (e.g., quantifying how many of a method’s correct answers reflect a genuinely specific detection versus a generic AudioSet bucket defaulting onto the right sub-category), not as an input to any reported metric. The complete lookup table, with everyapproximateentry and the reason it could not be taggedexact, is given in Appendix C. For four sub-category distinctions (clap count, doorbell type, knock surface, water-running distance), Au- dioSet’s ontology has no label capable of expressing the distinction at all, only a category-level one (e.g. a generic "Doorbell" label with no digital/mechanical/wireless variants). When a Tier B method’s top-ranked mapped prediction resolves to one of these category-only entries, it is scored as an ordinary Fine-Grained miss against whichever true sub-category the sample actually had — the same treatment any other incorrect prediction receives, with no special exclusion or partial credit. This keeps Tier B’s Fine-Grained metric on the same footing as Tier A and Tier C rather than granting it credit for distinctions its underlying vocabulary cannot express. 4.2.2. Reasoning-strategy analysis (Gemini models only).This secondary analysis covers only the four Gemini models, not Kimi-Audio: it requires a substantial, consistently-produced chain-of-thought to classify, and (as noted above) the large majority of Kimi-Audio’s responses are a bare letter with no reasoning text at all, making it a poor candidate for this specific pass. We saved the full reasoning trace for all2,242×4 = 8,968 Gemini responses and passed each one through a second LLM classification pass (gemini-2.5-flash-lite) that labeled its primary reasoning strategy (detailed acoustic analysis,holistic pattern match,process of elimination,direct assertion, ornon-grounded) and its apparent confidence level (high,medium,low, or hedging), based purely on the language used — not a self-reported score, since the original prompt never asked for one. We combined this with simple lexical statistics (word count, hedge-term and acoustic-term frequency) computed directly from the text. Sound Source Identification Benchmark6 August 4, 2026 5.Results 5.1.Category-Level Results (11 classes) Table2reports macro Precision, Recall (TPR), F1, and FNR for all eleven methods at the category level, grouped by evaluation tier. Table 2:Category-level (11-class) macro-averaged results, by evaluation tier.N=2,242for every method except Kimi- Audio (N=2,206; 36 samples produced no scoreable answer — see Section4, marked‡). BAT (Tier D) has no discrete predicted class, so Precision and F1 are not computable (†); only Recall/TPR (fraction of each true class the judge scored correct) and FNR are reported. TierMethodPrecisionRecall (TPR)F1FNR gemini-3.1-pro- preview 86.4%87.4%85.6%12.6% gemini-2.5-pro79.0%78.3%77.0%21.7% gemini-3.5-flash78.1%78.9%76.7%21.1% kimi-audio-7b- instruct ‡ 71.0%70.3%67.5%29.7% A gemini-2.5-flash69.5%66.3%65.7%33.7% B sslam 87.9% 87.8% 87.3% 12.2% panns84.6%83.9%83.2%16.1% whisper-at78.5%76.8%76.2%23.2% yamnet62.2%53.4%52.4%46.6% Cclap89.1%87.3%87.8%12.7% Dbatn/a † 57.7%n/a † 42.3% 5.2.Fine-Grained Results (23 classes) Table3reports the same metrics at the fine-grained level, where the model must distinguish the exact sub-category (e.g.,Siren - Policevs.Siren - Ambulance), not just the broad category. Sound Source Identification Benchmark7 August 4, 2026 Table 3:Fine-grained (23-class) macro-averaged results, by evaluation tier.N=2,242for every method except Kimi- Audio (N=2,206, marked‡; see Table2). Row order within Tier A matches Table2(category-level F1 rank) for easy cross-reference. TierMethodPrecisionRecall (TPR)F1FNR gemini-3.1-pro- preview 60.7%60.8%56.7%39.2% gemini-2.5-pro51.2%53.3%50.1%46.7% gemini-3.5-flash57.4%52.2%47.9%47.8% kimi-audio-7b- instruct ‡ 38.7%39.6%32.9%60.4% A gemini-2.5-flash38.9%41.2%37.0%58.8% B sslam40.3%39.6%38.0%60.4% panns37.7%37.0%35.9%63.0% whisper-at33.6%31.4%30.7%68.6% yamnet27.8%26.1%23.7%73.9% Cclap61.7%54.2%50.6%45.8% Dbatn/a † 35.4%n/a † 64.6% Two patterns stand out immediately. First, within Tier A,gemini-3.1-pro-previewleads at both gran- ularities by a comfortable margin (8+ F1 points at the category level over the next-best Gemini model); kimi-audio-7b-instruct, the one open-weight model in Tier A, lands fourth of five at both the category level — ahead ofgemini-2.5-flashdespite being a much smaller, openly-released model, but last of five at the fine-grained level, behind all four Gemini models. Second, the category-to-fine-grained F1 drop is universal and large: every method loses 25–50 F1 points moving from 11-way to 23-way discrimination, and the ranking of methods shifts substantially between the two tables (e.g. CLAP’s fine-grained F1 of 50.6% is only 6.1 points behind the best Gemini model, despite CLAP never performing any generative reasoning). 5.3.Where the Category-to-Fine-Grained Gap Comes From The gap is not diffuse noise; it concentrates in a small number of structurally hard distinctions, visible consistently across methods and confirmed in the per-class confusion matrices in Appendix D: •Water Running - Distant Sourceis essentially unsolvablefor all five Tier A models (F1≈0 throughout, including Kimi-Audio’s F1=0.02), with 57–83% of these samples predicted asWater Run- ning - Close Sourceacross all five (Kimi-Audio: 65%). Models correctly detect running water; they cannot judge recording distance from the audio alone. •Doorbell - Wirelessis similarly unsolvablefor four of five Tier A models (all exceptgemini- 2.5-flash), with errors split betweenDoorbell - MechanicalandDoorbell - Digital, and 11–35% spilling intoAppliance Alarm— a wireless doorbell chime is acoustically closer to an electronic alarm tone than to a mechanical door-knocker. Kimi-Audio shows the strongest version of this particular con- fusion (35% of itsDoorbell - Wirelesspredictions land onAppliance Alarm, versus 11–19% for the four Gemini models). •Siren sub-types show a model-family default bias.All five Tier A models struggle to separate Ambulance/Fire/Police/Other, butgemini-3.1-pro-previewandgemini-2.5-prodefault toward Policeunder uncertainty, whilegemini-3.5-flashandgemini-2.5-flashdefault towardAmbulance — the same underlying acoustic confusion, resolved by a different learned prior per model family. Kimi- Audio joins the Ambulance-defaulting group but far more extremely: 98% of itsSiren - Policesamples and 86% of itsSiren - Firesamples are predicted asSiren - Ambulance, effectively collapsing all siren sub-types into a single answer rather than showing a softer bias. Sound Source Identification Benchmark8 August 4, 2026 Figure 1:Category-level confusion matrix (row-normalized recall) forgemini-3.1-pro-preview, the best-performing Tier A model. The full 23-class fine-grained matrix, where the confusions above are visible in detail, is given in AppendixD (Figure6). 5.4.Top-K Accuracy (Tier B) Tables2and3score every method on its single best answer (Top-1). Tier B methods are unique among the four tiers in that they naturally produce a fullrankedlist of candidates rather than a single forced choice (Tier A and Tier C commit to one answer; Tier D produces free text with no ranking at all), so we additionally ask: does the true label appearanywherein the model’s top-Kranked, taxonomy-mapped predictions, for K∈1,3,5?K=1reproduces the raw accuracy already reported;K=5is the deepest ranking collected at data-preparation time, so it is the most lenient, best-case figure available from this data. Sound Source Identification Benchmark9 August 4, 2026 Table 4:Top-1/3/5 accuracy for all four Tier B methods, both granularities.N=2,242throughout. MethodK=1K=3K=5 Category-Level (11 classes) sslam88.4%95.9%96.9% panns84.3%94.0%96.5% whisper-at77.8%90.3%93.7% yamnet52.1%64.4%66.4% Fine-Grained (23 classes) sslam40.1%50.3%54.2% panns37.3%48.8%53.5% whisper-at31.7%44.5%50.8% yamnet26.5%35.1%40.7% The gap betweenK=1andK=5is large and consistent: every Tier B method gains roughly 14–19 Fine- Grained points byK=5(e.g. SSLAM: 40.1%→54.2%; Whisper-AT: 31.7%→50.8%), indicating the correct answer is frequently present in these models’ ranked output even when their single top choice misses. Most of that gain arrives byK=3; going fromK=3toK=5adds a further 4–6 points, a diminishing but still real return. AtK=5, SSLAM’s Fine-Grained accuracy (54.2%) approaches CLAP’s Top-1 figure (54.5%, Table 12), is on par with the strongest non-preview Tier A models (gemini-2.5-pro, 54.1%) and trails onlygemini-3.1-pro-preview(61.5%) - though this is not an apples-to-apples comparision:K=5credits a method for having the right answer anywhere in five guesses, a substantially easier bar than the single forced answer every other tier is held to. For the detailed per-class view, we focus theK=5(best-case) confusion matrices in Appendix D, alongside the standard Top-1 matrices, sinceK=5is the most informative depth for characterizing what each Tier B method’s ranked output is actually capable of resolving. 5.5.Reasoning-Strategy Analysis (Gemini Models) 5.5.1. Verbosity does not predict accuracy.Table5and Figure2show average chain-of-thought response length by model. The most accurate model,gemini-3.1-pro-preview, is also the most concise — nearly 7×shorter thangemini-2.5-pro, which writes exhaustive, structured, multi-step analyses (87.6% of its responses use explicit numbered steps, vs. 0% forgemini-3.1-pro-preview) without converting that extra length into better fine-grained F1 than the far tersergemini-3.5-flash(Table 3). Response length and accuracy are, if anything, inversely related across these four models. Table 5:Average chain-of-thought response length by model. ModelAvg. words / response gemini-3.1-pro-preview52 gemini-3.5-flash67 gemini-2.5-flash269 gemini-2.5-pro360 Sound Source Identification Benchmark10 August 4, 2026 Figure 2:Average chain-of-thought response length by model (same data as Table5). 5.5.2. An apparent strategy-accuracy pattern is a diﬀiculty confound.Each response was tagged with a primary reasoning strategy. Figure 3shows the distribution:detailed acoustic analysis(explicit discussion of frequency, timbre, envelope, pitch) dominates for every model, from 78.1% (gemini-2.5-flash) up to 95.2% (gemini-3.5-flash) of responses.Holistic pattern match(a quick ”this sounds like X” judgment with no feature breakdown) is the clear second-most-common strategy for all four models, but its share varies considerably:gemini-3.1-pro-previewuses it in 16.5% of responses, three to eight times more often than the other three models (1.9–5.5%). The remaining strategies (process of elimination,direct assertion,non- grounded) are minor and inconsistent across models, appearing in at most 13.9% of any one model’s responses. Naively cross-referencing strategy against fine-grained accuracy (Table 6, visualized in Figure4) shows the same pattern in all four models: responses taggedholistic pattern matchare correct 67.7–79.1% of the time, versus only 39.7–59.1% fordetailed acoustic analysis— a 14.3–28.0 point gap in every model, with no exceptions. Sound Source Identification Benchmark11 August 4, 2026 Figure 3:Primary reasoning strategy distribution by model.Detailed acoustic analysisdominates throughout (78.1– 95.2%);holistic pattern matchis a distant second everywhere, but far more common forgemini-3.1-pro-preview (16.5%) than the other three models (1.9–5.5%). Table 6:Fine-grained accuracy by primary reasoning strategy, for the two dominant strategies, per model.nis the number of responses tagged with that strategy (out of 2,242 per model); the holistic-pattern-match bar is taller than the detailed-acoustic-analysis bar in every model. ModelDetailed-analysis acc. (n) Holistic-match acc. (n)Gap (points) gemini-3.1-pro-preview59.1% (n=1,871)73.4% (n=369)14.3 gemini-2.5-pro53.5% (n=2,122)79.1% (n=43)25.6 gemini-3.5-flash52.4% (n=2,135)78.8% (n=85)26.4 gemini-2.5-flash39.7% (n=1,750)67.7% (n=124)28.0 Sound Source Identification Benchmark12 August 4, 2026 Figure 4:Fine-grained accuracy conditioned on primary reasoning strategy, per model (exact values in Table6). In every model the holistic-pattern-match bar is taller than the detailed-acoustic-analysis bar. Read at face value, this looks like ”less analysis produces better answers.” We do not believe that is the correct interpretation. Restricting the comparison togemini-3.1-pro-preview— the model with by far the highest holistic-match rate (Figure3) — and conditioning on individual classes (Table7) shows that the two strategies score about the samewithina given class (e.g., 100.0% vs. 98.9% onYell); the apparent strategy effect disappears once class diﬀiculty is held fixed. Table 7:Fine-grained accuracy by reasoning strategy, within class, forgemini-3.1-pro-preview. These three classes account for 47% of allholistic pattern matchresponses from this model. ClassOverall accuracyDetailed-analysis acc. (n) Holistic-match acc. (n) Yell - Yell99.0%100.0% (n=9)98.9% (n=89) Appliance Alarm93.0%90.7% (n=54)95.7% (n=46) Water Running - Close Source 80.0%78.7% (n=61)82.1% (n=39) The more plausible explanation, consistent with Table7: models default to a quick holistic judgment precisely when a sound is unambiguous, and reach for detailed acoustic analysis when it is genuinely hard to place. The reasoning strategy is asymptomof per-sample diﬀiculty, not a cause of the outcome — but it remains a useful signal: a model defaulting to detailed analysis is implicitly flagging ”this one was hard.” 5.5.3. Confidence calibration is essentially absent.We measured how often a model’s language reads as confident even when its final answer is wrong (Table 8, Figure5). Confidence level is inferred purely from phrasing (e.g., ”clearly,” ”definitely” vs. ”possibly,” ”might be”) by the second-pass classifier — models were never asked to self-report a confidence score. Sound Source Identification Benchmark13 August 4, 2026 Table 8:Share ofwrongfine-grained answers still stated with high-confidence language. ModelWrong answers stated with high-confidence language gemini-3.5-flash100.0% gemini-3.1-pro-preview99.9% gemini-2.5-pro97.8% gemini-2.5-flash92.1% Figure 5:Share of wrong answers still stated with high-confidence language, by model (same data as Table8). As Table8and Figure5show, all four models are essentially uniform on this measure — the range spans only 92.1% to 100.0%, with no model showing meaningfully better calibration than the others. Hedging language (”possibly,” ”might,” ”uncertain”) appears less than once per response on average for every model. There is effectively no verbal signal in these chain-of-thought traces that reliably distinguishes a correct answer from an incorrect one; a system that needs to know when to defer to a human cannot obtain that signal from a model’s stated confidence on this task. 6.Discussion 6.1.What Can and Cannot Be Compared Directly The four-tier structure in Section 4exists because a single flat ranking across all eleven methods would overstate precision. The defensible comparisons arewithina tier: which Tier A method is best (four Gemini models plus Kimi-Audio, all given the identical closed-set prompt), and which fixed-vocabulary tagger is best (Tier B). Tier C (CLAP) and Tier D (BAT) are each a category of one in this study and are reported for directional context rather than as ranked competitors. That said, one cross-tier observation is robust enough to be worth surfacing directionally: at the category level, task-agnostic methods that never saw the candidate list (SSLAM, PANNs in Tier B; CLAP in Tier C) match or exceed the best task-aware Tier A model’s F1, and PANNs approches it. At the fine-grained level, this reverses — the best Tier A model and CLAP (Tier C) pull ahead of every Tier B tagger by 12+ F1 Sound Source Identification Benchmark14 August 4, 2026 points. A plausible reading is that coarse categorization is largely an acoustic pattern-matching problem, at which purpose-built audio encoders (with or without a candidate list) are already strong, while fine-grained sub-type disambiguation increasingly draws on the kind of contextual, elimination-based reasoning that Tier A’s language-generation step provides and Tier B’s fixed classification head does not. We stress this is a hypothesis suggested by the data, not a controlled causal claim, since Tier A and Tier B differ in task framing as well as in architecture. Even within Tier A, the five methods are not architecturally uniform: the four Gemini models are large proprietary systems of undisclosed scale, while Kimi-Audio-7B-Instruct is a 7B-parameter open-weight model. That it lands mid-pack in Tier A rather than at the bottom is itself informative — task-aware closed-set prompting appears to narrow, though not eliminate, the gap between an open 7B audio-native model and much larger proprietary multimodal systems on this task — but its lower reasoning-instruction compliance (Section4) means its Tier A ranking reflects both classification ability and instruction- following, in a way the four Gemini models’ more consistent compliance does not. 6.2.Practical Guidance For a use case that only needs the broad category (is this a siren or a doorbell?), several method families are viable, and a task-agnostic tagger (SSLAM, PANNs) or a zero-shot model (CLAP) may be suﬀicient without needing an LLM call per sample. For a use case needing the exact sub-type, the strongest current methods top out around 51–57% F1, and Tier A (task-aware LLM with an explicit candidate list) or Tier C (CLAP) are the strongest available options; expect specific, predictable failure modes — recording-distance and directionality judgments, and acoustically adjacent sub-types within a category — rather than random error. If an open-weight, self-hostable option is a requirement, Kimi-Audio-7B-Instruct is a usable Tier A alternative to the Gemini models, at a cost in fine-grained F1 (32.9% vs. 56.7% for the best Gemini model) and in reliability (1.6% of samples yield no scoreable answer at all, versus 0% for the Gemini models), both worth budgeting for in a production deployment. 6.3.Limitations •Tier C’s task framing partially overlaps Tier A’s.CLAP is given the same 23 candidates as the Gemini models (as text), which is part of why its numbers are more directly comparable to Tier A than Tier B’s are; this is noted explicitly rather than treated as a hidden asymmetry, but it means CLAP’s strong fine-grained score should not be read as evidence that zero-shot similarity generally rivals LLM reasoning on unconstrained audio tagging. •Tier D’s scoring depends on a judge model.BAT’s free-text answers are graded bygemini-2.5- flash-literather than exact match; judge leniency or strictness is a source of variance this study does not independently calibrate against a second judge or human raters. •Tier B’s mapping is post-hoc.The AudioSet-to-taxonomy mapping used for YAMNet, PANNs, Whisper-AT, and SSLAM is hand-constructed and tagged exact/approximate at data-preparation time; a different, equally reasonable mapping could shift these numbers somewhat, though the relative ordering within Tier B was stable under manual review. •BAT’s spatial-audio design is not exercised here.BAT’s Spatial-AST encoder expects binaural input with directional cues; our clips are effectively mono, so the channel was duplicated rather than fabricating synthetic spatial cues. This is an honest ”no directional information” representation but means BAT is evaluated outside its primary designed use case. •Kimi-Audio’s 36 excluded samples are a genuine reliability difference, not a scoring artifact. We confirmed this by re-attempting extraction with three successive rounds of parser improvements and by re-running the 36 samples under identical settings, which reproduced the same failures rather than resolving them (Kimi-Audio’s text decoding is greedy, so a same-settings retry has limited ability to escape a repetition-loop or refusal outcome). We did not pursue a targeted fix (e.g., raising the text repetition penalty, currently uninformatively set to 1.0) specifically for these samples, since doing so would apply different generation settings to a subset of Kimi-Audio’s data than to the rest, undermining the like-for-like comparison this study is built around. Sound Source Identification Benchmark15 August 4, 2026 •Single dataset, single run.All numbers reflect one balanced 23-class dataset and a single evaluation pass per method (no repeated sampling or multiple-seed variance estimates), and the 11-category taxon- omy reflects the specific sound sources of interest to this study rather than a general-purpose acoustic scene ontology. 6.4.Future Work Natural extensions include: a second, independent LLM judge (or a small human-rated subset) to quantify Tier D’s judge-variance; testing whether few-shot prompting can induce BAT toward closed-set letter selection rather than its default binary yes/no bias; and extending the tiered framework to acoustic scenes with genuine spatial/directional structure, which would let BAT’s native architecture be evaluated on-task rather than degraded to mono input. A further direction motivated by concurrent work (Section 2) is to manipulate the audio front end itself as an independent variable within Tier A. All Tier A methods in this study are encoder-based to some degree — Kimi-Audio explicitly tokenizes audio through a pretrained Whisper-derived encoder before its LLM core ever sees it, and Gemini’s proprietary audio pathway is almost certainly similar in spirit. Fan et al. [ 4] report that removing this encoder entirely (projecting Mel-spectrogram patches directly into the LLM) improves performance specifically on tasks carried by low-level acoustic structure, including environmental sound classification, at some cost to knowledge-intensive tasks. Our own fine-grained results are dominated by exactly this kind of acoustic discrimination — distinguishingSiren - PolicefromSiren - Ambulance, or a wireless doorbell chime from a mechanical one, plausibly turns on preserved pitch contour and timbre rather than semantic content. Whether an encoder-free architecture narrows Kimi-Audio’s fine-grained gap to the Gemini models specifically on these acoustically-driven distinctions, or whether Gemini’s advantage instead reflects sheer model scale or training data, is a testable question our tiered framework is directly set up to answer, given access to a Mel-LLM-style checkpoint evaluated under the same closed-set prompt. 7.Conclusion Comparing eleven audio classification methods spanning four fundamentally different evaluation paradigms on the identical 2,242-sample, 23-class dataset shows that no single method dominates once category and fine- grained granularity, and task framing, are both accounted for. Task-aware closed-set LLM selection (gemini- 3.1-pro-preview) gives the best fine-grained F1 (56.7%) of any method tested, but task-agnostic fixed- vocabulary taggers and zero-shot audio-text similarity are competitive or superior at the coarser category level despite never seeing the candidate list. The open-weight Kimi-Audio-7B-Instruct, evaluated under the identical closed-set prompt, reaches a competitive 67.5%/32.9% category/fine- grained F1 for its size, though with a measurably higher rate of unscoreable responses (1.6% of samples) than the four Gemini models (0%), reflecting less consistent instruction-following rather than a scoring artifact. The category-to-fine-grained accuracy gap is concentrated in a small number of structurally hard, consistent failure modes rather than diffuse noise, and these generalize to Kimi-Audio, in some cases (Siren sub-types) more extremely than any Gemini model. Separately, analyzing the Gemini models’ own chain-of-thought reveals that more reasoning text does not buy more accuracy, that an apparent reasoning-strategy effect on accuracy is confounded by per-sample diﬀiculty, and — most consequential for downstream use — that these models give almost no usable signal, in their stated confidence, of when they are wrong. Ethics Statement This study evaluates model performance on a curated set of short environmental sound recordings (e.g., doorbells, sirens, appliance sounds) and involves no human subjects, personal data, or sensitive content. Sound Source Identification Benchmark16 August 4, 2026 References [1]Sajjad Abdoli, Patrick Cardinal, and Alessandro Lameiras Koerich. End-to-end environmental sound classification using a 1d convolutional neural network.Expert Systems with Applications, 136:252–263, 2019. [2]Tony Alex, Sara Atito, Armin Mustafa, Muhammad Awais, and Philip J. B. Jackson. Sslam: Enhancing self-supervised models with audio mixtures for polyphonic soundscapes. InInternational Conference on Learning Representations (ICLR), 2025. [3]Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general- purpose audio representations. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 336–340, 2024. [4]Ruchao Fan, Yiming Wang, Yuxuan Hu, Bo Ren, Yufei Xia, Xiaofei Wang, Yao Qian, Shujie Liu, and Jinyu Li. Llm can read spectrogram: Encoder-free speech-language modeling, 2026. [5]Google Gemini Team, Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. Gemini 2.5: Push- ing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. [6]Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780, 2017. [7]Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. InProc. Interspeech 2023, pages 2798– 2802, 2023. [8]Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. InInternational Conference on Learning Representations (ICLR), 2024. [9]Google Research. Yamnet.https://github.com/tensorflow/models/tree/master/research/au dioset/yamnet, 2020. Pretrained audio event classification model released in the TensorFlow Models repository, trained on AudioSet using the architecture of Hershey et al. [10]. [10]Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. Cnn architectures for large-scale audio classification. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 131–135, 2017. [11]KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, et al. Kimi-audio technical report, 2025. [12]Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020. [13]Karol J. Piczak. Esc: Dataset for environmental sound classification. InProceedings of the 23rd ACM International Conference on Multimedia, pages 1015–1018, 2015. [14]Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learning to reason about spatial sounds with large language models. InProceedings of the 41st International Conference on Machine Learning (ICML), volume 235 ofProceedings of Machine Learning Research, pages 61454–61469, 2024. Sound Source Identification Benchmark17 August 4, 2026 A.The 23 Fine-Grained Candidate Classes Table9lists the exact 23 lettered candidates given to every Tier A (Gemini) model for every sample, and to Tier C (CLAP) as natural-language phrasings of the same 23 classes. These collapse into the 11 broader Category-Level classes: Appliance Alarm, Cat Sounds, Clap, Dog Sounds, Doorbell, Hairdryer, Kettle Boiling, Knock, Siren, Water Running, Yell. Table 9:The 23 fine-grained candidate classes. Lettersound_typesub_category AAppliance AlarmAppliance Alarm BCat SoundsMeow CCat SoundsPurr DClapDouble Clap EClapSingle Clap FClapTriple Clap GDog SoundsGrowl HDog SoundsHowl IDoorbellDigital JDoorbellMechanical KDoorbellWireless LHairdryerHair Dryer MKettle BoilingBoiling NKettle BoilingWhistling OKnockDoor Knock PKnockSurface Knock QSirenAmbulance RSirenFire SSirenOther TSirenPolice UWater RunningClose Source VWater RunningDistant Source WYellYell B.Exact Prompts Used This appendix gives the verbatim text prompts used for every method that consumes a text input (Tier A, Tier C, and Tier D; Tier B methods receive no text prompt at all and are omitted). All prompts are reproduced exactly as sent, with no paraphrasing. B.1.Tier A: Candidate-Selection Prompt (Gemini and Kimi-Audio) The identical prompt below was used for all four Gemini models and for Kimi-Audio-7B-Instruct (Section4). candidate_listis substituted with the 23-line lettered list given in AppendixA. Listen carefully to the audio file provided. Tell me what you hear. You are free to use any signal processing capability or internal sound classification system you may have access to. Tell me what approach you used, and share your full chain of thought. Choose the single candidate below that best identifies the true source of the sound. You MUST pick exactly one letter from this list (use the exact letter): candidate_list After your analysis, provide your final answer on the very last line in this exact JSON format (no markdown, no extra text after it): Sound Source Identification Benchmark18 August 4, 2026 "answer": "X" Where X is the single capital letter of your chosen candidate. B.2.Tier C: CLAP Candidate Phrasings CLAP does not receive a generative prompt; instead, each of the 23 candidates (same letter assignment as AppendixA) is given to CLAP as a short natural-language phrase, embedded once, and compared against the audio embedding via cosine similarity. Table 10:The 23 candidate phrasings given to CLAP. Lettersound_typesub_categoryCLAP phrase AAppliance AlarmAppliance Alarmthe sound of an appliance alarm beeping BCat SoundsMeowa cat meowing CCat SoundsPurra cat purring DClapDouble Claptwo hands clapping twice in quick succession EClapSingle Clapa single hand clap FClapTriple Claphands clapping three times in quick succession GDog SoundsGrowla dog growling HDog SoundsHowla dog howling IDoorbellDigitala digital doorbell chime JDoorbellMechanicala mechanical doorbell ringing KDoorbellWirelessa wireless doorbell chime LHairdryerHair Dryera hair dryer blowing MKettle BoilingBoilingwater boiling in a kettle NKettle BoilingWhistlinga kettle whistling OKnockDoor Knocksomeone knocking on a door PKnockSurface Knocksomeone knocking on a surface QSirenAmbulancean ambulance siren RSirenFirea fire truck siren SSirenOthera generic emergency siren TSirenPolicea police car siren UWater RunningClose Sourcewater running close to the microphone VWater RunningDistant Sourcewater running far from the microphone WYellYella person yelling B.3.Tier D: BAT’s Open-Vocabulary Question Unlike Tier A, BAT receives no candidate list and no format constraint. The entire prompt is a fixed, generic question, identical for every sample: Enumerate the sound occurrences in the audio clip. B.4.Tier D: LLM Judge Prompt BAT’s free-text answers are not scored by exact match; they are graded by an independent LLM judge ( gemini-2.5-flash-lite ), given the true label and BAT’s raw answer, with the following prompt: Sound Source Identification Benchmark19 August 4, 2026 You are evaluating whether an audio model's free-text description of a sound correctly identifies its true source. True label: sound_type = "true_sound_type" sub_category = "true_sub_category" The model's free-text answer: --- answer --- Does this answer correctly identify the sound as being a "true_sound_type", specifi- cally "true_sub_category"? Consider synonyms and paraphrases. Judge category-level (broad sound_type match) and fine-grained (specific sub_category match) separately. Respond with ONLY this JSON on the last line (no markdown): "category_correct": true_or_false, "finegrained_correct": true_or_false, "reasoning": "one sentence" C.Complete AudioSet-to-Taxonomy Mapping (Tier B) Table11lists every entry in the lookup table used to translate YAMNet/PANNs/Whisper-AT/SSLAM’s native AudioSet labels into our 23-class taxonomy, as described in Section4.2.1. Sub-category “—” denotes a category-only entry (the class collapses to Category-Level only; see Section4). Sound Source Identification Benchmark20 August 4, 2026 Table 11:Complete AudioSet-to-taxonomy mapping, all 22 entries. AudioSet labelsound_type sub_category QualityReason (if approxi- mate) "Alarm"Appliance Alarm Appliance AlarmapproximateGeneric parent class; also covers car/smoke/clock alarms "Meow"Cat SoundsMeowexact— "Purr"Cat SoundsPurrexact— "Clapping"Clap—exactAudioSet has no clap- count distinction (sin- gle/double/triple) "Growling"Dog SoundsGrowlexact— "Howl"Dog SoundsHowlexact— "Bark"Dog Sounds—approximateGeneric bark, not spe- cific to growl/howl "Doorbell"Doorbell—exactAudioSet has no dig- ital, mechanical, or wireless distinction "Hair dryer"HairdryerHair Dryerexact— "Boiling"Kettle Boiling Boilingexact— "Steam whistle"Kettle BoilingWhistlingexact— "Whistling"Kettle Boiling Whistlingapproximate Generic whistling, not specific to a boiling kettle "Knock"Knock—exactAudioSet has no door/surface distinc- tion "Police car (siren)"SirenPoliceexact— "Ambulance (siren)"SirenAmbulanceexact— "Fire engine, fire truck (siren)" SirenFireexact— "Siren"SirenOtherapproximateGeneric siren, not vehicle-type-specific "Civil defense siren"SirenOtherapproximate A distinct siren type with no closer match available "Water tap, faucet"Water Running—exactAudioSet has no recording-distance distinction "Yell"YellYellexact— "Shout"YellYellapproximateNear-synonym, not an identical acoustic concept "Bellow"YellYellapproximate Near-synonym, not an identical acoustic concept D.Complete Results: Accuracy, Per-Class Metrics, and Confusion Matrices D.1.Raw Accuracy (for reference) Table12reports raw accuracy for all eleven methods, for direct comparability with the project’s earlier accuracy-based reporting. As discussed in Section4, we treat macro F1/TPR/FNR (Tables2and3) as the primary metrics; accuracy is provided here as a secondary, widely-recognized reference point. Sound Source Identification Benchmark21 August 4, 2026 Table 12:Raw accuracy, all eleven methods, both granularities.N=2,242for every method except Kimi-Audio (N=2,206, marked‡). TierMethodCategory Acc.Fine-Grained Acc. Agemini-3.1-pro-preview86.8%61.5% Agemini-2.5-pro78.5%54.1% Agemini-3.5-flash78.4%53.4% Akimi-audio-7b-instruct ‡ 71.8%40.8% Agemini-2.5-flash68.6%42.0% Bsslam88.4%40.1% Bpanns84.3%37.3% Bwhisper-at77.8%31.7% Byamnet52.1%26.5% Cclap89.1%54.5% Dbat (judge, micro)62.4%35.9% D.2.Confusion Matrices Figures7–28give the row-normalized (recall) confusion matrix for every method with a discrete predicted class (all methods except BAT, Tier D, for which no confusion matrix can be constructed — see Section4), at both the category and fine-grained level. Figure 6:Fine-grained (23-class) confusion matrix (row-normalized recall) forgemini-3.1-pro-preview, the best- performing Tier A model. This is the detailed view behind the class-level failures discussed in Section5.3. Sound Source Identification Benchmark22 August 4, 2026 Figure 7:gemini-2.5-flash — category-level confusion matrix (recall). Sound Source Identification Benchmark23 August 4, 2026 Figure 8:gemini-2.5-flash — fine-grained confusion matrix (recall). Sound Source Identification Benchmark24 August 4, 2026 Figure 9:gemini-2.5-pro — category-level confusion matrix (recall). Sound Source Identification Benchmark25 August 4, 2026 Figure 10:gemini-2.5-pro — fine-grained confusion matrix (recall). Sound Source Identification Benchmark26 August 4, 2026 Figure 11:gemini-3.5-flash — category-level confusion matrix (recall). Sound Source Identification Benchmark27 August 4, 2026 Figure 12:gemini-3.5-flash — fine-grained confusion matrix (recall). Sound Source Identification Benchmark28 August 4, 2026 Figure 13:kimi-audio-7b-instruct (Tier A) — category-level confusion matrix (recall).N=2,206; 36 samples with no scoreable answer excluded (see Section4). Sound Source Identification Benchmark29 August 4, 2026 Figure 14:kimi-audio-7b-instruct (Tier A) — fine-grained confusion matrix (recall).N=2,206. Sound Source Identification Benchmark30 August 4, 2026 Figure 15:YAMNet (Tier B) — category-level confusion matrix (recall). Sound Source Identification Benchmark31 August 4, 2026 Figure 16:YAMNet (Tier B) — fine-grained confusion matrix (recall), Top-1. Sound Source Identification Benchmark32 August 4, 2026 Figure 17:YAMNet (Tier B) — fine-grained confusion matrix (recall),Top-5 credited: diagonal counts a sample as correct if the true label appears anywhere in the model’s top-5 ranked mapped predictions, otherwise its actual Top-1 guess is shown (see Section5.4). Sound Source Identification Benchmark33 August 4, 2026 Figure 18:PANNs (Tier B) — category-level confusion matrix (recall). Sound Source Identification Benchmark34 August 4, 2026 Figure 19:PANNs (Tier B) — fine-grained confusion matrix (recall), Top-1. Sound Source Identification Benchmark35 August 4, 2026 Figure 20:PANNs (Tier B) — fine-grained confusion matrix (recall),Top-5 credited(see Section5.4). Sound Source Identification Benchmark36 August 4, 2026 Figure 21:Whisper-AT (Tier B) — category-level confusion matrix (recall). Sound Source Identification Benchmark37 August 4, 2026 Figure 22:Whisper-AT (Tier B) — fine-grained confusion matrix (recall), Top-1. Sound Source Identification Benchmark38 August 4, 2026 Figure 23:Whisper-AT (Tier B) — fine-grained confusion matrix (recall),Top-5 credited(see Section5.4). Sound Source Identification Benchmark39 August 4, 2026 Figure 24:SSLAM (Tier B) — category-level confusion matrix (recall). Sound Source Identification Benchmark40 August 4, 2026 Figure 25:SSLAM (Tier B) — fine-grained confusion matrix (recall), Top-1. Sound Source Identification Benchmark41 August 4, 2026 Figure 26:SSLAM (Tier B) — fine-grained confusion matrix (recall),Top-5 credited(see Section5.4). Sound Source Identification Benchmark42 August 4, 2026 Figure 27:CLAP (Tier C) — category-level confusion matrix (recall). Sound Source Identification Benchmark43 August 4, 2026 Figure 28:CLAP (Tier C) — fine-grained confusion matrix (recall). Sound Source Identification Benchmark44

Instant research discovery