Paper deep dive
Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models
Yujun Lee, Joonhyeok Shin, Hyoeun Kim, Kyuhong Shim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 7/5/2026, 6:13:54 AM
Summary
The paper introduces a diagnostic benchmark sequence derived from OpenMIC to evaluate instrument grounding in music audio-language models. It argues that high accuracy in simple binary instrument-presence QA can mask systematic failure modes such as genre-prior exploitation, option-position bias, instrument-label preference, and temporal response bias. The authors evaluate several models (MF, MF-Think, Qwen2.5-Omni, AF3, GPT-4o-audio, Gemini 2.5 Pro/Flash) across five progressive tasks: Binary QA, Genre-Prior-Reduced QA, Confusion-Aware Instrument Discrimination, Long-Context Multi-label Recognition, and Temporal Instrument Localization.
Entities (9)
Relation Signals (4)
Music-Flamingo → exhibits → Option-Position Bias
confidence 100% · MF-Think and AF3 strongly prefer the first option, with position gaps above 36 pp
Audio-Flamingo-3 → exhibits → Temporal Response Bias
confidence 100% · AF3 shows the most extreme time-range preference, predicting 20–30 seconds for 96.87% of examples.
OpenMIC-2018 → issourcefor → Instrument Grounding Benchmark
confidence 100% · Using OpenMIC-2018 (Humphrey et al., 2018), we construct an instrument-grounding benchmark sequence
Gemini 2.5 Pro → performswellon → Temporal Localization
confidence 90% · Gemini 2.5 Pro shows the most favorable pattern, combining the highest accuracy with prediction rates closest to the balanced gold distribution.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark sequence for instrument grounding in music audio-language models, extending binary instrument-presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer audio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can exhibit option-position bias, confusable-instrument errors, and temporal response bias. These results suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2606.31338v1
- Canonical: https://arxiv.org/abs/2606.31338v1
Trouble viewing inline? Open PDF directly →
Full Text
42,904 characters extracted from source content.
Expand or collapse full text
Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models Yujun Lee 1 Joonhyeok Shin 1 Hyoeun Kim 1 Kyuhong Shim 1 Abstract Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark- specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark se- quence for instrument grounding in music audio- language models, extending binary instrument- presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer au- dio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can ex- hibit option-position bias, confusable-instrument errors, and temporal response bias. These re- sults suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy. 1. Introduction Recent audio-language models have rapidly expanded from general audio understanding to music-specific reasoning, building on a growing line of audio-language models (Chu et al., 2023; 2024; Kong et al., 2024; Ghosh et al., 2025b; Tang et al., 2024; Gong et al., 2024; 2023). Recent systems further extend this trend: Qwen2.5-Omni (Xu et al., 2025) processes audio as part of multimodal interaction, Audio Flamingo 3 (AF3) (Ghosh et al., 2026) targets broad au- dio understanding across speech, sound, and music, and Music Flamingo (MF) (Ghosh et al., 2025a) specializes in music understanding tasks such as instrumentation, tim- bre, harmony, structure, lyrics, and temporal reasoning. As these models become more capable, how to evaluate musi- cal understanding and interpret benchmark results becomes increasingly important (Choi et al., 2017). 1 Sungkyunkwan University. Correspondence to: Yujun Lee <yj090744@g.skku.edu>, Kyuhong Shim <khshim@skku.edu>. Workshop on Machine Learning for Audio,43 rd International Conference on Machine Learning, Seoul, South Korea. 2026. Copy- right 2026 by the author(s). Prior audio-language and music-language evaluation has used datasets and benchmarks for audio captioning, event recognition, and music understanding (Gemmeke et al., 2017; Kim et al., 2019; Drossos et al., 2020; Elizalde et al., 2023; Doh et al., 2023; Agostinelli et al., 2023; Weck et al., 2024; Zhao et al., 2024). For instrument-level evaluation, a common benchmark format is instrument-presence question- answering (QA), where a model answers whether a target instrument is present. However, high accuracy in binary QA does not necessar- ily imply grounded instrument understanding. A model can answer correctly by exploiting genre-instrument pri- ors, response-format shortcuts, or short context cues, with- out reliably distinguishing instruments from the audio it- self (Geirhos et al., 2020; Gururangan et al., 2018; McCoy et al., 2019). The limitation becomes more important when evaluation moves beyond isolated yes/no questions; robust instrument grounding should require a model to distinguish acoustically aor musically confusable instruments, recog- nize multiple instruments in longer mixtures, and localize when a target instrument appears. In this paper, we investigate whether binary instrument-QA performance remains reliable under more diagnostic eval- uation formats. Using OpenMIC-2018 (Humphrey et al., 2018), we construct an instrument-grounding benchmark sequence, which starts from binary QA and progressively introduces genre-prior-reduced presence QA, confusion- aware instrument discrimination, multi-label recognition, and temporal instrument localization. Our contributions are threefold. First, we introduce an OpenMIC-derived diagnostic benchmark sequence for prob- ing instrument grounding in music audio-language mod- els. Second, we evaluate recent general-purpose and music- specialized models and show that similar binary-QA accu- racy can conceal substantially different failure modes. Third, we analyze model behavior beyond aggregate accuracy, re- vealing option-position biases, instrument-label preferences, and temporal-range response biases that are not visible in standard instrument-presence binary QA. We will release the benchmark metadata, prompt templates, and evaluation code to support reproducible comparison. 1 arXiv:2606.31338v1 [cs.SD] 30 Jun 2026 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models Table 1. Overview of the diagnostic benchmark sequence. Binary QA (9,332) and genre-prior-reduced QA (590) report overall, Yes-QA, and No-QA accuracy. Instrument discrimination task (1,051) reports multiple-choice accuracy. Long-context multi-label benchmark (1,028) reports exact-set accuracy and F1. Temporal localization task (3,579) reports accuracy over three candidate time ranges. Model Binary QAPrior-Reduced QADiscriminationLong-ContextTemp. Loc. OverallYesNoOverallYesNoAcc.ExactF1Acc. MF87.6086.8488.3685.7678.6691.1044.4332.3974.7952.14 MF-Think81.7568.0595.4681.0260.8796.1447.7635.8072.7244.90 Qwen2.5-Omni88.7488.0089.4887.9785.7789.6186.1156.6184.6772.90 AF387.1981.2993.1084.7574.3192.5868.4124.0374.8033.70 GPT-4o-audio–87.7354.1882.7057.50 Gemini 2.5 Pro–83.8251.2680.3686.28 Gemini 2.5 Flash–83.9244.9470.5770.30 2. Benchmark Construction and Evaluation We construct an OpenMIC-derived diagnostic se- quence (Humphrey et al., 2018) using the relevance annotations described in Appendix A.1: binary instrument- presence QA, genre-prior-reduced presence QA, confusion- aware instrument discrimination, long-context multi-label recognition, and temporal instrument localization. This design follows the broader view that benchmark accuracy alone can miss systematic failure modes, motivating targeted diagnostic and behavioral tests (Ribeiro et al., 2020; Geirhos et al., 2020; Srivastava et al., 2023). 2.1. Binary Instrument-Presence QA We begin with binary instrument-presence QA. Each ex- ample consists of a 10-second OpenMIC clip and a target instrument, and the model is asked whether the instrument is present. From 4,666 clips, we generate one positive and one negative QA pair per clip, producing 9,332 examples (Ap- pendix A.3). As shown in Table 1, this setting yields high ac- curacy across all evaluated models: MF, AF3, and Qwen2.5- Omni exceed 87%, and MF-Think 1 reaches 81.75%. The results indicate that binary QA is a relatively permissive evaluation format for current audio-language models. This observation raises a diagnostic question: are models truly identifying the target instrument from the audio, or are they leveraging genre-instrument associations and the simplicity of yes/no responses? We therefore next reduce the influence of genre-level priors. 2.2. Genre-Prior-Reduced Presence QA To examine the influence of genre-instrument priors, we construct a hard set from the 9,108 binary QA examples with usable genre metadata. The examples are split into 70% training and 30% test data, stratified by the gold yes/no label. 1 MF-Think denotes the same Music Flamingo model evaluated with reasoning-enabled inference. A simple genre-prior baseline estimates the positive-answer rate for each genre-instrument pair from the training split, using an instrument-level fallback for unseen pairs. Test examples incorrectly answered by this baseline are selected as hard cases, resulting in 590 examples (see Appendix A.4). Overall accuracy decreases only moderately on this hard set. This subset does not eliminate all genre-related cues, but reduces examples that are solved by a simple genre- instrument prior. Nevertheless, reducing such genre-favored cases does not fully expose model limitations, since the task remains binary (Gardner et al., 2020). We therefore re- move the yes/no response format and require discrimination between confusable instruments. 2.3. Confusion-Aware Instrument Discrimination The third benchmark changes the task from binary presence detection to two-way instrument discrimination. Each exam- ple contains a 10-second music clip and two candidate in- struments sampled from a predefined confusable instrument group(see Appendix A.2), producing 1,051 multiple-choice examples (see Appendix A.5). Note that the confusable groups are not part of the original OpenMIC annotations, but are manually added during benchmark construction to define musically or acoustically related candidate sets. Thus, the task should be interpreted as related-instrument discrimina- tion rather than as a perceptually validated human-confusion benchmark. This multiple-choice format evaluates whether models can distinguish between related candidate instruments rather than merely answer whether a named instrument is plausi- ble. This also enables response-format analysis by varying the answer interface, such as A/B labels, X/Y labels, or di- rect instrument-name output. As shown in Section 3, this benchmark reveals substantial performance gaps and re- sponse biases that are hidden by binary QA. Since the input is still a single 10-second clip, we next extend the setting to longer 30-second concatenated music examples. 2 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models 2.4. Long-Context Multi-label Instrument Recognition The fourth benchmark evaluates multi-label instrument recognition in a longer music context. Multi-label evalu- ation is commonly used when multiple target classes can be simultaneously present, requiring metrics beyond single- label accuracy (Zhang & Zhou, 2013). Each example is a 30-second music input generated by concatenating three 10-second OpenMIC clips. Four candidate instruments are sampled from the same confusable group, two are present and the other two are absent. The model must select all candidate instruments that appear in the 30-second input, producing 1,028 multi-label examples (see Appendix A.7). The results show a clear gap between exact-set accuracy and partial recognition performance. Exact-set accuracy ranges from 24.03% to 56.61%, while F1 is much higher at 70.57%– 84.67%. This indicates that models often recover part of the correct set but struggle to identify all present instruments exactly. Since the task still asks only whether instruments ap- pear somewhere in the input, the final benchmark introduces explicit temporal localization. 2.5. Temporal Instrument Localization The final benchmark evaluates whether models can localize an instrument in time. Unlike the preceding benchmarks, this task requires temporal grounding rather than only pres- ence detection. Each example is a 30-second input con- structed by concatenating three 10-second OpenMIC clips. For a target instrument, exactly one segment has a high- confidence positive label, while the other two have negative labels. The model must choose the time range in which the target instrument appears. This benchmark includes 3,579 examples, with 1,190 labeled as 0–10 sec, 1,195 as 10–20 sec, and 1,194 as 20–30 sec (see Appendix A.8). Because the three time ranges are nearly balanced, a model cannot obtain high accuracy by exploiting a majority class. 3. Analysis In this section, we analyze model behavior beyond aggre- gate accuracy, focusing on response imbalance and struc- tured confusions. The diagnostic tasks introduced above are designed not only to change difficulty, but also to expose failure modes and confusion patterns that binary instrument- presence QA does not reveal. 3.1. Response Bias Beyond Accuracy To quantify response imbalance, we use a simple max–min prediction-rate gap. Given a set of possible responses and the model’s prediction ratep i for each responsei, the bias score is defined as:Bias = max i p i − min i p i .For the multiple-choice benchmark, responses correspond to the Table 2. Response-bias analysis on the confusion-aware instrument discrimination benchmark. Option rates report how often the model selects the first or second candidate. Position gap is the absolute difference between the two rates. ModelAcc.Option 1Option 2Pos. Gap MF44.4355.4744.5310.94 MF-Think47.7668.1331.8736.25 Qwen2.5-Omni86.1153.9546.057.90 AF368.4168.4131.5936.82 GPT-4o-audio87.7349.8650.140.29 Gemini 2.5 Pro83.8248.8551.152.31 Gemini 2.5 Flash83.9247.3852.625.23 first and second candidate positions. For temporal localiza- tion, responses correspond to the three time ranges. A larger value indicates that the model disproportionately selects a subset of available responses, even when the benchmark is balanced by construction (Zang et al., 2025). 3.2. Option-Position Bias in Multiple Choice The confusion-aware instrument discrimination benchmark reveals performance gaps hidden by binary QA. Among models evaluated on binary QA, all achieve high accu- racy; however, the Flamingo-family models degrade sub- stantially when asked to distinguish confusable instruments: MF reaches 44.43%, MF-Think 47.76%, and AF3 68.41%. Table 2 shows that the performance degradation of Flamingo-family models is accompanied by option-position bias, a known robustness issue in multiple-choice evalua- tion (Zheng et al., 2024; Pezeshkpour & Hruschka, 2024). MF-Think and AF3 strongly prefer the first option, with position gaps above 36 p, while MF shows a smaller but visible gap. In contrast, GPT-4o-audio (Hurst et al., 2024), Gemini models (Comanici et al., 2025), and Qwen2.5-Omni maintain more balanced option distributions. Moving from yes/no QA to two-way discrimination therefore exposes both weaker instrument discrimination and sensitivity to candidate order. Appendix A.6 provides an additional MF analysis under alternative answer interfaces and candidate orders, where the model exhibits a strong first-option prefer- ence. 3.3. Instrument Confusion Structure Figure 1 shows row-normalized confusion matrices for the three lowest-performing models on the discrimination benchmark: MF, MF-Think, and AF3. Errors are not uni- formly distributed across instruments; each model exhibits structured confusion patterns within the predefined groups. MF and MF-Think show clear instrument-label prefer- ences; MF over-selects ukulele and accordion, while MF- Think over-selects trumpet, ukulele, and mallet percussion. 3 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models Figure 1. Row-normalized instrument confusion matrices for MF, MF-Think, and AF3 on the confusion-aware instrument discrimination benchmark. Rows denote gold instruments and columns denote predicted instruments. Boundary lines (black) indicate predefined confusable instrument groups. Table 3. Temporal localization accuracy and prediction distribu- tion on the temporal instrument localization benchmark. Bias is computed as the difference between the maximum and minimum prediction rates across the three time ranges. ModelAcc.0–10s10–20s20–30sBias MF52.1453.6510.5335.8243.11 MF-Think44.9034.3729.2336.417.18 Qwen2.5-Omni72.9033.7851.4114.8136.60 AF333.701.151.9896.8795.73 GPT-4o-audio57.5014.5329.0053.1438.61 Gemini 2.5 Flash70.3029.9039.9327.6612.27 Gemini 2.5 Pro86.2830.4033.8134.033.63 AF3 performs better overall, but still shows uneven group- specific errors, such as over-prediction of clarinet in the brass/wind group. These patterns show that multiple-choice failures cannot be explained by option-position bias alone (Naik et al., 2018), but also reflect label-level preferences and group-specific confusions. 3.4. Temporal Response Bias Table 3 reports temporal localization accuracy and predic- tion rates over the three time ranges. Because the bench- mark is balanced by construction, strong deviations from one-third indicate time-range response bias. The results reveal distinct temporal failure modes. AF3 shows the most extreme time-range preference, predicting 20–30 seconds for 96.87% of examples. GPT-4o-audio also favors the final segment, while MF over-selects 0–10 sec- onds and rarely predicts 10–20 seconds. Qwen2.5-Omni achieves higher accuracy, but its predictions concentrate on 10–20 seconds and under-represent 20–30 seconds. The comparison also shows that response bias and temporal grounding are separate axes. MF-Think has the most bal- anced prediction distribution, with a bias score of only 7.18, but its accuracy remains low at 44.90%. Gemini 2.5 Pro shows the most favorable pattern, combining the highest accuracy with prediction rates closest to the balanced gold distribution. Overall, the temporal benchmark reveals errors hidden by instrument-presence QA: a model may recognize that an in- strument appears somewhere, yet fail to localize when it ap- pears. Reporting prediction distributions alongside accuracy is therefore important for separating temporal grounding from time-range response bias. 4. Conclusion and Future Work We presented an OpenMIC-derived diagnostic sequence for probing instrument grounding in music audio-language models. Starting from binary instrument-presence QA, we extended the evaluation to genre-prior-reduced presence QA, confusion-aware instrument discrimination, long-context multi-label recognition, and temporal instrument localiza- tion. Across these settings, we showed that high binary QA accuracy can hide systematic failure modes, including option-position, instrument-label, and temporal-range bi- ases. These findings suggest that instrument-centered music understanding should be evaluated through multiple diagnos- tic axes rather than a single aggregate accuracy. In summary, our results caution against interpreting strong binary-QA performance as sufficient evidence of grounded instrument understanding. Future work should extend this diagnostic approach to broader dimensions of music understanding, including harmony, structure, lyrics, and fine-grained tem- poral reasoning. 4 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models References Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. Choi, K., Fazekas, G., Cho, K., and Sandler, M. A tutorial on deep learning for music information retrieval. arXiv preprint arXiv:1709.04396, 2017. Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing universal au- dio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Doh, S., Choi, K., Lee, J., and Nam, J. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint arXiv:2307.16372, 2023. Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 736–740. IEEE, 2020. Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H. Clap learning audio concepts from natural language su- pervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. IEEE, 2023. Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, p. 1307–1323, 2020. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 776–780. IEEE, 2017. Ghosh, S., Goel, A., Koroshinadze, L., Lee, S.-g., Kong, Z., Santos, J. F., Duraiswami, R., Manocha, D., Ping, W., Shoeybi, M., et al. Music flamingo: Scaling music understanding in audio language models. arXiv preprint arXiv:2511.10289, 2025a. Ghosh, S., Kong, Z., Kumar, S., Sakshi, S., Kim, J., Ping, W., Valle, R., Manocha, D., and Catanzaro, B. Audio flamingo 2: An audio-language model with long-audio un- derstanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983, 2025b. Ghosh, S., Goel, A., Kim, J., Kumar, S., Kong, Z., Lee, S.- g., Yang, C.-H., Duraiswami, R., Manocha, D., Valle, R., et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Advances in Neural Information Processing Systems, 38:41819– 41886, 2026. Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., and Glass, J. Joint audio and speech understanding. In 2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU), p. 1–8. IEEE, 2023. Gong, Y., Luo, H., Liu, A., Karlinsky, L., and Glass, J. R. Listen, think, and understand. In International Con- ference on Learning Representations, volume 2024, p. 18516–18545, 2024. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., and Smith, N. A. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), p. 107– 112, 2018. Humphrey, E., Durand, S., and McFee, B. Openmic-2018: An open data-set for multiple instrument recognition. In ISMIR, p. 438–444, 2018. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Rad- ford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), p. 119–132, 2019. 5 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B.Audio flamingo: A novel audio lan- guage model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831, 2024. McCoy, R. T., Pavlick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 3428–3448, 2019. Naik, A., Ravichander, A., Sadeh, N., Rose, C., and Neubig, G. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, p. 2340–2353, 2018. Pezeshkpour, P. and Hruschka, E. Large language models sensitivity to the order of options in multiple-choice ques- tions. In Findings of the Association for Computational Linguistics: NAACL 2024, p. 2006–2017, 2024. Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. Beyond accuracy: Behavioral testing of nlp models with check- list. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 4902–4912, 2020. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023. Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Salmonn: Towards generic hear- ing abilities for large language models. In International Conference on Learning Representations, volume 2024, p. 16607–16629, 2024. Weck, B., Manco, I., Benetos, E., Quinton, E., Fazekas, G., and Bogdanov, D. Muchomusic: Evaluating music un- derstanding in multimodal audio-language models. arXiv preprint arXiv:2408.01337, 2024. Xu, J., Guo, Z., Hu, H., Chu, Y., Wang, X., He, J., Wang, Y., Shi, X., He, T., Zhu, X., et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025. Zang, Y., O’Brien, S., Berg-Kirkpatrick, T., McAuley, J., and Novack, Z. Are you really listening? boosting percep- tual awareness in music-qa benchmarks. arXiv preprint arXiv:2504.00369, 2025. Zhang, M.-L. and Zhou, Z.-H. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 26(8):1819–1837, 2013. Zhao, M., Zhong, Z., Mao, Z., Yang, S., Liao, W.-H., Taka- hashi, S., Wakaki, H., and Mitsufuji, Y. Openmu: Your swiss army knife for music understanding. arXiv preprint arXiv:2410.15573, 2024. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. Large language models are not robust multiple choice selectors. In International Conference on Learning Rep- resentations, volume 2024, p. 19426–19454, 2024. 6 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models A. Appendix A.1. OpenMIC-2018 Data Format OpenMIC-2018 provides 10-second music clips with instrument-level relevance annotations. The aggregated label file used in this work contains 41,534 clip–instrument annotations over 20,000 unique clips. As summarized in Table 4, each row corresponds to one observed clip–instrument pair and includes a clip identifier, an instrument label, an aggregated relevance score, and the number of annotator responses used for aggregation. Table 4. Fields in the OpenMIC-2018 aggregated label file. ColumnDescription sample_keyOpenMIC clip identifier instrumentAnnotated instrument class relevanceAggregated confidence score for the clip–instrument pair num_responsesNumber of annotator responses used for aggregation Table 5 shows example annotations for individual clips. A single clip can have multiple observed instrument annotations, each stored as a separate row with its own relevance score. Table 5. Example clip–instrument annotations from the OpenMIC-2018 aggregated label file. sample_key instrument relevance num_responses 000046_3840clarinet0.171053 000046_3840flute0.000003 000046_3840trumpet0.000003 000135_483840saxophone0.147053 000135_483840voice1.000003 000135_483840trumpet0.000003 000182_145920piano0.000003 000182_145920voice1.000003 The instrument vocabulary contains 20 classes: accordion, banjo, bass, cello, clarinet, cymbals, drums, flute, guitar, mallet percussion, mandolin, organ, piano, saxophone, synthesizer, trombone, trumpet, ukulele, violin, and voice. We use only extreme relevance values as high-confidence labels. A relevance score of 1.0 is treated as a positive label, indicating that the instrument is present, while a relevance score of 0.0 is treated as a negative label, indicating that the instrument is absent. Intermediate relevance values are excluded from gold-label construction to reduce annotation ambiguity. OpenMIC also provides clip-level metadata, including track and genre information, which is used only for the genre-prior-reduced hard set. A.2. Confusable Instrument Groups The confusable instrument groups are manually defined during benchmark construction and are not part of the original OpenMIC-2018 annotation file. OpenMIC provides clip–instrument relevance annotations, while thegroup_namefield is added in our benchmark metadata to support confusion-aware candidate sampling. The manually defined groups are summarized in Table 6. These groups are used in the confusion-aware two-choice benchmark and the strict 30-second choose-all benchmark. In both cases, candidates are sampled within the same group to make the alternatives acoustically or musically related. In the two-choice benchmark, one positive and one negative instrument are sampled from the same group. In the strict 30-second benchmark, all four candidate instruments are sampled from the same group, so that the model must distinguish between related candidates rather than choose from unrelated instruments. These groups should not be interpreted as perceptually validated human-confusion groups. They are intended to create related candidate sets that are more diagnostic than randomly sampled unrelated instruments. Accordingly, the resulting 7 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models confusion matrices characterize model behavior within our manually defined candidate groups, rather than human perceptual confusability. Future work should validate such groupings through controlled listening studies. Table 6. Manually defined confusable instrument groups used for candidate sampling. GroupInstruments string_folkbanjo, cello, guitar, mandolin, ukulele, violin brass_windclarinet, flute, saxophone, trombone, trumpet keyboard_synthaccordion, organ, piano, synthesizer percussioncymbals, drums, mallet percussion A.3. Binary Instrument-Presence QA Benchmark The binary instrument-presence QA benchmark contains 9,332 yes/no question-answer pairs derived from 4,666 OpenMIC- 2018 clips. Each clip contributes one positive and one negative question, yielding a balanced set with 4,666 “Yes” and 4,666 “No” answers. Positive questions use instrument labels with relevance score 1.0, while negative questions use labels with relevance score 0.0. Each instance consists of an audio clip, a target instrument, and a yes/no question of the form: “Is there a [instrument] in this audio clip?” Model outputs are evaluated by exact-match accuracy after answer normalization. The core CSV fields are summarized in Table 7, and representative examples are shown in Table 8. Table 7. Core CSV fields for the binary instrument-presence QA benchmark. ColumnDescription qa_idUnique identifier of the QA instance sample_keyOpenMIC-2018 clip identifier audio_pathPath to the audio file instrumentTarget instrument in the question questionNatural-language yes/no question gold_answerGround-truth answer, Yes or No label_typePositive or negative label type relevanceOpenMIC relevance score source_datasetSource dataset name settingBenchmark construction setting Table 8. Representative examples from the binary instrument-presence QA benchmark. sample_keyinstrumentgoldlabel typerelevance 000135_483840voiceYespositive1.0 000135_483840trumpetNonegative0.0 000182_145920voiceYespositive1.0 000182_145920pianoNonegative0.0 This construction reduces answer-prior bias by balancing positive and negative questions at the clip level. Since each selected clip contributes both a positive and a negative query, models cannot achieve high performance by always favoring one answer class. A.4. Genre-Prior-Reduced Presence QA The genre-prior-reduced hard set contains 590 binary QA examples selected from the main benchmark. It is designed to reduce cases that can be answered using simple genre-instrument associations. We use examples with available genre metadata, split them into 70% training and 30% test partitions, and fit a genre-prior baseline on the training split. The 8 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models baseline estimates the positive-answer rate for each genre-instrument pair, with an instrument-level fallback for unseen pairs. Test examples incorrectly answered by this baseline are retained as hard cases. Each row follows the binary QA format and additionally stores the metadata used by the genre-prior baseline. The core CSV fields are summarized in Table 9, and representative examples are shown in Table 10. All examples use the same prompt template as the main binary benchmark: “Is there a [instrument] in this audio clip?” Table 9. Core CSV fields for the genre-prior-reduced hard set. ColumnDescription qa_idUnique identifier of the QA instance sample_keyOpenMIC-2018 clip identifier audio_pathPath to the audio file instrumentTarget instrument in the question questionNatural-language yes/no question gold_answerGround-truth answer, Yes or No label_typePositive or negative label type relevanceOpenMIC relevance score primary_genreGenre used by the prior baseline yes_rateGenre-instrument positive rate inst_yes_rateInstrument-level fallback positive rate final_yes_rateFinal positive rate used for prediction genre_prior_predGenre-prior baseline prediction genre_prior_correctWhether the baseline prediction is correct Table 10. Representative examples from the genre-prior-reduced hard set. sample_keyinstrumentgoldgenreprior pred.final rate 111817_268800cymbalsYesInternationalNo0.333 016747_334080violinYesAfrobeatNo0.000 017608_506880celloYesAsia-Far EastNo0.418 057853_0violinNoPopYes0.667 This subset does not remove all possible shortcuts. Instead, it specifically filters out examples solved by a simple genre- instrument prior, thereby increasing the need for audio-grounded instrument recognition. A.5. Confusion-Aware Instrument Discrimination The confusion-aware two-choice benchmark contains 1,051 instrument discrimination examples. Each example presents two candidate instruments from a predefined confusable group: one positive instrument with relevance score 1.0 and one negative instrument with relevance score 0.0. The model must output the name of the instrument that is present. Each instance stores the present instrument, the absent confusable candidate, and the candidate order shown to the model. Although option-position metadata is retained for analysis, evaluation is based on matching the predicted instrument name to the gold instrument name rather than on A/B labels. The core CSV fields are summarized in Table 11, and representative examples are shown in Table 12. All examples use the prompt template: “Which instrument is present in this audio clip? Candidate instruments: [instrument 1], [instrument 2]. Answer with only one instrument name from the candidates.” This benchmark removes the yes/no response format and tests whether models can discriminate between acoustically or semantically confusable instruments. 9 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models Table 11. Core CSV fields for the confusion-aware two-choice name-answer benchmark. ColumnDescription mc_idUnique identifier of the instance sample_keyOpenMIC-2018 clip identifier audio_pathPath to the audio file group_nameConfusable instrument group gold_instrumentPresent instrument with relevance score 1.0 negative_instrumentAbsent candidate with relevance score 0.0 optionsCandidate instruments shown to the model gold_answerCorrect instrument name ab_gold_answerPosition metadata of the correct candidate label_typeName-answer evaluation format Table 12. Representative examples from the confusion-aware two-choice benchmark. sample_keygroupcandidatesgold instrument 000386_65280brass_windclarinet|saxophonesaxophone 000739_0string_folkguitar|violinguitar 001430_291840keyboard_synthorgan|synthesizersynthesizer 001378_34560percussiondrums|mallet_percussiondrums A.6. Answer-Interface and Candidate-Order Variation We conduct a targeted prompt-variation analysis on MF to examine whether the option-position behavior observed in Section 3.2 persists under different candidate orders and answer interfaces. We use the same confusion-aware two-choice benchmark, keeping the audio clips, candidate instruments, and gold labels fixed. We vary only the displayed candidate order and the answer format. The main discrimination experiment in Sections 2.3 and 3.2 uses the direct instrument-name format, where the model outputs the name of the candidate instrument that is present. Here, we compare four variants: (i) the original direct instrument-name prompt, (i) a direct instrument-name prompt with the candidate order swapped, (i) an A/B answer prompt with the original candidate order, and (iv) an A/B answer prompt with the candidate order swapped. For all variants, model outputs are mapped back to instrument names before evaluation. The results are shown in Table 13. Table 13. Prompt-variation analysis on MF for the confusion-aware two-choice benchmark. Option rates report how often the model selects the first or second displayed candidate. Gap denotes the absolute difference between the two option rates. Prompt/interfaceAcc.Opt. 1Opt. 2UnknownGap Name, original order44.4355.4744.530.0010.94 Name, swapped order42.0658.7141.290.0017.41 A/B, original order47.8692.587.420.0085.16 A/B, swapped order44.8193.246.760.0086.49 The original direct instrument-name condition reproduces the MF result reported in Table 2, confirming that the prompt- variation pipeline is comparable to the main experiment. Across all four variants, MF predicts the first displayed candidate more often than the second, indicating that the first-option preference is not limited to a single candidate ordering. The bias becomes especially severe under the A/B answer interface: MF selects the first displayed candidate in more than 92% of examples under both original and swapped candidate orders. These results show that MF’s multiple-choice behavior is sensitive not only to the candidate order, but also to the response interface itself. 10 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models A.7. Long-Context Multi-label Instrument Recognition The long-context multi-label instrument recognition benchmark contains 1,028 multi-label instrument recognition examples. Each example is constructed by concatenating three 10-second OpenMIC-2018 clips into a 30-second input. Four candidate instruments are shown to the model: two positive instruments with relevance score 1.0 and two negative instruments with relevance score 0.0. The model must select all candidate instruments that appear anywhere in the 30-second audio. Each row stores the three source clip identifiers, their original audio paths, the positive and negative candidate instruments, the full candidate list, and the complete gold answer set. Evaluation is performed using exact-set accuracy, precision, recall, and F1. The core CSV fields are summarized in Table 14, and representative examples are shown in Table 15. All examples use the prompt template: “Listen carefully to the 30-second audio clip. Which instruments are present in this audio clip? Candidate instruments: [instrument 1], [instrument 2], [instrument 3], [instrument 4]. Answer with all instrument names from the candidates that are present, separated by commas.” Table 14. Core CSV fields for the long-context multi-label benchmark. ColumnDescription concat_idUnique identifier of the concatenated instance group_nameConfusable instrument group source_sample_keysThree OpenMIC clip identifiers positive_instrumentsPresent candidate instruments negative_instrumentsAbsent candidate instruments optionsFour candidate instruments shown to the model gold_answersComplete set of correct instrument names num_source_clipsNumber of source clips, fixed to 3 duration_secDuration of the concatenated audio, fixed to 30 num_positiveNumber of positive instruments, fixed to 2 num_negativeNumber of negative instruments, fixed to 2 gold_positionsOption positions of the gold instruments audio_pathPath to the concatenated 30-second audio file Table 15. Representative examples from the long-context multi-label benchmark. concat_idgroupcandidatesgold instruments 000000string_folkmandolin | ukulele | violin | banjomandolin | violin 000001string_folkguitar | cello | violin | ukulelecello | violin 000002string_folkmandolin | ukulele | cello | guitarcello | guitar 000003string_folkukulele | violin | guitar | cellocello | violin This benchmark is stricter than two-choice discrimination because the model must recover the complete set of present instruments. Exact-set accuracy measures full recovery, while precision, recall, and F1 capture partial recognition. A.8. Temporal Instrument Localization The temporal localization benchmark contains 3,579 examples constructed from 30-second concatenated audio inputs. Each input consists of three 10-second OpenMIC-2018 clips arranged into non-overlapping temporal segments. For a target instrument, exactly one segment has a high-confidence positive label with relevance score 1.0, while the other two segments have negative labels with relevance score 0.0. The model must choose the time range in which the target instrument appears, rather than only deciding whether the instrument is present somewhere in the audio. Each row stores an anonymized audio identifier, the path to the 30-second audio file, the target instrument, the gold time range, and the positive segment index. Segment indices 0, 1, and 2 correspond to 0–10 seconds, 10–20 seconds, and 20–30 seconds. This makes the gold answer directly recoverable from the position of the positive source segment in the concatenated input. The core CSV fields are summarized in Table 16, and representative examples are shown in Table 17. 11 Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models All examples use the prompt template: “Listen to the full audio carefully and identify the time range where the target instrument is heard. Target instrument: [instrument]. Choose exactly one time range: 0–10 seconds, 10–20 seconds, 20–30 seconds. Answer with only the selected time range.” Table 16. Core CSV fields for the temporal localization benchmark. ColumnDescription item_idUnique identifier of the temporal instance anon_audio_idAnonymized audio filename audio_pathPath to the 30-second audio file instrumentTarget instrument to localize group_nameInstrument group of the target instrument questionNatural-language temporal localization prompt gold_time_rangeCorrect time range answer positive_segment_indexSegment index containing the target instrument num_source_clipsNumber of source clips, fixed to 3 duration_secDuration of the concatenated audio, fixed to 30 Table 17. Representative examples from the temporal localization benchmark. item_idinstrumentgroupgold time range item_000000mandolinstring_folk20–30 seconds item_000001ukulelestring_folk20–30 seconds item_000002accordionkeyboard_synth0–10 seconds item_003578mallet_percussionpercussion20–30 seconds The benchmark is nearly balanced across the three candidate time ranges, with 1,190 examples labeled as 0–10 seconds, 1,195 as 10–20 seconds, and 1,194 as 20–30 seconds. This balance prevents accuracy from being dominated by a majority time range and allows temporal response bias to be analyzed separately from localization accuracy. 12