Paper deep dive
UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition
Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang, Yujun Cai, Hongli Cao, Yiwei Wang, Xiaoyan Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing \textbf{UA-Bench}, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0\%--13.1\% leaf accuracy, against a random reference of 7.7\%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.
Tags
Links
- Source: https://arxiv.org/abs/2610.16942v1
- Canonical: https://arxiv.org/abs/2610.16942v1
Trouble viewing inline? Open PDF directly β
Full Text
99,855 characters extracted from source content.
Expand or collapse full text
UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang, Yujun Cai, Hongli Cao, Yiwei Wang, Xiaoyan Wang Β§ Southeast University zhe.sun@seu.edu.cn xyan@seu.edu.cn Abstract Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Underwater acoustics is a case in point. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Building a benchmark for this setting is not straightforward. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing UA-Bench, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0%β13.1% leaf accuracy, against a random reference of 7.7%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work. Source Code and Data:: https://sunzhe12.github.io/ua-bench/ 1 Introduction Large audio-language models (LALMs) pair an audio encoder with a language model and are applied to a widening range of audio understanding tasks [1β3]. Benchmarks for these models have centered on speech, music, and environmental sound [4β6]. Signals generated by physical or engineered processes have received less attention. Their categories are typically defined by temporal and spectral structure rather than by semantic content, so recognizing them may depend on cues that everyday sound categories seldom require. How LALMs perform on such signals remains largely unexamined. Underwater acoustic recognition provides a concrete setting, as illustrated in Figure 1. It supports vessel monitoring, maritime safety, and ocean observation [7β11], and its signals fall into three kinds, each with its own structure. Active Β§ Corresponding author. arXiv:2610.16942v1 [cs.SD] 12 Oct 2026 An AUV, ROV or even submarine with aIntelligent Passive Sonar System. Detection Pulses: Who is searching? Interpret sonar pulses from nearby platforms. Communication Signals: Who is talking? Identify waveforms to capture communication messages. Ship Noise Signals: What is passing by? Classify vessels for traffic awareness and safe routing. Can large audio-language models(LALMs)listen to the ocean? 13.1% Best Accuracy of 9 LALMs random 7.7% Figure 1 UA-Bench at a glance. An underwater platform (AUV, ROV, or submarine) equipped with an intelligent passive sonar system must interpret the sound field around it. We build UA-Bench, the first underwater acoustic understanding benchmark, to evaluate large audio-language models (LALMs) on this task. Despite their strong general-purpose audio understanding, the best of the nine evaluated LALMs reaches only 13.1% accuracy at the finest level (L3), compared with 7.7% for the random-guess reference, which shows that current LALMs are far from ready to listen to the ocean. detection pulses are distinguished by their frequency trajectories, communication waveforms by their modulation, and ship-radiated noise by its tonal and broadband composition, with considerable overlap among vessel categories. Multipath propagation, which depends on the sound-speed profile and on the source and receiver geometry, further reshapes each signal before it reaches the receiver [12]. Labeled data are also scarce because annotated recordings are costly to acquire [13]. Existing resources address this setting only in part. General benchmarks such as AIR-Bench and MMAU cover a wide range of audio tasks, but they contain no engineered detection pulses or underwater communication classes and provide no controlled channel metadata [4,5]. Underwater datasets such as ShipsEar, DeepShip, QiandaoEar22, and Oceanship offer valuable field recordings, yet most support supervised, closed-set vessel recognition [7β9,14]. And the closest marine-domain example, WoW-Bench, relies on marine mammal vocalizations [15]. What is needed is a resource that brings engineered underwater signals, known generation conditions, and a question-answering interface together in one evaluation. Building such a resource and evaluating LALMs with it raise difficulties on two sides. The first lies in the data. A field recording reflects the source and the propagation path together, and the conditions behind it are seldom fully known, so an error is difficult to trace to either. Generating signals under known conditions can supply the missing control, but it involves many interacting factors. The three signal families follow different generative models, from fixed-carrier and swept pulses [10] to digitally modulated waveforms [11] and the tonal and broadband radiation of vessels [16,17]. The received waveform further depends on the sound-speed profile, the source and receiver geometry, and the multipath arrivals they produce [12]. The second lies in the evaluation. Recognition proceeds from coarse choices, such as whether a signal is transmitted deliberately (active) or radiated by a platform (passive), to fine ones, such as the modulation scheme or the vessel type, and an overall accuracy does not show at which step a model breaks down. A model may also answer from the wording of a question and from class priors, so a correct answer does not by itself show that the recording was used [18]. We introduce UA-Bench, a benchmark for evaluating LALMs on underwater acoustic signals, meant to complement field recordings (Figure 2). It addresses the two sets of difficulties through its data and its evaluation protocol. To make errors attributable, each source is generated from a parameterized model of its class, covering three kinds of detection pulses, five communication modulations, and noise radiated by five vessel types, and is propagated with BELLHOP under sound-speed profiles drawn from ocean climatology and perturbed geometry [19,20]. The generating parameters are stored with every result, which yields 375,051 audio clips in 13 leaf classes, each traceable to the source and channel conditions that produced it and paired with a templated three-turn dialogue. Clips of the same source are kept in the same partition, so that propagation variants of one source cannot leak across partitions. A balanced subset of 2,600 clips, UA-Bench-Eval, is fixed for evaluation, and the training partition is released to support later adaptation studies. To locate errors, evaluation follows a gated coarse-to-fine dialogue over a three-level label that runs from the source 2 3 Signal Families 13 Leaf Classes 375,051 Audio Records 2,600 Eval Examples S i g n a l C o v e r a g e H i e r a r c h i c a l R e c o g n i t i o n D i a g n o s t i c E v a l u a t i o n P r o p a g a t i o n C o n t r o l Underwater Communications Sonar Interpretation AUV Acoustic Awareness Vessel Monitoring Dataset Composition Audio + dialogue + physical metadata Label Space & Benchmart Item Active Detection pulses CWLFMHFM Communication 2FSK4FSKBPSKQPSKOFDM Passive Ship noise CargoCruiseFishingNavalUnderwater vehicle Concrete example: 2FSK communication signal F r e q u e n c y ( k H z ) Time (s) Route: Active Classify: Communication signal β 2FSK Explain: Alternating frequency states with symbol-wise switching. Metadata: source parameters Β· sound-speed profile Β· source depth Β· receiver depth Β· range Β· seabed Measures: hierarchical accuracy Β· routing-error decomposition Β· metadata-stratified analysis Does the model use acoustic evidence? H1 Hierarchical consistency β Correct route Active β Detection pulse Passive β Ship noise H2 Acoustic reliance β Original audio Communication β 2FSK Misclassified route H3 Explanation grounding β Supported explanation Consistent with audioInconsistent with audio Γ Cascade error Silent audio Γ Γ Contradictory cue Figure 2 Overview of UA-Bench. Detection pulses, communication waveforms, and ship-radiated noise are synthesized under traceable source and propagation conditions. Each record includes a three-level label and a reference dialogue that proceeds from source category to leaf class and then requests an explanation. The benchmark measures recognition accuracy, separates routing errors from downstream errors, and compares performance across acoustic conditions. It also checks whether each explanation agrees with the predicted label. category through the signal family to the leaf class. A model names the source category, selects a complete class path within the chosen branch, and then explains its choice. A wrong answer at the first turn ends the dialogue, so every error is assigned to source routing (the active or passive decision), fine-grained classification, or output format. To test reliance on the waveform, each evaluation recording is paired with silence of equal duration under identical questions, and the two sets of answers are compared. We evaluate nine open LALMs without task-specific adaptation. Leaf accuracy, the share of clips whose complete class path is correct, ranges from 2.0% to 13.1%, against a random reference of 7.7%, and no model is best on all three signal families. The models fail in different ways: some at the answer format, some at source routing, and some by concentrating their answers on a few classes. In the silent control, the recording improves source routing for some models and changes it little for others, whereas the gain in leaf accuracy is at most about 4.7 points for any model. For the models that benefit, the contribution of the waveform appears mainly at the coarse level, a pattern that an overall accuracy would have concealed. Our contributions are as follows. β’A controlled benchmark. UA-Bench brings detection pulses, communication waveforms, and ship-radiated noise into one audio-language task, with hierarchical labels, templated dialogues, traceable source and channel metadata, and source-grouped partitions. β’ A diagnostic evaluation design. A gated coarse-to-fine protocol, metadata-conditioned analysis, and a paired silent-audio control separate signal from channel, locate the failing decision, and test reliance on the waveform. β’ An empirical account of current LALMs. Nine open models show low leaf accuracy, divergent failure modes, and limited use of the waveform at the leaf level. 3 ResourceSignal focusScalePrimary targetLALMHier.Dialogue/QASource meta.Prop. ctrl. Audio-language benchmarks AIR-Bench [4]Speech, sound, music21k QFound. + chatββββ MMAU [5]Speech, sound, music10k QAPercept. + reason.ββββ Dynamic-SUPERB P2 [6] Speech, music, sound180 tasksCls. / reg. / gen. βΌββΌβ AudioBench [21]Speech, voice, scenes100k+ samp.8 tasks / 26 setsβββΌβ WoW-Bench [15]Marine mammal calls1,777 QPercept. + cognit.ββββ Underwater acoustic resources ShipsEar [7]Recorded vessel noise90 rec.11 vessel typesβββ DeepShip [8]Recorded vessel noise47 h4 vessel classesββΌβ QiandaoEar22 [14]Recorded vessel noise10,611 rec. Detection + IDβββ Oceanship [9]Recorded vessel noise121 h15 classes + retr.βββ UA-Bench (ours)Pulses, comms., vessel noise375,051 rec.13-leaf hier. rec.β βdenotes supported,βΌdenotes partial or dataset-dependent support, andβdenotes no support. Q denotes questions, samp. denotes samples, rec. denotes recordings, and retr. denotes retrieval. βSource meta.β refers to source-condition metadata. βProp. ctrl.β refers to independently specified propagation conditions. Table 1 Comparison with representative audio-language benchmarks and underwater acoustic resources. 2 Related Work Research related to UA-Bench spans general audio-language evaluation, the transfer of pretrained audio models to specialized acoustics, and underwater recognition with measured or simulated data. Table 1 compares UA-Bench with representative work in these areas. Existing audio-language benchmarks rarely include underwater signals, while underwater resources usually evaluate task-specific models rather than LALMs. WoW-Bench is the closest marine-domain LALM benchmark and contains 1,777 validated questions about marine mammal vocalizations [15]. UA-Bench instead covers three underwater signal families and 13 leaf classes in 375,051 records, together with a fixed evaluation suite of 2,600 records. It is the first large-scale, multi-category benchmark for evaluating LALMs on underwater acoustic signal recognition, with controlled source and propagation metadata for analysis under known conditions. 2.1 Audio Understanding Benchmarks General audio benchmarks measure capabilities across spoken content, instruments, sound events, perception, and reasoning. AIR-Bench [4] includes foundational tasks in speech, sound, and music, and it also supports open-ended interaction. MMAU [5] places greater emphasis on expert-level perception and reasoning across speech, environmental audio, and music. AudioBench [21] evaluates speech, voice, and diverse acoustic scenes. Broader evaluation frameworks include AHELM [22], which combines several datasets and metrics, and VoiceBench [23], which focuses on voice assistant capabilities. Recent work has moved beyond conventional sound-event recognition. Dynamic-SUPERB Phase-2 [6] extends instruction-based evaluation to classification, regression, and sequence generation, with some tasks examining acoustic properties such as pitch and rhythm. GAMA [3] studies more complex audio reasoning through the synthetic CompA-R instruction-tuning dataset and the human-annotated CompA-R-test. WoW-Bench [15] further extends fine-grained perception and cognition to marine mammal vocalizations. These studies broaden the scope of audio evaluation, but engineered underwater signals and variation caused by propagation remain largely outside it. 2.2 General Audio Pretraining in Specialized Acoustics Pretrained audio representations are increasingly used in domains that require specialized acoustic evidence. BioLingual applies contrastive audio and language pretraining to more than one million captioned animal recordings, supporting zero-shot species recognition and natural-language retrieval across taxa [24]. General audio models have also been tested on respiratory and heart sounds, where frozen encoders provide features for downstream classifiers. Their performance can approach task-specific systems on clean recordings but declines on noisier tasks [25]. These results demonstrate both the value and the limits of broad audio pretraining. Most studies in specialized domains use a learned probe, fine-tuning, or further domain pretraining, so they do not show whether an instruction-following LALM can recognize the same signals without task-specific adaptation. Source 4 structure and acquisition conditions are also rarely varied independently. UA-Bench focuses directly on this transfer setting by keeping the model fixed while varying the source family and propagation conditions in a controlled manner. The values of these variables are retained for diagnostic analysis. 2.3 Underwater Acoustic Recognition, Data, and Simulation Underwater target recognition has traditionally relied on task-specific spectral and modulation features. Work on ShipsEar [7] combines cepstral representations with Gaussian mixture models for vessel classification. DeepShip [8] compares time-domain and frequency-domain representations and evaluates several learning methods, including a separable convolutional autoencoder. QiandaoEar22 [14] uses LOFAR and DEMON spectra to capture narrowband and modulation patterns for individual ship identification. Cross-domain contrastive learning has also been used to address limited annotations and shifts between recording environments [13]. The datasets differ in annotation detail and task coverage. ShipsEar and DeepShip mainly provide vessel-category labels, while QiandaoEar22 supports the identification of individual targets. Oceanship [9] adds vessel coordinates, velocity, and timestamps to its recordings and supports both classification and retrieval between audio and text. These resources preserve the complexity of real underwater environments, but the source and its propagation effects are observed together and cannot be controlled independently. It is therefore difficult to isolate the effect of a specific channel on a known source. Physical simulation provides a complementary approach because it separates source generation from channel trans- formation. Classical ship-noise models describe the broadband, tonal, and modulation components of radiated waveforms [16,17,26]. The Acoustics Toolbox provides several propagation solvers, including BELLHOP based on Gaussian beams and KRAKEN based on normal modes [19,27,28]. Their inputs specify the sound-speed profile, boundaries, source and receiver depths, and range, which allows researchers to recompute arrivals or acoustic fields under different conditions. These simulators are forward models rather than evaluation resources, and they do not define a source inventory, label space, leakage-safe partition, dialogue task, or diagnostic protocol. UA-Bench connects these elements in a single evaluation framework by combining parameterized source synthesis with BELLHOP propagation and validating each received waveform. It also retains the generating metadata and uses source-grouped data partitions, allowing model behavior to be related to known source and channel variables. 3 UA-Bench UA-Bench is built to show where a LALM goes wrong when it listens to an underwater signal. It combines three components. A synthesis pipeline generates each source waveform from known parameters, propagates it through a simulated channel, and pairs the received audio, called a clip, with labels, a templated dialogue, and the source and channel conditions that produced it (Sections 3.2 and 3.3). A source-grouped partition, together with a balanced, fixed subset of the test partition, keeps training, validation, and evaluation data cleanly separated (Section 3.4). A gated coarse-to-fine dialogue assigns each error to a decision stage, and a paired silent-audio control tests whether the waveform contributes to a score (Sections 3.5 and 3.6). 3.1 Task Scope and Label Hierarchy L1L2L3 classes Active source category ActiveDetection pulseCW, LFM, HFM ActiveCommunication signal2FSK, 4FSK, BPSK, QPSK, OFDM Passive source category PassiveShip-radiated noiseCargo vessel, Cruise ship, Fishing vessel, Naval vessel, Underwater vehicle Table 2 UA-Bench label hierarchy. A source category, signal family, and specific class form each label path. UA-Bench poses recognition as a short dialogue about a single clip. The model receives the clipβs mono 16 kHz waveform with the first question and keeps it in the dialogue history, then answers three questions in turn. The first 5 Signal Synthesis Detection Pulses Communication Signals Ship-radiated Noise 1. Data Preparation Acoustic profile Location Depth Salinity, Temperature 3. Quality Control 2. Bellhop Underwater Channel Simulation Bottom reflection Surface reflection Direct path Active transmission Passive ship- radiated noise Receiver Sample Filtering Final output Arrival check Waveform check Valid sample No arrival Low SNR Filtered out Processed signal Retained audio JSON files Figure 3 Source generation and propagation in UA-Bench. Parameterized underwater signals are propagated with BELLHOP under sampled environmental and geometric conditions. Arrival and waveform checks determine the retained audio and metadata. two are multiple choice, answered with a single option letter: Turn 1 names the source category, and Turn 2 names a complete class path. The third asks for a brief explanation in free text. No task examples are given, and labels and metadata are withheld from the model. The answers follow a three-level hierarchy, which allows each failure to be assigned to a coarse or a fine decision. Every example carries a label pathν¦ = (ν¦ (1) , ν¦ (2) , ν¦ (3) )for source category, signal family, and specific class. At L1, Active denotes signals transmitted for detection or communication, and Passive denotes noise radiated by an operating vessel or underwater vehicle. At L2, the Active branch separates detection pulses from communication waveforms, and the Passive branch contains ship-radiated noise. These branches define the thirteen L3 classes in Table 2. Turn 1 determines L1. Turn 2 lists the complete paths of the branch reached at Turn 1, eight for Active and five for Passive, so a single letter fixes L2 and L3 together. Classification and explanation are scored separately (Section 3.5). The classes are defined by the generator. Active classes follow waveform structure. Vessel classes follow class-specific ranges of spectral level, shaft frequency, blade count, tonal frequency, and modulation strength. The underwater vehicle class, a generic propeller-driven platform, uses five or seven blades, tonal frequencies of 10β200 Hz, and principal modulation coefficients of 0.06β0.08, while the surface-vessel classes use three to five blades, tonal frequencies up to 1000 Hz, and larger coefficients, with overlapping ranges among them (Appendix A). UA-Bench therefore measures recognition of generator-defined categories under known conditions, not identification of real vessel types. 3.2 Controlled Source and Channel Synthesis To attribute a failure to signal structure or to propagation, the two have to be generated separately. Each clip is therefore built in three steps: a class-conditioned source, a sampled propagation channel, and output normalization. For sourceν and propagation realization ν , ν₯ ν = ν ν¦ (3) ν (ν ν ), ν’ νν =H ν νν [ν₯ ν ], ν νν = 0.95 ν’ νν β₯ν’ νν β₯ β ,(1) where ν ν holds the source parameters and ν νν the channel parameters: the sound-speed profile, the source and receiver geometry, and the bottom properties (Figure 3). 6 00.751.5 Aligned time (s) 4650 4800 4950 Frequency (Hz) (a) LFM Β· Source Unpropagated 00.751.5 Aligned time (s) 4650 4800 4950 Frequency (Hz) (b) LFM Β· ch0 r = 4.24 km 00.751.5 Aligned time (s) 4650 4800 4950 Frequency (Hz) (c) LFM Β· ch1 r = 4.24 km β50 β25 0 Power (dB) 03672 Window time (ms) 5200 5600 6000 Frequency (Hz) (d) 2FSK Β· Source Unpropagated 03672 Window time (ms) 5200 5600 6000 Frequency (Hz) (e) 2FSK Β· ch0 r = 2.45 km 03672 Window time (ms) 5200 5600 6000 Frequency (Hz) (f) 2FSK Β· ch1 r = 2.54 km β50 β25 0 Power (dB) 05001000 Frequency (Hz) β60 β45 β30 PSD (dB/Hz) (g) Cruise Β· Source Unpropagated 05001000 Frequency (Hz) β60 β45 β30 PSD (dB/Hz) (h) Cruise Β· ch0 r = 0.68 km 05001000 Frequency (Hz) β60 β45 β30 PSD (dB/Hz) (i) Cruise Β· ch1 r = 0.80 km Each WAV peak-normalized to 1; STFT 0 dB = maximum within each row. Received time is arrival-aligned; these panels compare structure, not propagation loss. Figure 4 Representative source and channel realizations. Each row follows one source waveform through two retained propagation realizations: (aβc) LFM, (dβf) 2FSK, and (gβi) cruise ship. The left column shows the unpropagated source; the middle and right columns show channel realizationsch0andch1, with source-to-receiver range reported in each panel. Received time axes are aligned to the first arrival, and each waveform is peak-normalized independently; the comparison therefore concerns temporal and spectral structure rather than absolute propagation loss. Spectrogram power is referenced to the maximum within each row. Source generation. All sources are synthesized at 16 kHz. CW pulses use a fixed carrier, and LFM and HFM pulses follow linear and hyperbolic frequency trajectories. Communication waveforms use binary and quaternary frequency-shift keying, binary and quadrature phase-shift keying, and orthogonal frequency-division multiplexing [10,11]. Carrier and center frequencies are drawn from subbands between 300 and 7500 Hz, while pulse width, sweep direction, symbol rate, symbol count, and modulation parameters vary across sources (Appendix A). A ship-radiated noise source combines a shaped continuous spectrum with discrete mechanical tones. The continuum is amplitude modulated at the shaft frequency, the blade-passing frequency, and one to four weaker modulation frequencies. Each source contains three to ten tones, of which one to three are tied to shaft or blade harmonics, and the ranges are class specific [16,17,26]. Continuum and tones are mixed before propagation, so both undergo the same channel transformation. Propagation. Each realization is simulated with BELLHOP [19]. The sound-speed profile is drawn from 100 annual-mean World Ocean Atlas profiles along a 15 km transect in the northern South China Sea. Source depth, receiver depth, and range are perturbed independently by up to 10%, and the seabed is silt in all simulations. Every source is propagated through up to two such realizations, each with its own deterministic random stream. BELLHOP returns the arrival amplitudes and delays, from which an impulse response is built (Appendix B). Pulse and communication signals are convolved with one impulse response, computed at the center or carrier frequency. Ship sources use responses at 50, 100, 200, 400, 800, and 1600 Hz, interpolated across frequency. The convolution is cropped to the source duration. Validation and normalization.A realization is retained only if it yields at least one usable arrival and a finite, nonzero waveform. A simulation that returns no arrivals is repeated once with a denser beam grid. Each retained waveform is then scaled to a peak amplitude of 0.95. This removes absolute level and keeps relative temporal and spectral structure, so 7 clips differ in structure rather than in propagation loss. The metadata record the source parameters, sampled environment, arrival information, delay reference, normalization gain, and duration. Figure 4 compares one source passed through two independent realizations. Multipath reshapes the local timeβfrequency pattern of the active signals and the spectral envelope of ship noise, while the class-defining structure stays recognizable. 3.3 QA Annotation Construction The synthesis stage produces clips with known label paths and source and channel conditions. We next pair each clip with questions and reference answers for the dialogue task defined in Section 3.1. The annotated dataset is D =(ν ν , y ν ,Q ν , m ν , ν ν ) ν ν=1 , Q ν = ((ν ν1 ,ν ν1 ),(ν ν2 ,ν ν2 ),(ν ν3 ,ν ν3 )). (2) Here theνretained clips are reindexed byν. The waveform is denoted byν ν , its three-level label path byy ν , its source and channel metadata bym ν , and its underlying source identifier byν ν . Each pair(ν νν‘ ,ν νν‘ )contains the question and reference answer for turn ν‘ β 1, 2, 3. Classification questions.The first two turns translate the label path into multiple-choice questions. Turn 1 asks for L1 using two source-category options. Turn 2 asks for L2 and L3 together by presenting eight complete paths on the Active branch or five on the Passive branch. Each question requires the uppercase letter of one option, and the reference letters are determined byy ν . The annotation procedure samples from eight Turn 1 templates, eight Active Turn 2 templates, and six Passive Turn 2 templates. The option order is fixed across templates. Reference explanations. The third turn links the class decision to acoustic structure by asking for a brief explanation of the preceding classification. Its reference answer is assembled from the source metadata and combines a source descriptor, a family descriptor, and three acoustic descriptors. Active-signal descriptors follow the generated waveform structure, with the LFM and HFM references also using the recorded sweep direction. Vessel descriptors use tonal extent, line count, harmonic tags, and modulation coefficients. Appendix D lists the templates. Record assembly.These annotations are packaged with each clip in a JSONL record containing its audio path, three-turn dialogue, canonical labels, source identity, and source and channel metadata. The annotation code maps every label to a valid three-level path; missing audio paths and unsupported label paths cause record construction to fail. At evaluation time, the model receives only the audio, stored questions, and earlier model answers. Ground-truth labels, reference answers, and metadata are withheld. The source identifierν ν is retained so that propagation realizations of the same source remain together when the records are partitioned in Section 3.4. 3.4 Dataset Partitions and Evaluation Selection The annotation stage produces records that are ready for use in the dialogue task, but multiple records can share one source waveform. We therefore split records by source identity to keep propagation variants of the same source from crossing training, validation, and test partitions. We then select a fixed, class-balanced subset of the test partition, UA-Bench-Eval, so that all models are compared on the same inputs with equal support for each leaf class. From sources to records. A source is a waveform before propagation; a record contains one retained clip and its annotations. The source collection contains 240,000 waveforms: 80,000 detection pulses, 80,000 communication waveforms, and 80,000 ship-radiated noise waveforms. The pulse set contains 24,000 CW signals and 28,000 signals for each chirp class. Each communication and vessel class contains 16,000 sources. Two configured propagation attempts per source allow at most 480,000 sourceβchannel realizations. After channel validation and annotation, UA-Bench contains 375,051 records. Each record represents one retained realization, so records that share a source must be grouped before splitting. Table 3 distinguishes the source waveforms, retained records, and selected evaluation examples. 8 StageCountUnit Source generation and propagation Configured sources240,000Waveforms before propagation Propagation attempts480,000Maximum at two realizations per source Released source-grouped records Retained records375,051Audio, labels, dialogue, and metadata Training262,580Records in training source groups Validation56,250Records in validation source groups Test56,221Records in test source groups UA-Bench-Eval2,600Selected test records, 200 per class Table 3 Dataset construction and partition sizes. Counts distinguish configured source waveforms, propagation attempts, retained audio records, source-grouped partitions, and the fixed evaluation subset. Source-grouped splitting.All records with the same source identifierν ν form one group and remain in the same partition. Within each L3 class, these groups are shuffled with seed 42 and allocated to training, validation, and test partitions in a nominal ratio of 70, 15, and 15 percent. The largest-remainder rule converts the ratios to integer group counts. Every record inherits the partition of its source. The resulting record counts are shown in Table 3, and UA-Bench-Eval is drawn only from the test partition. Class-specific ranking.To construct the evaluation subset, we rank test records within each leaf class using a statistic from the generation metadata. The ranking variable depends on the source branch. Active signals use the impulse- response energy gainνΊ β , computed before output normalization. Ship-radiated noise uses the source tonal-to-continuum power ratio ν src , computed before propagation: νΊ β = 10 log 10 βοΈ ν |β[ν]| 2 , ν src = 10 log 10 mean ν ν₯ tonal [ν] 2 mean ν ν₯ continuum [ν] 2 .(3) Hereβ[ν]is the active-signal channel impulse response,ν₯ tonal [ν]is the complete tonal component of a ship source, and ν₯ continuum [ν]is its modulated continuous component;νindexes discrete-time samples. The ship statistic compares the mean-square powers of these two complete source components. It differs from the local line-to-continuum ratio used to set each tone during synthesis. Appendix C defines both quantities. Fixed evaluation subset.Within each leaf class, test records are ordered by decreasingνΊ β orν src , with sample identifiers resolving ties. The first 400 records form a ranked candidate pool. We sample 200 records without replacement from each pool with seed 42. This gives 2,600 evaluation records with equal support across the thirteen leaf classes: 600 detection-pulse records, 1,000 communication-signal records, and 1,000 ship-radiated noise records. The selection code verifies record identifiers, label paths, metadata, audio paths, and per-class counts. Properties of the selected inputs. Ranking and sampling determine both the class counts and the acoustic conditions represented in UA-Bench-Eval. Figure 5 compares the selected records with the full test partition. The selectedνΊ β values for active signals andν src values for ship noise are concentrated at larger values, and active center or carrier frequencies shift upward. Duration changes depend on the signal family: selected pulses are more concentrated at shorter durations, while the communication and ship-noise curves remain closer to their full-test counterparts. The appendix galleries show the acoustic structures present in these evaluation inputs. Appendix Figure 10 shows stationary, linear, and curved pulse trajectories, together with discrete frequency switching and multicarrier occupancy. The power-only views of BPSK and QPSK are not interpreted as direct measurements of their phase states. Appendix Figure 11 shows low-frequency tonal and continuous ship-noise structure. The displayed spectral-centroid intervals overlap strongly across the five ship-noise leaf classes. With the evaluation inputs fixed, Section 3.5 defines how their stored dialogues are executed and how the classification decisions are scored. 9 1eβ01110 Duration (s) 0.00 0.25 0.50 0.75 1.00 ECDF (a) Audio duration Valid n: 56,221 / 2,600 30006000 Signal center/carrier frequency (Hz) 0.00 0.25 0.50 0.75 1.00 ECDF (b) Active signal frequency Valid n: 32,223 / 1,600 β180β120β60 Channel energy gain, G h (dB) 0.00 0.25 0.50 0.75 1.00 ECDF (c) Active channel gain Valid n: 32,223 / 1,600 β24β16β8 Source tonal-to-continuum power ratio, S src (dB) 0.00 0.25 0.50 0.75 1.00 ECDF (d) Ship source power ratio Valid n: 23,998 / 1,000 PulseCommunicationShip noiseFull testUA-Bench-Eval Figure 5 Distributions in the full test partition and UA-Bench-Eval. Empirical cumulative distribution functions compare (a) audio duration, (b) active-signal center or carrier frequency, (c) active channel energy gainνΊ β , and (d) ship source tonal-to-continuum power ratioν src . Colors identify signal families; dashed and solid curves denote the full test partition and UA-Bench-Eval, respectively. Valid record counts are reported as full test / UA-Bench-Eval. 3.5 Gated Evaluation Protocol and Classification Metrics Each model receives the audio and stored questions from UA-Bench-Eval without task examples in the prompt. The first two turns yield predictions at the three label levels; the third supplies an explanation for separate analysis. Gated execution. At Turn 1, the evaluator compares the predicted source category with L1. An invalid-format or incorrect answer ends the dialogue and assignscascadeerrorto L2 and L3. A correct answer opens the corresponding branch at Turn 2, where one complete-path option determines both L2 and L3. Turn 3 is executed for every example that passes Turn 1, including examples with an invalid-format or incorrect Turn 2 answer. Answer decoding. For the two classification turns, the decoder removes surrounding whitespace and accepts only a single ASCII letter that matches a displayed option, treating lowercase and uppercase forms equivalently. Free-form text and expanded class names are not recovered. The raw answer and decoding status are preserved for error analysis. Appendix Figure 12 illustrates a 2FSK dialogue. Correct source routing opens the Active branch, where option D identifies the Communication family and the 2FSK leaf; Turn 3 then requests an acoustic explanation. Accuracy and conditional accuracy.For this evaluation,ν =2,600 andνindexes the records in UA-Bench-Eval. Let ν (β) ν be one when the prediction at label levelβ β 1,2,3matches its reference label and zero otherwise. Unresolved and cascade outcomes count as incorrect. Defineν§ (β) ν = Γ β ν=1 ν (ν) ν to indicate that the full path through levelβis correct. 10 The metrics are Acc νΏβ = 1 ν ν βοΈ ν=1 ν (β) ν ,(4) Acc(νΏβ | parent path) = Γ ν ν§ (β) ν Γ ν ν§ (ββ1) ν (β = 2, 3).(5) We abbreviate the conditional quantities as L2|L1 and L3|L2. The first conditions on correct L1; the second conditions on correct L1 and L2. An empty conditional denominator yields a score of zero, following the convention in Appendix F. Class balance and failure outcomes.L3 Macro F1 averages the F1 scores of the thirteen leaf classes. Per-class precision, recall, and confusion matrices summarize class-specific errors. Invalid-format and cascade outcomes contribute false negatives to their true classes and appear as separate prediction columns; they are not additional classes in the Macro F1 average. For stage-wise analysis, we partition records into five mutually exclusive outcomes: invalid-format or valid-but-wrong answers at Turn 1, the same two outcomes at Turn 2 after a correct route, and correct L3 decisions. Under the execution gate and strict decoder, correct L3 implies correct L1 and L2, so a separate joint score would duplicate L3 accuracy. Appendix F gives the numerical conventions. Random reference. The random reference chooses L1 uniformly and then chooses a leaf uniformly from the selected branch. The balanced evaluation set has eight Active and five Passive leaf classes, so its expected leaf accuracy is E[Acc νΏ3 ] = 8 13 1 2 1 8 + 5 13 1 2 1 5 = 1 13 β 7.69%.(6) Including the probability of choosing the correct source category, the expected leaf accuracies are 6.25% on Active examples and 10% on Passive examples. A predictor that always selects Active has an L1 accuracy of 8/13, or 61.54%. Appendix F derives the family-level and conditional references. 3.6 Diagnostic Analyses We supplement classification scores with a paired audio control, analysis across generation conditions, and a check of explanation terminology. Silent-audio control.Each evaluation record is paired with an equal-duration silent waveform while its questions and labels remain fixed. We compare L3 accuracy and Macro F1 across the two conditions and report original-minus-silent differences. We obtain 95% percentile intervals for the scores and differences by bootstrapping source clusters, keeping channel variants of each source together. Appendix F gives the resampling settings and additional interval checks. Metadata-conditioned performance.Within each leaf class, we divide examples into low, middle, and high strata using empirical tertiles ofνΊ β for active signals orν src for ship-radiated noise. The thresholds are fixed across models. For each stratum, we report L3 recall averaged with equal leaf weight within each branch. Appendix G gives the quantile conventions. Lexical explanation consistency.Each executed Turn 3 response is matched against a supporting set of terms compatible with the predicted leaf and a comparison set containing the remaining inventory terms. Shared terms support every compatible leaf. The retained matches define four categories: Aligned for supporting terms only, Concept Confusion for comparison terms only, Vague for neither, and Term Stacking for both. The contradiction rate is the sum of the confusion and stacking rates. All rates use the number of executed Turn 3 responses for that model as the denominator. Appendix E specifies the inventory, matching rules, and treatment of invalid outputs and empty responses. The rates describe consistency with the predicted class at the level of terminology. 11 4 Experiments We evaluate recognition without task examples across detection pulses, communication waveforms, and ship-radiated noise. The analysis begins with performance over the complete label hierarchy. It then examines source families, decision-stage errors, metadata strata, and explanation vocabulary. This order connects the aggregate scores with the predictions from which they are derived. 4.1 Experimental Setup Models and inference. We evaluate nine released instruction-following checkpoints. They are Gemma-4-12B-IT and Gemma-4-E2B-IT [29], Audio-Flamingo-Next [30], MiDashengLM-7B-1021 [31], Voxtral-Small-24B-2507 [32], Voxtral-Mini-3B-2507 [33], Aero-1-Audio [34], Qwen2.5-Omni-7B [35], and Qwen2-Audio-7B-Instruct [36]. Each model uses its native audio processor and chat template. The audio is attached to the first user turn and retained in the dialogue history. Text generation uses greedy decoding when the backend supports it. Each turn permits at most 256 new tokens, and audio generation is disabled. Appendix G lists the checkpoints and settings. Evaluation records.All results use the fixed UA-Bench-Eval suite defined in Section 3.4. The training and validation partitions are not used for model evaluation. All nine checkpoints have complete, integrity-verified strict-decoder runs and paired silent-audio controls. Run-specific batch sizes and software environments are preserved in the protocol records, so the reported comparisons should be interpreted as checkpoint-configuration results rather than architecture-only effects. 4.2 Overall and Family Performance Table 4 reports accuracy at all three levels, L3 Macro F1, and two conditional accuracies. The level accuracies measure predictions at individual levels. The conditional scores separate source routing from classification within a correctly reached path. Macro F1 measures the balance of terminal predictions across the thirteen leaves. A separate joint column is omitted because, under the strict gated decoder, a correct L3 decision necessarily identifies the complete path. Model Classification qualityConditional accuracy L1 Acc.βL2 Acc.βL3 Acc.βMacro F1βL2|L1βL3|L2β Gemma-4-12B-IT [29]27.69.32.00.02333.921.0 Gemma-4-E2B-IT [29]51.630.76.50.04959.521.2 Audio-Flamingo-Next [30]87.640.38.10.06146.020.0 MiDashengLM-7B-1021 [31]57.547.010.30.08381.721.9 Voxtral-Small-24B-2507 [32]56.826.97.70.05647.428.7 Voxtral-Mini-3B-2507 [33]59.125.16.10.04642.524.2 Aero-1-Audio [34]61.324.3 6.90.03339.628.3 Qwen2.5-Omni-7B [35]72.554.313.10.08274.824.2 Qwen2-Audio-7B-Instruct [36]38.538.57.30.063100.019.0 Gated random reference50.035.67.7N/A71.221.6 Table 4 Hierarchical classification on 2,600 examples without task demonstrations. All rows use the strict single-letter decoder. Accuracy values are percentages, and Macro F1 is reported on a scale from 0 to 1. Bold values mark the best model result, and underlining marks the second-best result. The shaded columns are the primary terminal-level metrics. Across the nine strict runs, L3 accuracy ranges from 2.0% to 13.1%. Qwen2.5-Omni obtains the highest accuracy, 5.4 percentage points above the gated random reference after rounding. MiDashengLM has the highest Macro F1 at 0.083, narrowly ahead of Qwen2.5-Omni at 0.082. Gemma-4-12B has the lowest L3 accuracy; its result is partly shaped by 1,156 Turn 1 refusals stating that no audio was available, even though the corresponding inputs passed the dataset and pipeline checks. Earlier levels produce different rankings. Audio-Flamingo-Next has the highest L1 accuracy at 87.6%, whereas Qwen2.5-Omni leads at L2 with 54.3%. Qwen2-Audio attains 100.0% L2|L1 because every correctly routed example also receives the correct family label, but its L1 accuracy is only 38.5%. Voxtral-Small instead has the highest L3|L2 12 Cascade CW LFM HFM 2FSK4FSK BPSK QPSK OFDM Cargo Cruise Fishing Naval Underw. Unres. Predicted class CW LFM HFM 2FSK 4FSK BPSK QPSK OFDM Cargo Cruise Fishing Naval Underwater True class 363230 373428 393426 6526 6822 6827 5630 5630 223036 223434 282733 303231 272832 (a) Confusion 0.00.10.20.30.4 Score CW LFM HFM 2FSK 4FSK BPSK QPSK OFDM Cargo Cruise Fishing Naval Underwater (b) Class metrics PRF1 0%50%100% True-class share (%) Figure 6 Class-level diagnostics for the strict MiDashengLM-7B-1021 run on 2,600 examples. (a) Row-normalized confusion matrix over the thirteen leaf classes, with separate columns for cascaded (Cascade) and unresolved (Unres.) predictions. Colors encode true-class shares; values of at least 20% are annotated as percentages rounded to integers. (b) Per-class precision (filled circles), recall (open triangles), and F1 (filled squares). The shortened labels Cargo, Cruise, Fishing, and Naval denote the corresponding taxonomy classes; both Underwater and Underw. denote Underwater vehicle. Model L3 accuracy by source family Detection pulse 3 classes, ν = 600 Communication waveform 5 classes, ν = 1,000 Ship-radiated noise 5 classes, ν = 1,000 Gemma-4-12B-IT [29]4.51.01.4 Gemma-4-E2B-IT [29]4.312.02.3 Audio-Flamingo-Next [30]0.06.2 14.8 MiDashengLM-7B-1021 [31]10.85.215.1 Voxtral-Small-24B-2507 [32]23.21.44.8 Voxtral-Mini-3B-2507 [33]16.82.23.5 Aero-1-Audio [34]22.54.40.0 Qwen2.5-Omni-7B [35]15.05.319.8 Qwen2-Audio-7B-Instruct [36]0.00.019.0 Gated random reference6.256.2510.0 Table 5 L3 accuracy by source family under the strict single-letter decoder. Bold values mark the best model result, and underlining marks the second-best result. value at 28.7%, narrowly followed by Aero-1-Audio at 28.3%. These differences show why reporting the intermediate routing metrics remains useful even though L3 accuracy already represents complete-path correctness under the strict protocol. Table 5 separates terminal accuracy across the three source families. Detection pulses, communication waveforms, and ship-radiated noise contain 600, 1,000, and 1,000 examples. Their different best-performing models reveal that aggregate scores combine distinct acoustic strengths. 13 0%25%50%75%100% All evaluation records (%) Qwen2-Audio Aero1-Audio Voxtral-Mini AF-Next Gemma4-12B Gemma4-E2B MiDaShengLM Qwen2.5-Omni Voxtral-Small Model 44.5 9.7 9.4 61.5 38.0 40.9 9.0 28.0 38.7 40.6 27.5 33.8 32.3 31.2 54.4 53.0 47.2 25.2 45.1 47.2 59.4 49.0 8.1 10.3 13.1 7.7 n = 2,600 n = 2,600 n = 2,600 n = 2,600 n = 2,600 n = 2,600 n = 2,600 n = 2,600 n = 2,600 Five mutually exclusive outcomes from saved parse and execution states T1 invalid T1 wrong T2 invalid T2 leaf wrong Leaf correct Figure 7 Decision-stage error decomposition under the strict single-letter decoder. Each bar partitions 2,600 examples into invalid-format and valid-but-incorrect outcomes at Turn 1, the corresponding outcomes at Turn 2 after a correct route, and correct L3 decisions. The five segments reproduce the counts in Table 6 as percentages. Model Turn 1 failureTurn 2 failureSuccess Invalid formatValid but wrongInvalid formatValid but wrongL3 correct Gemma-4-12B-IT [29]11567271065651 Gemma-4-E2B-IT [29]252100601173169 Audio-Flamingo-Next [30]902338411226210 MiDashengLM-7B-1021 [31]49105601227268 Voxtral-Small-24B-2507 [32]24587901275201 Voxtral-Mini-3B-2507 [33]0106401378158 Aero-1-Audio [34]1898801415179 Qwen2.5-Omni-7B [35]071401545341 Qwen2-Audio-7B-Instruct [36]016000810190 Table 6 Decision-stage outcomes under the strict single-letter protocol. Counts sum to 2,600 in every row. The shaded column is the terminal success outcome. Voxtral-Small has the highest result on detection pulses, Gemma-4-E2B-IT on communication waveforms, and Qwen2.5- Omni on ship-radiated noise. None of these models maintains its lead across the other two families. The strict runs are also highly concentrated: Aero-1-Audio reaches 61.5% recall on CW but zero recall on every ship-noise leaf, Qwen2.5-Omni reaches 91.5% on Naval vessel, and Qwen2-Audio obtains all of its correct terminal predictions on ship-noise examples. The family averages also conceal substantial variation among leaf classes. Figure 6 illustrates this variation for the strict MiDashengLM-7B-1021 run. Correct L3 predictions occur only for CW, 2FSK, Fishing vessel, Naval vessel, and Underwater vehicle; the remaining eight classes have zero recall. Cascade outcomes account for 53.0% of active-signal examples and 25.7% of ship-noise examples. Among correctly routed examples, active-signal predictions concentrate on CW and 2FSK, while ship-noise predictions concentrate on Naval vessel and Underwater vehicle. These results reveal both source-routing failures and limited discrimination among leaf classes within the correctly reached branches. 14 The strict decoder distinguishes invalid-format answers from valid but incorrect decisions. Table 6 partitions each run into five outcomes: two at source routing, two at terminal classification after a correct route, and a correct terminal decision. Because every valid Turn 2 option identifies a complete path, no separate leaf-only category is needed. The strict-run error composition varies substantially. Gemma-4-12B produces 1,156 invalid-format Turn 1 answers, all of which are no-audio refusals, and only 51 correct terminal predictions. Audio-Flamingo-Next instead reaches Turn 2 more often but produces 841 invalid-format answers at that stage. Qwen2.5-Omni reaches Turn 2 on 1,886 examples without an invalid-format decision and produces 341 correct terminal predictions. Voxtral-Mini also produces no invalid-format answer, yet 1,378 examples are assigned a valid but incorrect terminal option. These differences illustrate why formatting failures and valid but incorrect decisions must be reported separately. Figure 7 visualizes the five strict-decoder outcomes in Table 6. It shows that the dominant failure mode differs across checkpoints: Gemma-4-12B is dominated by Turn 1 invalid-format responses, Audio-Flamingo-Next has a substantial Turn 2 invalid-format segment, and several other models primarily produce valid but incorrect decisions. The terminal-success segment remains small for every model. 4.3 Associations with Generation Metadata Table 7 reports the metadata-stratified analysis defined in Section 3.6. Active records are grouped byνΊ β , while ship-radiated noise records are grouped byν src . Thresholds are computed within each leaf class and shared across models. Recall is first computed for each leaf and stratum. It is then averaged with equal leaf weight within each branch. The final column in each group reports the high-minus-low difference. Model Active signals using νΊ β Ship-radiated noise using ν src LowMiddleHighΞLowMiddleHighΞ Gemma-4-12B-IT [29]1.51.73.7+2.20.92.11.2+0.3 Gemma-4-E2B-IT [29]6.910.410.1+3.21.53.02.4+0.9 Audio-Flamingo-Next [30]4.33.63.7β0.614.314.515.5+1.2 MiDashengLM-7B-1021 [31]6.76.48.8+2.115.514.515.2β0.3 Voxtral-Small-24B-2507 [32]9.59.79.50.05.13.65.7+0.6 Voxtral-Mini-3B-2507 [33]8.49.35.4β3.03.64.22.7β0.9 Aero-1-Audio [34]10.611.011.9+1.30.00.00.00.0 Qwen2.5-Omni-7B [35]9.38.78.8β0.619.719.420.3+0.6 Qwen2-Audio-7B-Instruct [36]0.00.00.0 0.016.419.421.2+4.8 Table 7 L3 recall across within-class metadata strata under the strict single-letter decoder. Recall values are percentages.Ξis the high-minus-low difference in percentage points. The shaded columns show the high-minus-low effects. The strata do not show a shared monotonic pattern. From the low to the high Active stratum, the endpoint difference is +3.2 percentage points for Gemma-4-E2B andβ3.0 points for Voxtral-Mini; both models deviate from a monotonic trend in the middle stratum. Qwen2-Audio has zero Active recall, whereas its ship-noise recall rises from 16.4% in the low stratum to 21.2% in the high stratum. The remaining models show smaller or non-monotonic endpoint differences. Figure 8 visualizes the same strict-run values and makes the lack of a common trend apparent across checkpoints. 4.4 Lexical Explanation Analysis Table 8 reports the lexical categories defined in Section 3.6. The denominator for each row is the number of Turn 3 responses executed after a correct Turn 1 answer. The four categories partition these responses and measure lexical consistency with the predicted leaf. The contradiction rate combines Concept Confusion and Term Stacking. 15 Low MiddleHigh Within-class stratum 0% 5% 10% 15% Class-balanced recall (%) (a) Active: G h Ξ 10.6 p Ξ 11.0 p Ξ 11.9 p Low MiddleHigh Within-class stratum 0% 5% 10% 15% 20% 25% Class-balanced recall (%) (b) Ship noise: S src Ξ 19.7 p Ξ 20.1 p Ξ 21.0 p Independent y ranges; shading: model minβmax; Ξ: gap (p) Archived predictions checked against their saved manifest and protocol Qwen2-AudioVoxtral-MiniGemma4-12BMiDaShengLM Aero1-AudioAF-NextGemma4-E2BQwen2.5-OmniVoxtral-Small Figure 8 Metadata-conditioned class-balanced recall under the strict single-letter decoder. Recall is averaged with equal class weight across (a) the eight active leaves stratified by channel energy gainνΊ β and (b) the five ship-noise leaves stratified by source tonal-to-continuum power ratio ν src . The plotted values reproduce Table 7. Modelν Without competing termsWith competing termsSummary AlignedβVagueβ Concept confusionβ Term stackingβ Contradiction rateβ Gemma-4-12B-IT [29]71787.66.05.21.36.4 Gemma-4-E2B-IT [29]134237.659.92.20.22.5 Audio-Flamingo-Next [30]227720.129.646.14.250.3 MiDashengLM-7B-1021 [31]149529.766.11.62.64.2 Voxtral-Small-24B-2507 [32]147661.05.622.810.533.3 Voxtral-Mini-3B-2507 [33]153662.617.817.81.819.6 Aero-1-Audio [34]15946.871.321.50.421.9 Qwen2.5-Omni-7B [35]188658.425.36.79.616.3 Qwen2-Audio-7B-Instruct [36]1000 37.88.919.234.153.3 Table 8 Lexical consistency with predicted leaf classes under the strict single-letter decoder. The four response categories sum to 100% up to rounding. Contradiction is the sum of Concept Confusion and Term Stacking. Bold values mark the best model result, and underlining marks the second-best result. Gemma-4-12B has the highest aligned rate at 87.6%, followed by Voxtral-Mini at 62.6%. This rate is conditional on an executed Turn 3 response: Gemma-4-12B contributes only 717 eligible responses because of its large Turn 1 failure count, so the high conditional rate does not imply reliable end-to-end performance. Voxtral-Small has the lowest vague rate at 5.6%, while MiDashengLM has the lowest concept-confusion rate at 1.6%. Lexical alignment therefore varies independently of terminal accuracy and captures a separate property of model responses. 4.5 Silent-Audio Control Table 9 reports the paired original-versus-silent analyses for all nine checkpoints. Each silent waveform has the same duration as its corresponding original input, while the questions and labels remain unchanged. The primary quantity is the original-minus-silent difference: a positive value indicates higher performance with the original waveform than with its paired silent input. The table reports source-cluster bootstrap 95% confidence intervals; Wilson accuracy intervals and the analytic paired-difference interval are retained in the machine-readable report as cross-checks. 16 0481216 Accuracy (%) Qwen2-Audio Aero1-Audio Voxtral-Mini AF-Next Gemma4-12B Gemma4-E2B MiDaShengLM Qwen2.5-Omni Voxtral-Small -0.42 -0.38 -1.42 +2.35 +0.23 +0.69 0.00 +4.73 +2.15 Ξ (p) (a) Leaf recognition 0102030 Difference (p) 0.00 38.46/38.46 -0.12 61.31/61.42 0.00 59.08/59.08 +29.54 87.58/58.04 +13.38 27.58/14.19 +0.62 51.62/51.00 0.00 57.50/57.50 +21.19 72.54/51.35 +9.42 56.77/47.35 Ξ (p)O/S (%) (b) Routing change Ξ = original β silence Original audioMatched silence Figure 9 Paired original-versus-silent control at the leaf and routing levels on UA-Bench-Eval. Each checkpoint is evaluated on 2,600 originalβsilent input pairs. (a) L3 leaf-recognition accuracy with original audio (filled blue circles) and duration-matched silence (open orange squares); lines connect the two accuracies for each checkpoint. (b) Original-minus-silent differences in L1 routing accuracy; O/S lists the original/silent accuracies in percent. AllΞvalues are computed before rounding and expressed in percentage points (p). The figure shows point estimates without confidence intervals; L3 accuracy intervals are reported in Table 9. Model L3 accuracyL3 Macro F1 Original SilentΞ95% CIOriginal SilentΞ95% CI Gemma-4-12B-IT [29]2.01.7+0.2[β0.4,+0.8]0.0230.023β0.001[β0.009,+0.007] Gemma-4-E2B-IT [29]6.55.8 +0.7[+0.04,+1.32]0.0490.046+0.003[β0.004,+0.010] Audio-Flamingo-Next [30]8.15.7+2.3[+1.3,+3.4]0.0610.052+0.009[+0.000,+0.018] MiDashengLM-7B-1021 [31]10.310.30.0[0.0, 0.0]0.0830.0830.000[0.000, 0.000] Voxtral-Small-24B-2507 [32]7.75.6+2.2[+1.2,+3.1]0.0560.035+0.020[+0.012,+0.028] Voxtral-Mini-3B-2507 [33]6.17.5β1.4[β2.2,β0.7]0.0460.048β0.002[β0.007,+0.003] Aero-1-Audio [34]6.97.3β0.4[β0.9,+0.2]0.0330.028+0.005[+0.001,+0.009] Qwen2.5-Omni-7B [35]13.18.4+4.7[+3.5,+6.0]0.0820.059+0.023[+0.013,+0.033] Qwen2-Audio-7B-Instruct [36]7.37.7β0.4[β1.1,+0.3]0.0630.064β0.002[β0.008,+0.005] Table 9 Paired original-versus-silent control on UA-Bench-Eval. All pairs pass the integrity checks. Differences are original minus silent; bracketed ranges are source-cluster bootstrap 95% confidence intervals based on 10,000 source-level resamples. Accuracy is reported in percent, with differences in percentage points, and Macro F1 on a scale from 0 to 1. The shaded column pairs report the paired effects and their confidence intervals. Qwen2.5-Omni shows the largest original-minus-silent L3 accuracy difference at 4.7 percentage points, followed by Audio-Flamingo-Next at 2.3 points and Voxtral-Small at 2.2 points. Gemma-4-E2B has a smaller difference of 0.7 points. The cluster-bootstrap intervals for all four positive L3 accuracy differences exclude zero; the Gemma-4-E2B interval is shown to two decimal places so that its positive lower bound remains visible. Voxtral-Mini instead has a β1.4-point difference with an interval ofβ2.2 toβ0.7 points, indicating higher L3 accuracy on silence in this diagnostic comparison. Gemma-4-12B, Aero-1-Audio, and Qwen2-Audio have small L3 accuracy differences whose intervals include zero, while MiDashengLMβs L3 accuracy and Macro F1 are unchanged. Figure 9 extends this comparison across the label hierarchy, juxtaposing end-to-end L3 leaf-recognition accuracy with coarse L1 routing accuracy for the same nine checkpoints. The point estimates show that larger gains in L1 routing can coexist with much smaller gains in end-to-end L3 leaf recognition. Audio-Flamingo-Next has an L1 accuracy difference of+29.54 percentage points, compared with+2.35 points at L3; the corresponding differences for Qwen2.5-Omni are+21.19 and+4.73 points. Gemma-4-12B provides a further contrast: its L1 difference is+13.38 points, whereas its L3 difference is only+0.23 points and the L3 confidence 17 interval includes zero. The L1 comparisons are descriptive; no confidence intervals for these differences are shown in the figure. Together, Table 9 and Figure 9 indicate that the observed contribution of the waveform depends on both the model and the level of the label hierarchy. These conclusions are specific to the evaluated configurations and should not be generalized beyond this paired control. 5 Discussion and Conclusion In this work, we connect hierarchical underwater acoustic recognition with controlled source and propagation metadata. Our benchmark combines thirteen leaf classes, source-grouped partitions, and a fixed evaluation suite within one workflow. We use a gated protocol to measure source routing and leaf classification, and we relate model predictions to signal families, decision stages, generation metadata, and explanation vocabulary. The evaluated models show limited fine-grained recognition on UA-Bench-Eval. Across the nine strict-decoder runs, L3 accuracy ranges from 2.0% to 13.1%, and the highest value is 5.4 percentage points above the gated random reference after rounding. MiDashengLM achieves the highest Macro F1 at 0.083, narrowly ahead of Qwen2.5-Omni at 0.082. The model rankings also change across the hierarchy and source families. Voxtral-Small leads on detection pulses, Gemma-4-E2B-IT leads on communication waveforms, and Qwen2.5-Omni leads on ship-radiated noise. No evaluated model provides a consistent advantage across all three families. Prediction-level results explain part of this variation. Some models fail because their source-routing answer has an invalid format or selects the wrong branch, while others reach the terminal decision but concentrate their predictions on a small number of leaves. Under the strict single-letter decoder, a correct terminal option identifies the full hierarchical path, which makes the stage-wise error partition directly interpretable. Explanation vocabulary supplies another view of response behavior: Gemma-4-12B has the highest conditional lexical alignment rate, but only 717 responses enter this analysis because its Turn 1 failure count is large. The paired silent-audio results further separate model behaviors. Qwen2.5-Omni, Audio-Flamingo-Next, Voxtral-Small, and Gemma-4-E2B show positive L3 differences whose cluster-bootstrap intervals exclude zero, whereas Voxtral-Mini performs better on silence in this diagnostic comparison. The remaining models show small or zero L3 differences. These outcomes provide evidence about model behavior under the present protocol, not a general claim that audio is or is not useful. The benchmark represents a synthetic distribution defined by its source ranges, propagation settings, evaluation selection, questions, and decoder. Field recordings will be needed to measure transfer beyond this setting. Future work can also expand environmental diversity, add independently validated labels, and compare model decisions under controlled changes to the audio. We provide UA-Bench as a common basis for these studies and for tracking progress in hierarchical underwater acoustic recognition. 18 References [1]Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. [2]Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. ICLR, 2024. [3]Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406.11768, 2024. [4]Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. ACL, 2024. [5]S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. ICLR (Spotlight), 2025. [6] Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, et al. Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. ICLR, 2025. [7]David Santos-Dom Μ Δ±nguez, Soledad Torres-Guijarro, Antonio Cardenal-L Μ opez, and Antonio Pena-Gimenez. Shipsear: An underwater vessel noise database. Applied Acoustics, 113:64β69, 2016. doi: 10.1016/j.apacoust.2016.06.008. [8]Muhammad Irfan, Jiangbin Zheng, Shahid Ali, Muhammad Iqbal, Zafar Masood, and Umar Hamid. Deepship: An underwater acoustic benchmark dataset and a separable convolution based autoencoder for classification. Expert Systems with Applications, 183:115270, 2021. doi: 10.1016/j.eswa.2021.115270. [9] Zeyu Li, Suncheng Xiang, Tong Yu, Jingsheng Gao, Jiacheng Ruan, Yanping Hu, Ting Liu, and Yuzhuo Fu. Oceanship: A large-scale dataset for underwater audio target recognition. In Advanced Intelligent Computing Technology and Applications, pages 475β486. Springer, 2024. [10] Yves Doisy, Laurent Deruaz, Sander P. van IJsselmuide, Sander P. Beerens, and Robert Been. Reverberation suppression using wideband doppler-sensitive pulses. IEEE Journal of Oceanic Engineering, 33(4):419β433, 2008. doi: 10.1109/JOE.2008. 2002582. [11]Milica Stojanovic. Recent advances in high-speed underwater acoustic communications. IEEE Journal of Oceanic Engineering, 21(2):125β136, 1996. doi: 10.1109/48.486787. [12]Milica Stojanovic and James Preisig. Underwater acoustic communication channels: Propagation models and statistical characterization. IEEE Communications Magazine, 47(1):84β89, 2009. doi: 10.1109/MCOM.2009.4752682. [13]Xiaodong Cui, Zhuofan He, Yangtao Xue, Keke Tang, Peican Zhu, and Jing Han. Cross-domain contrastive learning- based few-shot underwater acoustic target recognition. Journal of Marine Science and Engineering, 12(2):264, 2024. doi: 10.3390/jmse12020264. [14]Xiaoyang Du and Feng Hong. Qiandaoear22: A high-quality noise dataset for identifying specific ship from multiple underwater acoustic targets using ship-radiated noise. EURASIP Journal on Advances in Signal Processing, 2024(1):96, 2024. doi: 10.1186/s13634-024-01181-9. [15]Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo, Chao-Han Huck Yang, and Gunhee Kim. WoW-Bench: Evaluating fine-grained acoustic perception in audio-language models via marine mammal vocalizations. arXiv preprint arXiv:2508.20976, 2025. URL https://arxiv.org/abs/2508.20976. [16] Donald Ross. Mechanics of Underwater Noise. Pergamon Press, New York, 1976. [17] Stephen C. Wales and Richard M. Heitmeyer. An ensemble source spectra model for merchant ship-radiated noise. The Journal of the Acoustical Society of America, 111(3):1211β1231, 2002. doi: 10.1121/1.1427355. [18]Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li, Ke-Han Lu, and Hung-yi Lee. All that glitters is not audio: Rethinking text priors and audio reliance in audio-language evaluation. arXiv preprint arXiv:2604.24401, 2026. [19] Michael B Porter. The bellhop manual and userβs guide: Preliminary draft. Heat, Light, and Sound Research, Inc., 2011. 19 [20]Alexey V. Mishonov, Christopher R. Paver, Dan Seidov, Zhankun Wang, and Dmitry Dukhovskoy. World ocean atlas 2023. NOAA National Centers for Environmental Information, Dataset, 2023. [21]Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. AudioBench: A universal benchmark for audio large language models. In Proceedings of NAACL-HLT, pages 4297β4316, 2025. doi: 10.18653/v1/2025.naacl-long.218. URL https://aclanthology.org/2025.naacl-long.218/. [22] Tony Lee, Haoqin Tu, Chi Heem Wong, Zijun Wang, Siwei Yang, Yifan Mai, Yuyin Zhou, Cihang Xie, and Percy Liang. AHELM: A holistic evaluation of audio-language models. arXiv preprint arXiv:2508.21376, 2025. URLhttps://arxiv. org/abs/2508.21376. [23]Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. VoiceBench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196, 2024. URL https://arxiv.org/abs/2410.17196. [24]David Robinson, Adelaide Robinson, and Lily Akrapongpisak. Transferable models for bioacoustics with human language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1316β1320, 2024. doi: 10.1109/ICASSP48485.2024.10447250. [25]Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Yasunori Ohishi, and Noboru Harada. Assessing the utility of audio foundation models for heart and respiratory sound analysis. In 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 1β4, 2025. doi: 10.1109/EMBC58623.2025.11254427. [26]Paul T. Arveson and David J. Vendittis. Radiated noise characteristics of a modern cargo ship. The Journal of the Acoustical Society of America, 107(1):118β129, 2000. doi: 10.1121/1.428344. [27]Michael B. Porter and Homer P. Bucker. Gaussian beam tracing for computing ocean acoustic fields. The Journal of the Acoustical Society of America, 82(4):1349β1359, 1987. doi: 10.1121/1.395269. [28]Michael B. Porter and Edward L. Reiss. A numerical method for ocean-acoustic normal modes. The Journal of the Acoustical Society of America, 76(1):244β252, 1984. doi: 10.1121/1.391101. [29] Gemma Team. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026. [30]Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Zhifeng Kong, Siddharth Gururani, Sang-gil Lee, Jaehyeon Kim, Aya Aljafari, et al. Audio flamingo next: Next-generation open audio-language models for speech, sound, and music. arXiv preprint arXiv:2604.10905, 2026. [31] Heinrich Dinkel, Gang Li, Jizhong Liu, Jian Luan, Yadong Niu, Xingwei Sun, Tianzi Wang, Qiyang Xiao, Junbo Zhang, and Jiahao Zhou. Midashenglm: Efficient audio understanding with general audio captions. arXiv preprint arXiv:2508.03983, 2025. [32]Mistral AI. Voxtral-small-24b-2507. Hugging Face model repository, 2025. URLhttps://huggingface.co/mistralai/ Voxtral-Small-24B-2507. Accessed 2026-10-09. [33] Mistral AI. Voxtral-mini-3b-2507: A compact audio-language model. Hugging Face model repository, 2025. URL https://huggingface.co/mistralai/Voxtral-Mini-3B-2507. Accessed 2026-10-09. [34] LMMs-Lab. Aero-1-audio: An open audio-language model. Hugging Face model repository, 2025. URLhttps:// huggingface.co/lmms-lab/Aero-1-Audio. Accessed 2026-10-09. [35] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025. [36]Yunfei Chu, Jin Xu, Qian Yang, Haoji Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024. 20 A Source Synthesis Settings Active signal sampling. The pulse generator creates 24,000 CW source waveforms and 28,000 source waveforms for each chirp class. The communication generator creates 16,000 source waveforms for each modulation class. These groups contain 160,000 active source waveforms in total. All waveforms use a sampling rate of 16 kHz. The occupied frequencies remain between 100 and 7900 Hz. Table 10 summarizes the main sampling rules. Pulse duration depends on the selected subband. Communication duration follows from the symbol count and symbol rate. OFDM also varies the cyclic prefix and the number of subcarriers. FamilyCenter-frequency subbands in Hz Duration or symbol ruleAdditional variation Detection pulses CW300β700, 700β1500, 1500β3500, 3500β7500 1β10, 0.5β10, 0.05β5, 0.02β2 sSingle-frequency pulse LFM and HFMSame four subbandsSame band-dependent duration ranges as CW Both sweep directions; minimum bandwidth 50, 50, 50, 100 Hz Communication waveforms 2FSK and 4FSKSame four subbands50 to max(50,β30ν ν β) symbols at rate ν ν Modulation index 0.5β5 in steps of 0.5 BPSK and QPSKSame four subbandsSame symbol-count rule as FSKRaised-cosine roll-off 0 or 0.5β0.9 OFDM1500β3500 and 3500β75001β10 symbols with a 30 s limit512β4096 subcarriers; cyclic-prefix fraction0, 0.125, 0.25, 0.5 Table 10 Active-signal synthesis settings by source family. Representative evaluation inputs preserve the waveform structures described above. Figure 10 provides the complete active-class gallery used to support the qualitative discussion in Section 3.4. Ship-radiated noise sampling.Each of the five vessel classes contains 16,000 source waveforms. A source waveform combines a shaped random continuum with discrete mechanical tones. Letν(ν‘)denote the continuum. Letν shaft andν΅ denote the shaft frequency and blade count. The blade-passing frequency is ν blade = ν΅ν shaft . The waveform is ν₯(ν‘) = ν(ν‘)ν(ν‘)+ νΎ βοΈ ν=1 ν ν cos(2ν ν ν ν‘+ ν ν ),(7) eν(ν‘) = 1+ ν ν cos(2ν ν shaft ν‘+ ν ν )+ ν ν cos(2ν ν blade ν‘+ ν ν )+ ν βοΈ ν=1 ν ν cos(2ν ν ν ν‘+ ν ν ),(8) ν(ν‘) = max(eν(ν‘), 0.05) mean ν‘ max(eν(ν‘), 0.05) .(9) The continuum is flat below 300 Hz and decreases by 6 dB per octave above that frequency. A 32nd-order autoregressive filter implements this shape. The sampled 1 kHz spectrum level sets its amplitude. Shaft and blade components share the principal modulation coefficient. One to four additional components use smaller coefficients. Each source contains between three and ten tones. The generator attempts to associate one to three tones with shaft or blade harmonics. Their frequency jitter is at most 0.3 Hz. The remaining tones are sampled within the class range. A spacing of 3 Hz is enforced when feasible. Each tone has a local line-to-continuum ratio between 6 and 10 dB in a 1 Hz reference bandwidth. Table 11 gives the class-specific ranges. Blade count is sampled uniformly from the listed choices. Shaft frequency, spectrum level, and principal modulation coefficient are sampled uniformly from their intervals. The internal source classunderwatertargetrepresents a generic propeller-driven underwater platform and is presented to evaluated models under the formal taxonomy name Underwater vehicle. 21 0.00.81.5 Time (s) 5000 5500 Frequency (Hz) (a) CW 0.00.71.3 Time (s) 4000 5000 6000 Frequency (Hz) (b) LFM 0.00.71.3 Time (s) 4500 6000 Frequency (Hz) (c) HFM 0.01.32.6 Time (s) 4500 6000 Frequency (Hz) (d) OFDM 0.019.539.0 Time (ms) 4000 4800 5600 Frequency (Hz) (e) 2FSK 0.099.0198.0 Time (ms) 4000 4800 5600 Frequency (Hz) (f) 4FSK 0.018.036.0 Time (ms) 4200 4800 5400 Frequency (Hz) (g) BPSK 0.0102.7205.5 Time (ms) 4050 4200 Frequency (Hz) (h) QPSK β50β250 Pulse power (dB) 0 dB = family maximum β50β250 Communication power (dB) 0 dB = family maximum Figure 10 Representative active-signal structure in UA-Bench-Eval. (aβc) Time-frequency power for CW, LFM, and HFM detection pulses. (dβh) Local views of OFDM, 2FSK, 4FSK, BPSK, and QPSK communication waveforms. Each panel shows one representative evaluation input with a frequency range and time window matched to the displayed structure. Power is referenced to the maximum within each family, as indicated by the separate color scales. Parameter Underwater vehicle Fishing vessel Cargo vessel Cruise ship Naval vessel Mechanical configuration Blade count ν΅5, 7 3, 4, 5 3, 4, 5 3, 4, 5 3, 4, 5 Shaft frequency in Hz1.5β183β352β252β222β28 Blade frequency in Hzν΅ν ν ν΅ν ν ν΅ν ν ν΅ν ν ν΅ν ν Principal modulation coefficient0.06β0.080.15β0.200.15β0.200.15β0.200.10β0.12 Spectral and tonal configuration 1 kHz spectrum level in dB re 1νPa 2 /Hz103β130118β136124β142120β148124β148 Continuum corner frequency in Hz300300300300300 Continuum slope in dB per octaveβ6β6β6β6β6 Number of tones3β103β103β103β103β10 Tone frequency range in Hz10β20010β100010β100010β100010β1000 Local line-to-continuum ratio in dB6β106β106β106β106β10 Geometry and source count Source and receiver depth in m20β3005β105β105β105β10 Source to receiver range in km0.2β30.2β30.2β30.2β30.2β3 Source waveforms16,00016,00016,00016,00016,000 Table 11 Ship-radiated noise settings before propagation. ν΅ denotes blade count and ν ν denotes shaft frequency. Figure 11 complements the parameter ranges with representative evaluation spectra and the distribution of spectral centroids across the five ship-noise leaf classes. 22 05001000 Frequency (Hz) β75 β60 β45 β30 PSD (dB) (a) Cargo vessel 05001000 Frequency (Hz) β75 β60 β45 β30 PSD (dB) (b) Cruise ship 05001000 Frequency (Hz) β75 β60 β45 β30 PSD (dB) (c) Fishing vessel 05001000 Frequency (Hz) β75 β60 β45 β30 PSD (dB) (d) Naval vessel 05001000 Frequency (Hz) β75 β60 β45 β30 PSD (dB) (e) Underwater vehicle 150200250300 Spectral centroid (Hz) Cargo Cruise Fishing Naval Underwater Class (f) Centroid: median and IQR (aβe) PSD reference: 1 digital amplitudeΒ²/Hz; (f) Eval, n = 200 per class Figure 11 Representative ship-radiated-noise structure in UA-Bench-Eval. (aβe) Low-frequency power spectral densities for one evaluation input from each ship-noise leaf class, using a common digital-amplitude-squared-per-hertz reference. (f) Median and interquartile range of spectral centroid over 200 evaluation examples per class. The shortened label Underwater denotes the formal taxonomy class Underwater vehicle. The spectra illustrate individual inputs rather than class-average templates. B Propagation Response Construction Each propagation realization selects a sound-speed profile and perturbs the source and receiver geometry. Source depth, receiver depth, and range are perturbed independently by up to 10%. Source and receiver depths remain within the water column. Table 12 summarizes the shared settings and the differences between source families. The default number of beams is ν ν = min8000, max1000,β100ν km β.(10) If this run returns no arrivals andν ν <8000, the simulator repeats it once withmin8000, max4000,2ν ν beams. The response metadata records the final beam count and whether the second run was required. Arrival response. An arrival is valid when its delay is finite and nonnegative and its amplitude is finite and positive. The parser retains at most twenty arrivals by amplitude while preserving the earliest arrival. At representative frequency ν ν , the response is β ν [ν] = βοΈ νβP ν ν΄ νν ν νβ ν ν (ν νν β ν 0 ) ,(11) whereν΄ νν andν νν are the arrival magnitude and delay. The functionνis a 33-tap windowed-sinc kernel with a Kaiser parameter of 6.0. The earliest retained arrival definesν 0 and provides a common reference for all usable frequency responses in one ship realization. Applying the response. Active waveforms are convolved with the response at their center or carrier frequency. For ship-radiated noise, letν» ν ( ν)denote the Fourier transform ofβ ν . The response between adjacent representative frequencies is ν»( ν) = (1β ν€)ν» ν ( ν)+ ν€ν» ν+1 ( ν), ν€ = ν β ν ν ν ν+1 β ν ν .(12) The nearest endpoint response is used outside the available frequency range. If only one frequency has usable arrivals, that response is used for the complete waveform. The inverse transform produces a linear convolution, which is cropped to the source frame count. The retained waveform is then scaled to a peak amplitude of 0.95. 23 ParameterActive signalsShip-radiated noise Response construction Response modelSingle frequencySix representative frequencies Frequency in HzSignal center or carrier frequency50, 100, 200, 400, 800, 1600 Response applicationConvolution with one impulse responseInterpolation of complex frequency responses Environment and numerical settings Sound-speed profiles100 annual-mean WOA profiles100 annual-mean WOA profiles Geometry perturbationUp to 10%Up to 10% Bottom typeSiltSilt Bottom sound speed1520 m/s1520 m/s Bottom density1.4 g/cm 3 1.4 g/cm 3 Bottom attenuation3.0 dB/ν3.0 dB/ν Beam countAdaptive from 1000 to 8000Adaptive from 1000 to 8000 Execution and selection Propagation attempts per source 22 Selection statisticImpulse-response energy gainSource tonal-to-continuum power ratio Table 12 BELLHOP settings for active and ship-radiated sources. Retained waveforms are scaled to a peak amplitude of 0.95. Retaining a propagation realization.An active realization requires one usable response. A ship realization requires at least one usable representative frequency. Both branches reject a nonfinite or zero output waveform. Audio and metadata are written only after these checks succeed. C Metadata, Records, and Dataset Selection The active fieldtldbstores the impulse-response energy gainνΊ β defined in Equation 3. It is computed before output scaling and is not conventional positive transmission loss. The ship fieldsnrdbstores the source statistic snraftermixdb. The metadata converter reconstructs the complete tonal component from its saved frequencies, amplitudes, and phases. It subtracts this component from the source waveform and compares the mean-square powers. This statistic differs from the local ratio used to set each individual tone. The annotation stage copies the source and channel metadata into each audio record. It also adds the canonical label path and source identity. The channel suffix is removed from the record identifier to recover the source identity. The released fields and selection settings are summarized in Table 13. (a) Released record fields FieldContent id, audioPropagation realization identifier and processed waveform path conversationsThree stored questions interleaved with three reference answers gtCanonical L1, L2, and L3 labels metaSource identity, source parameters, channel pa- rameters, and derived statistics qa promptversionVersion of the stored question specification recordschema- version Version of the record structure (b) Partition and evaluation selection SettingValue Split unitSource identity Nominal ratio70 / 15 / 15 Split seed42 Candidate pool400 per leaf Evaluation sample200 per leaf Leaf classes13 Selection seed42 Table 13 Released record fields and dataset selection settings. The two panels connect the stored record structure with the source-grouped split and fixed evaluation subset. Source groups are formed within each leaf class. They are shuffled with seed 42 and assigned to training, validation, and test partitions in a nominal ratio of 70, 15, and 15 percent. Integer group counts are obtained with the largest-remainder rule. All propagation realizations of one source inherit the same partition. 24 UA-Bench-Eval is drawn from the test partition. Records are ranked within each leaf class by the corresponding finite selection statistic. The first 400 records form the candidate pool. The procedure samples 200 records without replacement from each pool with seed 42. It then verifies record identifiers, label paths, metadata, waveform paths, and class counts. D Question and Reference Construction The question pool contains eight Turn 1 templates, eight Active Turn 2 templates, and six Passive Turn 2 templates. Annotation selects the wording once and stores it in the record. Evaluation reuses the stored questions. All classification templates end with the same instruction. Select the option that you consider the best match for the audio. You must choose exactly one of the listed options. Reply with the uppercase option letter only, without explanation or any additional text. The eight Turn 1 templates are summarized below using lightly simplified renderings. The released records preserve the exact stored wording. Each template is followed by the same two source-category options and the common answer instruction. Turn 1 question stems 1. Determine whether this underwater sound is a deliberately transmitted signal or source-radiated noise. 2. Is the sound intentionally transmitted for detection or communication, or radiated as noise by an operating platform? 3. Classify the sound by its source, either deliberate transmission or platform-radiated noise. 4. Does this audio contain an intentionally transmitted waveform or noise radiated by a vessel or underwater vehicle? 5. Identify the source category of this sound, either a deliberate acoustic transmission or incidental platform noise. 6. Is this sound produced by deliberate signal transmission or by the operation of a vessel or underwater vehicle? 7. Distinguish a deliberately transmitted acoustic signal from noise radiated by an operating platform. 8. Which source category best describes this underwater sound, intentional transmission or source-radiated noise? A. Active, deliberately transmitted signal B. Passive, source-radiated noise The Active branch uses eight second-turn templates, summarized below using lightly simplified renderings. The field L1 is replaced by the confirmed source category before the question is stored. Turn 2 Active question stems 1. The signal isL1. Determine whether it is a detection pulse or communication signal, and specify the exact type. 2. The signal isL1. Identify its subcategory. 3. The signal isL1. What is its specific signal type? 4. The source category isL1. Determine whether it is a detection pulse or communication signal, and identify the specific type. 5. The signal is confirmed asL1. Identify the specific signal type. 6. Confirmed asL1. Which specific signal type is it? 7. The first classification step is complete. Further classify thisL1 signal. 8. Within theL1 category, which specific signal type is it? The Passive branch uses six second-turn templates, again shown as lightly simplified renderings. 25 Turn 2 Passive question stems 1. The signal isL1. Identify which type of vessel or underwater vehicle produced this radiated noise. 2. Confirmed asL1. Identify the specific type of noise source. 3. The signal isL1. What kind of platform emitted this noise? 4. For thisL1 signal, determine the target type. 5. What type of target is this radiated noise most likely from? 6. Further classify the noise source of thisL1 signal. Turn 1 presents the two source categories. Turn 2 presents complete paths within the branch reached at Turn 1. Table 14 gives the fixed option mapping. The displayed Underwater vehicle option maps to underwatertarget. TurnOptionLabel levelDisplayed choice 1AL1Active, deliberately transmitted signal 1BL1Passive, source-radiated noise 2 ActiveAL2 and L3Detection pulse, CW 2 ActiveBL2 and L3Detection pulse, LFM 2 ActiveCL2 and L3Detection pulse, HFM 2 ActiveDL2 and L3Communication signal, 2FSK 2 ActiveEL2 and L3Communication signal, 4FSK 2 ActiveFL2 and L3Communication signal, BPSK 2 ActiveGL2 and L3Communication signal, QPSK 2 ActiveHL2 and L3Communication signal, OFDM 2 PassiveAL2 and L3Ship-radiated noise, Cargo vessel 2 PassiveBL2 and L3Ship-radiated noise, Cruise ship 2 PassiveCL2 and L3Ship-radiated noise, Fishing vessel 2 PassiveDL2 and L3Ship-radiated noise, Naval vessel 2 PassiveEL2 and L3Ship-radiated noise, Underwater vehicle Table 14 Fixed classification options. Turn 1 uses the first two rows. Turn 2 uses the branch-specific rows. The decoder removes surrounding whitespace and accepts only one ASCII letter corresponding to a displayed option; lowercase and uppercase forms are equivalent. It does not recover labels from expanded or free-form answers. An invalid-format or incorrect Turn 1 answer stops the dialogue. After a correct Turn 1 route, each valid Turn 2 letter resolves a unique L2βL3 path. The raw answer and parsing status are retained. Turn 3 prompt. Briefly explain your previous classification using acoustic cues you can identify in the audio. Focus on the observed cues and how they support your choice. Do not list, compare, or rule out alternative classes, and do not merely repeat the class name or its definition. If the acoustic evidence is insufficient, state this explicitly rather than inventing supporting details. The first two reference answers contain the correct option letters. The Turn 3 reference combines a source descriptor, a signal-family descriptor, and three acoustic descriptors. The construction procedure selects one of the following sentence patterns. β’ The signal isL1 and exhibitsL2; observable evidence includesL3. β’ Based on its time-frequency and modulation structure, the signal isL1 withL2; supporting cues areL3. β’ Spectral and temporal analysis indicatesL1 withL2, supported byL3. β’ Based on its spectral and modulation structure, the signal isL1 withL2; supporting cues areL3. 26 CW, FSK, PSK, and OFDM use descriptors defined by their waveform structures. LFM and HFM use two trajectory descriptors and one direction descriptor selected from the stored sweep direction. Ship-radiated noise uses tonal extent, tone count, harmonic tags, and modulation coefficients. For vessel sources, a maximum tone frequency at or below 250 Hz selects the low-frequency description. A maximum at or below 1200 Hz with a minimum below 250 Hz selects the low-to-mid-frequency description. Other ranges select the broad-frequency description. At most four tones indicate sparse structure. Five to seven indicate moderate density, and more than seven indicate dense structure. The largest principal modulation coefficient determines weak, moderate, or strong wording at thresholds of 0.09 and 0.14. Reference answers and metadata are not sent to the model. Evaluation supplies the waveform, stored questions, and the modelβs earlier answers. The generated Turn 3 response is used for the lexical analysis below. 27 ν 1 ν 2 Time Shared audio context Audio is supplied in Turn 1 and retained in the dialogue history. Illustrative 2FSK example. Left: schematic frequency states. Instruction appended to both classification prompts Select the option that you consider the best match for the audio. You must choose exactly one of the listed options. Reply with the uppercase option letter only, without explanation or any additional text. TURN 1 SOURCE CATEGORY Stored question Determine whether this underwater sound is a deliberately transmitted signal or source-radiated noise. A.Active β Deliberately transmitted signal B.Passive β Source-radiated noise Illustrative response A Evaluator Parsed L1: Active. Matches the true L1. Continue to Turn 2. TURN 2 SIGNAL FAMILY AND SPECIFIC CLASS Stored question Confirmed as active. Which specific signal type is it? Active / Detection pulse A.CW B.LFM C.HFM Active / Communication D.2FSK E.4FSK F.BPSK G.QPSK H.OFDM Illustrative response D Evaluator Parsed L2: Communication. Parsed L3: 2FSK. One answer supplies both labels. Continue to Turn 3. TURN 3 AUXILIARY EXPLANATION Stored question Briefly explain your previous classification using acoustic cues you can identify in the audio. Focus on the observed cues and how they support your choice. Do not list, compare, or rule out alternative classes, and do not merely repeat the class name or its definition. If the acoustic evidence is insufficient, state this explicitly rather than inventing supporting details. Illustrative response The audio alternates between two discrete frequency states, with symbol-wise frequency switching. Evaluator Compare matched terms with the pre- dicted 2FSK inventory. Supporting terms: two discrete frequency states symbol-wise frequency switching Lexical outcome: Aligned. Classification labels remain unchanged. Parsed path: Active / Communication / 2FSK Figure 12 Illustrative three-turn dialogue in UA-Bench. The 2FSK example follows source routing, complete-path classification, and an auxiliary explanation. 28 E Lexical Explanation Analysis Letνbe the complete term inventory. Letν ν β νcontain the terms compatible with predicted leafν. Shared cues belong to every compatible leaf. Matching is case insensitive and uses letter and digit boundaries. Candidate phrases are ordered by decreasing length. A phrase is retained when its span does not overlap a previously retained phrase. This rule prevents a shorter phrase from being counted inside a longer match. Table 15 summarizes the inventory. The five vessel leaves share the vessel terms and add their displayed class name. Table 15 Term inventory used for lexical explanation analysis. Class or groupTerms Shared inventory All classesnarrowband, narrowband structure Detection pulses CW constant frequency, continuous wave, fixed frequency, single tone, single-frequency, stable single- frequency tone, steady frequency, unchanging frequency, unmodulated LFM chirp, constant sweep rate, frequency decreases linearly with time, frequency increases linearly with time, frequency ramp, linear chirp, linear downsweep, linear frequency modulation, linear sweep, linear upsweep, straight time-frequency trajectory HFMchirp, curved time-frequency trajectory, frequency decreases nonlinearly with time, frequency increases nonlinearly with time, hyperbolic chirp, hyperbolic frequency modulation, hyperbolic sweep, nonlinear chirp, nonlinear downsweep, nonlinear sweep, nonlinear upsweep, time-varying sweep rate Communication waveforms 2FSKbinary FSK, binary frequency-shift keying, frequency switching, symbol-wise frequency switching, two discrete frequency states, two frequency states 4FSKfour discrete frequency states, four frequency states, frequency switching, multi-level FSK, quaternary FSK, quaternary frequency-shift keying, symbol-wise frequency switching BPSK approximately 180-degree phase transitions, binary phase, binary phase-shift keying, binary PSK, phase flip, phase inversion, phase reversal, symbol-wise phase-state transitions, two phase states QPSKapproximately 180-degree phase transitions, four phase states, four-phase, phase flip, phase inversion, phase reversal, quadrature modulation, quadrature phase-shift keying, quadrature PSK, symbol-wise phase-state transitions OFDMdense carriers, multi-carrier, multiple subcarriers, orthogonal carriers, orthogonal frequency-division multiplexing, regularly spaced subcarriers, simultaneous multicarrier structure, subcarriers Ship-radiated noise Shared vessel termsblade-rate-related harmonic lines, broadband-plus-tonal vessel noise, dense discrete tonal structure, discrete machinery tonal components, low-frequency tonal structure, mechanical tonal components without a dominant harmonic family, moderate periodic amplitude modulation, moderate-density discrete tonal structure, periodic propeller-related modulation, shaft- and blade-rate-related harmonic lines, shaft-rate-related harmonic lines, sparse discrete tonal structure, strong periodic amplitude modulation, tonal components concentrated in the low-frequency band, tonal components spanning a broad frequency range, tonal components spread across the low-to-mid-frequency band, weak periodic amplitude modulation, weak periodic propeller-related modulation Cargo vesselcargo vessel Cruise shipcruise ship Fishing vesselfishing vessel Naval vesselnaval vessel Underwater vehicleunderwater vehicle For responseν, letν ν indicate at least one retained match inν Λν¦ (3) ν . Letν ν indicate at least one retained match in its 29 complement. If ν Turn 3 responses are executed, the rates are ν aligned = ν β1 βοΈ ν ν ν (1β ν ν ),ν confusion = ν β1 βοΈ ν (1β ν ν )ν ν ,(13) ν vague = ν β1 βοΈ ν (1β ν ν )(1β ν ν ),ν stacking = ν β1 βοΈ ν ν ν ν ν ,(14) ν contradiction = ν β1 βοΈ ν ν ν = ν confusion + ν stacking .(15) The first four categories partition the executed responses. An invalid-format Turn 2 prediction has an empty supporting set and usesνas its comparison set. Such a response is Concept Confusion when an inventory term is present and Vague otherwise. An empty explanation is also Vague. F Classification Metrics and Random References For leaf class ν, precision, recall, and F1 are ν ν = νν ν νν ν + νΉν ν , ν ν = νν ν νν ν + νΉν ν , νΉ ν = 2ν ν ν ν ν ν + ν ν .(16) Invalid-format and cascade predictions contribute false negatives to their true classes. They also appear as separate prediction outcomes in the confusion matrix. L3 Macro F1 is the mean F1 over the thirteen leaf classes. A ratio with an empty denominator is recorded as zero. The evaluator rounds each per-class precision, recall, and F1 value to four decimal places. Macro F1 is the unweighted mean of the thirteen stored four-decimal class F1 values. Runs without an executed Turn 3 response record a zero response count and zero lexical rates. For a valid Turn 2 option, leaf correctness implies correct source and signal-family labels. Consequently, L3 accuracy equals complete-path accuracy under the strict decoder and no duplicate joint metric is reported. The random reference chooses each source category with probability 1/2. On the Active branch, three of the eight options belong to detection pulses and five belong to communication signals. The Passive branch contains one signal family. The expected L2 accuracy is E[Acc νΏ2 ] = 3 13 1 2 3 8 + 5 13 1 2 5 8 + 5 13 1 2 = 37 104 β 35.58%.(17) The corresponding conditional references are 37/52β71.15% for L2|L1 and 8/37β21.62% for L3|L2. Together with L1 accuracy of 1/2, they yield the 1/13 leaf reference in Equation 6. For the paired silent-audio control, the original and silent L3 accuracies receive Wilson score intervals. The original- minus-silent accuracy difference also receives an analytic paired-proportion interval based on the two discordant-pair counts. The reported 95% intervals for original accuracy, silent accuracy, their difference, and the corresponding Macro F1 quantities use a percentile cluster bootstrap. One cluster is one source identity, obtained by removing a terminalchN channel suffix from the sample identifier. Channel variants are therefore resampled together. The evaluator uses 10,000 replicates and seed 20261008 by default. G Inference and Reproducible Evaluation We identify every evaluated checkpoint by its repository namespace and checkpoint identifier in Table 16. Across these backends, the shared execution, input, and preprocessing settings are collected in Table 17. Each backend uses its native processor and chat template. The waveform enters the first user turn and remains in the dialogue history. Later turns use the questions stored in each evaluation record. Each prediction record preserves the questions, raw answers, parsed labels, and turn status. Each protocol file records the model settings, taxonomy, evaluation manifest hash, and implementation hashes. Aggregation requires complete shard coverage and a shared run identity. It also checks sample coverage and duplicate identifiers before calculating metrics. 30 ModelRepositoryCheckpoint identifier Gemma-4-12B-IT [29] google gemma-4-12B-it Gemma-4-E2B-IT [29] google gemma-4-E2B-it Audio-Flamingo-Next [30] nvidia audio-flamingo-next-hf MiDashengLM-7B-1021 [31] mispeech midashenglm-7b-1021-bf16 Voxtral-Small-24B-2507 [32] mistralai Voxtral-Small-24B-2507 Voxtral-Mini-3B-2507 [33] mistralai Voxtral-Mini-3B-2507 Aero-1-Audio [34] lmms-lab Aero-1-Audio Qwen2.5-Omni-7B [35] Qwen Qwen2.5-Omni-7B Qwen2-Audio-7B-Instruct [36] Qwen Qwen2-Audio-7B-Instruct Table 16 Evaluated model repositories and checkpoint identifiers. Generation and executionInput and preprocessing ParameterValueParameterValue Numerical precisionbfloat16 where supportedBenchmark waveformMono at 16 kHz Maximum text output256 new tokens per turnModel preprocessingNative model processor Text decodingGreedy decoding where supportedBatch sizeBackend-specific Audio outputDisabledEvaluation questionsStored questions from each record Table 17 Shared inference settings without task demonstrations. Metadata strata use empirical one-third and two-thirds quantiles within each leaf class. A value equal to a boundary enters the higher stratum. Recall is computed for each leaf and stratum. The branch result is the unweighted mean of its leaf recalls. The evaluation records and thresholds remain fixed across models. 31