Paper deep dive
UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation
Yuxuan Chen, Peize He, Haoyuan Yu, Junzi Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 12:13:12 PM
Summary
The paper introduces UniWhisper, a framework for efficient continual multi-task training of universal audio representations. It unifies heterogeneous audio tasks (speech, environmental sound, music) into a single instruction-answer format, enabling next-token prediction without task-specific heads. Using a Whisper Large-v3 encoder initialized backbone and a compact pretrained language model (Qwen3-0.6B) as the decoder, UniWhisper is trained on 38k hours of public audio. It achieves superior performance on 20 tasks compared to baselines like Whisper, HuBERT, and CLAP, balancing speech and non-speech capabilities.
Entities (12)
Relation Signals (13)
UniWhisper → supportstasks → ASR
confidence 95% · including ASR, speech translation, audio captioning
UniWhisper → trainedwith → Instruction-Answer Format
confidence 95% · casts heterogeneous audio tasks into a unified instruction and answer format
UniWhisper → uses → Whisper large-v3
confidence 95% · The encoder is initialized from Whisper Large-v3
UniWhisper → usesmethod → Continual Multi-task Training
confidence 95% · We propose UniWhisper, an efficient continual multi-task training framework
UniWhisper → uses → Qwen3 0.6B
confidence 92% · the autoregressive decoder is a pretrained language model Qwen3-0.6B
UniWhisper → evaluatedon → HEAR
confidence 90% · we build our evaluation set on HEAREval [14]
UniWhisper → outperforms → Whisper
confidence 90% · UniWhisper reaches normalized weighted averages of 0.81 ... compared to 0.64 ... for Whisper
UniWhisper → supportstasks → Audio Captioning
confidence 90% · including ASR, speech translation, audio captioning
UniWhisper → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.
Tags
Links
- Source: https://arxiv.org/abs/2602.21772v2
- Canonical: https://arxiv.org/abs/2602.21772v2
Trouble viewing inline? Open PDF directly →
Full Text
39,498 characters extracted from source content.
Expand or collapse full text
UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation Yuxuan Chen ID 1,∗,∗ , Peize He 2,∗ , Haoyuan Yu 3,∗ , Junzi Zhang 4 1 Jilin University 2 University of Electronic Science and Technology of China 3 Hunan University 4 Shandong University yxchen5522@mails.jlu.edu.cn, 2023300904027@std.uestc.edu.cn, y15352176976@hnu.edu.cn, 202300800615@mail.sdu.edu.cn Abstract A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. However, prior encoders often excel in one domain but degrade in others. We propose Uni- Whisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music with the entire framework trained on only 38k hours of public audio. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance. Index Terms: post-training of speech foundation models, uni- versal audio representation, continual learning and adaptation 1. Introduction A universal audio representation aims to support speech, envi- ronmental sounds, and music with a single encoder. Large-scale pretraining has produced strong backbones for individual do- mains. Wav2vec 2.0 [1], HuBERT [2], WavLM [3], and Whis- per [4] improve robustness and performance on speech-related tasks. BEATs [5] and CLAP [6] perform strongly on audio event recognition and audio-text semantic matching. However, this training progress also highlights a persistent domain imbal- ance. Speech-focused encoders often lack semantic coverage for complex non-speech scenes. In contrast, general audio models often fail to preserve the fine-grained temporal cues required by speech. This split becomes especially costly in large audio language models (LALMs). A common recipe aligns a pretrained audio encoder with a large language model using paired audio and text data. When the encoder is speech-centric, broader audio cov- erage is often achieved through continual training on large and diverse datasets, which increases training cost and data require- ments [7–11]. Dual-encoder systems such as SALMONN [12] and Kimi-Audio [13] improve domain coverage by fusing en- coders from different domains. However, they require addi- tional coordination across temporal resolutions and representa- tion spaces, and they often need extra alignment data. More importantly, concatenating features from multiple encoders in- creases the number of audio tokens and consumes the limited * These authors contributed equally. ** indicates the corresponding author. ASV2015 CREMA-D Fluent Speech LibriCount LibriSpeech-100h LibriSpeech-MF RAVDESS Speech Commands VocalSound VoxCeleb1 VoxLingua33 Free Music Archive GTZAN Genre NSynth-Instruments DESED ESC-50 FSD18-Kaggle FSD50k UrbanSound 8k Vocal Imitation wav2vec HuBERT WavLM BEATs CLAP Whisper UniWhisper Figure 1: Normalized per-task performance of UniWhisper on our 20-task extended HEAR [14] spanning speech, environmen- tal sound, and music. Full results are reported in Table 3. context window of the language model, while broader coverage often comes with longer token sequences. To avoid architectural redundancy, we adopt a single- encoder design and focus on unifying supervision. Whisper already learns rich acoustic perception from large-scale weakly supervised transcription, but transcription dominated training biases the representation toward speech. We propose prompt guided continual multi-task training, where diverse objectives are expressed with a shared instruction and answer format. This ex- pands coverage across speech, environmental sound, and music without task specific heads or multi encoder feature concatena- tion. Since the audio prefix always comes from one encoder stream, token redundancy is removed at the source. Using this framework, we train UniWhisper, a unified encoder backbone for multi-domain audio understanding that strengthens non-speech semantics while preserving speech ca- pability such as ASR. We also identify an efficiency bottleneck in the original Whisper decoder under instruction-style align- ment. In our setting, the decoder converges slowly and requires substantially more updates to reach competitive performance. We replace it with a compact pretrained language model that serves as the semantic interface during instruction-style train- ing. The compact decoder provides strong language priors that better match instruction-following targets and can accelerate convergence. As a result, UniWhisper can be adapted with a sub- stantially smaller training corpus than recent LALMs. We train on 38k hours of public audio, while Qwen2-Audio [9] reports 520k hours for pre-training. The pipeline is simple because all arXiv:2602.21772v2 [cs.SD] 9 Mar 2026 Whisper Encoder Audio Wavform x(t) log-Mel Conv1D [Prompt u] Pretrained LM Decoder Adapter A(·) [Audio tokens Z] Logits Next token prediction trainable frozen CE [Answer v] Loss CE cross entropy Enc speech Enc sound Enc music [Zsound] [Z speech ] [Zmusic]Z =++ long context, redundant Multi-encoder fusion: Enc base Task-specific design: Head ASR Head AAC Head SV complex recipes, hard to maintain L ASR L AAC L SV Head MGR L MGR [Z base ] UniWhisper: Enc unified [Z unified ]Head unified L unified single recipe, lower compute (C) ILLUSTRATION OF EFFICIENCY Speech: Sound: Music: Mixed public audio corpora (~38k h) Audio Wavform Unified Prompt Template Builder Your task is [Control tags] + [Natural-language instruction] Non-structured text fields [Prompt u][Answer v] Example (same skeleton for all tasks): [Audio tokens Z] ASR / SV / SD / S2T ... AAC / AQA / SED ... MQA / MGR / MIC / MC ... (A) DATA → UNIFIED INSTRUCTION FORMAT u: <TASK=CAPTION> <A_LANG=ANY> <T_LANG=EN> <STYLE=short> "Describe the audio." v: "A dog barks while traffic passes by." u: U: <TASK=ASR> <A_LANG=ZH> <T_LANG=EN> <TS=ON> "Transcribe the speech." v: "[00:01-00:03] Nice to meet you. [00:04-00:06] Where are you ..." (B) MODEL TRAINING PIPELINE Figure 2: Overview of the UniWhisper continual multi-task training framework. (A) Converting heterogeneous datasets into a unified instruction and answer format. (B) Training with a single audio encoder and next-token prediction on answer tokens. (C) Comparison with common alternatives, highlighting reduced audio token redundancy and a unified supervision interface. tasks share the same templates and data format, so we can mix multiple open-source datasets with a single recipe. We assess representations using shallow MLP probing and non-parametric kNN evaluation, with ablations on the decoder and backbone. Ex- periments show that UniWhisper is competitive across 20 tasks spanning speech, environmental sound, and music. Under our evaluation protocol, it outperforms Whisper, HuBERT, BEATs, WavLM and CLAP on average and shows no clear catastrophic forgetting. Code and pretrained weights will be released. 2. Method 2.1. Backbone and initialization Whisper uses an encoder-decoder transformer model. Given au- diox, we extract log-Mel features and encode them into acoustic representations. The decoder predicts the next text token condi- tioned on the audio representation and previous tokens, trained with a standard cross-entropy loss. UniWhisper retains the standard next-token prediction ob- jective, but revises the decoder to better fit instruction-style su- pervision. The encoder is initialized from Whisper Large-v3 [4], while the autoregressive decoder is a pretrained language model Qwen3-0.6B [15]. A lightweight adapter maps the encoder hid- den states to the decoder hidden dimension. LetH∈R T ′ ×d w denote the Whisper encoder outputs and let d q be the hidden size of the decoder. The adapter produces Z = A(H)∈R T ′ ×d q ,(1) whereA(·)is a small MLP that performs a learned projection fromd w tod q . Using a pretrained LM as the decoder provides strong language priors that match instruction-following targets and can ease alignment between audio representations and text semantics. The pretrained LM decoder keeps frozen throughout training while the encoder and projection modules updating. 2.2. Unified instruction style multitask training We express diverse audio tasks in a unified instruction and an- swer format, including ASR, speech translation, audio caption- ing, keyword or attribute prediction, audio question answering, and audio-text matching. As illustrated in Fig. 2, each example is represented by a prompt that combines control tags, such as task type, audio language, text language, optional timestamps, and output constraints, with a natural language instruction. Su- pervision is provided as a text answer. Different tasks therefore differ only in the prompt content and the target answer, while sharing the same training and decoding interface. Given the audio prefixZ, prompt tokensu, and answer tokens v = (v 1 , . . . , v |v| ), the decoder models p θ (v t | v <t , u, Z),(2) and we optimize next token cross entropy on the answer tokens L CE =− X |v| t=1 log p θ (v t | v <t , u, Z).(3) Since UniWhisper uses a single audio token streamZas the prefix, it avoids multi-encoder feature concatenation and the resulting audio token redundancy. Multi domain capability is obtained primarily through unified templates and mixed task training data under continual multi-task training. 3. Experimental Setup 3.1. Datasets Training: we train UniWhisper on a mixture of open-source audio datasets summarized in Table 1. We preserve the original supervision signals while converting all datasets into a unified Table 1: Public datasets used for continual multi-task training, grouped by domain. Duration denotes the total audio hours used after filtering and deduplication. TypeNameTaskDuration General AudioCaps [16]Captioning142.5 h AudioSet [17]Tagging5.8k h LAION-Audio [18]Captioning4.3k h WavCaps [19]Captioning7.6k h Speech AISHELL-1 [20]ASR & SID178 h GigaSpeech [21]ASR10k h Libri-adhoc40 [22]ASR4.5k h LibriSpeech [23]ASR & SV860 h Sound Clotho [24]Captioning43.6 h Seeing Sound [25]SED0.2 h SONYC-UST [26]Tagging51.5 h TAU-ASC2020 [27]ASC64 h URBAN-SED [28]SED27.8 h VGGSound [29]ASC553.3 h Music GuitarSet [30]Guitar Trans3 h MAESTRO [31]Piano Trans200 h MedleyDB [32]AMT7.3 h MTG-Jamendo [33]Tagging3.8k h MusicCaps [34]Captioning15.3 h MusicNet [35]AMT34 h Slakh2100 [36]AMT145 h SongDescriber [37]Captioning23 h YT8M-MTC [38]Captioning11.7 h instruction–answer format. To prevent train–eval leakage, we apply identifier-based filtering and content-level deduplication using hashing and acoustic fingerprinting. Evaluation: as shown in table 2, we build our evaluation set on HEAREval [14]and evaluate on 20 tasks spanning speech, envi- ronmental sound, and music. Since HEAR provides limited cov- erage of human voice processing [39], we add speech-oriented tasks. More details refer to HEAR [14] and X-ARES [39]. 3.2. Training details Audio preprocessing: We resample all audio to 16 kHz and extract 128-bin log-Mel features using a 25 ms window and a 10 ms hop. We use 30 s clips and pad shorter clips with zeros. To re- duce the audio sequence length, we apply an additional temporal strided convolution with stride 2. Together with Whisper encoder subsampling, each output frame represents approximately 40 ms of the input waveform. We propagate padding masks through the encoder so that padded frames do not contribute to attention. Optimization: We minimize next-token cross-entropy loss only on answer tokens while masking prompt tokens. We use 8-bit AdamW with cosine decay (LR2× 10 −5 , weight decay0.01), a 1,500-step warm-up to align the audio and text representation spaces, and train for 30,000 update steps in total. Training uses bf16 and DDP under 24 wall-clock hours on 8 A800 GPUs with batch size 32 per GPU (global batch size 256). Unless otherwise specified, we update only the Whisper encoder and the projection adapter, keeping the pretrained LM decoder frozen. 3.3. Evaluation protocols We evaluate encoder representations with two protocols: super- vised probing with a shallow MLP and non-parametric kNN. Tasks are divided into clip tasks that use a single embedding per example and frame tasks that use a sequence of embeddings. We also include an ASR task 1 on LibriSpeech-100h subset to check whether continual multi-task training preserves the speech recog- 1 Inspired by X-ARES [39], we use Qwen2.5 0.5B as the text decoder for ASR, and keep it frozen during training. Table 2: Evaluation tasks, metrics, and weights used to compute normalized weighted averages. Weights are proportional to the number of test examples. TypeTaskMetric Weight MLPkNN Speech ASV2015Acc20002000 CREMA-DAcc11161116 Fluent Speech CommandsAcc20002000 LibriCountAcc11441144 LibriSpeech-100hiWER10000- LibriSpeech-MFAcc26202620 RAVDESSAcc360360 Speech Commands V1Acc20002000 VocalSoundAcc20002000 VoxCeleb1Acc20002000 VoxLingua33Acc16091609 Music Free Music Archive SmallAcc800800 GTZAN GenreAcc100100 NSynth-InstrumentsAcc20002000 Sound DESEDSeg-F11153- ESC-50Acc400400 FSD18-KagglemAP1600- FSD50kmAP2000- UrbanSound 8kAcc873873 Vocal ImitationAcc18671867 nition ability. Embedding: We use frame embeddings from the final encoder layer. For clip tasks, we mean pool over time to ob- tain one clip embedding. For frame tasks, we keep the full frame sequence and align frame labels by padding when needed. MLP: We freeze the encoder and train a shallow MLP on top of the clip embedding or the frame sequence for each task. kNN: We per- form classification or retrieval directly in the pooled embedding space without training a classifier. Metrics: We map each metric to [0, 1] via min–max normalization using best/worst attainable values (Acc/mAP/Seg-F1/Recall@1 use1/0); for ASR we use iWER = max(1− WER, 0). We computeS = P i n i ˆ M i P i n i and report it for MLP and kNN in table 2. 4. Results We report results under the same protocols for several widely used pretrained audio encoders, including wav2vec 2.0-Large [1], HuBERT-Large [2], WavLM-Large [3], Whisper-Large-v3 [4], BEATs-iter3 [5], and CLAP-HTSAT [6]. Per task results are provided in Table 3 and confidence intervals in Table 4. 4.1. MLP results Table 3 shows that UniWhisper achieves weighted averages of 0.81 with MLP and 0.61 with kNN. This gain suggests that continual multi-task training with unified instruction supervision increases the amount of linearly accessible information in the representation. We observe the largest improvements on non- speech tasks that rely on global semantic cues, such as audio tagging and captioning. Besides, UniWhisper maintains strong performance on speech-oriented tasks, including speaker and paralinguistic classification. Compared with domain specialized baselines, UniWhisper narrows the gap on environmental sound and music tasks without requiring multi encoder feature fusion. We also note that speech only encoders such as wav2vec and WavLM remain strong on speech heavy subsets, but they are less consistent on music and complex sound scene tasks under our unified evaluation. Conversely, CLAP and BEATs provide strong performance on tasks aligned with their pretraining objec- tives, but they tend to underperform on fine grained speech tasks that depend on phonetic detail. UniWhisper improves the cross domain balance by building on Whisper acoustics and adding supervision signals that explicitly target non-speech semantics. Table 3: Per-task normalized results for MLP and kNN on 20 tasks. Scores are normalized to[0, 1]using task-specific bounds so higher is better and reported as percentages (×100). CLAP-U replaces the Whisper encoder with CLAP-HASAT under the same training recipe. Whisper-U and Whisper-U-3 use the original Whisper Large-v3 decoder and train with one pass and three-pass replay over the same data, respectively. Highest score is in bold. Domain Task wav2vec 2.0 HuBERTWavLMBEATsCLAPWhisperCLAP-U Whisper-U Whisper-U-3Ours MLP kNNMLP kNNMLP kNNMLP kNNMLP kNNMLP kNNMLP kNNMLP kNNMLP kNNMLP kNN Speech ASV2015 94.0 95.8 95.1 88.5 97.4 94.6 91.5 81.0 95.1 71.2 97.5 90.3 92.3 82.9 98.3 95.0 94.7 90.6 99.0 92.9 CREMA-D 55.5 21.8 64.5 34.0 71.3 28.0 66.5 39.7 36.9 25.4 57.5 38.2 54.3 49.0 64.3 17.8 66.3 30.0 84.4 67.9 FSC 47.2 1.6 96.5 3.6 97.5 4.3 47.3 5.5 3.6 1.6 97.8 23.4 17.7 5.4 97.2 1.0 98.0 53.1 82.7 14.6 LibriCount 58.0 20.8 58.2 18.2 65.1 38.8 68.2 37.8 38.8 28.1 59.4 45.2 60.9 54.0 62.8 9.1 53.7 41.4 64.4 47.2 LS-100h 16.0– 83.4– 64.1– 15.8– 0.2– 62.5– 1.5– 79.0– 89.9– 91.6– LS-MF 95.3 70.3 97.8 79.9 97.9 77.2 97.5 92.3 90.2 86.8 90.8 58.0 97.1 96.4 97.2 53.9 82.7 55.9 98.1 91.3 RAVDESS 44.1 15.8 58.1 32.9 68.0 25.2 66.6 37.9 23.7 21.9 61.3 38.5 55.9 45.5 59.4 23.9 65.6 36.8 88.2 70.0 GSC 71.2 23.2 95.7 28.9 96.3 58.3 88.0 49.6 25.1 13.7 97.0 74.8 67.0 49.1 96.7 9.8 94.9 73.0 95.5 65.4 VocalSound 77.3 24.1 85.5 35.7 89.3 31.1 91.4 75.2 42.6 25.9 89.3 57.1 88.5 85.8 90.7 39.0 90.6 43.3 93.0 90.9 VoxCeleb1 34.0 0.2 58.2 6.5 65.4 8.9 41.3 13.2 4.3 0.5 22.0 4.1 13.6 7.8 40.9 0.9 33.8 4.1 45.5 4.5 VoxLingua33 55.8 0.6 75.0 6.3 86.2 29.0 41.9 14.8 7.7 3.5 97.6 95.8 18.8 13.4 97.6 53.6 94.0 62.8 89.5 71.6 Music FMA 47.8 21.1 50.5 22.6 52.6 31.0 66.0 61.0 30.0 25.7 57.4 51.4 66.0 62.2 61.0 40.8 61.2 46.5 68.9 67.3 GTZAN 63.7 31.1 69.6 20.7 74.0 43.6 89.8 85.2 40.4 34.9 71.2 53.3 84.2 79.5 70.5 11.2 82.3 64.1 94.5 89.5 NSynth 31.6 25.1 37.7 35.5 40.9 25.0 64.8 61.6 24.5 20.1 46.0 12.4 77.3 78.0 47.0 15.0 49.7 38.5 70.7 70.0 Sound DESED 31.7– 33.5– 38.8– 4.2– 2.4– 29.4– 20.2 0.0 31.0– 44.3– 57.0– ESC-50 51.9 7.9 56.9 13.2 65.7 19.2 95.3 85.5 16.3 18.2 62.4 29.5 97.0 95.6 67.0 3.5 80.6 46.4 95.8 93.7 FSD18-Kaggle 23.1– 34.9– 36.4– 79.1– 6.4– 36.0– 87.6 0.0 21.8– 55.8– 90.5– FSD50k 16.5– 21.9– 27.4– 56.7– 4.5– 32.1– 59.1 0.0 34.6– 50.3– 60.0– UrbanSound 8k 66.7 34.8 65.8 34.3 69.1 30.6 89.3 81.4 36.4 33.8 71.9 45.5 85.4 83.1 73.4 11.5 76.4 54.4 84.4 78.3 Vocal Imitation 14.5 0.9 17.4 2.4 24.5 4.3 24.1 13.4 2.5 1.7 24.1 5.2 17.2 11.4 26.8 1.6 28.5 10.9 25.7 14.7 Weighted Avg. 42.8 27.7 69.2 32.6 67.2 37.1 52.4 49.2 22.0 27.6 64.2 45.7 43.9 53.1 70.3 27.8 74.4 47.0 80.9 60.4 Table 4: Reported deviations correspond to the maximum absolute dif- ference between the mean and the endpoints of the 95% percentile CI, estimated using hierarchical bootstrap resampling over seeds and evalu- ation examples with replacement. Task Whisper UniWhisper CLAP-Uni Whisper-Uni-1 Whisper-Uni-3 MLP kNN MLP kNNMLP kNN MLPkNNMLPkNN ASV2015 ±1.7±1.4 ±2.0 ±1.8 ±1.1 ±1.3 ±2.5 ±2.4 ±1.4 ±1.3 CREMA-D ±0.9±0.8 ±1.4 ±1.2 ±0.9 ±1.1 ±1.7 ±0.3 ±1.3 ±0.4 FSC ±1.8±0.4 ±1.6 ±0.2 ±0.4 ±0.1 ±1.8 ±0.3 ±2.5 ±0.7 LibriCount ±0.9±0.8 ±1.0 ±0.9 ±1.5 ±0.6 ±1.6 ±0.2 ±1.2 ±0.8 LS-100h ±1.0 – ±1.8– ±0.3– ±1.9– ±1.8– LS-MF ±1.8±1.0 ±2.0 ±2.1 ±2.4 ±2.5 ±1.6 ±1.1 ±2.3 ±1.0 RAVDESS ±1.1±0.7 ±1.9 ±1.1 ±1.0 ±0.7 ±1.3 ±0.4 ±1.0 ±1.0 GSC ±1.4±1.8 ±2.5 ±0.9 ±1.1 ±1.2 ±2.1 ±0.2 ±2.3 ±1.2 VocalSound ±1.8±0.8 ±1.8 ±1.5 ±1.4 ±1.2 ±1.9 ±0.7 ±1.9 ±0.7 VoxCeleb1 ±0.4±0.1 ±1.1 ±0.1 ±0.2 ±0.2 ±0.8 ±0.5 ±0.8 ±0.1 VoxL33 ±1.5±1.6 ±1.7 ±0.9 ±0.3 ±0.2 ±1.4 ±1.2 ±2.5 ±1.2 FMA ±1.1±0.8 ±1.2 ±1.3 ±1.2 ±1.3 ±1.3 ±0.7 ±1.2 ±1.0 GTZAN ±1.3±0.9 ±1.8 ±2.3 ±1.4 ±1.5 ±1.1 ±0.2 ±1.2 ±1.2 NSynth ±1.3±0.2 ±1.2 ±1.4 ±1.7 ±1.5 ±0.5 ±0.2 ±1.2 ±0.6 DESED ±0.6 – ±1.3– ±0.2– ±0.5– ±1.0– ESC-50 ±1.3±0.5 ±1.8 ±2.1 ±2.0 ±2.4 ±1.6 ±0.1 ±1.9 ±1.0 FSD18-K ±0.7 – ±1.2– ±1.0– ±0.5– ±1.4– FSD50k ±0.7 – ±1.1– ±1.3– ±0.7– ±1.2– UB 8k ±1.4±0.9 ±1.4 ±1.2 ±2.1 ±1.4 ±1.4 ±0.2 ±2.2 ±1.1 Vocal Im ±0.3±0.1 ±0.5 ±0.3 ±0.3 ±0.2 ±0.5 ±0.5 ±0.8 ±0.2 Avg. ±0.5±0.3 ±0.6 ±0.5 ±0.2 ±0.4 ±0.7 ±0.3 ±0.8 ±0.2 4.2. kNN results kNN evaluation in Table 3 largely mirrors the MLP trends while emphasizing embedding-space geometry without learning a task- specific head. UniWhisper improves kNN performance over Whisper, suggesting that instruction-style continual training yields a better organized representation space in addition to higher probe accuracy. Across baselines, we find that models optimized for global alignment such as CLAP can perform well on retrieval oriented tasks but remains weaker on fine-grained speech transfer, consistent with a retrieval-optimized encoder whose representations favor global semantic alignment over dense temporal detail. Bootstrap intervals in table 4 for weighted averages are narrow, supporting the stability of these trends. 4.3. Ablation We ablate the encoder choice and decoder design under the same instruction-style continual training pipeline. Encoder choice: Inspired by CLIP [40], we test a CLIP- style contrastive encoder by replacing our encoder with CLAP- HASAT [41] while keeping the rest of the pipeline unchanged. For fairness, CLAP-Uni bypasses CLAP’s final clip-level mean pooling and uses penultimate-layer dense features: each 10 s segment yieldsh ∈R 64×2048 (stride≈ 156ms). We also remove the extra temporal convolution used in UniWhisper. As shown in Table 3, our instruction-style continual train- ing substantially improves CLAP: CLAP-Uni nearly doubles the weighted average and is competitive with UniWhisper on CLAP-aligned semantic audio tasks. However, its gains are less pronounced on temporally precise speech understanding. This suggests that native temporal granularity of the backbone remains a key factor even after continual training. Decoder design: We compare our pretrained LM decoder with the original Whisper decoder under the same pipeline. As shown in Table 3, the Whisper decoder yields slower gains and weaker semantic alignment: Whisper-Uni-1 reaches0.70in MLP but drops to0.28in kNN, below the original Whisper kNN score0.46, indicating a less well-structured embedding space despite higher probe accuracy. Multi-pass replay improves results, but at much higher cost and still below UniWhisper. This supports using a pretrained LM decoder, which better aligns instruction-style semantics and preserves representation structure under a fixed compute budget. 5. Conclusion We presented UniWhisper, an efficient continual multi-task train- ing framework for universal audio representation trained with a unified instruction and answer format for continual multi-task supervision. Starting from Whisper Large v3, we replace the original decoder with a compact pretrained language model con- nected through a lightweight projection adapter, enabling effi- cient next-token training on 38k hours of public audio. On 20 tasks spanning speech, environmental sound, and music, Uni- Whisper achieves weighted averages of 0.81 (MLP) and 0.61 (kNN), outperforming strong pretrained baselines while main- taining competitive speech capability. 6. Generative AI Use Disclosure We used a generative AI tool to assist with language editing and polishing of the manuscript, including improving grammar, clarity, and readability. The tool was not used to generate scien- tific content, experimental results, or conclusions. All coauthors reviewed the final manuscript and take full responsibility for it. 7. References [1]A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems, vol. 33, 2020. [Online]. Available: https://proceedings.neurips.c/paper/2 020/hash/92d1e1eb1cd6f9fba3227870b6d7f07-Abstract.html [2] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representa- tion learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, p. 3451–3460, 2021. [3]S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self- supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, p. 1505–1518, 2022. [4] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” arXiv preprint arXiv:2212.04356, 2022. [5] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: Audio pre-training with acoustic tokenizers,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202.PMLR, 2023, p. 5178–5193. [Online]. Available: https://proceedings.mlr.press/v202/chen23ag.html [6]B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “CLAP: Learning audio concepts from natural language supervision,” in ICASSP 2023, 2023, p. 1–5. [7]S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio Flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, p. 19 358–19 405. [Online]. Available: https://proceedings.mlr.press/v267/ghosh25b. html [8] Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023. [9]Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-Audio technical report,” arXiv preprint arXiv:2407.10759, 2024. [10] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin, “Qwen2.5-Omni technical report,” arXiv preprint arXiv:2503.20215, 2025. [11]StepFun Audio Team, “Step-Audio 2 technical report,” arXiv preprint arXiv:2507.16632, 2025. [Online]. Available: https: //arxiv.org/abs/2507.16632 [12]C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023. [13]KimiTeam, “Kimi-Audio technical report,” arXiv preprint arXiv:2504.18425, 2025. [14] J. Turian, J. Shier et al., “HEAR: Holistic evaluation of audio representations,” in Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, ser. Proceedings of Machine Learning Research, vol. 176.PMLR, 2022, p. 120–136. [Online]. Available: https://proceedings.mlr.press/v176/turian22a.html [15] Qwen Team, “Qwen3 technical report,” arXiv preprint arXiv:2505.09388,2025.[Online].Available:h t t p s : //arxiv.org/abs/2505.09388 [16]C. D. Kim, B. Kim, H. Lee, and G. Kim, “AudioCaps: Generating captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, p. 119–132. [Online]. Available: https://aclanthology.org/N19-1011/ [17] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, p. 776–780. [18]Y. Wu, K. Chen, T. Zhang, Y. Hui, M. Nezhurina, T. Berg- Kirkpatrick, and S. Dubnov, “Large-scale contrastive language- audio pretraining with feature fusion and keyword-to-caption augmentation,” in 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. [Online]. Available: https://arxiv.org/abs/2211.06687 [19]X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang, “WavCaps: A ChatGPT-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 32, p. 3339–3354, 2024. [20]H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “AISHELL-1: An open-source mandarin speech corpus and a speech recognition baseline,” in 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), 2017, p. 1–5. [21]G. Chen, S. Chai, G.-B. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, M. Jin, S. Khudanpur, S. Watanabe, S. Zhao, W. Zou, X. Li, X. Yao, Y. Wang, Z. You, and Z. Yan, “GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio,” in Proceedings of INTERSPEECH 2021, 2021. [Online]. Available: https://w.isca -archive.org/interspeech2021/chen21ointerspeech.html [22]S. Guan, S. Liu, J. Chen, W. Zhu, S. Li, X. Tan, Z. Yang, M. Xu, Y. Chen, C. Liang, J. Wang, and X.-L. Zhang, “Libri-adhoc40: A dataset collected from synchronized ad-hoc microphone arrays,” in 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Tokyo, Japan, Dec. 2021, p. 1116–1120. [Online]. Available: https://dblp.org/rec/conf/apsipa/GuanLCZLTYXCLWZ21 [23] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, p. 5206–5210. [24]K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio caption- ing dataset,” in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, p. 736–740. [25]M. Cartwright, A. Seals, J. Salamon, A. Williams, S. Mikloska, D. MacConnell, E. Law, J. P. Bello, and O. Nov, “Seeing sound: Investigating the effects of visualizations and complexity on crowd- sourced audio annotations,” Proceedings of the ACM on Human- Computer Interaction, vol. 1, no. 2, 2017. [26]M. Cartwright, A. Cramer, A. E. Mendez Mendez, Y. Wang, H.-H. Wu, V. Lostanlen, M. Fuentes, G. Dove, C. Mydlarz, J. Salamon, O. Nov, and J. P. Bello, “SONYC-UST-V2: An urban sound tagging dataset with spatiotemporal context,” in Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), 2020. [Online]. Available: https://dcase.community/documents/workshop2020/proceedings /DCASE2020WorkshopCartwright68.pdf [27]T. Heittola, A. Mesaros, and T. Virtanen, “TAU urban acoustic scenes 2020 mobile, development dataset,” Zenodo, 2020. [Online]. Available: https://zenodo.org/records/3670167 [28]“URBAN-SEDdataset,”Zenodo,2018,syntheticur- bansoundscapeswithsoundeventannotations(Sca- per; UrbanSound8K source material). [Online]. Available: https://zenodo.org/records/1324404 [29] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, “VGGSound: A large-scale audio-visual dataset,” in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, p. 721–725. [30]Q. Xi, R. M. Bittner, J. Pauwels, X. Ye, and J. P. Bello, “GuitarSet: A dataset for guitar transcription,” in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), 2018, p. 453–460. [31]C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in International Conference on Learning Representations (ICLR), 2019. [Online]. Available: https://openreview.net/forum?id=r1lYRjC9F7 [32]R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “MedleyDB: A multitrack dataset for annotation- intensive MIR research,” in Proceedings of the International Soci- ety for Music Information Retrieval Conference (ISMIR), 2014. [33]D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The MTG-jamendo dataset for automatic music tagging,” in Machine Learning for Music Discovery Workshop (ML4MD), International Conference on Machine Learning (ICML), Long Beach, CA, USA, 2019. [Online]. Available: http://hdl.handle.net/10230/42015 [34]A. Agostinelli, T. I. Denk, Z. Borsos, J. H. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. H. Frank, “MusicLM: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023, introduces the MusicCaps dataset. [Online]. Available: https://arxiv.org/abs/2301.11325 [35]J. Thickstun, Z. Harchaoui, and S. M. Kakade, “Learning features of music from scratch,” in International Conference on Learning Representations (ICLR), 2017. [Online]. Available: https://openreview.net/forum?id=rkFBJv9g [36] E. Manilow, G. Wichern, P. Seetharaman, and J. Le Roux, “Cut- ting music source separation some Slakh: A dataset to study the impact of training data quality and quantity,” in 2019 IEEE Work- shop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019, p. 45–49. [37] B. Weck, I. Manco et al., “The song describer dataset: a corpus of audio captions for music-and-language evaluation,” arXiv preprint arXiv:2311.10057, 2023. [Online]. Available: https://arxiv.org/abs/2311.10057 [38]D. McKee, J. Salamon, J. Sivic, and B. Russell, “Language-guided music recommendation for video via prompt analogies,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p. 14 784–14 793, introduces the YouTube8M-MusicTextClips dataset. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2023/html/McKe eLanguage-GuidedMusicRecommendationforVideoviaPro mptAnalogiesCVPR2023paper.html [39]J. Zhang, H. Dinkel, Y. Niu, C. Liu, S. Cheng, A. Zhao, and J. Luan, “X-ARES: A comprehensive framework for assessing audio encoder performance,” arXiv preprint arXiv:2505.16369, 2025. [40] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 139. PMLR, 2021, p. 8748–8763. [Online]. Available:https: //proceedings.mlr.press/v139/radford21a.html [41] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dub- nov, “Large-scale contrastive language-audio pretraining with fea- ture fusion and keyword-to-caption augmentation,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, p. 1–5. [42]H. Dinkel, Z. Yan, T. Wang, Y. Wang, X. Sun, Y. Niu, J. Liu, G. Li, J. Zhang, and J. Luan, “Glap: General contrastive audio- text pretraining across domains and languages,” arXiv preprint arXiv:2506.11350, 2025. [43]H. Dinkel et al., “Scaling up masked audio encoder learning for general audio classification,” in Proc. Interspeech 2024, 2024. [Online]. Available: https://w.isca-archive.org/interspeech202 4/dinkel24binterspeech.html [44]P.-H. Yang et al., “SUPERB: Speech processing universal performance benchmark,” in Proc. Interspeech 2021, 2021, p. 1194–1198. [Online]. Available: https://w.isca-archive.org/int erspeech2021/yang21cinterspeech.html [45]J. Zhang and Y. Niu, “Librispeechmalefemale in webdataset format,” Zenodo, Jan. 2025, version v2. A WebDataset-format repackaging of LibriSpeech (train-clean-100, dev-clean, test- clean) with speaker gender metadata. [Online]. Available: https://zenodo.org/records/14716252 [46]S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” PLOS ONE, vol. 13, no. 5, p. e0196391, 2018. [47]A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A large- scale speaker identification dataset,” in Interspeech 2017, 2017, p. 2616–2620. [48]Z. Wu, T. Kinnunen, N. Evans, J. Yamagishi, C. Hanilc ̧i, M. Sahidullah, and A. Sizov, “ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,” in Interspeech 2015, 2015, p. 2037–2041. [49] S. Liu, A. Mallol-Ragolta, E. Parada-Cabaleiro, K. Qian, X. Jing, A. Kathan, B. Hu, and B. W. Schuller, “Audio self-supervised learning: A survey,” Patterns, vol. 3, no. 12, p. 100616, 2022. [Online]. Available: https://w.sciencedirect.com/science/articl e/pii/S2666389922002410 [50] P. He, Z. Wen, Y. Wang, Y. Wang, X. Liu, J. Huang, Z. Lei, Z. Gu, X. Jin, J. Yang et al., “Audiomarathon: A comprehensive bench- mark for long-context audio understanding and efficiency in audio llms,” arXiv preprint arXiv:2510.07293, 2025. [51]D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, H. Guo, X. Chang, J. Shi, S. Zhao, J. Bian, Z. Zhao, X. Wu, and H. M. Meng, “UniAudio: Towards universal audio generation with large language models,” in Proceedings of the 41st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, Eds., vol. 235. PMLR, 21–27 Jul 2024, p. 56 422–56 447. [Online]. Available: https://proceedings.mlr.press/v235/yang24x.html [52] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Towards a universal audio pre-training framework,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, p. 2391–2406, 2024. [Online]. Available: https://doi.org/10.1109/TASLP.2024.3389636 [53]C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech representation learning with labeled and unlabeled data,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, p. 10 937–10 947. [Online]. Available: https://proceedings.mlr.press/v139/wang21y.html [54]M. Ni, H. Huang, L. Su, E. Cui, T. Bharti, L. Wang, D. Zhang, and N. Duan, “M3P: learning universal representations via multitask multilingual multimodal pre-training,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021.Computer Vision Foundation / IEEE, 2021, p. 3977–3986. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2021/html/NiM3P LearningUniversalRepresentationsviaMultitaskMultilingua lMultimodalPre-TrainingCVPR2021paper.html [55]G. Kim, H. Wu, L. Bondi, and B. Liu, “Multi-modal continual pre-training for audio encoders,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024.IEEE, 2024, p. 691–695. [Online]. Available: https: //doi.org/10.1109/ICASSP48485.2024.10446424 [56]N. M. Selvaraj, X. Guo, A. Kong, B. Shen, and A. Kot, “Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers,” in Interspeech 2023, 2023, p. 909–913. [57] Z. Li, W. Wang, Y. Cai, Q. Xu, P. Wang, D. Zhang, H. Song, B. Jiang, Z. Huang, and T. Wang, “UnifiedMLLM: Enabling unified representation for multi-modal multi-tasks with large language model,” in Findings of the Association for Computational Linguistics: NAACL 2025.Albuquerque, New Mexico: Association for Computational Linguistics, Apr. 2025, p. 334–344. [Online]. Available: https://aclanthology.org/2025. findings-naacl.19/