Paper deep dive
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami, Dinesh Manocha
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/15/2026, 7:46:39 AM
Summary
The paper introduces TEMPO, a unified Large Audio-Language Model (LALM) capable of temporally-grounded multi-task post-training for audio, speech, and music. TEMPO addresses the limitation of existing LALMs that lack timestamping capabilities by employing a supervised fine-tuning (SFT) stage with atomic timestamp tokens, a time-aware projector, and a distance-aware Gaussian loss, followed by a reinforcement learning (RL) refinement stage using Group Relative Policy Optimization (GRPO). The model outperforms state-of-the-art baselines like Audio Flamingo Next and Qwen3-Omni on a benchmark of 10K samples across five tasks: multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning.
Entities (23)
Relation Signals (20)
TEMPO → isbasedon → Audio-Flamingo-3
confidence 95% · Tempo is built on Audio Flamingo 3 (AF3) (Goel et al., 2025)
TEMPO → performstask → Timestamped Music Captioning
confidence 95% · Tempo handles all five tasks with a single decoder... timestamped music captioning
TEMPO → performstask → Multi-speaker ASR
confidence 95% · Tempo handles all five tasks with a single decoder... multi-speaker automatic speech recognition (ASR)
TEMPO → performstask → Speaker Diarization
confidence 95% · Tempo handles all five tasks with a single decoder... speaker diarization
TEMPO → performstask → Audio Temporal Grounding
confidence 95% · Tempo handles all five tasks with a single decoder... audio temporal grounding
TEMPO → performstask → Dense Audio Captioning
confidence 95% · Tempo handles all five tasks with a single decoder... dense audio captioning
TEMPO → usesalgorithm → GRPO
confidence 95% · We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO
TEMPO → usescomponent → Time-Aware Projector
confidence 95% · Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: ... a time-aware projector...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.
Tags
Links
- Source: https://arxiv.org/abs/2608.29999v1
- Canonical: https://arxiv.org/abs/2608.29999v1
Trouble viewing inline? Open PDF directly →
Full Text
78,867 characters extracted from source content.
Expand or collapse full text
Tempo: Temporally-grounded Multi-task Post-training for Large Audio-Language Models Apoorva Kulkarni11footnotemark: 1 Affiliation: University of Maryland, College Park, USA Kaousheik Jayakumar11footnotemark: 1 Affiliation: University of Maryland, College Park, USA Sreyan Ghosh Affiliation: University of Maryland, College Park, USA Utathya Aich Affiliation: CNH Industrial IndiaCorrespondence:apoorvak@umd.eduProject: https://kaousheik-26.github.io/tempo/ Ramani Duraiswami22footnotemark: 2 Affiliation: University of Maryland, College Park, USA Dinesh Manocha22footnotemark: 2 Affiliation: University of Maryland, College Park, USA Abstract Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present Tempo (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, Tempo outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements. †footnotetext: ∗Equal Contribution. †Equal Advising. 1 Introduction Audio signals carry rich temporal information. Answering what happens when in an audio stream is fundamental to applications such as meeting transcription and music analysis, which require associating specific events with precise time intervals. Yet most existing Large Audio-Language Models (LALMs) treat audio understanding as a clip-level task, mapping a recording to a single caption or answer and discarding temporal structure (Ghosh et al., 2024; Goel et al., 2025; Kong et al., 2024; Ghosh et al., 2025b). Timestamping is therefore a foundational capability that current LALMs lack. In this work, we focus on five timestamping tasks that span speech, music and audio: multi-speaker automatic speech recognition (ASR), speaker diarization, audio temporal grounding, dense audio captioning and timestamped music captioning. Despite differing in their outputs, all five share a common structure: each requires segmenting an audio into intervals and assigning a textual label to each one. Recent LALM-based approaches such as TAC (Kumar et al., 2026) and TimeAudio (Wang et al., 2025) address timestamping within a unified model but remain restricted to limited tasks. We present Tempo, the first unified LALM for timestamping across audio, speech, and music. Tempo handles all five tasks with a single decoder, emitting text with timestamp tokens distinguished only by a task tag in the prompt. Our supervised fine-tuning (SFT) recipe builds on three components: atomic timestamp tokens, a time-aware multi-modal projector that injects sinusoidal wall-clock encodings into audio frame embeddings and a distance-aware Gaussian loss. Training follows a two-stage synthetic-to-real curriculum. Building on this, we present the first application of reinforcement learning to timestamped audio understanding, using Group Relative Policy Optimization (GRPO) (Guo et al., 2025) with verifiable rewards to train directly against the temporal metrics used at evaluation. On a benchmark constructed from established corpora, Tempo substantially outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Gains are largest on speech: Tempo cuts WER from 0.70 to 0.44 on multi-speaker ASR and improves diarization mIoU from 0.44 to 0.71 over Qwen3-Omni, with consistent improvements across audio and music tasks as well. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements, suggesting that careful post-training design is the primary lever for adding timestamping capabilities to LALMs. Our main contributions include: • First unified LALM for timestamping across speech, sound, and music. Tempo performs speaker diarization, multi-speaker ASR, audio temporal grounding, dense audio captioning, and timestamped music captioning in a single model, outperforming Qwen3-Omni, Audio Flamingo Next and other LALM-based timestamping baselines. • A post-training SFT recipe that teaches timestamping from scratch. Three innovations, namely atomic timestamp tokens at 0.1 s resolution, a time-aware multi-modal projector with sinusoidal wall-clock encodings, and a distance-aware Gaussian loss, together teach a model with no prior temporal supervision to predict accurate timestamps. • First unified RL approach across speech, audio, and music timestamping. We apply GRPO with task-specific verifiable temporal rewards jointly across speech, sound, and music. Starting from the SFT checkpoint, GRPO serves as a refinement that directly optimizes the evaluation objectives, yielding modest additional gains on the speech and sound tasks. • Multi-task benchmark and training corpus. 119K training examples (51K synthetic, 68K real-world) and 10K evaluation examples drawn from established corpora, covering all five timestamping tasks under a unified output format. 2 Related Work 2.1 Large Audio-Language Models and Timestamping Recent LALMs including SALMONN (Tang et al., 2024), Qwen2-Audio (Chu et al., 2024), the Audio Flamingo series (Kong et al., 2024; Ghosh et al., 2025b; Goel et al., 2025; Ghosh et al., 2026), GAMA (Ghosh et al., 2024), and Kimi-Audio (KimiTeam et al., 2025) have demonstrated strong capabilities on captioning, question answering, and open-ended reasoning, but temporal grounding remains a persistent weakness. Recent studies further show that even strong LALMs struggle with fine-grained temporal reasoning, including event ordering, duration, and onset/offset estimation (Bhattacharya et al., 2025), (Kulkarni et al., 2026). These findings further motivate explicit temporal supervision beyond clip-level audio understanding. The closest prior work, TAC (Kumar et al., 2026) and TimeAudio (Wang et al., 2025), address timestamping within a unified model but are restricted to limited tasks and rely on supervised training. Tempo differs from its closest prior work, TAC, in both training data and task coverage. TAC uses only synthetic data for dense audio captioning and delegates speech transcription to Whisper (Radford et al., 2023), whereas Tempo uses publicly available human-annotated data and jointly models five tasks spanning speech, sound, and music. Consequently, TAC does not support speaker-attributed multi-speaker ASR or speaker diarization. TAC also identifies timestamped chord progression generation as a limitation, whereas Tempo directly predicts structured timestamped chord progressions. We do not report a direct numerical comparison because public TAC model weights and inference code are unavailable. Task Question Ground Truth Response ASR Provide a timestamped transcription of the speech in this audio. [speech:asr] <|0.0|> The memory uh <|2.2|> [speech:asr] <|3.4|> Pops up the options <|4.5|> [speech:asr] <|4.9|> Yeah yeah that would be possible yeah sure <|7.1|> [speech:asr] <|7.5|> I th dont think thats uh that takes a lot of storage space …<|14.0|> Diarization How many speakers are in this audio and when does each one speak? Include timestamps. [speech:diar] <|0.0|> Speaker 1 <|6.2|> [speech:diar] <|4.6|> Speaker 2 <|6.1|> [speech:diar] <|5.9|> Speaker 3 <|16.9|> [speech:diar] <|13.5|> Speaker 1 <|27.9|> Audio Grounding What is the time interval (start and end) for the query ‘a bird tweets quietly and repeatedly outdoors’ in the audio? [audio:ground] <|0.9|> to <|1.4|> [audio:ground] <|2.3|> to <|2.8|> [audio:ground] <|3.7|> to <|4.3|> [audio:ground] <|5.2|> to <|5.7|> Dense Audio Captioning Which sound events occur, and what are their time intervals and descriptions? [audio:caption] <|0.0|> A river flows loudly outdoors. <|23.9|> [audio:caption] <|0.9|> A bird tweets quietly and repeatedly outdoors. <|1.4|> [audio:caption] <|2.3|> A bird tweets quietly and repeatedly outdoors. <|2.8|> [audio:caption] <|3.7|> A bird tweets quietly …<|4.3|> Music Captioning Caption this music with timestamps. Describe the instrument, tempo, chord progression, dynamics, and note statistics. [audio:music][instrument] Organ enters at <|0.0|> exits at <|30.0|> [tempo] 120.0 BPM at <|0.0|> [tempo] 125.0 BPM at <|9.5|> [chord] B:min from <|0.0|> to <|3.5|> [chord] D:sus2 from <|3.5|> to <|21.0|> [chord] C:sus2 from <|21.0|> to <|28.5|> [chord] D:min7 from <|28.5|> to <|29.0|> [chord] G:sus2 from <|29.0|> to <|29.5|> [chord] D:sus2 from <|29.5|> to <|30.0|> [stats] Note density:0.6 notes/sec, range: D5 to G6 Table 1: Example instances from each evaluation task. Ground truth responses are truncated for brevity. 2.2 Speech: Diarization and Multi-Speaker ASR Diarization has evolved from clustering pipelines to end-to-end and target-speaker models; multi-speaker ASR extends this with joint transcription via serialized output training. Recent LLM-based methods include DiarizationLM (Wang et al., 2024b), which refines outputs via post-processing and SpeakerLM (Yin et al., 2026) jointly models speaker and language. Unlike these speech-only systems, Tempo jointly predicts who, what, and when within a general-purpose framework trained across non-speech tasks as well. 2.3 Sound: Event Detection, Temporal Grounding and Dense Captioning Sound event detection (SED) and the DCASE challenge series provide established formulations and evaluation protocols for temporally localizing acoustic events (Mesaros et al., 2021; Politis et al., 2021). Recent work has also connected SED with language models for generating event labels and temporal locations (Wang et al., 2024a), (Yang et al., 2026). Audio grounding localizes a text query to a time interval, while dense captioning requires segmenting an audio stream and describing each event with timestamps. Standard benchmarks like AudioCaps (Kim et al., 2019) and Clotho (Drossos et al., 2019) provide only clip-level annotations, inducing semantic collapse where distinct events are compressed into global summaries (Kumar et al., 2026). Recent efforts address this via synthetic dense annotation pipelines and frame-level temporal indexing (Wang et al., 2025). 2.4 Music: Chord Estimation and Captioning Automatic chord estimation traditionally combines acoustic classifiers with harmonic language models, evaluated on McGill Billboard (Burgoyne et al., 2011) using MIREX-standard metrics, but these systems are specialized classifiers that do not generalize beyond chord recognition. Music Flamingo (Ghosh et al., 2025a) introduces rotary time embeddings for temporal localization and produces theory-aware captions that may reference chord progressions in natural language, but does not generate structured, timestamped chord labels. To our knowledge, no existing audio-language model produces structured, timestamped chord progressions from audio; we integrate this as one of five tasks in Tempo. 2.5 Reinforcement Learning for LALMs The DeepSeek-R1 pipeline (Guo et al., 2025), which combines an SFT cold start with Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and verifiable rewards, has proven effective for text and vision-language models. Recent audio work has begun to apply RL post-training to LALMs: Audio-Thinker (Wu et al., 2025) uses GRPO to guide audio reasoning while R1-AQA (Li et al., 2025) explores RL post-training for audio QA. However, these efforts target reasoning or classification accuracy and optimize text-level metrics. We are the first to apply GRPO with temporal verifiable rewards across a unified multi-task LALM. 3 Tasks and Datasets We train Tempo on five timestamping tasks spanning speech, sound, and music using a combination of real and synthetic data. Table 1 shows example inputs and outputs for each task. 3.1 Real Data Multi-Speaker ASR and Diarization. We use AMI (Carletta et al., 2005), ICSI (Shriberg et al., 2004), and Switchboard (Godfrey et al., 1992), three conversational corpora with speaker-labeled transcripts and precise timestamps, enabling evaluation of both joint transcript-timing prediction and speaker turn segmentation, yielding 16,045 ASR and 16,053 diarization samples. Audio Temporal Grounding. We use AudioSet Strong (Hershey et al., 2021), a strongly-labeled subset of AudioSet (Gemmeke et al., 2017) that provides onset and offset times for over 500 sound classes. To convert the categorical event labels into natural-language queries, we prompt GPT-4o to generate paraphrased queries for each class, yielding 13,500 training samples. Dense Audio Captioning. We directly use TACOS (Primus et al., 2025), which pairs audio recordings with captions and onset/offset timestamps, yielding 9,858 data points for this task. Timestamped Music Captioning. We use Slakh2100 (Manilow et al., 2019), a collection of 2,100 multi-track songs synthesized from MIDI, from which we derive timestamped chord, tempo, and instrument annotations via the aligned MIDI files, yielding 13,000 samples. In total, the real-world corpus contains 68,456 training examples. Figure 1: Overview of Tempo, built on Audio Flamingo 3. Top: Three-stage curriculum from synthetic SFT (51K) through real-data SFT (33K) to GRPO with verifiable rewards (35K). Bottom-left: Atomic timestamp tokens replace BPE fragmentation with ∼601 601 dedicated tokens at 0.1 s resolution. Bottom-right: Time-aware projector injects sinusoidal wall-clock encodings φ(t) (t) into frozen audio encoder features before the MLP. 3.2 Synthetic Data To improve temporal calibration before real-data training, we also construct a synthetic Stage-1 corpus from LibriSpeech (Panayotov et al., 2015) and ESC-50 (Piczak, 2015) covering ASR, diarization, DAC, and AG tasks, with timestamp tokens balanced uniformly across all possible values. We generate 8,783 ASR samples, 8,783 diarization samples, 11,962 AG samples and 11,984 DAC samples. We add 10,000 samples from Slakh2100 to this stage to ensure exposure to all tasks. This yields 51,512 training examples in total. Full details are in Appendix A. 4 Method Tempo is built on Audio Flamingo 3 (AF3) (Goel et al., 2025), which couples a Whisper-large audio encoder to a Qwen2-7B language model through a two-layer MLP multi-modal projector. The audio encoder is frozen throughout training, and the five tasks are distinguished by task-specific tags in the prompt: [speech:asr], [speech:diar], [audio:caption], [audio:ground], and [audio:music]. Figure 1 gives an overview of TEMPO’s architecture and training pipeline. 4.1 Supervised Fine-Tuning We begin with a supervised fine-tuning (SFT) stage that teaches the model the timestamped output format and provides the temporal calibration needed before reinforcement learning. A standard teacher-forced cross-entropy objective is insufficient for fine-grained timestamping for three reasons. First, numeric timestamps rendered as text are tokenized inconsistently by the BPE vocabulary, fragmenting supervision across subword pieces. Second, the AF3 audio encoder produces features indexed by frame rather than wall-clock time, requiring the model to learn the correspondence implicitly. Third, token-level cross-entropy penalizes all incorrect timestamps equally, which is misaligned with the downstream temporal IoU (tIoU) metric. Our SFT recipe addresses each of these in turn through a corresponding architectural or objective change: atomic timestamp tokens, a time-aware multi-modal projector, and a distance-aware Gaussian loss. 4.1.1 Atomic Timestamp Tokens To prevent BPE from fragmenting timestamps across subword pieces, we extend the tokenizer with atomic timestamp tokens of the form ⟨|t|⟩ |t| , where t∈0.0,0.1,…,60.0t∈\0.0,0.1,…,60.0\ at 0.10.1 s resolution. Each timestamp is represented by a single token, so a timestamp prediction becomes a single categorical decision over approximately 600 candidates. New token embeddings are initialized as the mean of the BPE decomposition of the corresponding numeric value. 4.1.2 Time-Aware Multi-Modal Projector To supply the wall-clock information that frame-indexed features lack, we replace the AF3 projector, whose positional embeddings encode frame index rather than elapsed time, with one that injects sinusoidal wall-clock encodings into each frame embedding: t=Linear2(σ(Linear1(t+Wτϕ(t)))),h_t=Linear_2\! (σ(Linear_1(x_t+W_τφ(t))) ), (1) where tx_t is the t-th audio frame, ϕ(t)φ(t) is a fixed sinusoidal encoding of wall-clock time t=i⋅0.04t=i· 0.04 s, and WτW_τ is a learned projection initialized from (0,0.012)N(0,0.01^2). We use dτ=64d_τ=64 with frequencies logarithmically spaced between periods of 0.080.08 s and 6060 s, so that a single projector can span both sub-second acoustic events and long-range musical structure without task-specific parameterization. Figure 2: Standard cross-entropy (top) uses a one-hot target, penalizing all errors equally. Our distance-aware Gaussian target (bottom, σt=0.3 _t=0.3 s) assigns partial credit to near-miss predictions, encouraging the model to learn the ordinal structure of the timestamp vocabulary. 4.1.3 Distance-Aware Timestamp Loss Token-level cross-entropy treats every wrong timestamp as equally wrong, discarding the ordinal structure of the timestamp vocabulary and decoupling the training signal from the tIoU metric, under which a near miss is far better than a distant one. We restore this structure with an auxiliary loss that places a Gaussian soft label over the timestamp vocabulary, centered at the ground-truth time (Figure 2). If the target is t⋆t and the timestamp vocabulary corresponds to times tkk=1K\t_k\_k=1^K, the soft-label distribution is qk∝exp(−(tk−t⋆)22σt2),∑k=1Kqk=1.q_k \! (- (t_k-t )^22 _t^2 ), _k=1^Kq_k=1. (2) The auxiliary loss is ℒtime=−∑kqklogpk,L_time=- _kq_k p_k, (3) where pkp_k is the model probability over the timestamp vocabulary obtained by restricting the output logits to timestamp tokens and applying a softmax. The overall training objective is ℒ=ℒCE+λtimeℒtimeL=L_CE+ _timeL_time (4) with λtime=0.5 _time=0.5 and σt=0.3 _t=0.3 s, where ℒCEL_CE is the standard cross-entropy over all output positions including timestamps. 4.1.4 Two-Stage Synthetic-to-Real Curriculum The three components above define what the model learns to predict; the curriculum determines the order in which it sees supervision. Stage 1 trains on the 51,51251,512 synthetic examples described in Section 3 to establish temporal calibration before exposure to real data. Stage 2 then fine-tunes the merged Stage 1 checkpoint on 32,72632,726 real-world examples taken from the real-world corpus. Implementation details. During SFT, LoRA adapters (r=128r=128, α=256α=256, dropout 0.10.1) are attached to the language-model attention and MLP projection matrices, while the multi-modal projector and timestamp token embeddings are trained directly. Stage 1 runs for two epochs at learning rate 1×10−41× 10^-4 with 150 warmup steps; Stage 2 runs for two additional epochs at learning rate 5×10−55× 10^-5 with 50 warmup steps. 4.2 Reinforcement Learning with GRPO Starting from the SFT checkpoint, we apply Group Relative Policy Optimization (GRPO) (Guo et al., 2025) with verifiable rewards designed to directly optimize the temporal metrics used at evaluation. For each task, we construct a reward that averages the eval metrics themselves, so that maximizing reward and improving eval performance are aligned. All rewards are bounded in [0,1][0,1] and gated by a binary format check that returns zero if the output does not parse into the expected task tag with valid timestamp tokens. ASR and diarization. Both tasks use equally-weighted average of three eval metrics. For ASR, the reward averages (1−WER)(1-WER), symmetric mIoU between predicted and ground-truth segments, and a boundary score exp(−MAE/τ) (-MAE/τ) (with τ=1.0τ=1.0 s) computed over greedy-matched pairs with IoU >> 0.3. For diarization, the same structure averages (1−DER)(1-DER), symmetric mIoU, and a soft speaker-labeled F1@IoU0.5 that requires exact speaker-label matches. Because both tasks frequently produce overlapping segments, we scale this averaged reward by a soft factor (0.8+0.2⋅p)(0.8+0.2· p), where p is the fraction of non-overlapping output time. Dense audio captioning. The reward averages three equally-weighted terms: soft event F1@IoU0.5, symmetric mIoU, and a per-pair METEOR computed over greedy-matched predicted and ground-truth segments. This rewards both correct localization of sound events and accurate textual descriptions of each, jointly aligning the model with the three eval metrics rather than treating them as independent objectives. Audio grounding. The reward averages soft F1@IoU0.5, symmetric mIoU, and a boundary score with τ=2.0τ=2.0 s (larger than ASR’s τ=1.0τ=1.0 s to accommodate the naturally noisier boundaries of environmental sound events). A light count-ratio term (weight 0.250.25 out of 3.253.25 total, or 7.7%) discourages large mismatches between predicted and ground-truth interval counts. The weight is intentionally small because precision and recall in the F1 term already penalize over- and under-prediction; the count ratio acts as a soft regularizer rather than a hard gate. Music captioning. The reward is a weighted sum of five terms gated by a format check (weight 0.050.05) that verifies all required section tags. The dominant component is chord accuracy (weight 0.450.45): each ground-truth chord is scored against its best-overlapping prediction as 0.3⋅tIoU+0.7⋅sim0.3·tIoU+0.7·sim, where similarity weights root pitch (0.600.60), quality (0.250.25), and extension (0.150.15), with a false-positive penalty (0.7+0.3⋅min(|G|/|P|, 1)) (0.7+0.3· (|G|/|P|,\,1) ). Instrument accuracy (weight 0.200.20) averages name matching with entry/exit timing error under a 2.02.0 s tolerance. Tempo accuracy (weight 0.150.15) combines 0.7⋅BPMaccuracy+0.3⋅timestampaccuracy0.7·BPM\;accuracy+0.3·timestamp\;accuracy with the same tolerance. Statistics accuracy (weight 0.150.15) averages relative note-density error and exact pitch-range matching. The chord boundary tolerance is relaxed to 1.01.0 s (from 0.50.5 s) to reduce reward sparsity early in training. Implementation details. GRPO training is initialized from the Stage 1+2 SFT checkpoint. We attach LoRA adapters (r=256r=256, α=512α=512) to the language model and train for 1,0001,000 steps at learning rate 1×10−51× 10^-5. Each step generates 88 completions per prompt at temperature 1.21.2, with a maximum completion length of 768768 tokens. The KL coefficient is set to β=0.01β=0.01. 5 Experiments and Results We evaluate Tempo on held-out test sets across all five tasks, using metrics that capture both the textual and temporal aspects of each output. 5.1 Evaluation Benchmarks We evaluate Tempo on held-out test sets across all five tasks. For dense audio captioning, we use the official TACOS test set: 2,000 question-answer pairs in which each query asks the model to caption all sound events in a clip along with their temporal boundaries. For audio temporal grounding, we reformat the TACOS test set into the grounding format used by TimeAudio (Wang et al., 2025), in which each strong caption becomes a natural-language query and its associated segment becomes the ground-truth interval, yielding 5,151 question-answer pairs. For speaker diarization and multi-speaker ASR, we evaluate on the AMI Meeting test set, segmented into chunks of up to 60 seconds for consistency with our training setup; this yields 1,181 question-answer pairs per task, with the same audio chunks used for both and the expected output differing only in whether transcription is required. For timestamped music captioning, we evaluate on the Slakh2100 test set (225 tracks), split into 30-second segments to obtain 1,000 question-answer pairs, where each segment requires the model to produce timestamped chord labels, instrument entry and exit points, and tempo. 5.2 Evaluation Metrics We report standard metrics for each task; full computational definitions are given in Appendix B. Speech metrics. For multi-speaker ASR we report Word Error Rate (WER), computed over predicted and ground-truth segments matched by temporal IoU (Appendix B). For diarization we report Diarization Error Rate (DER) using the standard pyannote.metrics implementation (Bredin, 2017) with a 0.250.25 s collar; speaker labels are optimally permuted between hypothesis and reference before scoring, so a model that produces consistent but arbitrarily named speakers is not penalized for the naming alone. Mean Intersection-over-Union (mIoU). For all tasks that produce time intervals, we report a symmetric mIoU computed as the harmonic mean of recall-side and precision-side mean-best-IoU. Unlike the more common recall-only mIoU, the symmetric formulation prevents over-prediction from inflating the score, since a model that emits many spurious intervals is penalized on the precision side. F1 at IoU threshold and boundary MAE. For dense audio captioning and audio grounding we report event F1@IoU0.5 (eF1@IoU), counting a predicted interval as a true positive if its IoU with a greedily-matched ground-truth interval is at least 0.50.5. For diarization we additionally report a speaker-labeled F1@IoU0.5 (sF1@IoU) that requires exact speaker-label agreement in addition to temporal overlap. For tasks with continuous time boundaries (ASR, grounding, dense captioning, music) we report boundary MAE in seconds, computed over greedy-matched pairs with IoU≥0.3IoU≥ 0.3. Caption and music metrics. For dense audio captioning we additionally report METEOR over matched segment pairs. For music captioning we report chord root and quality accuracy, chord F1@IoU0.5 and chord mIoU, instrument-name F1 and instrument boundary MAE, and tempo accuracy within a 4%4\% tolerance. Full definitions of all caption and music metrics are in Appendix B. 5.3 Compared Methods We compare against three baseline families, frontier models, and ablations of our own pipeline. Zero-shot LALMs (Audio Flamingo 3, Audio Flamingo Next, and Qwen3-Omni) are evaluated without task-specific training. We additionally evaluate two frontier models, Gemini 2.5 Flash and Gemini 2.5 Pro (Comanici et al., 2025), across all supported tasks. TimeAudio (Wang et al., 2025), the closest prior open-source LALM-based timestamping approach, is evaluated on the two tasks it supports: dense captioning and audio grounding. Naive SFT follows our two-stage curriculum without the three proposed SFT innovations, isolating the effect of the training recipe from the data. Stage 1, Stage 2, and Stage 1+2 denote the SFT checkpoints obtained after the corresponding curriculum stages. Both RL variants are initialized from the Stage 1+2 checkpoint: single-task RL performs a separate GRPO stage for each task using task-specific data and rewards, whereas multi-task RL jointly optimizes all tasks in a single GRPO stage. We also compare against specialized systems, using one off-the-shelf dedicated model per task: Parakeet-TDT-0.6b-v2 (Xu et al., 2023) for ASR, pyannote-3.1 (Bredin, 2023) for diarization, PretrainedSED/BEATs (Chen et al., 2022) for dense captioning, CLAP-window (Wu et al., 2024) for audio grounding, Chordino (Mauch and Dixon, 2010) for chord estimation, and madmom (Böck et al., 2016) for tempo estimation. All systems are evaluated on the same held-out test sets where applicable, and all generative-model results are obtained using deterministic greedy decoding. 5.4 Main Results Table 2: Performance across all evaluation tasks. Speech: WER: word error rate (%); DER: diarization error rate (collar=0.25s) (%). Sound: eF1: event F1@IoU (%); MET: METEOR (%); F1: F1@IoU0.5 (%). Music: Root/Qual: chord root/quality accuracy (%); Ch.F1: chord F1@IoU0.5 (%); Inst F1: instrument name F1 (%); Tempo: tempo accuracy within 4% tolerance (%). Shared: MAE: boundary MAE (s, over matches with IoU≥ 0.3); mIoU: symmetric mean IoU (%); sF1: speaker-labeled F1@IoU (%). ↓ : lower is better; ↑ : higher is better. †Specialist systems (off-the-shelf, one dedicated model per task): Parakeet-TDT-0.6b-v2 (ASR); pyannote-3.1 (diarization); PretrainedSED/BEATs (dense captioning); CLAP-window (audio grounding, instrument); Chordino (chord); madmom (tempo). Model Speech Sound Music ASR Diarization Dense Captioning Audio Grounding Chord Instrument Tempo MAE↓ mIoU↑ WER↓ DER↓ mIoU↑ sF1↑ eF1↑ mIoU↑ MET↑ F1↑ MAE↓ mIoU↑ Root↑ Qual↑ F1↑ mIoU↑ F1↑ MAE↓ Acc↑ Audio Flamingo 3 0.99 19.4 209.7 103.2 2.9 1.0 3.8 5.7 2.7 3.3 1.25 5.6 0.4 0.9 0.1 2.1 33.7 2.84 18.0 Audio Flamingo Next 1.39 11.5 167.1 106.9 12.4 7.6 11.6 22.3 11.9 3.6 0.95 7.0 0.4 1.2 0.0 3.2 33.5 2.77 26.2 Qwen3-Omni 1.12 31.6 69.7 44.2 44.4 31.5 57.9 65.4 19.0 47.4 1.10 49.3 1.1 2.8 0.5 6.9 25.4 2.51 40.3 TimeAudio – – – – – – 43.6 64.6 19.9 40.1 1.32 46.3 – – – – – – – SFT (Naive) - Stage 1 + 2 0.84 64.0 46.0 27.4 67.2 52.3 52.4 64.5 22.7 37.1 1.44 40.3 0.3 0.3 0.3 0.3 0.5 3.95 37.8 SFT (Ours) - Stage 1 2.42 41.7 94.0 74.3 43.3 25.1 47.6 50.9 10.5 34.3 3.10 36.2 19.4 32.4 5.8 75.5 96.4 2.71 47.5 SFT (Ours) - Stage 2 0.85 64.8 47.0 25.1 70.0 54.5 55.2 67.8 20.2 44.6 1.14 44.7 17.4 31.9 4.6 76.0 89.4 2.78 41.7 SFT (Ours) - Stage 1 + 2 0.77 63.8 44.7 25.4 70.5 54.9 58.5 68.3 22.3 46.2 1.19 48.5 19.6 32.7 6.2 76.2 96.3 2.63 47.3 SFT + Single-task RL 0.84 66.1 46.7 26.8 69.2 54.0 58.6 67.8 24.4 45.5 1.21 48.0 19.0 32.2 6.0 75.5 96.3 2.63 47.0 SFT + Multi-task RL 0.76 65.8 43.5 25.4 71.1 56.3 59.3 68.5 23.5 46.5 1.32 49.4 18.7 31.7 5.5 75.3 96.0 2.62 46.9 Gemini 2.5 Flash 1.01 43.6 49.4 50.2 34.8 21.6 21.5 52.1 10.7 27.4 1.20 42.5 8.6 19.2 1.1 30.4 65.3 3.41 11.2 Gemini 2.5 Pro 0.99 42.2 46.9 27.6 51.7 42.0 41.5 63.4 10.7 36.9 1.09 48.7 8.8 15.5 0.8 25.0 77.4 3.03 12.9 Specialized systems† 1.27 42.7 31.4 19.6 55.3 48.8 35.0 48.8 3.9 30.8 1.83 43.6 18.8 23.4 2.0 23.8 61.2 – 45.0 Multi-speaker ASR. Existing LALMs perform poorly on multi-speaker ASR with timestamps under zero-shot evaluation. Qwen3-Omni performs best among baseline LALMs (WER of 69.7%) but still fails to align transcripts to accurate temporal boundaries (MAE of 1.12 s, mIoU of 31.6%). Tempo with full SFT (Stage 1+2) achieves a WER of 44.7%, a 25-point absolute improvement over Qwen3-Omni, alongside the boundary MAE of 0.77 s and mIoU of 63.8%. Adding multi-task RL on top of the SFT checkpoint yields further improvements: WER drops to 43.5%, MAE to 0.76 s, and mIoU rises to 65.8%. Speaker diarization. Qwen3-Omni achieves a reasonable DER of 44.2% but speaker-labeled F1 (sF1) remains low at 31.5%. Tempo reduces DER to 25.4% and raises speaker-labeled F1 to 54.9% and mIoU to 70.5%. The Stage 1+2 curriculum provides a clear gain over Stage 2 alone (54.9% vs 54.5% sF1, 70.5% vs 70.0% mIoU), validating the synthetic-to-real curriculum design. Multi-task RL further improves sF1 to 56.3% and mIoU to 71.1% at the same DER. Dense audio captioning. Tempo surpasses Qwen3-Omni on both metrics with 58.5% eF1 and 68.3% mIoU, while matching it closely on METEOR (22.3% vs 19.0%). TimeAudio, the closest prior LALM-based approach, achieves 43.6% eF1 and 64.6% mIoU on the same evaluation, which Tempo exceeds by 14.9 and 3.7 absolute points respectively. Multi-task RL pushes performance further to 59.3% eF1 and 68.5% mIoU. Audio temporal grounding. Audio grounding shows the largest gap between zero-shot baselines and trained models. Audio Flamingo 3 and Audio Flamingo Next achieve F1@IoU0.5 of only 3.3% and 3.6%, respectively, indicating that these models lack the ability to localize natural-language queries to specific time intervals. Qwen3-Omni performs better at 47.4% F1 but boundary MAE remains high at 1.10 s. Tempo with full SFT reaches 46.2% F1 with boundary MAE of 1.19 s and mIoU of 48.5%. TimeAudio achieves 40.1% F1 and 46.3% mIoU on the same eval, which Tempo exceeds by 6.1 and 2.2 absolute points. Multi-task RL improves F1 to 46.5% and mIoU to 49.4%. Timestamped music captioning. On Slakh2100, the zero-shot baselines fail almost completely on chord-related metrics: all three models produce chord F1@IoU below 1% and chord root accuracy below 2%, indicating that none has learned to associate harmonic events with precise timestamps. Tempo accuracy is also poor for the LALM baselines (18.0% for Audio Flamingo 3, 40.3% for Qwen3-Omni). Tempo substantially closes these gaps: Stage 1+2 reaches 19.6% chord root accuracy, 32.7% chord quality accuracy, 6.2% chord F1@IoU0.5, and 76.2% chord mIoU, alongside 96.3% instrument F1 and 47.3% tempo accuracy. 5.5 Ablation Study Effect of the synthetic-to-real curriculum. Comparing Stage 1 alone and Stage 1+2 reveals the contribution of the two-stage curriculum. Stage 1 trained on synthetic data alone is insufficient for downstream tasks (94.0% WER for ASR, 74.3% DER for diarization, 34.3% F1 for grounding). Stage 2 fine-tuning on real annotated data brings dramatic improvements across the board, with the largest gains on audio grounding (F1 from 34.3% to 46.2%) and ASR (WER from 94.0% to 44.7%). Stage 1+2 also outperforms Stage 2 alone in most cases (0.77 s vs 0.85 s ASR MAE, 70.5% vs 70.0% diarization mIoU), indicating that the synthetic warmup provides priors that transfer beneficially to real data. Effect of the full SFT recipe. On the full Stage 1+2 corpus, the Naive SFT baseline isolates the contribution of our three SFT innovations under the complete training budget. The full SFT recipe outperforms Naive SFT: 1.3 points lower WER on ASR, 2.0 points lower DER on diarization, +6.1 event F1 on dense captioning, and +9.1 F1 on audio grounding. A controlled study isolating each component individually is provided in Appendix C. Effect of reinforcement learning. Adding GRPO on top of the Stage 1+2 SFT checkpoint produces consistent but moderate improvements across 4 out of the 5 tasks (Table 2). Multi-task RL improves ASR WER by 1.2 points (44.7 to 43.5), diarization sF1 by 1.4 points (54.9 to 56.3), dense captioning eF1 by 0.8 points (58.5 to 59.3), and audio grounding mIoU by 0.9 points (48.5 to 49.4). Single-task RL, in which a separate RL stage is run for each task, yields gains of similar magnitude but does not consistently match multi-task RL across all metrics. However, RL did not help on music captioning, where nearly every metric declines slightly relative to the SFT checkpoint. We hypothesize that this lack of improvement on music is because of the lack of pretrained music data in the AF3’s training curriculum. The smaller magnitude of these RL gains relative to the SFT gains supports our overall finding: careful SFT design is the primary lever for installing timestamping in LALMs, with RL providing additional refinement of timestamp boundaries. 6 Baseline format-failure analysis To ensure that the large gaps to zero-shot LALMs are not artifacts of output formatting, we use model-specific tolerant parsers and separately evaluate each baseline on only its parseable outputs. The performance gap remains substantial even after removing format failures, indicating that the dominant errors arise from incorrect content and temporal localization rather than parsing; detailed prompts, valid-output analysis, and qualitative failure modes are provided in Appendix D. Representative qualitative examples in Appendix E show that TEMPO’s residual errors are dominated by segment merging and boundary misalignment, even when the underlying semantic content is correct. 7 Transfer to real world music All our training data consist of human annotated labels from real-world recordings except music. Slakh2100 (Manilow et al., 2019) dataset comprises of human annotated MIDI files from Lakh Midi (Raffel, 2016) but the recordings are synthesized from the MIDI files. To evaluate whether training on synthetic data transfers to real world recordings, we evaluate our trained model on Maestro dataset (Hawthorne et al., 2019) which consists of real-world piano recordings. We use the same data processing pipeline from the testset of the Maestro dataset. We randomly pick 1000 samples to evaluate our final checkpoint. The final numbers are in Table 3. Our multi-task RL-trained model consistently outperforms the base Audio Flamingo 3 model and Gemini 2.5 Flash across all metrics. Table 3: Out-of-domain music evaluation on real acoustic piano recordings from MAESTRO. Model Root↑ Qual↑ Inst F1↑ Tempo↑ AF3 0.0 0.0 98.7 19.5 Gemini 2.5 Flash 9.1 22.4 96.3 17.9 TEMPO 23.9 27.3 99.8 24.8 8 Conclusion We presented Tempo, the first unified LALM for timestamped generation across speech, sound, and music, covering multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning in a single decoder. Our SFT recipe combines atomic timestamp tokens, a time-aware multi-modal projector, and a distance-aware Gaussian loss, trained under a synthetic-to-real curriculum and refined with GRPO against verifiable temporal rewards. Tempo substantially outperforms Audio Flamingo Next and Qwen3-Omni, while remaining competitive with Gemini 2.5 models on a benchmark spanning five tasks, with the largest gains on multi-speaker ASR (WER from 69.7% to 43.5%) and diarization (mIoU from 44.4% to 71.1%). Ablations show that careful SFT design is the primary lever for installing timestamping in LALMs, with RL providing additional refinement. Limitations TEMPO covers five timestamping tasks but does not address all forms of temporal audio understanding. In particular, we do not handle overlapping multi-domain events (e.g., simultaneous music, background sounds, and speech), which would require the model to jointly segment and label multiple concurrent streams. Second, TEMPO uses a fixed temporal resolution of 0.1 s. This resolution is too coarse for applications requiring sub-100 ms precision, such as phoneme-level alignment or fine-grained percussive onset detection, which are outside the scope of this work. For the five tasks considered here, however, timestamp quantization is not the dominant source of error: a 0.1 s grid introduces at most 0.05 s of quantization error, whereas the observed boundary MAE is approximately 0.77 - 1.19 s for the corresponding temporal tasks. Diarization is additionally evaluated with a 0.25 s collar. Thus, finer timestamp resolution is unlikely to materially affect the reported results, although adaptive or higher-resolution timestamp representations may be necessary for tasks requiring substantially finer temporal precision. Third, TEMPO is built on Audio Flamingo 3 with a frozen Whisper-large encoder, inheriting any limitations of that encoder’s spectral and temporal representations. Our results are thus conditioned on this specific encoder and may not generalize to other audio frontends. Fourth, we do not perform multi-seed training due to the computational cost of repeating the full multi-stage 7B training pipeline; however, the large margins over the strongest baselines and the consistent ordering across our controlled ablations (Appendix C) support the robustness of our main conclusions. Finally, the music captioning component is trained exclusively on Slakh2100, a synthesized MIDI dataset. We additionally evaluate the trained checkpoint on the real-world MAESTRO dataset, demonstrating transfer beyond synthesized MIDI recordings. However, performance on broader real-world music with natural acoustics, mixing artifacts, and production effects may differ substantially. References Bhattacharya et al. (2025) D. Bhattacharya, A. Kulkarni, and S. Ganapathy Benchmarking and confidence evaluation of lalms for temporal reasoning. External Links: 2505.13115, Link Cited by: §2.1. Böck et al. (2016) S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer Madmom: a new python audio and music signal processing library. External Links: 1605.07008, Link Cited by: §5.3. Bredin (2017) H. Bredin pyannote.metrics: A Toolkit for Reproducible Evaluation, Diagnostic, and Error Analysis of Speaker Diarization Systems. In Interspeech 2017, p. 3587–3591. External Links: Document, ISSN 2958-1796 Cited by: Appendix B, §5.2. Bredin (2023) H. Bredin pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Interspeech 2023, p. 1983–1987. External Links: Document, ISSN 2958-1796 Cited by: §5.3. Burgoyne et al. (2011) J. A. Burgoyne, J. Wild, and I. Fujinaga An expert ground truth set for audio chord recognition and music analysis. In ISMIR, Cited by: §2.4. Carletta et al. (2005) J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner The AMI meeting corpus: a pre-announcement. In Machine Learning for Multimodal Interaction (MLMI), Lecture Notes in Computer Science, Vol. 3869, p. 28–39. External Links: Document Cited by: §3.1. Chen et al. (2022) S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei BEATs: audio pre-training with acoustic tokenizers. External Links: 2212.09058, Link Cited by: §5.3. Chu et al. (2024) Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou Qwen2-audio technical report. External Links: 2407.10759, Link Cited by: §2.1. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §5.3. Drossos et al. (2019) K. Drossos, S. Lipping, and T. Virtanen Clotho: an audio captioning dataset. External Links: 1910.09387, Link Cited by: §2.3. Gemmeke et al. (2017) J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter Audio set: an ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 776–780. External Links: Document Cited by: §3.1. Ghosh et al. (2026) S. Ghosh, A. Goel, K. Jayakumar, L. Koroshinadze, N. Anand, Z. Kong, S. Gururani, S. Lee, J. Kim, A. Aljafari, C. H. Yang, S. Kim, R. Duraiswami, D. Manocha, M. Shoeybi, B. Catanzaro, M. Liu, and W. Ping Audio flamingo next: next-generation open audio-language models for speech, sound, and music. External Links: 2604.10905, Link Cited by: §2.1. Ghosh et al. (2025a) S. Ghosh, A. Goel, L. Koroshinadze, S. Lee, Z. Kong, J. F. Santos, R. Duraiswami, D. Manocha, W. Ping, M. Shoeybi, and B. Catanzaro Music flamingo: scaling music understanding in audio language models. External Links: 2511.10289, Link Cited by: §2.4. Ghosh et al. (2025b) S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro Audio flamingo 2: an audio-language model with long-audio understanding and expert reasoning abilities. External Links: 2503.03983, Link Cited by: §1, §2.1. Ghosh et al. (2024) S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha GAMA: a large audio-language model with advanced audio understanding and complex reasoning abilities. External Links: 2406.11768, Link Cited by: §1, §2.1. Godfrey et al. (1992) J. J. Godfrey, E. C. Holliman, and J. McDaniel SWITCHBOARD: telephone speech corpus for research and development. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. 1, p. 517–520. External Links: Document Cited by: §3.1. Goel et al. (2025) A. Goel, S. Ghosh, J. Kim, S. Kumar, Z. Kong, S. Lee, C. H. Yang, R. Duraiswami, D. Manocha, R. Valle, and B. Catanzaro Audio flamingo 3: advancing audio intelligence with fully open large audio language models. External Links: 2507.08128, Link Cited by: §1, §2.1, §4. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. External Links: ISSN 1476-4687, Link, Document Cited by: §1, §2.5, §4.2. Hawthorne et al. (2019) C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck Enabling factorized piano music modeling and generation with the MAESTRO dataset. In International Conference on Learning Representations, External Links: Link Cited by: §7. Hershey et al. (2021) S. Hershey, D. P. W. Ellis, E. Fonseca, A. Jansen, C. Liu, R. C. Moore, and M. Plakal The benefit of temporally-strong labels in audio event classification. External Links: 2105.07031, Link Cited by: §3.1. Kim et al. (2019) C. D. Kim, B. Kim, H. Lee, and G. Kim AudioCaps: generating captions for audios in the wild. In NAACL-HLT, Cited by: §2.3. KimiTeam et al. (2025) KimiTeam, D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tang, Z. Wang, C. Wei, Y. Xin, X. Xu, J. Yu, Y. Zhang, X. Zhou, Y. Charles, J. Chen, Y. Chen, Y. Du, W. He, Z. Hu, G. Lai, Q. Li, Y. Liu, W. Sun, J. Wang, Y. Wang, Y. Wu, Y. Wu, D. Yang, H. Yang, Y. Yang, Z. Yang, A. Yin, R. Yuan, Y. Zhang, and Z. Zhou Kimi-audio technical report. External Links: 2504.18425, Link Cited by: §2.1. Kong et al. (2024) Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro Audio flamingo: a novel audio language model with few-shot learning and dialogue abilities. External Links: 2402.01831, Link Cited by: §1, §2.1. Kulkarni et al. (2026) A. Kulkarni, K. Jayakumar, S. Ghosh, S. Wiegreffe, D. Manocha, and R. Duraiswami A closer look at failure modes in temporal understanding of large audio-language models. External Links: 2606.17417, Link Cited by: §2.1. Kumar et al. (2026) S. Kumar, P. Seetharaman, K. Chen, O. Nieto, J. Su, Z. Wang, R. Kumar, D. Manocha, N. J. Bryan, Z. Jin, and J. Salamon TAC: timestamped audio captioning. External Links: 2602.15766, Link Cited by: §1, §2.1, §2.3. Li et al. (2025) G. Li, J. Liu, H. Dinkel, Y. Niu, J. Zhang, and J. Luan Reinforcement learning outperforms supervised fine-tuning: a case study on audio question answering. External Links: 2503.11197, Link Cited by: §2.5. Manilow et al. (2019) E. Manilow, G. Wichern, P. Seetharaman, and J. L. Roux Cutting music source separation some slakh: a dataset to study the impact of training data quality and quantity. External Links: 1909.08494, Link Cited by: §3.1, §7. Mauch and Dixon (2010) M. Mauch and S. Dixon Approximate note transcription for the improved identification of difficult chords.. In ISMIR 2010, p. 135–140. Cited by: §5.3. Mesaros et al. (2021) A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley Sound event detection: a tutorial. IEEE Signal Processing Magazine 38 (5), p. 67–83. External Links: ISSN 1558-0792, Link, Document Cited by: §2.3. OpenAI et al. (2024) OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Ł. Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Ł. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph GPT-4 technical report. External Links: 2303.08774, Link Cited by: Appendix A. Panayotov et al. (2015) V. Panayotov, G. Chen, D. Povey, and S. Khudanpur Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 5206–5210. External Links: Document Cited by: §3.2. Piczak (2015) K. J. Piczak ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, p. 1015–1018. External Links: Link, Document, ISBN 978-1-4503-3459-4 Cited by: §3.2. Politis et al. (2021) A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen Overview and evaluation of sound event localization and detection in dcase 2019. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, p. 684–698. External Links: ISSN 2329-9304, Link, Document Cited by: §2.3. Primus et al. (2025) P. Primus, F. Schmid, and G. Widmer TACOS: temporally-aligned audio captions for language-audio pretraining. External Links: 2505.07609, Link Cited by: §3.1. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. In International conference on machine learning, p. 28492–28518. Cited by: §2.1. Raffel (2016) C. Raffel Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. Ph.D. Thesis, Columbia University. Cited by: §7. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §2.5. Shriberg et al. (2004) E. Shriberg, R. Dhillon, S. Bhagat, J. Ang, and H. Carvey The ICSI meeting recorder dialog act (MRDA) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, Cambridge, Massachusetts, USA, p. 97–100. External Links: Link Cited by: §3.1. Tang et al. (2024) C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang SALMONN: towards generic hearing abilities for large language models. External Links: 2310.13289, Link Cited by: §2.1. Wang et al. (2025) H. Wang, Y. Li, S. Ma, H. Liu, and X. Wang Listening between the frames: bridging temporal gaps in large audio-language models. External Links: 2511.11039, Link Cited by: §1, §2.1, §2.3, §5.1, §5.3. Wang et al. (2024a) H. Wang, J. Mao, Z. Guo, J. Wan, H. Liu, and X. Wang Leveraging language model capabilities for sound event detection. External Links: 2308.11530, Link Cited by: §2.3. Wang et al. (2024b) Q. Wang, Y. Huang, G. Zhao, E. Clark, W. Xia, and H. Liao DiarizationLM: speaker diarization post-processing with large language models. In Interspeech 2024, interspeech_2024, p. 3754–3758. External Links: Link, Document Cited by: §2.2. Wu et al. (2025) S. Wu, C. Li, W. Wang, H. Zhang, H. Wang, M. Yu, and D. Yu Audio-thinker: guiding audio language model when and how to think via reinforcement learning. External Links: 2508.08039, Link Cited by: §2.5. Wu et al. (2024) Y. Wu, K. Chen, T. Zhang, Y. Hui, M. Nezhurina, T. Berg-Kirkpatrick, and S. Dubnov Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. External Links: 2211.06687, Link Cited by: §5.3. Xu et al. (2023) H. Xu, F. Jia, S. Majumdar, H. Huang, S. Watanabe, and B. Ginsburg Efficient sequence transduction by jointly predicting tokens and durations. External Links: 2304.06795, Link Cited by: §5.3. Yang et al. (2026) C. H. Yang, S. Ghosh, Q. Wang, J. Kim, H. Hong, S. Kumar, G. Zhong, Z. Kong, S. Sakshi, V. Lokegaonkar, O. Nieto, R. Duraiswami, D. Manocha, G. Kim, J. Du, R. Valle, and B. Catanzaro Multi-domain audio question answering benchmark toward acoustic content reasoning. External Links: 2505.07365, Link Cited by: §2.3. Yin et al. (2026) H. Yin, Y. Chen, C. Deng, L. Cheng, H. Wang, C. Tan, Q. Chen, W. Wang, and X. Li SpeakerLM: end-to-end versatile speaker diarization and recognition with multimodal large language models. External Links: 2508.06372, Link Cited by: §2.2. Appendix A Dataset and Preprocessing Details Multi-Speaker ASR and Diarization. AMI, ICSI, and Switchboard feature spontaneous, overlapping, naturalistic speech. Recordings are segmented into chunks of up to 60 seconds while preserving multi-speaker structure within each chunk. Audio Temporal Grounding. AudioSet Strong provides categorical event labels rather than natural-language queries. We use GPT-4o (OpenAI et al., 2024) to generate diverse paraphrased queries for each event class (e.g., for dog_bark: “When does a dog start barking?”), with the strong temporal labels serving as ground-truth intervals. Dense Audio Captioning. TACOS contains 12,358 FreeSound recordings (15–30 seconds each) annotated with 47,748 timestamped captions. Timestamped Music Captioning. Slakh2100 contains 145 hours of audio split into 30-second segments. Chord labels are extracted by quantizing simultaneously sounding notes into standard chord symbols (e.g., C major, G7); instrument entries/exits, tempo, dynamics, and note statistics are read directly from the MIDI metadata. Synthetic speech compositions. We synthesize multi-speaker audio by randomly selecting 1-4 speakers from LibriSpeech and concatenating their utterances onto a shared 60-second canvas with small gaps between segments. Speaker slots are balanced across 1-4 speakers in equal proportions. Segment start and end times are chosen to avoid over-representing any particular timestamp value, and the final audio is peak-normalized. Synthetic audio event compositions. We synthesize audio event recordings by randomly placing ESC-50 sound clips onto a 60-second canvas, with each composition containing 1-6 overlapping or non-overlapping events drawn from different sound categories. Separate pools are generated for DAC and AG to decouple their category distributions. For AG, we ensure at least 50% of compositions contain repeated occurrences of the same sound category, so that the model must learn to localize multiple spans of the same event. A random 10% of AG compositions ground the full clip rather than a specific event. As with speech, timestamp values are balanced uniformly across all compositions. Appendix B Evaluation Metric Details This appendix gives the full computational definitions of the evaluation metrics. All interval matching uses the greedy IoU-based matching described below unless stated otherwise. Word Error Rate (WER). For multi-speaker ASR, we first greedily match predicted segments to ground-truth segments by temporal IoU. For each matched pair we compute the word-level Levenshtein distance between the normalized predicted and reference transcripts, and accumulate it into a total edit count. Unmatched ground-truth segments contribute all of their reference words as deletions, and unmatched predicted segments contribute all of their words as insertions. The final WER is the total edit count divided by the total reference word count across all segments. Diarization Error Rate (DER). We use the standard DER as implemented in pyannote.metrics (Bredin, 2017), with a collar of 0.250.25 s. Speaker labels are optimally permuted between hypothesis and reference before scoring, so that a hypothesis with consistent but arbitrarily named speakers incurs no penalty from the labeling alone. DER aggregates missed speech, false alarms, and speaker confusion as a fraction of total reference speech duration. Symmetric mIoU. For a set of reference intervals and a set of predicted intervals, we define the recall-side mean-best-IoU as the average over reference intervals of the maximum IoU with any predicted interval, and the precision-side mean-best-IoU symmetrically as the average over predicted intervals of the maximum IoU with any reference interval. The reported symmetric mIoU is the harmonic mean of these two quantities. The harmonic mean prevents over-prediction from inflating the score: emitting many spurious intervals raises recall-side IoU but lowers precision-side IoU, and the harmonic mean is dominated by the smaller of the two. F1 at IoU threshold. For dense audio captioning and audio grounding, a predicted interval is a true positive if its IoU with a greedily-matched ground-truth interval is at least 0.50.5; precision, recall, and F1 are computed from the resulting true-positive, false-positive, and false-negative counts. For diarization, the speaker-labeled F1@IoU0.5 additionally requires that the predicted and ground-truth segments carry the same speaker label, so a temporally correct segment with the wrong speaker is not counted as a true positive. Boundary MAE. For tasks with continuous time boundaries (ASR, grounding, dense captioning, music), we report the mean absolute error of segment onsets and offsets in seconds. The MAE is computed only over greedy-matched pairs with IoU≥0.3IoU≥ 0.3, so that boundary error is measured only where the prediction and reference refer to the same underlying event. METEOR. For dense audio captioning, caption quality is measured with METEOR computed over greedy-matched predicted and ground-truth segment pairs and averaged across matched pairs. Matching uses the same IoU-based greedy procedure as the other interval metrics. Music-specific metrics. For chords, we report two frame-level accuracies sampled at 0.50.5 s resolution: root accuracy (fraction of sampled frames with the correct chord root) and quality accuracy (fraction with the correct chord quality label). We additionally report chord F1@IoU0.5, which greedily matches predicted and ground-truth chord segments requiring both IoU≥0.5IoU≥ 0.5 and an exact chord-label match, and chord mIoU, the symmetric mIoU over all chord segments. For instruments, we report instrument-name F1 (token-level F1 between the predicted and ground-truth instrument-name sets) and instrument boundary MAE (onset/offset error in seconds over matched instruments). Tempo accuracy is the fraction of examples whose predicted BPM falls within 4%4\% of the ground truth. Appendix C Per-Component SFT Ablation Table 4: SFT component ablation on a controlled 2,000-example subset, reported on three representative tasks. Loss: distance-aware Gaussian timestamp loss. Proj: time-aware multi-modal projector. DER, eF1, F1, mIoU, and MET (METEOR) are reported as percentages; MAE in seconds. ↓ : lower is better; ↑ : higher is better. Best result per column in bold. Variant Diarization Dense Captioning Audio Grounding DER↓ mIoU↑ eF1↑ mIoU↑ MET↑ F1↑ MAE↓ mIoU↑ Naive 98.4 23.8 35.3 43.3 20.5 33.9 2.05 36.1 + Loss 86.2 37.5 49.9 60.2 21.5 38.0 1.73 41.9 + Projector 104.2 22.2 34.2 42.8 20.5 34.0 2.02 36.4 + Both (Ours) 79.3 37.5 49.7 60.1 21.9 38.7 1.69 42.5 To isolate the contribution of each SFT innovation, we train four variants on a controlled subset of 2,000 real-world examples: (i) Naive, with neither the distance-aware loss nor the time-aware projector; (i) + Loss, adding only the distance-aware Gaussian loss; (i) + Projector, adding only the time-aware multi-modal projector; and (iv) + Both, the full SFT recipe with both components. Due to compute constraints, we report three representative tasks spanning speech and audio (diarization, dense audio captioning, and audio temporal grounding) in Table 4. An interesting pattern emerges. The distance-aware loss, in isolation, provides large improvements across all three tasks, including a 14.6-point gain in dense captioning event F1 and a corresponding gain in diarization speaker-labeled F1 and grounding F1. In contrast, the time-aware projector in isolation is not useful, suggesting that wall-clock positional information requires a corresponding learning signal to be effective. However, when paired with the loss, the projector contributes a clear additional benefit on the two tasks with the densest temporal structure: diarization (DER improves from 86.2% to 79.3%) and audio grounding (F1 from 38.0% to 38.7%, MAE from 1.73 s to 1.69 s, mIoU from 41.9% to 42.5%). Appendix D Baseline Prompts and Format failure analysis Tolerant output parsing. The zero-shot LALMs use different native timestamp conventions. AF3 and AF-Next are prompted using the <t>start-end</t> text format, whereas Qwen3-Omni uses timestamp tokens of the form <|start|> text <|end|>. We therefore use a separate, deliberately tolerant parser for each model. The parsers accept common model-specific variants, including bracketed intervals, paired timestamp tokens, silence markers, and speaker-label variants. Outputs from which no timestamped interval can be recovered are treated as format failures and scored as maximally incorrect rather than being discarded. For each zero-shot baseline, we evaluated three closely related prompt scaffolds and report the best-performing configuration; Exact baseline prompts are provided in Table 7. Format validation. To determine whether poor zero-shot performance is primarily caused by failure to follow the requested output syntax, we recompute ASR and diarization metrics using only examples with parseable baseline outputs. For comparison, TEMPO is evaluated on exactly the same valid-example subset for each baseline. Table 5: ASR format-validation analysis. “Valid” denotes evaluation after excluding outputs from which no timestamped interval can be recovered. TEMPO is evaluated on the identical subset. WER is reported in %. Model Invalid WER WER TEMPO (%) all valid valid Audio Flamingo 3 5.1 209.7 206.7 44.7 Audio Flamingo Next 5.8 167.1 162.2 44.1 Qwen3-Omni 18.9 69.7 62.7 45.5 Table 6: Diarization format-validation analysis. DER is reported in %. TEMPO is evaluated on the same valid-example subset as each baseline. Model Invalid DER DER TEMPO (%) all valid valid Audio Flamingo 3 21.7 103.2 101.2 27.2 Audio Flamingo Next 23.6 106.9 103.6 31.8 Qwen3-Omni 0.0 44.2 44.2 28.0 Restricting evaluation to parseable outputs changes the baseline scores only modestly. For example, AF3’s ASR WER decreases from 209.7% to 206.7%, while its diarization DER remains above 100% (103.2% to 101.2%) even when all format failures are excluded. Similarly, Qwen3-Omni has no diarization format failures, yet its DER remains 44.2%, compared with 28.0% for TEMPO on the same examples. Thus, the large baseline errors primarily reflect incorrect content and temporal predictions rather than strict output parsing. Failure analysis. TEMPO’s remaining errors are dominated by temporal granularity and segment-boundary mistakes rather than degenerate generations. In dense audio captioning, the 10.2% of examples with severe temporal failure (mIoU<0.1mIoU<0.1) still obtain a METEOR score of 0.28, compared with a dataset-wide average of 0.23, indicating that the event descriptions are often semantically correct even when their temporal boundaries are inaccurate. For multi-speaker ASR, only 7.1% of examples have WER above 100%; these examples also exhibit substantially lower segment overlap (mean IoU 0.34 versus 0.58 overall), suggesting that the dominant failure mode is merging adjacent speaker turns or misplacing their boundaries. The zero-shot baselines exhibit qualitatively different failures. AF3 frequently generates plausible but unsupported dialogue or caption-like content instead of faithful transcription, AF-Next can enter repetitive generation loops, and Qwen3-Omni sometimes collapses an entire clip into a single untimestamped segment. These behaviors explain why WER and DER can exceed 100% even when the outputs are otherwise parseable. Table 7: Exact task-specific prompt suffixes used for the zero-shot LALM baselines. For each example, the corresponding evaluation question is followed by the model-specific suffix shown below. No system prompt is used for these models. Task Audio Flamingo 3 Audio Flamingo Next Qwen3-Omni ASR Transcribe the speech with timestamps. Break it into short segments (a few words each). Use <t>start-end</t> format for each segment, one per line. Example: <t>0.0-2.5</t> Hello world <t>3.1-4.8</t> How are you <t>5.2-7.0</t> I am fine thank you Provide accurate start and end times in seconds. Transcribe the input speech. If multiple speakers are present, provide diarized transcripts with speaker labels. Include timestamps for each utterance using <t>start-end</t> format. Example: <t>0.0-2.5</t> [Speaker 1] Hello world <t>3.1-4.8</t> [Speaker 2] How are you <t>5.2-7.0</t> [Speaker 1] I am fine thank you Break into multiple short segments with accurate timestamps. Respond using ONLY the following format. Split the transcription into SHORT segments (a few words each) with precise start AND end timestamps in seconds. Each segment MUST have both <|start|> and <|end|> timestamps. Output one segment per line: [speech:asr] <|start_seconds|> transcribed text <|end_seconds|> Example: [speech:asr] <|0.0|> Hello world <|2.5|> [speech:asr] <|3.1|> How are you <|4.8|> [speech:asr] <|5.2|> I am fine thank you <|7.0|> Do NOT put all text into a single segment. Break it into multiple short segments with accurate timestamps. Diarization Identify each speaker and when they speak. Use <t>start-end</t> format with speaker labels, one turn per line. Example: <t>0.0-3.5</t> [Speaker 1] okay hi <t>3.8-6.2</t> [Speaker 2] hello how are you <t>6.5-9.0</t> [Speaker 1] i’m good thanks Transcribe the input audio. If multiple speakers are present, provide diarized transcripts with speaker labels. Include timestamps for each speaker turn using <t>start-end</t> format. Example: <t>0.0-3.5</t> [Speaker 1] okay hi <t>3.8-6.2</t> [Speaker 2] hello how are you <t>6.5-9.0</t> [Speaker 1] i’m good thanks Respond using ONLY the following format, one segment per line: [speech:diar] <|start_seconds|> Speaker N <|end_seconds|> where N is the speaker number (1, 2, 3, ...). Example: [speech:diar] <|0.0|> Speaker 1 <|3.5|> [speech:diar] <|3.8|> Speaker 2 <|6.2|> Dense Captioning Write a time-segmented caption of the audio, capturing sound, speech, and music details. Use <t>start-end</t> format for each event, one per line. Example: <t>0-3</t> A dog barks loudly in the background. <t>2-8</t> Wind blows steadily outdoors. <t>5-10</t> A car engine revs and drives away. Write a time segmented caption of the input audio capturing sound, speech and music details. Use <t>start-end</t> format for each segment. Example: <t>0-3</t> A dog barks loudly in the background. <t>2-8</t> Wind blows steadily outdoors. <t>5-10</t> A car engine revs and drives away. Respond using ONLY the following format, one event per line: [audio:caption] <|start_seconds|> description of sound event <|end_seconds|> Example: [audio:caption] <|0.0|> A dog barks loudly. <|3.5|> [audio:caption] <|2.1|> Wind blows in the background. <|8.0|> Audio Grounding Provide the exact time intervals where the queried sound event occurs. Use <t>start-end</t> format, one interval per line. Example: <t>0.9-1.4</t> <t>3.7-4.3</t> <t>6.5-7.5</t> Provide the exact time intervals where the queried sound event occurs in the audio. Use <t>start-end</t> format, one interval per line. Example: <t>0.9-1.4</t> <t>3.7-4.3</t> <t>6.5-7.5</t> Respond using ONLY the following format, one interval per line: [audio:ground] <|start_seconds|> to <|end_seconds|> Example: [audio:ground] <|0.9|> to <|1.4|> [audio:ground] <|3.7|> to <|4.3|> Music Captioning Describe the music in detail: instruments, tempo (BPM), key, chord progressions, dynamics, and note statistics. Include timestamps using <t>start-end</t> format. Mention style, arrangement, production, and emotions conveyed. Summarize the track with precision: mention its musical style, BPM, key, arrangement, production choices, and the emotions or story it conveys. Include timestamps using <t>start-end</t> format. Also describe instruments, tempo changes, chord progressions, dynamics, and note statistics (density and range). Respond using ONLY the following format on a single line: [instrument] InstrumentName enters at <|start|> exits at <|end|> [tempo] BPM at <|time|> [chord] ChordName from <|start|> to <|end|> [dynamics] description [note] density: X range: NoteA to NoteB Example: [instrument] Piano enters at <|0.00|> exits at <|10.00|> [tempo] 120.0 BPM at <|0.00|> [chord] C from <|0.00|> to <|2.50|> [chord] G from <|2.50|> to <|5.00|> [dynamics] moderate and steady [note] density: 4.5 range: C3 to C5 Appendix E Qualitative Diagnostic Analysis Aggregate metrics do not reveal whether poor temporal overlap arises from incorrect content, collapsed temporal structure, or boundary misalignment. We therefore inspect representative held-out examples and visualize the input log-mel spectrogram together with the ground-truth and TEMPO prediction intervals. The examples are selected using per-example task metrics, and all predictions are taken directly from the final GRPO checkpoint without re-inference or manual editing. Figure 3 presents representative success and failure cases for ASR and dense audio captioning. In Figure 3(a), TEMPO accurately recovers both the ASR transcript and its temporal structure, achieving 0 WER and 0.99 mIoU, with onset and offset errors below 0.05 s. In (b), five reference ASR segments collapse into two predictions, including three adjacent utterances merged into a single interval; consequently, WER rises to 125% and mIoU falls to 0.21. In (c), TEMPO successfully preserves the temporal granularity of repeated sound events in dense audio captioning, achieving 0.75 mIoU. In contrast, (d) shows a granularity collapse in which seven reference events are merged into a single predicted interval. Despite this severe temporal error, the predicted event description remains semantically correct (METEOR = 0.99), while mIoU drops to 0.04. These examples illustrate that TEMPO’s residual errors can arise from segment merging and boundary misalignment even when the underlying semantic content is correctly identified. Figure 3: Qualitative diagnostic examples for ASR and dense audio captioning. Each panel shows the log-mel spectrogram together with ground-truth intervals and TEMPO predictions. (a) ASR success with accurate transcript content and temporal boundaries. (b) ASR failure in which adjacent reference utterances are merged. (c) Dense-captioning success with repeated events individually localized. (d) Dense-captioning failure in which seven reference events collapse into a single prediction despite high semantic agreement. Appendix F Potential Risks We do not foresee significant risks specific to this work beyond those already inherent to existing large audio-language models and speech recognition systems. Tempo performs temporal localization of events in audio and does not generate speech, identify individual speakers by name, or enable capabilities not already present in publicly available audio understanding systems. All training and evaluation data are drawn from established, publicly released research corpora. Appendix G License for Artifacts We use a number of publicly released scientific artifacts in this work. Below, we summarize the licenses under which they are distributed and confirm that our use is consistent with their intended terms. Datasets. AMI and ICSI are released under the C BY 4.0 license for research use. Switchboard is distributed by the Linguistic Data Consortium (LDC) under a research license. AudioSet and AudioSet Strong are released under C BY 4.0. TACOS is released under C BY 4.0. Slakh2100 is released under C BY 4.0. LibriSpeech is released under C BY 4.0. ESC-50 is released under the C BY-NC 3.0 license, which permits non-commercial research use. Models. Audio Flamingo 3, our base model, is released under a research-only license by NVIDIA. Baselines we evaluate against (Audio Flamingo Next, Qwen3-Omni, TimeAudio) are accessed through their official releases under their respective research licenses. Compatibility with intended use. All artifacts are used for non-commercial academic research, which is consistent with the intended use specified by each provider. Appendix H Computational Resources Tempo is built on Audio Flamingo 3, which consists of a frozen Whisper-large audio encoder (approximately 0.6B parameters) and a Qwen2-7B language model (approximately 7B parameters), connected by a two-layer MLP projector. The audio encoder remains frozen throughout training. All experiments are run on NVIDIA A6000 GPUs. SFT runs on 8 GPUs on a single node, with Stage 1 covering 51,512 synthetic examples over two epochs and Stage 2 covering 32,726 real examples over two additional epochs. GRPO training (1,000 steps with 8 completions per prompt) runs on 8 GPUs distributed across 2 nodes (4 GPUs per node) using DDP via HuggingFace Accelerate. Evaluation across all five tasks (10,513 question-answer pairs) is run on a single A6000. Appendix I Use of AI Assistants We utilized AI assistants to help clarify explanations, suggest concise phrasing, and organize text for readability. These tools were used exclusively for linguistic support and were not used to generate scientific results or formulate claims. Acknowledgement The research is partially supported by Adobe, Amazon, NVIDIA, and Sesame