Paper deep dive
Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing
Ilpo Viertola, Giulio Cengarle, Gouthaman KV, Daniel Arteaga, Lie Lu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/25/2026, 5:04:23 AM
Summary
The paper introduces Spot, Separate, and Enhance (SSE), a multimodal, user-guided generative model for audio remixing and enhancement. SSE utilizes video, audio, and textual inputs to rebalance audio mixes, remove unwanted sources, and reduce reverberation. The authors propose a new dataset, DegradedMix, built on MuddyMix, and a new neural audio codec, SkipDACVAE. Experiments show SSE outperforms baselines like VisAH and VisAH-FM in both controllability and remixing quality, particularly under severe audio degradation.
Entities (10)
Relation Signals (8)
DegradedMix → isbasedon → MuddyMix
confidence 95% · we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix.
SSE → uses → SkipDACVAE
confidence 95% · Finally, the generated audio is decoded into a waveform representation with the SkipDACVAE decoder.
SSE → uses → DiT
confidence 95% · The fused and textual features are then passed to the DiT. The DiT generates enhanced audio...
SSE → outperforms → VisAH
confidence 92% · SSE outperforms existing baselines in both controllability and remixing quality... SSE exceeds all the compared methods across all the metrics except PaSST FD.
SSE → uses → PE
confidence 90% · We utilize a pretrained PE [4] as the visual encoder.
SSE → uses → CFM
confidence 90% · We propose a generative approach for audio remixing based on CFM [18]...
SSE → uses → T5-base
confidence 90% · Text descriptions are encoded with a pretrained T5-base encoder [25].
SSE → outperforms → VisAH-FM
confidence 85% · SSE outperforms existing baselines... SSE exceeds all the compared methods... subjective evaluation... SSE-L is preferred over VisAH with a large margin.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2609.29169v1
- Canonical: https://arxiv.org/abs/2609.29169v1
Trouble viewing inline? Open PDF directly →
Full Text
31,762 characters extracted from source content.
Expand or collapse full text
Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing Ilpo Viertola †thanks: This work was conducted during an internship at Dolby Laboratories. Giulio Cengarle Gouthaman KV Daniel Arteaga Lie Lu Abstract We introduce SSE (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: sse-ai.notion.site Index Terms: Generative Learning, Audio Remixing, Multimodal Learning †address: 1Dolby Laboratories 2Tampere University 1 Introduction Recent advances in deep learning and generative modeling have opened new possibilities for enhancing UGC (UGC). A large portion of UGC consists of videos captured in everyday environments, where audio often contains multiple sound sources, background noise, reverberation, and other recording artifacts. Despite improvements in video capture hardware, post-processing remains essential for improving audio quality and intelligibility. For example, rebalancing the audio mix to emphasize the primary subject, suppressing background noise, or mitigating reverberation can substantially improve the viewing experience. To enable automatic audio remixing, Huang et al. [11] introduced VGAH (VGAH), which rebalances the audio mix to emphasize the main subject based on visual information. While an important step toward automated audio enhancement, VGAH primarily considers loudness adjustments and does not address other common degradations such as background noise or reverberation. Together with the task, Huang et al. [11] proposed an evaluation framework based primarily on differences between the generated remix and ground-truth balanced mix in waveform or spectral representations. However, audio remixing is inherently creative, with multiple perceptually valid solutions, and reconstruction-based evaluation may therefore penalize valid outputs that differ from a particular ground truth. These limitations motivate a more general remixing framework that can address diverse degradations and acoustic conditions while allowing greater flexibility in the solution space. Existing approaches [11, 22] use an encoder-decoder model [26] where a Transformer [32] processes the latent audio representation conditioned on video frames, and the decoder generates the remixed audio. Huang et al. [11] propose VisAH, which is trained end-to-end in a discriminative manner using a reconstruction loss between the automatic remix and the ground truth balanced mix. Malard et al. [22] propose VisAH-FM, a concurrent work that adopts the same architecture with VisAH but replaces the discriminative training objective with a generative CFM (CFM) [18] formulation. They further introduce a rollout loss [30, 31], which performs full generation during training and computes an MSE (MSE) against the ground-truth target, exposing the model to its own intermediate predictions and encouraging self-correction. However, both approaches have limitations. First, VisAH inherently struggles to generate new audio as it predicts a spectral mask applied to the input mixture. When the target source is heavily buried in noise or reverberation, the model cannot generate new content to replace the degraded audio. Second, both approaches train the audio codec jointly with the generative backbone which limits scalability [19, 7]. Third, neither approach provides user control beyond the video conditioning. Finally, the rollout loss in VisAH-FM introduces overhead by requiring full generation for every training sample. To address these limitations, we propose SSE (SSE), a multimodal, user-guided generative framework for audio remixing. Unlike existing approaches that primarily modify the input mixture, SSE can generate new audio to replace severely degraded target-source regions, while explicit user guidance enables control over the desired remixing behavior. We also introduce a new evaluation framework based on principles from generative modeling, which are better suited to a creative task such as audio remixing. Rather than requiring close reconstruction of a single ground-truth target, our evaluation considers the perceptual quality and effectiveness of the resulting remix. We further define a controlled set of audio degradations to simulate common real-world recording defects, enabling systematic evaluation under challenging acoustic conditions. Extensive experiments demonstrate that SSE achieves superior remixing quality compared to existing approaches while providing greater user controllability and the ability to recover target-source content under severe audio degradation. 2 Method Figure 1: Overview of SSE. Given a video with degraded and unbalanced audio, SSE enhances it according to the given textual description. First, all the modalities are encoded with specific encoders, and the visual and audio features are aligned and fused. The fused and textual features are then passed to the DiT. The DiT generates enhanced audio that closely follows the original content and user guidance. Finally, the generated audio is decoded into a waveform representation with the SkipDACVAE decoder. SSE, presented in Fig. 1, is a multimodal generative approach for UGC enhancement. Given a video from an everyday environment, SSE enhances the audio according to the user’s preferences. All modalities are first encoded separately. Then, audio and visual features are temporally aligned and fused. The fused features are passed to a generative model, where text guides generation via cross-attention, and the output is decoded into a waveform. SSE can rebalance the audio mix, remove unwanted sources, and generate new audio content to replace degraded audio. 2.1 Neural Audio Codec – SkipDACVAE We propose SkipDACVAE, adapted from the DACVAE [24, 16] audio codec. DACVAE [24] replaces the RVQ (RVQ) bottleneck of the original DAC [16] with a VAE (VAE) bottleneck. This gives DACVAE a continuous latent space which is well suited for continuous flow matching [18]. SkipDACVAE adds skip connections from the encoder to the decoder. This improves reconstruction quality, since the decoder can access the original audio features directly. Each skip connection adds encoder features to decoder features at matching temporal resolution. Before addition, a 1D convolutional layer with kernel size of 1 maps the encoder features to the decoder’s channel dimension. Each connection uses its own convolutional layer. The number of connections equals the number of downsampling layers in the encoder. We initialize SkipDACVAE with pretrained DACVAE weights and freeze all shared layers, leaving only the added 1D convolutional layers trainable. These layers are zero-initialized, so SkipDACVAE behaves identically to DACVAE at the start of training. During training, we obtain a degraded version of the input audio and pass it through SkipDACVAE to compute the skipped values. We then encode the original, undegraded audio and decode it using skipped values computed from corresponding degraded version. We compute the reconstruction loss between the decoded output and the original audio, and backpropagate it to update SkipDACVAE. Using the degraded audio to compute skipped values forces SkipDACVAE to transfer high-level information from encoder to decoder, rather than simply copying the original audio features. We use the original training configuration of DAC [16]. During training and inference of SSE, we use a pretrained SkipDACVAE. We encode the input audio mixture, yielding a latent sequence mix∈ℝT×Caud x_mix ^T× C_aud at 25 Hz, where T is the number of audio frames and Caud=128C_aud=128 is the channel dimension. 2.2 Visual Encoder We utilize a pretrained PE (PE) [4] as the visual encoder. PE’s large scale contrastive training on vision-language pairs allows it to learn semantically rich representations for actions and scene context. We encode the video per-frame, yielding a sequence of visual features vis∈ℝCvis×Tvis x_vis ^C_vis× T_vis, where TvisT_vis is the number of visual frames and Cvis=1024C_vis=1024 is the channel dimension. 2.3 Text Encoder Text descriptions are encoded with a pretrained T5-base encoder [25]. Text feature sequence text∈ℝN×Ctext x_text ^N× C_text, where N is the number of text tokens and Ctext=768C_text=768 is the channel dimension, is obtained from the last hidden layer of the encoder. Before feeding the text features to the cross-attention layers of the generative model, we project them to the model’s channel dimension C using a linear layer. 2.4 Generative Conditional Flow Matching Model We propose a generative approach for audio remixing based on CFM [18] and recent audio source separation techniques [29]. Our model learns a continuous vector field that transports a Gaussian prior sample 0∼N(0,) x_0 N(0, I) to a target data sample 1 x_1 over t∈[0,1]t∈[0,1], where t is the timestep. At each timestep, the model predicts a velocity field θ(t,,t) v_θ(t, C, x_t), where =mix,vis,text C=\ x_mix, x_vis, x_text\ is the conditioning signal, t x_t is the flow state at time t, and θ are the model parameters. By integrating θ v_θ over time, we obtain the final prediction ^1 x_1. Modality Aligner. The target data sample 1∈ℝT×2Caud x_1 ^T× 2C_aud stacks the target audio tgt x_tgt and the residual signal mix−tgt x_mix- x_tgt along the channel dimension. The Gaussian prior sample 0∈ℝT×2Caud x_0 ^T× 2C_aud stacks noise ∼N(0,) z N(0, I) and an all-zero tensor in the same way. Following [29], we condition the model on the audio mixture mix x_mix for content-faithful generation. At each flow step, we concatenate the flow state t x_t with the fixed mixture mix x_mix, then project to the model’s channel dimension C using a linear layer, yielding ∈ℝT×C h ^T× C. We also add visual information to h. First, to temporally align the audio and visual features, we match the frame rate of vis x_vis to 25 Hz using nearest-neighbor interpolation, yielding a visual feature sequence vis∈ℝCvis×T x_vis ^C_vis× T. Then, visual information is projected and added via a gated summation: =+tanh(σ)⊙LN(Conv1D(vis)⊤) h= h+ (σ) (Conv1D( x_vis) ), where σ is a learnable gating parameter, ⊙ denotes elementwise (broadcasted) multiplication, LNLN is layer normalization, and Conv1DConv1D is a 1D convolutional layer with kernel size of 1, that projects the visual features to the model’s channel dimension C. Diffusion Transformer. We represent the velocity field with a DiT [23] and previously introduced Modality aligner. SSE adopts an architecture where each Transformer [32] block is modulated by the flow time embedding via scale-and-shift operations on normalization and residual layers. To create the flow time embedding, a shared MLP maps t to six modulation parameters (four scales, two biases), reused across blocks with layer-specific biases added for depth-dependent effects, reducing model size. In practice, we sample t as t=k/(K−1)t=k/(K-1), where k∼0,…,K−1k \0,…,K-1\ and K=4K=4. As shown in previous works [6, 5, 21, 28], selecting a few timesteps during training may be beneficial during inference time in image-to-image generation tasks. Also, a concurrent work by Malard et al. [22] utilize a similar timestep sampling approach. K=4K=4 yielded the best overall results in our testing, and we use this value for all experiments. During inference, we use Euler integration with the same amount of steps. We initialize the model with pretrained weights of SAM-Audio [29]. We argue that pretraining the model on a large-scale audio source separation task is beneficial for remixing and enhancement, as it allows the model to learn general audio representations and source separation capabilities. In our experiments, training the query, key, value, and output projections across all DiT blocks together with the Modality aligner and the linear layer projecting the text x_text features (Section 2.3) yielded the best results. We compared different selective unfreezing strategies, LoRA [10] and full finetuning. 3 Experiments 3.1 Dataset – DegradedMix We base our data on the MuddyMix dataset [11], also used by the compared approaches [11, 22]. MuddyMix consists of 15,078/1,927/1,789 train/validation/test 10-second clips extracted from movies [3], each with a professionally mixed audio track. Each track is separated into stems via a pretrained source separation model [35], s=s^h+s^m+s^e+s^rs= s_h+ s_m+ s_e+ s_r, denoting speech, music, effects, and residual signals. MuddyMix degrades these stems with random discrete gain changes, then remixes them into a degraded audio track. To mimic real-world UGC enhancement scenarios, we propose DegradedMix which extends MuddyMix by adding background noise [36] and reverberation [2] to the audio mixes. We train on DegradedMix and evaluate on both DegradedMix (Tables 1 and 2) and MuddyMix (Tables 4 and 3) test sets, enabling comparison to prior work under standard and more challenging degradations. Note that MuddyMix [11] does not support background noise removal, as all the stems in the degraded mix are also present in the ground-truth mix. We also add audio-based captions to the DegradedMix dataset, which describe the original, non-degraded audio content. One-to-two sentence audio captions are generated automatically with Qwen2-Audio [8]. This mimics the real-world scenario where the user provides a textual description of the desired enhancement. We release the code to generate DegradedMix data. 3.2 Compared Methods The proposed model is compared against two recent approaches, namely VisAH [11] and VisAH-FM [22]. VisAH is a discriminative approach that uses a Transformer-based [32] encoder-decoder model to predict a mask in spectral space which is applied to the input audio to generate the remixed audio. VisAH-FM is a closed-source concurrent work that was published after the original VisAH which uses a conditional flow matching approach to generate the remixed audio, following a similar architecture as VisAH. 3.3 Implementation and Training Details We train on 25 FPS video with 48 kHz audio, resampled to 44.1 kHz for evaluation to match prior work [11, 22]. Video frames are resized to 336×336336× 336, and SSE uses a 10-second context length, following [11, 22]. We generate degraded audio on-the-fly during training, applying gain changes to the stems and adding random background noise and reverberation. The model is initialized from pretrained weights [29] and trained for approximately 67K steps with batch size 26 on two NVIDIA RTX PRO 6000 Blackwell GPUs, using AdamW [20] (β=[0.9,0.95]β=[0.9,0.95], weight decay 0.1, learning rate 1e−41e^-4) with a cosine annealing schedule and 2K warmup steps. 3.4 Evaluation Metrics The evaluation protocol proposed by Huang et al. [11] is largely based on comparing the automatic remix with the ground truth balanced mix. These metrics include a) magnitude [38] and envelope [17] distance between the mixes, b) time alignment using Wasserstein distance, and c) semantic alignment using KL (KL) [19, 34, 15] and IB (IB) [9, 33]. In particular, a) and b) are ill-suited for creative tasks like remixing, where multiple valid solutions can exist. We instead adopt evaluation metrics from generative modeling, which better suit the setting. Following evaluation practices in multimodal audio generation [19, 7], we propose to use the following metrics to evaluate the remixing performance: a) FD (FD) [13] and KL (KL) [19, 34] for distribution matching, c) IS (IS) [27] to measure audio quality, d) IB (IB) [9] for semantic alignment, e) Sync (Sync) [12], and f) CLAP (CLAP) [37]. For distribution matching scores, we utilize PANNs [14] and PaSST [15] pretrained audio classifiers to extract features from the generated remixes and the ground truth mixes. For IS, we use the same PANNs classifier. 3.5 Results Table 1: Proposed metrics calculated for DegradedMix test data. We could not evaluate VisAH-FM [22] as the model is closed-source. *: retrained with the DegradedMix data for fair comparison. PANNs PaSST Model FD ↓ KL ↓ FD ↓ KL ↓ IS ↑ IB ↑ Sync ↓ CLAP ↑ Input 3.55 0.35 43.39 0.26 3.16 30.49 51.80 40.56 VisAH* [11] 2.97 0.31 40.63 0.21 3.04 31.08 51.80 44.06 SSE 1.26 0.14 41.30 0.12 3.13 32.44 48.67 45.06 Table 2: VGAH [11] metrics calculated for DegradedMix test data. We could not evaluate VisAH-FM [22] as the model is closed-source. *: retrained with the DegradedMix data for fair comparison. Variant Env ↓ Mag ↓ Was ↓ KLPaSST↓KL_PaSST Input 6.79 23.13 1.95 25.65 VisAH* [11] 5.16 15.70 1.22 21.08 SSE 3.65 10.87 0.90 11.60 Audio Enhancement. Table 1 reports model performance on DegradedMix, which includes gain imbalance, background noise, and reverberation as degradations, using the proposed metrics. For a fair comparison, we retrain VisAH [11] on DegradedMix, and also report metrics for the unprocessed degraded Input. We could not evaluate VisAH-FM [22], as its code is not publicly available. Our model outperforms VisAH across all metrics except PaSST [15] FD, highlighting its effectiveness in enhancing degraded audio. The high CLAP and IB scores further show the model’s ability to follow conditional guidance, which is crucial for user-guided audio enhancement. Table 2 reports the model performance on DegradedMix using the VGAH metrics [11]. Our model clearly outperforms VisAH. Table 3: Proposed metrics calculated for MuddyMix [11] test data. We could not evaluate VisAH-FM [22] as the model is closed-source and the authors do not provide samples for the MuddyMix test set. PANNs PaSST Model FD ↓ KL ↓ FD ↓ KL ↓ IS ↑ IB ↑ Sync ↓ CLAP ↑ Input 2.30 0.29 31.86 0.21 3.11 31.19 51.19 38.30 VisAH [11] 1.26 0.18 18.06 0.11 3.06 31.81 49.71 39.56 SSE 1.04 0.13 35.78 0.10 3.13 32.74 47.69 46.78 Table 4: VGAH [11] metrics calculated for MuddyMix [11] test data. Model Env ↓ Mag ↓ Was ↓ KLPaSST↓KL_PaSST Input 6.29 22.69 1.96 20.74 VisAH [11] 3.38 9.99 0.84 11.37 VisAH-FM [22] 2.74 8.28 0.63 9.70 SSE 3.46 9.86 0.88 9.61 Audio Remixing. Table 3 reports the model performance on MuddyMix, which includes only gain imbalance as a degradation, using the proposed metrics. We could not evaluate VisAH-FM [22] as its code or samples are not available. SSE exceeds all the compared methods across all the metrics except PaSST [15] FD. However, we clearly achieve the best balance between all the metrics. Table 4 reports model performance on MuddyMix [11] using VGAH metrics. When compared against a single ground-truth solution (Env, Mag, Was), SSE falls short of the concurrent work VisAH-FM [22], though we achieve the best KLPaSSTKL_PaSST score. We argue this reflects a limitation of the metric rather than the model: in a creative task like remixing, where multiple valid solutions can exist, comparing generated results to a single ground truth is not ideal. This is further highlighted by the subjective evaluation in Table 5, where our approach (SSE-L) is preferred over VisAH with a large margin, despite scoring only marginally better or on par under VGAH metrics. Table 5: Pairwise preference test results for UGC enhancement. Preference rate is the proportion of trials in which the first condition was preferred over the second, with 95% confidence intervals. p-values are from a two-sided binomial test against chance (50%). Comparison (A vs. B) Wins (A–B) Pref. rate p-value SSE-L vs. SSE-S 68–22 75.6% <0.001<0.001 SSE-L vs. Original audio 72–18 80.0% <0.001<0.001 SSE-L vs. VisAH [11] 80–10 88.9% <0.001<0.001 SSE-S vs. Original audio 47–43 52.2% 0.7520.752 SSE-S vs. VisAH [11] 57–33 63.3% 0.0150.015 Original audio vs. VisAH [11] 62–28 68.9% <0.001<0.001 Subjective Analysis. We conduct a subjective evaluation on UGC audio enhancement. We select 10 UGC videos (presented at the project page) and generate enhanced audio using the small- and large-variants of SSE (SSE-S and SSE-L), and VisAH [11]. 10 participants select the preferred audio for a video between two samples, presented in random order. Preference rate tells how many times on average a particular audio was preferred over other options. Table 5 shows the final rates, where SSE-L is preferred in 81% of the cases, demonstrating its effectiveness. SSE-S is preferred in 47% and the original audio is preferred in 46% of the cases. VisAH’s limited UGC enhancement and remixing capabilities result in a lower preference rate of 26%. 3.6 Ablations Table 6: Ablation on different model variants. We use MuddyMix [11] data. Preferred configuration is highlighted in yellow. PANNs PaSST Model FD ↓ KL ↓ FD ↓ KL ↓ IS ↑ IB ↑ Sync ↓ CLAP ↑ Small 1.12 0.14 37.81 0.11 3.13 32.76 47.82 47.02 Base 1.06 0.13 37.28 0.10 3.12 32.64 48.84 47.10 Large 1.04 0.13 35.78 0.10 3.13 32.74 47.69 46.78 Model Size. Table 6 presents the performance of different model sizes. Small-variant has 500M parameters, base-variant has 1B parameters, and large-variant has 3B parameters. The approach is scalable and we observed that the large-variant yields the best performance across the majority of the metrics. We conduct all the experiments using the large-variant of the model if not otherwise specified. Table 7: Effect of using SkipDACVAE. We use MuddyMix [11] data. Preferred configuration is highlighted in yellow. PANNs PaSST Variant FD ↓ KL ↓ FD ↓ KL ↓ IS ↑ IB ↑ Sync ↓ CLAP ↑ DACVAE 3.38 0.21 143.55 0.28 3.03 32.50 48.35 48.13 SkipDACVAE 1.04 0.13 35.78 0.10 3.13 32.74 47.69 46.78 SkipDACVAE. As shown in Table 7, adding skip connections boosts the performance significantly. We conduct the experiments using the MuddyMix [11] data. The added representation capacity of the neural audio codec allows for better reconstruction of the audio which is reflected in the improved performance across all metrics. Note that the model is not retrained for the SkipDACVAE as these two neural audio codecs share the same latent space. Table 8: Ablation on different caption types. We use small-variant and MuddyMix [11]. Preferred configuration is highlighted in yellow. PANNs PaSST Type FD ↓ KL ↓ FD ↓ KL ↓ IS ↑ IB ↑ Sync ↓ CLAP ↑ Original 1.24 0.16 39.15 0.12 3.13 32.58 49.97 44.82 Summarized 1.22 0.16 38.47 0.12 3.12 32.66 49.11 44.78 Audio-based 1.12 0.14 37.81 0.11 3.13 32.76 47.82 47.02 Captions. We ablate on different styles of textual descriptions in Table 8 and use the small-variant of the model with MuddyMix [11]. MuddyMix dataset provides captions for each video frame, extracted with 1 FPS. These Original captions are long and contain unrelated information for audio enhancement. Summarized captions are generated by summarizing the original ones using Claude Sonnet 5 [1]. Finally, Audio-based captions are generated by describing the original, non-degraded, audio content of the video using Qwen2-Audio [8]. We use audio-based captions, as they mimic the real-world scenario where users provide a description of the desired enhancement. 4 Conclusion SSE shows that a single multimodal generative model can enhance UGC audio. It outperforms VisAH in controllability and quality under both generative and reconstruction-based metrics, though results are more mixed under single-ground-truth metrics, underscoring the need for evaluation that embraces remixing’s one-to-many nature. We hope DegradedMix and our evaluation framework support this shift. References [1] Anthropic (2026) Claude Sonnet 5. Note: https://w.anthropic.com/claudeLarge language model Cited by: §3.6. [2] C. Bagwell and SoX Contributors (2015) SoX - Sound eXchange. Note: http://sox.sourceforge.netVersion 14.4.2 Cited by: §3.1. [3] M. Bain, A. Nagrani, A. Brown, and A. Zisserman (2020) Condensed movies: story based retrieval with contextual embeddings. In ACCV, Cited by: §3.1. [4] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, et al. (2025) Perception encoder: the best visual embeddings are not at the output of the network. arXiv:2504.13181. Cited by: §2.2. [5] C. Chadebec, O. Tasar, E. Benaroche, and B. Aubin (2025) Flash diffusion: accelerating any conditional diffusion model for few steps image generation. In AAAI, Cited by: §2.4. [6] C. Chadebec, O. Tasar, S. Sreetharan, and B. Aubin (2025) LBM: latent bridge matching for fast image-to-image translation. In ICCV, Cited by: §2.4. [7] H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y. Mitsufuji (2025) MMAudio: taming multimodal joint training for high-quality video-to-audio synthesis. In CVPR, Cited by: §1, §3.4. [8] Y. Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y. Leng, Y. Lv, J. He, J. Lin, C. Zhou, and J. Zhou (2024) Qwen2-Audio technical report. arXiv:2407.10759. Cited by: §3.1, §3.6. [9] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra (2023) ImageBind one embedding space to bind them all. In CVPR, Cited by: §3.4, §3.4. [10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §2.4. [11] C. Huang, R. Gao, J.M.F. Tsang, J. Kurcius, C. Bilen, C. Xu, A. Kumar, and S. Parekh (2025) Learning to highlight audio by watching movies. In CVPR, Cited by: §1, §1, §3.1, §3.1, §3.2, §3.3, §3.4, §3.5, §3.5, §3.5, §3.5, §3.6, §3.6, Table 1, Table 2, Table 2, Table 2, Table 3, Table 3, Table 3, Table 4, Table 4, Table 4, Table 5, Table 5, Table 5, Table 6, Table 6, Table 7, Table 7, Table 8, Table 8. [12] V. Iashin, W. Xie, E. Rahtu, and A. Zisserman (2024) Synchformer: efficient synchronization from sparse cues. In ICASSP, Cited by: §3.4. [13] K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi (2018) Fréchet audio distance: a metric for evaluating music enhancement algorithms. arXiv:1812.08466. Cited by: §3.4. [14] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley (2020) PANNs: large-scale pretrained audio neural networks for audio pattern recognition. TASLP. Cited by: §3.4. [15] K. Koutini, J. Schlüter, H. Eghbal-zadeh, and G. Widmer (2022) Efficient training of audio transformers with patchout. In Interspeech, Cited by: §3.4, §3.4, §3.5, §3.5. [16] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar (2023) High-fidelity audio compression with improved RVQGAN. In NeurIPS, Cited by: §2.1, §2.1. [17] S. Liang, C. Huang, Y. Tian, A. Kumar, and C. Xu (2023) Av-nerf: learning neural fields for real-world audio-visual scene synthesis. In NeurIPS, Cited by: §3.4. [18] Y. Lipman, R. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In ICLR, Cited by: §1, §2.1, §2.4. [19] H. Liu, Y. Yuan, X. Liu, X. Mei, Q. Kong, Q. Tian, Y. Wang, W. Wang, Y. Wang, and M. D. Plumbley (2024) AudioLDM 2: learning holistic audio generation with self-supervised pretraining. TASLP. Cited by: §1, §3.4, §3.4. [20] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In ICLR, Cited by: §3.3. [21] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao (2023) Latent consistency models: synthesizing high-resolution images with few-step inference. arXiv:2310.04378. Cited by: §2.4. [22] H. Malard, G.L. Lan, D. Wong, D.L. L. Alon, Y. Wu, and S. Parekh (2026) Conditional flow matching for visually-guided acoustic highlighting. arXiv:2602.03762. Cited by: §1, §2.4, §3.1, §3.2, §3.3, §3.5, §3.5, §3.5, Table 1, Table 1, Table 2, Table 2, Table 3, Table 3, Table 4. [23] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In ICCV, Cited by: §2.4. [24] A. Polyak, A. Zohar, A. Brown, A. Tjandra, A. Sinha, A. Lee, A. Vyas, B. Shi, C. Ma, C. Chuang, et al. (2024) Movie Gen: a cast of media foundation models. arXiv:2410.13720. Cited by: §2.1. [25] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. Cited by: §2.3. [26] S. Rouard, F. Massa, and A. Défossez (2023) Hybrid transformers for music source separation. In ICASSP, Cited by: §1. [27] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen (2016) Improved techniques for training GANs. In NeurIPS, Cited by: §3.4. [28] T. Salimans and J. Ho (2022) Progressive distillation for fast sampling of diffusion models. In ICLR, Cited by: §2.4. [29] B. Shi, A. Tjandra, J. Hoffman, H. Wang, Y.-C. Wu, L. Gao, J. Richter, M. Le, A. Vyas, S. Chen, C. Feichtenhofer, P. Dollár, W.-N. Hsu, and A. Lee (2025) SAM audio: segment anything in audio. arXiv:2512.18099. Cited by: §2.4, §2.4, §2.4, §3.3. [30] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In ICLR, Cited by: §1. [31] Y. Song and P. Dhariwal (2024) Improved techniques for training consistency models. In ICLR, Cited by: §1. [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In NeurIPS, Cited by: §1, §2.4, §3.2. [33] I. Viertola, V. Iashin, and E. Rahtu (2025) Temporally aligned audio for video with autoregression. In ICASSP, Cited by: §3.4. [34] A. Vyas, B. Shi, M. Le, A. Tjandra, Y. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan, et al. (2023) AudioBox: unified audio generation with natural language prompts. arXiv:2312.15821. Cited by: §3.4, §3.4. [35] K. N. Watcharasupat, C. Wu, and I. Orife (2024) Remastering Divide and Remaster: a cinematic audio source separation dataset with multilingual support. In 5th IEEE International Symposium on the Internet of Sounds (IS2), Cited by: §3.1. [36] G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow, and J. Le Roux (2019) WHAM!: extending speech separation to noisy environments. In Interspeech, Cited by: §3.1. [37] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov (2023) Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, Cited by: §3.4. [38] X. Xu, H. Zhou, Z. Liu, B. Dai, X. Wang, and D. Lin (2021) Visually informed binaural audio generation without binaural audios. In CVPR, Cited by: §3.4.