Paper deep dive
VIBE: Video Instruction-aligned Background music gEneration
Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj, Gouthaman KV, Sreyan Ghosh, Ramani Duraiswami, Lie Lu, Dinesh Manocha
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/15/2026, 8:03:20 AM
Summary
The paper introduces VIBE, a text-and-video-to-music generation model that addresses the lack of semantic control in existing video-to-music models. VIBE utilizes a novel 'Conditioning Connection' mechanism for dynamic cross-layer conditioning and a comprehensive reward modeling taxonomy to optimize for both hard constraints (tempo, key) and soft subjective qualities (musicality, alignment) via a 5-stage training curriculum.
Entities (10)
Relation Signals (6)
Vibe → uses → Conditioning Connection
confidence 95% · VIBE... leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism
Vibe → optimizes → Reward Modeling Taxonomy
confidence 92% · VIBE... leverages... (2) a comprehensive reward modeling taxonomy, optimizing for both hard... and soft... qualities
Conditioning Connection → addresses → Representational Bottleneck
confidence 90% · Conditioning Connection... addresses the representational bottleneck of static conditioning
Vibe → trainswith → DiffusionNFT
confidence 88% · integrated with a DiffusionNFT-style reinforcement learning objective
CMI-RM → measures → musicality
confidence 85% · supervise musicality... via CMI-RM... a cross-modal instruction-following reward model
Qwen2.5-Omni → measures → Multimodal Alignment
confidence 85% · omni-modal reward extracted using Qwen2.5-Omni... as a judge... rV↔Malign
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment.
Tags
Links
- Source: https://arxiv.org/abs/2608.30125v1
- Canonical: https://arxiv.org/abs/2608.30125v1
Trouble viewing inline? Open PDF directly →
Full Text
64,718 characters extracted from source content.
Expand or collapse full text
VIBE: Video Instruction-aligned Background music gEneration Aryan Vijay Bhosale Vaibhavi Lokegaonkar Vishnu Raj Affiliation: Dolby Laboratories, USA Gouthaman KV Affiliation: Dolby Laboratories, USA Sreyan Ghosh Affiliation: University of Maryland, College Park, USA Ramani Duraiswami Affiliation: University of Maryland, College Park, USA Lie Lu Affiliation: Dolby Laboratories, USA Dinesh Manocha Email: aryan.vi.bhosale, vaibhavilokegaonkar@gmail.com Project Page Code Affiliation: University of Maryland, College Park, USA Abstract Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment. †footnotetext: ∗Equal contribution. 1 Introduction Music and imagery jointly shape the emotional experience of short-form video, driving creators to seek background scores that align rhythmically and semantically with their visual content. Background scores, added in post-production, have been shown to reinforce the emotional tone, narrative pacing, and aesthetic of a video Huh et al. (2026). However, curating the perfect background score for a video is a highly nuanced task with no single correct answer Hammad et al. (2025); Frid et al. (2020). The same video may be paired with acoustic guitar, chill lo-fi beats or suspenseful piano. Visual signals alone are insufficient to capture creator intent, and therefore, supplemental fine-grained textual prompts are required Hammad et al. (2025); Melechovsky et al. (2024); Huang et al. (2023) to disambiguate and accurately capture creator intent. Existing video-to-music (V2M) works such as video-only models Tian et al. (2025); Zuo et al. (2025); Ji et al. (2025), models that accept minimal auxiliary inputs Liu et al. (2024); Kang et al. (2024) and text++video-to-music models Lokegaonkar et al. (2026); Kim et al. (2025); Kim et al. (2026), rely majorly on visual cues for music generation and provide limited semantic and stylistic controllability to the end user. Even when text is provided as input to Video-Robin Lokegaonkar et al. (2026), it is used for high-level style steering only and not for fine-grained attribute specification of tempo, key, musical genre and mood. This is because most existing models are trained on reconstruction objectives over paired video-music data with no mechanism to explicitly penalise misalignment in specific parts of the input text instruction. This lack of instruction following can be attributed to clear architectural and training limitations in such models. More specifically, this can be attributed to the architectural bottleneck of static cross-modal conditioning, especially in diffusion autoregressive models and the absence of a principled training framework for decomposed, fine-grained instruction adherence. To overcome these issues, we present VIBE (Video Instruction-aligned Background music gEneration), a novel joint text++video-to-music multi-preference-optimised generation technique with a structured training curriculum that provides a mechanism for dynamic conditioning to the DiT, enabling highly controllable music generation that aligns seamlessly with the textual and visual prompts. To tackle static conditioning, we propose Conditioning Connection, a depth-wise conditioning mechanism that passes the intermediate hidden states from the Autoregressive-Head as multimodal context, along with the autoregressive audio generation history to each layer of the Refinement-Head. This provides a clear abstraction of global planning and refinement tasks across the architecture rather than collapsing the LM’s representational hierarchy into a single embedding. We also address the issue that no existing video-conditioned music generation system is explicitly trained to follow individual components of music instructions, such as tempo, key, genre, and mood, as specified in the input text prompts. Our approach audits the full spectrum of musical instruction components, partitions them into objectively verifiable aspects and subjective perceptual qualities, and develops dedicated reward models for each attribute class to directly supervise instruction adherence via reinforcement learning. We also introduce a verifiability-driven taxonomy of rewards for preference optimization combining hard signal-processing rewards for objective attributes with learned compositional multi- and omni-modal reward models for subjective attributes. Overall, our approach can generate music that semantically and rhythmically adheres to fine-grained text instructions while remaining temporally coherent with the input video. Our novel contributions include: 1. We propose VIBE, a text++video-to-music generation model for multimodally aligned, high fidelity music generation featuring Conditioning Connection, a novel depth-wise cross-layer conditioning mechanism that addresses the representational bottleneck of static conditioning by passing a learned linear combination of the multimodal LM’s hidden states to each layer of the LocDiT, enabling rich multimodal grounding and smoother propagation of multimodal context throughout the denoising hierarchy. 2. We introduce a systematic taxonomy of musical attributes and perform holistic reward modeling for each spanning hard, verifiable rewards for objective instructions and learned cross-modal and omni-modal reward models for subjective perceptual qualities. 3. We demonstrate how the compositional reward signal can be integrated with a DiffusionNFT-style reinforcement learning objective through a comprehensive, multi-stage training recipe and precise data mixtures for preference optimization of Diffusion Autoregressive architectures under multimodal instruction-following constraints. Multiple experiments across objective metrics on Reelbench benchmarks and human evaluation demonstrate consistent gains in fine-grained instruction adherence over prior text- and video-conditioned systems, while remaining competitive on generation fidelity. 2 Related Work 2.1 Video-to-Music Generation Video-conditioned music generation has garnered significant research interest, with approaches centering on visual feature extraction and alignment with musical rhythm. Foundational works such as CMT (Di et al., 2021), and Video2Music (Kang et al., 2024), generate symbolic music in the MIDI format by using motion and semantic video features. Whereas VidMuse Tian et al. (2025) generates high-fidelity music waveforms from video using local and global features, while MuMu-LLaMA (interchangeably referred to as M2UGen in literature) Liu et al. (2024) broadens this with visual and textual encoders to allow text prompt conditioning. Works like GVMGen (Zuo et al., 2025), and Diff-V2M (Ji et al., 2025) address alignment through hierarchical attention over spatio-temporal features and specialized encoders for different aspects of music. More recently, Video-Robin (Lokegaonkar et al., 2026) and V2M-ZERO (Lin et al., 2026) have advanced the field by extending conditioning capabilities beyond discrete token spaces and eliminating the need for paired training data, respectively. A parallel line of work bridges vision and music explicitly rather than architecturally: Visuals-Music Bridge (VMB) (Wang et al., 2024) converts video into textual descriptions that condition a text-to-music model, so every visual cue must survive a discrete textual bottleneck. Conditioning Connection bridges the two implicitly instead, propagating a fused multimodal representation through the depth of the diffusion stack. We test the assumption behind the VMB idea in Section E. Broader still, any-modality-to-audio systems such as AudioX (Tian et al., 2026) generate audio (including sound effects) from video, text or their combination, but target general audio plausibility rather than adherence to specified musical attributes, placing them adjacent to—rather than in competition with—the instruction-following setting we study. Despite these advances, none of these works optimize for human perceptual preferences during training, resulting in generated music that lacks the expressivity characteristic of human-composed music. Further, capturing relative quality across a broader candidate pool and reducing dependence on large-scale preference datasets remain open challenges, as existing approaches rely on pairwise comparisons or fixed datasets. Furthermore, diffusion-based approaches such as Diff-V2M and Video-Robin suffer from static conditioning Li et al. (2026), where the conditioning inputs do not adapt across the depth of the Diffusion Transformer. This causes a mismatch between the requirements of each layer and the fixed conditioning signal. In this work, we address these limitations by introducing Conditioning Connection for dynamic conditioning and DiffusionNFT-based online preference optimization. 2.2 Preference Optimization for Conditional Music Generation Imbuing human preferences into the music generation process continues to be a prevalent challenge. CMI-Reward Bench (Ma et al., 2026) addresses the holistic nature of music quality evaluation by providing a reward ecosystem containing large-scale human-annotated preference datasets alongside trained reward models. Building on such foundations, recent works have explored popular preference optimisation techniques to directly align generative models with human judgement. MR-FlowDPO (Ziv et al., 2025) extends DPO to flow-matching text-to-music models using multi-dimensional automated rewards across text alignment, audio quality, and semantic consistency. LeVo (Lei et al., 2025) constructs a semi-automatic preference dataset of approximately 60K win-lose pairs and applies interpolation-based multi-preference DPO to jointly optimise all objectives without retraining. HeartMuLa (Yang et al., 2026) takes a different approach, constructing separate preference datasets scored by distinct music quality metrics, training independent DPO models on each, and linearly merging the resulting checkpoints to jointly optimise style adherence, lyric clarity, and audio quality. Beyond DPO, ACE-Step v1.5 (Gong et al., 2025) applies GRPO to a language model planner for intrinsic reward-based alignment, deriving rewards from the model’s own internal consistency, while SymphonyGen (He et al., 2026) applies GRPO with a cross-modal audio-perceptual reward to refine symbolic orchestral generation. In this work, we extend the paradigm of preference-aligned music generation conditioned on video and fine-grained text instructions. 3 Methodology Generating music that simultaneously adheres to fine-grained textual instructions and aligns rhythmically with video requires solving two distinct problems. First, the architecture must propagate rich multimodal context throughout the entire denoising hierarchy. Second, the training objective must explicitly penalize violation of individual instruction attributes, a signal that reconstruction-based losses are blind to. We address both through VIBE: a Diffusion Autoregressive multimodal music generation model with Conditioning Connectors for depth-wise dynamic conditioning, trained via a structured 5-stage curriculum with diverse, composite reward formulation for preference optimization. 3.1 Architecture Figure 1: VIBE Architecture. Video frames, text prompt, and previously generated patch embeddings are passed to the Multimodal Semantic LM. Layer-wise hidden states are linearly combined via learnable per-DiT-layer coefficients to form Conditioning Connectors, which are routed to every LocDiT layer alongside residual-integrated embeddings to generate each music patch. VIBE adopts a Diffusion Autoregressive architecture Lokegaonkar et al. (2026) that decomposes music generation into two global planning (AR-Head) and patch-wise refinement (Refinement-Head) as shown in Figure 1. The AR-Head performs global semantic planning and integrates video frames (encoded by a frozen CLIP encoder), fine-grained text instructions, and the autoregressive patch history through a Multimodal Semantic LM. The resulting representation is compressed via a Finite Scalar Quantization (FSQ) bottleneck and enriched with residual acoustic detail through a Residual Integration Transformer Encoder (RITE), producing a per-patch planning embedding. The Refinement-Head, implemented as a local Diffusion Transformer (LocDiT), denoises each latent patch conditioned on this embedding and the previously generated patch. Generated patches are re-encoded and fed back into the AR-Head autoregressively. Once all patches are complete, a pretrained VAE decoder reconstructs them as a full waveform. In vanilla Diffusion Autoregressive models, the planning embedding is passed as a single static tensor to the LocDiT. This collapses the full representational hierarchy of the multimodal semantic LM into one embedding, ignoring the functional stratification of deep transformer stacks, wherein earlier layers encode global structural properties and later layers encode fine-grained local detail Li et al. (2026). Conditioning Connection. We address this bottleneck by introducing Conditioning Connection, an architectural mechanism that carries the learned, weighted representation conditioning vectors between the multimodal semantic LM and the LocDiT. At each generation step, we compute a linear combination of hidden states from all layers of the Multimodal Semantic LM, using learnable coefficients, and pass the result to each layer of the Refinement Head. We refer to these linearly combined vectors as Conditioning Connectors. We compute a Conditioning Connector vector i(k)c^(k)_i for each LocDiT layer k as a linear combination of hidden states across all LM layers using learnable coefficients: i(k)=∑l=1Lαl(k)i(l),∑l=1Lαl(k)=1c^(k)_i=W\, _l=1^Lα^(k)_l\,h^(l)_i, _l=1^Lα^(k)_l=1 (1) where αl(k)∈ℝα^(k)_l are learnable scalar weights specific to LocDiT layer k, and ∈ℝdDiT×dLMW ^d_DiT× d_LM is a learnable projection, dDiTd_DiT being the hidden dimension of the LocDiT and dLMd_LM that for Multimodal Semantic LM. Since each autoregressive step also receives previously generated patches as input, the LocDiT layer k conditions on both current multimodal context i(k)c^(k)_i & autoregressive musical patch history i−1m_i-1 at every layer, promoting patch-to-patch continuity and strengthening rhythmic and semantic coherence across the generated sequence. ~i(k)=LocDiT(k)(t,i(k),i−1,t) h^(k)_i=LocDiT^(k)\! (x^t,\;c^(k)_i,\;m_i-1,\;t ) (2) The combination coefficients are learned independently per DiT layer, allowing each layer to attend to the level of semantic abstraction most suited to its denoising responsibility, i.e. shallower LM representations for early LocDiT layers handling global structure and deeper representations for later layers handling fine acoustic detail. Conditioning Connectors replace static single-vector cross-head conditioning with a structured representational hierarchy that flows through the full depth of the diffusion stack. 3.2 Training Objectives Pre-training and Supervised Finetuning (SFT). Both pre-training and SFT optimize the LocDiT via a flow-matching diffusion loss over the velocity field θv_θ: ℒdiff=t,0,ϵ‖θ(t,p,i−1)−α˙t0−σ˙tϵ‖22L_diff=E_t,x^0, ε \|v_θ(x^t,E_p,m_i-1)- α_tx^0- σ_t ε \|_2^2 (3) where t=αt0+σtϵx^t= _tx^0+ _t ε, p˙t p_t indicates dpdt dpdt for any p, ϵ∼(0,) ε (0,I), and θv_θ is the LocDiT velocity field. See Table 6 for notation. Preference Optimization. We adopt DiffusionNFT Zheng et al. (2026) to perform online RL directly on the forward diffusion process. At each iteration, G candidates 0gg=1G\x_0^g\_g=1^G are generated per conditioning input c and scored with an optimality probability r∈[0,1]r∈[0,1]: ℒNFT=,t0∼πold[r‖θ+−‖22+(1−r)‖θ−‖22] -14.0ptL_NFT=E_ subarraycc,\,t\\ x_0 π^old subarray\! [r\|v_θ^+-v\|_2^2+(1-r)\|v_θ^--v\|_2^2 ] (4) where the implicit positive and negative policies are: θ+(t,p,i−1,t) _θ^+(x^t,E_p,m_i-1,t) :=(1−β)old+βθ :=(1-β)\,v^old+β\,v_θ (5) θ−(t,p,i−1,t) _θ^-(x^t,E_p,m_i-1,t) :=(1+β)old−βθ :=(1+β)\,v^old-β\,v_θ (6) with =α˙t0+σ˙tϵv= α_tx^0+ σ_t ε the flow-matching target, β the guidance strength, and oldv^old the frozen sampling policy. Refer to appendix for further training details. 3.3 Reward Modelling We partition musical instruction attributes by verifiability into two classes: hard verifiable attributes (tempo and key) whose adherence is directly measurable from the audio signal, and soft perceptual attributes (musical genre and mood) which resist objective evaluation and require learned judgement. We develop dedicated reward models for each class and combine them into task-specific reward signals for preference optimization in text-to-music and video-to-music models respectively. Hard Verifiable Rewards. These rewards are developed for instruction components (tempo and key) that have discrete values and can be objectively evaluated. Tempo. We estimate the BPM b b of the generated audio and parse the target tempo into one of three forms: exact value, range, or blanket descriptor (e.g. slow, fast). For exact targets we apply a Gaussian reward; for range and blanket targets a trapezoidal reward: tempo=exp(−(b^−b∗)22σb2)exactmax(0,min(1, 1−eδ))range r_tempo= cases \! (- ( b-b^*)^22 _b^2 )& 5.0ptexact\\ \! (0,\, \! (1,\,1- eδ ) )& 5.0ptrange cases (7) where e=max(blo−b^,b^−bhi, 0)e= (b_lo- b,\; b-b_hi,\;0). To correct for octave errors common in BPM estimators, we evaluate rtempor_tempo at b b, 2b^2 b, and b^/2 b/2 and take the maximum. Key. We average two complementary tonal alignment signals. The Circle-of-Fifths (CoF) reward detects the predicted key kpredk_pred and penalises arc distance darcd_arc from the target k∗k^*, incorporating a mode penalty for major/minor mismatch: Δk _k =darc(kpred,k∗)+[modepred≠mode∗], =d_arc(k_pred,k^*)+1[mode_pred ^*], (8) rCoF r_CoF =s⋅exp(−Δk22σk2) =s· \! (- _k^22 _k^2 ) where the reward is Gaussian weighted by the key detector confidence s∈[0,1]s∈[0,1] and σk=2.0 _k=2.0 which naturally downweights the reward when the detected key is ambiguous. The Krumhansl-Schmuckler (KS) reward correlates the Harmonic Pitch Class Profile (HPCP) ∈ℝ12h ^12 against the psychoacoustically derived tonal hierarchy ∗∈ℝ12p^* ^12 for the target key. The CoF reward is interpretable but depends on key detector accuracy whereas the KS reward operates directly on the spectrum and remains robust under detector failure. rKS=ρ(,∗)+12r_KS= ρ(h,\,p^*)+12 (9) key=rCoF+rKS2 r_key= r_CoF+r_KS2 (10) Soft Rewards. These are developed to penalize subjective aspects of musical instruction (genre, mood) and multimodal alignment (text-music, video-music) Cross-Modal Reward (Text-to-Music). We supervise musicality rmusicalityr_musicality (genre, mood) and text-music alignment rT↔Malignr_T Malign via CMI-RM Ma et al. (2026), a cross-modal instruction-following reward model. softCM=rmusicality+rT↔Malign2 R^CM_soft= r_musicality+r_T Malign2 (11) Omni-Modal Reward (Text++Video-to-Music). Since we need to simultaneously judge musicality and alignment between text, video and generated music, we develop an omni-modal reward extracted using Qwen2.5-Omni Qwen et al. (2025) as a judge as follows: softomni=rmusicality+rT↔Malign+rV↔Malign3 -10.0pt R^omni_soft= r_musicality+r_T +r_V 3 (12) Resultant Composite Reward Signal For the text-to-music (13) and text++video-to-music (14) task respectively may be formulated as follows: T→M=RsoftCM+rtempo+rkey⏟Rhard R^T =R^CM_soft+ r_tempo+r_key_R^hard (13) T+V→M=Rsoftomni+rtempo+rkey⏟Rhard R^T+V =R^omni_soft+ r_tempo+r_key_R^hard (14) 3.4 Training Curriculum In this section, we describe each stage in our training curriculum. Figure 2 illustrates this curriculum. Figure 2: Overview of our training curriculum Stage 1 (Music generation pre-training) The model is trained on large-scale text-music pairs (JamendoMaxCaps Roy et al. (2025)) without video conditioning, serving as a projector alignment phase where the AR-Head learns to map multimodal hidden states to musical latents via the Refinement-Head. This establishes a strong generative backbone for subsequent stages. Stage 2 (Text-to-Music Instruction Finetuning). We finetune on a curated, instruction-rich data from MusicBench Melechovsky et al. (2024) to improve instruction-following and retain the architecture and training configuration as Stage 1. Stage 3 (Text-to-Music Preference Optimization). We apply DiffusionNFT-based online RL using the composite reward RT→MR^T , combining CMI-RMMa et al. (2026) for subjective qualities with hard verifiable rewards for tempo and key on captions from the CMI-Pref Dataset Ma et al. (2026). Multiple rollouts per prompt are scored and used for negative-aware finetuning, expanding the model’s controllability and musicality beyond what reconstruction-based training permits. Stage 4 (Text++Video-to-Music Supervised Instruction Finetuning). We introduce visual conditioning through the video encoder (frozen) and trainable projection layer which can enable text++video-to-music generation. The inputs at this stage are primarily video ++ music pairs from HarmonySet Zhou et al. (2025) and V2M Tian et al. (2025) datasets with curated fine-grained textual prompts. The projection layer is initialized and trained from scratch and while all other layers are initialized and fine-tuned from the Stage 3 checkpoint. Stage 5 (Omni-modal alignment preference tuning). Analogous to Stage 3, we apply DiffusionNFT using the composite V2M reward RT+V→MR^T+V , which replaces CMI-RM with an omni-modal judge to additionally supervise video-music rhythmic and thematic alignment. Stage Training Dataset(s) Size Music Generation Pretraining JamendoMaxCaps ≈ 1.6M Text-to-Music Instruction SFT MusicBench 52,768 Text-to-Music Preference Opt. CMI-Pref 3,527 Text+Video-to-Music SFT V2M + HarmonySet 60,000 Omni-Modal Preference Opt. V2M + HarmonySet 8,000 Table 1: Training datasets used at each stage of the curriculum. All splits are training splits unless otherwise noted. 4 Experiments Model FAD (↓)( ) FD (↓)( ) KL (↓)( ) IS (↑)( ) IB (↑)( ) Density (↑)( ) Coverage (↑)( ) GT – – – – 0.1417 0.9900 0.8800 Video-Only to Music CMT Di et al. (2021) 8.7522 37.7945 1.7329 1.2243 ± 0.0147 0.1119 0.1084 0.0614 GVMGen Zuo et al. (2025) 3.5729 16.2638 1.5573 1.7085 ± 0.0281 0.0957 0.0835 0.3881 VidMuse Tian et al. (2025) 2.3022 14.5385 1.3194 1.4549 ± 0.0281 0.1233 0.1377 0.5213 Video + Auxiliary Inputs to Music Video2Music Kang et al. (2024) 22.6459 73.0670 1.8839 1.0233 ± 0.0014 0.0473 0.1647 0.0084 M2UGen Liu et al. (2024) 4.5767 27.4208 1.5301 1.6499 ± 0.0567 0.0722 0.1094 0.2761 Text + Video to Music Video-Robin Lokegaonkar et al. (2026) 1.5110 10.9020 1.2556 2.0586 ± 0.0472 0.1017 0.1384 0.5259 VIBE (Ours) 1.5829 10.1282 1.3205 2.3578 ± 0.0798 0.0888 0.6741 0.6095 Table 2: Results on the ReelBench Lokegaonkar et al. (2026) dataset. Bold indicates best and underline indicates second-best. Components Metrics Pre-train C T2M RL V2M RL FAD↓ FD↓ KL↓ IS↑ IB↑ Den.↑ Cov.↑ Base architecture ✗ ✗ ✗ ✗ 3.044 22.592 1.343 1.588 ± 0.131 0.113 0.258 0.133 + Pretraining ✓ ✗ ✗ ✗ 2.538 16.446 1.349 1.743 ± 0.057 0.0802 0.149 0.175 + Conditioning Connection ✓ ✓ ✗ ✗ 1.724 10.199 1.346 1.833 ± 0.069 0.0835 0.541 0.523 Preference optimization w/o C ✓ ✗ ✓ ✓ 1.618 17.920 1.442 1.760 ± 0.039 0.0578 0.570 0.500 w/o V2M RL ✓ ✗ ✓ ✗ 1.728 11.372 1.354 1.763 ± 0.045 0.0794 0.290 0.309 + C & preference optimization = VIBE (Ours) ✓ ✓ ✓ ✓ 1.583 10.128 1.321 2.358 ± 0.080 0.0888 0.674 0.610 Table 3: Component analysis of VIBE on ReelBench. The first block isolates the architectural contribution, building the model up one component at a time with no preference optimization. The second block isolates the training contribution, applying the preference optimization stages (Stage 3 and Stage 5) without C. VIBE proposes the architectural design additions and training paradigm together as a comprehensive solution for fine-grained instruction following in text+video-to-music generation. Bold indicates best. Model Rhythm Theme Emotion Culture Temporal Instr. Fit Overall MuMuLLaMa 2.490 2.731 3.000 3.255 2.143 3.100 2.724 Video2Music 2.622 2.744 2.744 3.293 2.268 3.090 2.677 Video-Robin 2.786 2.890 3.100 3.565 2.472 3.200 2.753 VIBE 2.816 2.910 3.028 3.581 2.589 3.240 2.843 Table 4: Gemini omni-modal judge results. Comparison to existing baselines along criteria defined in Table 13 4.1 Datasets We summarise the datasets used during each training stage in Table 1. Pretraining uses JamendoMaxCaps Roy et al. (2025), a large-scale instrumental music dataset. Text-to-Music Instruction Finetuning uses MusicBench Melechovsky et al. (2024), whose tempo and key annotations support fine-grained instruction following; vocals are removed via Demucs Rouard et al. (2023). Text-to-Music Preference Optimization uses CMI-Pref Ma et al. (2026), a human-annotated preference dataset whose prompts lie in the CMI-RM training distribution. Text+Video-to-Music Instruction Finetuning samples 30,000 pairs each from V2M Tian et al. (2025) and HarmonySet Zhou et al. (2025); since neither provides instruction-level prompts, we use Gemini to extract tempo, key, instruments, genre, and atmosphere from each clip. Omni-Modal Alignment Preference Tuning resamples 8,000 unique videos in a 1:2 ratio of V2M to HarmonySet, providing each without its background score to Gemini-2.5-Flash to generate 4 diverse instructional prompts per video. For evaluation, we use ReelBench Lokegaonkar et al. (2026), obtained directly from the authors. 4.2 Implementation Details Our model operates in the latent space of a pretrained SongBloom Yang et al. (2025) audio VAE at 48 kHz, frozen throughout training. The SemanticLM is initialised from MiniCPM4-0.5B Team et al. (2025) with 24 transformer layers, hidden dimension 896, and 16 attention heads. Conditioning Connectors consist of 4 learnable routing vectors of size 24 and a shared 896→ 1024 linear projection, initialised to zeros. The AR-Head uses an FSQ bottleneck with latent dimension 256 followed by an 8-layer RITE transformer. The Refinement Head is a 4-layer LocDiT Jia et al. (2025) trained with flow matching and a patch size of 4. We use frozen CLIP-ViT-Base (Radford et al., 2021) as the vision encoder. For SFT stages, we use AdamW with learning rate 1×10−41× 10^-4, weight decay 0.01, and warmup ratio 0.1. For preference optimization, we apply LoRA (r=8r=8, α=16α=16) on q_proj and v_proj of both LM and DiT, with learning rates 2×10−72× 10^-7 and 1×10−71× 10^-7 respectively, a group size of 8, β=0.5β=0.5, and 20 inference timesteps. For Stage 5, we increase LoRA rank to 64 and use Qwen2.5-Omni-7B as the multimodal judge with 4 video frames per sample. 212223242522.023.224.2(a) Tempo Exact ↑ 2930313229.729.631.4(b) Tempo ± ↑ 2.42.73.03.32.72.63.1(c) Key Exact ↑ 9.510.010.511.010.19.910.9(d) Key Loose ↑ 32.232.733.233.733.4332.7732.42(e) Tempo MAE ↓ Video-Robin VIBE w/o Stage 5 VIBE Figure 3: Instruction-following evaluation across tempo and key metrics. Panels (a)-(d) are higher-is-better, (e) is lower-is-better. 4.3 Evaluation Metrics We evaluate using standard audio quality metrics: FAD, FD, KL Zhang et al. (2023), IS Salimans et al. (2016), Density, and Coverage Naeem et al. (2020). Audio-visual alignment is measured via ImageBind Score (IB) Girdhar et al. (2023) and Gemini as an omni-judge Liang et al. (2026) across seven axes including rhythmic sync, emotion alignment, and overall alignment. For instruction following, we report Tempo Accuracy and Key Accuracy against ReelBench ground-truth annotations (see Appendix D). 4.4 Comparison Models We compare against three categories of baselines. Video-only models include CMT Di et al. (2021), GVMGen Zuo et al. (2025), and VidMuse Tian et al. (2025). Video + auxiliary input models include Video2Music Kang et al. (2024) and M2UGen Liu et al. (2024), for which we extract required attributes directly from our unified text prompt. Text + video models include Video-Robin Lokegaonkar et al. (2026). Models that do not support detailed text input receive the generic prompt: “Generate background music that aligns with the visual content of the video.” 4.5 Performance Evaluation In this section, we evaluate the performance of our model through ablations and comparisons to existing models detailed above. 4.5.1 Instruction Alignment Figure 3 reports instruction-following results across tempo and key metrics on ReelBench. VIBE consistently outperforms both Video-Robin and the Stage 5 ablation (VIBE w/o S5) across all five metrics, confirming that omni-modal preference tuning is the primary driver of instruction adherence. Tempo accuracy improves progressively from Video-Robin to VIBE under both exact and octave-equivalent tolerances, while Tempo MAE decreases monotonically, indicating that preference optimization produces generations that are not only more frequently correct but also closer to the target BPM on average. Key accuracy follows the same trend, with VIBE achieving the highest exact and loose key accuracy among all three models. 4.5.2 Quantitative comparison with baselines. Table 2 reports results on ReelBench. VIBE achieves the best IS, FD Density, and Coverage among all models, demonstrating strong perceptual quality and generative diversity. The marginal increase in FAD and KL and increase in IB relative to Video-Robin is expected, as RL finetuning trades distributional proximity for improved perceptual quality and instruction adherence. Among baselines, VidMuse achieves competitive audio fidelity but trails on diversity, while CMT and GVMGen lag across nearly all metrics due to their lack of fine-grained conditioning. M2UGen outperforms Video2Music but remains well below VIBE, highlighting the gap between structured auxiliary inputs and free-form text conditioning paired with preference optimization. Given the limitations of the objective metrics, we resort to 7 distinct axes to adequately measure our model’s performance following Video-Robin Lokegaonkar et al. (2026). VIBE outperforms baselines on 5 of the 6 ability axes and outperforms all evaluated baselines on overall omni-judge multimodal alignment as can be seen in Table 4. The definitions of the metrics may be found in Table 13. A recent study demonstrates that omni-modal models achieve correlations to human judgments comparable to or exceeding traditional metrics (ImageBind/CLAP) on semantic alignment axes such as audio-text alignment and tri-modal coherence, citing Qwen3-Omni as an example. That being said, we would like to note that we in no way imply that such omni-judges can perfectly represent human preference and also perform subjective human evaluation as presented in Table 5. (a) Audio quality 1 2 3 4 5 6 7 1 CMT – 51 57 38 34 30 29 2 GVMGen 49 – 43 37 42 23 21 3 M2UGen 43 57 – 41 27 34 32 4 Video2Music 62 63 59 – 49 30 28 5 VidMuse 66 58 73 51 – 48 46 6 Video-Robin 70 77 66 70 52 – 30 7 VIBE (ours) 71 79 68 72 54 70 – (b) Musicality 1 2 3 4 5 6 7 1 CMT – 47 59 40 40 25 24 2 GVMGen 53 – 39 37 38 33 31 3 M2UGen 41 61 – 41 29 38 38 4 Video2Music 60 63 59 – 37 34 32 5 VidMuse 60 62 71 63 – 43 41 6 Video-Robin 75 67 62 66 57 – 47 7 VIBE (ours) 76 69 62 68 59 53 – (c) Video-music alignment 1 2 3 4 5 6 7 1 CMT – 51 45 57 28 17 16 2 GVMGen 49 – 43 53 36 25 23 3 M2UGen 55 57 – 55 27 31 30 4 Video2Music 43 47 45 – 29 27 25 5 VidMuse 72 64 73 71 – 41 39 6 Video-Robin 83 75 69 73 59 – 31 7 VIBE (ours) 84 77 70 75 61 69 – (d) Overall 1 2 3 4 5 6 7 1 CMT – 53 53 50 34 19 18 2 GVMGen 47 – 43 44 34 30 28 3 M2UGen 47 57 – 45 27 31 30 4 Video2Music 50 56 55 – 29 30 28 5 VidMuse 66 66 73 71 – 41 39 6 Video-Robin 81 70 69 70 59 – 35 7 VIBE (ours) 82 72 70 72 61 65 – Table 5: Results of the Human A/B evaluation. Win rates (%) from pairwise subjective comparisons across 4 criteria. Fleiss κ = 0.249, Krippendorff α = 0.245. Each cell is the win rate of the row model against the column model. Darker shade indicates higher win rate. 4.5.3 Human Evaluation: A/B Testing. We conducted A/B testing (shown in Table 5) with 18 participants aged 18–30, representative of the short-form video audience. Each evaluator compared music from two randomly selected models paired with the same video across four criteria, yielding four binary judgements per comparison. Across 20 videos and 7 models, each pair was assessed by three judges with majority vote determining the win. The criteria, also adopted in Tian et al. (2025) are: Audio Quality (signal fidelity and absence of artifacts), Musicality (standalone coherence of melody, harmony, and rhythm), Video-Music Alignment (temporal and semantic correspondence between music and video), and Overall Assessment (holistic audiovisual preference). 4.6 Ablations Effect of removing Conditioning Connections. Table 3 demonstrates the importance of the conditioning connection. Adding it to the pretrained model reduces FAD by 32.07%. The base architecture attains a marginally higher IB, but its poor FAD indicates the model does not possess strong priors for high fidelity music generation. Stage-based ablations. Table 3 shows the effects of removing certain training stages from the pipeline. The table shows that all the training stages combined with conditioning connection provide us with better performance on most audio quality metrics. Further, Table 12 shows our Stage 3 model, a text-to-music generation model, (text-to-music Preference Optimization) outperforms ACE-Step v1.5 Gong et al. (2025) when given video captions as input instructions. Visuals Music Bridge (VMB). To evaluate our omni-modal reward model for text+video-to-music generation, we compare it against a cross-modal baseline inspired by Visual-Music-Bridge (VMB) Wang et al. (2024) due to the lack of dedicated reward models in this space. VMB generates detailed video-music alignment captions using Gemini-2.5-Flash and fine-tunes a text-to-music model on them, assuming the text fully captures the visual context. As shown in Table 10, our model trained using Qwen2.5-Omni judge clearly outperforms VMB across all qualitative metrics. 5 Conclusion We presented VIBE, a text-and-video-to-music model that addresses static cross-modal conditioning and the absence of instruction-following supervision via Conditioning Connection and a structured 5-stage training curriculum with holistic reward modeling. Experiments demonstrate strong performance on perceptual quality, generative diversity, and instruction-following metrics, with human evaluation confirming preference for VIBE across all assessment axes. Limitations VIBE inherits the frozen VAE and encoder constraints of its base architecture, which may limit expressivity in niche genres. ImageBind, used as an audio-visual alignment proxy, is not trained on music data and may not fully capture semantic correspondence between generated music and video. VIBE currently targets 10-second instrumental clips and does not support vocal music, long-form scoring, or interactive editing. Finally, genre and mood rewards rely entirely on CMI-RM, whose own limitations propagate into the reward signal, and omni-modal reward extraction via Qwen2.5-Omni adds non-trivial inference overhead during Stage 5 training. Acknowledgement The research at the University of Maryland is partially supported by Adobe, Amazon, NVIDIA, and Sesame. References Di et al. (2021) S. Di, Z. Jiang, S. Liu, Z. Wang, L. Zhu, Z. He, H. Liu, and S. Yan Video background music generation with controllable music transformer. In Proceedings of the 29th ACM International Conference on Multimedia, M ’21, p. 2037–2045. Cited by: §2.1, §4.4, Table 2. Frid et al. (2020) E. Frid, C. Gomes, and Z. Jin Music creation by example. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, New York, NY, USA, p. 1–13. External Links: ISBN 9781450367080, Link, Document Cited by: §1. Girdhar et al. (2023) R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra ImageBind: one embedding space to bind them all. External Links: 2305.05665 Cited by: §4.3. Gong et al. (2025) J. Gong, S. Zhao, S. Wang, S. Xu, and J. Guo ACE-step: a step towards music generation foundation model. External Links: 2506.00045 Cited by: §2.2, §4.6. Hammad et al. (2025) N. Hammad, C. A. Fraser, E. Harpstead, J. Hammer, and M. Dontcheva “It’s more of a vibe i’m going for”: designing text-to-music generation interfaces for video creators. In Proceedings of the 2025 ACM Designing Interactive Systems Conference, DIS ’25, New York, NY, USA, p. 2738–2754. External Links: ISBN 9798400714856, Link, Document Cited by: §1. He et al. (2026) X. He, N. Nan, Z. Wang, Z. Kang, Z. Mo, A. Li, Y. Pan, X. Li, F. Yu, and X. Guan SymphonyGen: 3d hierarchical orchestral generation with controllable harmony skeleton. External Links: 2604.25498, Link Cited by: §2.2. Huang et al. (2023) Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, J. Engel, Q. V. Le, W. Chan, Z. Chen, and W. Han Noise2Music: text-conditioned music generation with diffusion models. External Links: 2302.03917, Link Cited by: §1. Huh et al. (2026) M. Huh, C. A. Fraser, D. Li, M. Dontcheva, and B. Wang VidTune: creating video soundtracks with generative music and contextual thumbnails. External Links: 2601.12180, Link Cited by: §1. Ji et al. (2025) S. Ji, Z. Wang, J. Yu, X. Yang, S. Li, S. Wu, and K. Zhang Diff-v2m: a hierarchical conditional diffusion model with explicit rhythmic modeling for video-to-music generation. External Links: 2511.09090 Cited by: §1, §2.1. Jia et al. (2025) D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang, and Y. Wang DiTAR: diffusion transformer autoregressive modeling for speech generation. External Links: 2502.03930 Cited by: §4.2. Kang et al. (2024) J. Kang, S. Poria, and D. Herremans Video2Music: suitable music generation from videos using an affective multimodal transformer model. Expert Systems with Applications 249, p. 123640. External Links: ISSN 0957-4174 Cited by: §1, §2.1, §4.4, Table 2. Kim et al. (2026) H. Kim, Z. Novack, J. McAuley, and H. Dong Dialogue-aware video-to-music generation using public domain film collections. External Links: 2608.11576, Link Cited by: §1. Kim et al. (2025) H. Kim, Z. Novack, W. Xu, J. McAuley, and H. Dong Video-guided text-to-music generation using public domain movie collections. ISMIR 2025. External Links: Link Cited by: §1. Lei et al. (2025) S. Lei, Y. Xu, Z. Lin, H. Zhang, W. Tan, H. Chen, J. Yu, Y. Zhang, C. Yang, H. Zhu, S. Wang, Z. Wu, and D. Yu LeVo: high-quality song generation with multi-preference alignment. External Links: 2506.07520, Link Cited by: §2.2. Li et al. (2026) B. Li, Y. Guan, H. Li, B. Zeng, Y. Ji, Y. Ding, P. Wan, K. Gai, Y. Zhang, and W. Zhang Semantic routing: exploring multi-layer llm feature weighting for diffusion transformers. External Links: 2602.03510, Link Cited by: §2.1, §3.1. Liang et al. (2026) S. Liang, C. Huang, F. Bellos, Y. Y. Tang, Q. Shen, J. Bi, L. Song, Z. Zhang, J. Corso, and C. Xu Omni-judge: can omni-llms serve as human-aligned judges for text-conditioned audio-video generation?. External Links: 2602.01623, Link Cited by: §4.3. Lin et al. (2026) Y. Lin, J. Casebeer, L. Mai, A. Mahapatra, G. Bertasius, and N. J. Bryan V2M-zero: zero-pair time-aligned video-to-music generation. External Links: 2603.11042, Link Cited by: §2.1. Liu et al. (2024) S. Liu, A. S. Hussain, Q. Wu, C. Sun, and Y. Shan MuMu-llama: multi-modal music understanding and generation via large language models. External Links: 2412.06660 Cited by: §1, §2.1, §4.4, Table 2. Lokegaonkar et al. (2026) V. Lokegaonkar, A. V. Bhosale, V. Raj, G. KV, R. Duraiswami, L. Lu, S. Ghosh, and D. Manocha Video-robin: autoregressive diffusion planning for intent-grounded video-to-music generation. External Links: 2604.17656, Link Cited by: Figure 5, Table 13, §1, §2.1, §3.1, §4.1, §4.4, §4.5.2, Table 2, Table 2. Ma et al. (2026) Y. Ma, H. Xia, H. Gao, W. Chen, Y. Ye, Y. Yang, S. Chang, M. Ding, Y. Li, R. Yuan, S. Dixon, and E. Benetos CMI-rewardbench: evaluating music reward models with compositional multimodal instruction. External Links: 2603.00610, Link Cited by: §2.2, §3.3, §3.4, §4.1. Melechovsky et al. (2024) J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria Mustango: toward controllable text-to-music generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 8293–8316. External Links: Link, Document Cited by: §1, §3.4, §4.1. Naeem et al. (2020) M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo Reliable fidelity and diversity metrics for generative models. Cited by: §4.3. Qwen et al. (2025) Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §3.3. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. External Links: 2103.00020 Cited by: §4.2. Rouard et al. (2023) S. Rouard, F. Massa, and A. Défossez Hybrid transformers for music source separation. In ICASSP 23, Cited by: §4.1. Roy et al. (2025) A. Roy, R. Liu, T. Lu, and D. Herremans Jamendomaxcaps: a large scale music-caption dataset with imputed metadata. In 2025 International Joint Conference on Neural Networks (IJCNN), p. 1–8. Cited by: §3.4, §4.1. Salimans et al. (2016) T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen Improved techniques for training gans. External Links: 1606.03498 Cited by: §4.3. Team et al. (2025) M. Team, C. Xiao, Y. Li, X. Han, Y. Bai, J. Cai, H. Chen, W. Chen, X. Cong, G. Cui, N. Ding, S. Fan, Y. Fang, Z. Fu, W. Guan, Y. Guan, J. Guo, Y. Han, B. He, Y. Huang, B. Ji, C. Kong, Q. Li, S. Li, W. Li, X. Li, Y. Li, Y. Li, Z. Li, D. Liu, B. Lin, Y. Lin, X. Long, Q. Lu, Y. Lu, P. Luo, H. Lyu, L. Ou, Y. Pan, L. Pu, Z. Qu, Q. Shi, Z. Song, J. Su, Z. Su, A. Sun, X. Sun, P. Tang, F. Wang, F. Wang, S. Wang, Y. Wang, Z. Wang, Y. Wu, Z. Xiao, J. Xie, Z. Xie, X. Xu, Y. Yan, J. Yuan, J. Zhang, K. Zhang, L. Zhang, L. Zhang, X. Zhang, Y. Zhang, H. Zhao, W. Zhao, W. Zhao, Y. Zhao, Z. Zheng, C. Zhou, G. Zhou, J. Zhou, W. Zhou, Y. Zhou, Z. Zhou, Z. Zhou, Z. Liu, G. Zeng, C. Jia, D. Li, and M. Sun MiniCPM4: ultra-efficient llms on end devices. External Links: 2506.07900, Link Cited by: §4.2. Tian et al. (2026) Z. Tian, Z. Liu, Y. Jin, R. Yuan, L. Xue, X. Tan, Q. Chen, W. Xue, and Y. Guo AudioX: a unified framework for anything-to-audio generation. In International Conference on Learning Representations (ICLR), Cited by: §2.1. Tian et al. (2025) Z. Tian, Z. Liu, R. Yuan, J. Pan, Q. Liu, X. Tan, Q. Chen, W. Xue, and Y. Guo VidMuse: a simple video-to-music generation framework with long-short-term modeling. External Links: 2406.04321 Cited by: §1, §2.1, §3.4, §4.1, §4.4, §4.5.3, Table 2. Wang et al. (2024) B. Wang, L. Zhuo, Z. Wang, C. Bao, W. Chengjing, X. Nie, J. Dai, J. Han, Y. Liao, and S. Liu Multimodal music generation with explicit bridges and retrieval augmentation. External Links: 2412.09428, Link Cited by: §E.2, §2.1, §4.6. Yang et al. (2025) C. Yang, S. Wang, H. Chen, W. Tan, J. Yu, and H. Li SongBloom: coherent song generation via interleaved autoregressive sketching and diffusion refinement. arXiv preprint arXiv:2506.07634. Cited by: §4.2. Yang et al. (2026) D. Yang, Y. Xie, Y. Yin, Z. Wang, X. Yi, G. Zhu, X. Weng, Z. Xiong, Y. Ma, D. Cong, J. Liu, Z. Huang, J. Ru, R. Huang, H. Wan, P. Wang, K. Yu, H. Wang, L. Liang, X. Zhuang, Y. Wang, Dingdong, Wang, H. Guo, J. Cao, Z. Ju, S. Liu, Y. Cao, H. Weng, and Y. Zou HeartMuLa: a family of open sourced music foundation models. External Links: 2601.10547 Cited by: §2.2. Zhang et al. (2023) Y. Zhang, W. Liu, Z. Chen, J. Wang, and K. Li On the properties of kullback-leibler divergence between multivariate gaussian distributions. External Links: 2102.05485 Cited by: §4.3. Zheng et al. (2026) K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu DiffusionNFT: online diffusion reinforcement with forward process. External Links: 2509.16117, Link Cited by: §3.2. Zhou et al. (2025) Z. Zhou, K. Mei, Y. Lu, T. Wang, and F. Rao HarmonySet: a comprehensive dataset for understanding video-music semantic alignment and temporal synchronization. External Links: 2503.01725 Cited by: §3.4, §4.1. Ziv et al. (2025) A. Ziv, S. Chen, A. Tjandra, Y. Adi, W. Hsu, and B. Shi MR-flowdpo: multi-reward direct preference optimization for flow-matching text-to-music generation. External Links: 2512.10264, Link Cited by: §2.2. Zuo et al. (2025) H. Zuo, W. You, J. Wu, S. Ren, P. Chen, M. Zhou, Y. Lu, and L. Sun GVMGen: a general video-to-music generation model with hierarchical attentions. External Links: 2501.09972 Cited by: §1, §2.1, §4.4, Table 2. Appendix Appendix A Potential Risks Potential Risks VIBE generates instrumental background music conditioned on video and text, which introduces several potential risks that warrant consideration. Misuse for Unauthorized Content. Automated music generation could be used to produce background scores for videos containing harmful, misleading, or manipulative content. Music is a powerful emotional amplifier, and pairing generated scores with propaganda, misinformation, or deceptive advertising could enhance the persuasive impact of such material. Copyright and Intellectual Property. Although VIBE generates novel audio rather than retrieving existing tracks, models trained on large-scale music corpora may inadvertently reproduce melodic fragments, harmonic progressions, or timbral characteristics closely resembling copyrighted works. Users may unknowingly deploy generated music that infringes on existing intellectual property, particularly in commercial contexts. Economic Displacement. Scalable, high-quality background music generation may reduce demand for human composers and music licensors in the short-form video ecosystem. While the tool is intended to complement creative workflows, widespread adoption without appropriate safeguards could disproportionately affect independent musicians who rely on licensing income from stock music and background scoring. Bias in Reward Models. Our reward modeling framework relies on CMI-RM and Qwen2.5-Omni as judges for subjective musical qualities. These models may encode cultural, stylistic, or genre biases present in their training data, potentially favoring Western tonal music conventions and underrepresenting non-Western musical traditions. This could lead VIBE to systematically produce outputs that lack diversity in cultural representation. Deepfake and Deceptive Media. Combined with advances in video generation and voice synthesis, automated music scoring could lower the barrier to producing convincing synthetic audiovisual media. We encourage the development of provenance tracking and watermarking mechanisms for AI-generated music to mitigate this risk. Appendix B License of Artifacts JamendoMaxCaps is derived from the Jamendo platform, where tracks are released under Creative Commons licenses permitting research use. MusicBench is released under a C-BY 4.0 license. CMI-Pref and CMI-RM are released for research purposes by their authors. V2M and HarmonySet are publicly available research datasets; we use them in accordance with their stated terms. ReelBench was obtained directly from its authors with permission for benchmarking. Pre-trained model weights are used under their respective licenses: MiniCPM4-0.5B under the Apache License 2.0, CLIP under the MIT License, SongBloom under its research license, and Qwen2.5-Omni under the Qwen License Agreement. The source code released is extended from VoxCPM under it’s Apache License. All artifacts are used in compliance with their distribution terms for non-commercial academic research. Appendix C Use of AI Assistants We used AI assistants at several stages of this work, detailed below. Data Preparation. In Stage 4 (Text+Video-to-Music Instruction Finetuning), we used Google Gemini to extract structured musical attributes — including tempo, key, instruments, genre, and atmosphere — from audio clips in the V2M and HarmonySet datasets, which do not provide instruction-level text prompts natively. In Stage 5 (Omni-Modal Alignment Preference Tuning), we used Gemini-2.5-Flash to generate 4 diverse instructional prompts per video for the 8,000 resampled training videos. All generated captions were used as training inputs and were not presented as human annotations. Reward Extraction. During preference optimization, we used Qwen2.5-Omni-7B as an omni-modal judge to extract reward signals for musicality, text-music alignment, and video-music alignment (Section 3.3). CMI-RM was used as a cross-modal reward model for text-to-music quality assessment. Both models were used programmatically as scoring functions within the training loop, not for subjective editorial decisions. Visuals Music Bridge (VMB) Baseline. For the VMB ablation (Section 4.6), we used Gemini-2.5-Flash to generate diverse music captions containing video-music alignment details for each training video. Writing Assistance. We used AI language models (ChatGPT and Claude) for proofreading, grammar correction, and minor stylistic editing of the manuscript text. All scientific content, experimental design, analysis, and claims were produced entirely by the authors. No AI assistant was used to generate or fabricate experimental results. Training Objective Details Notation. Table 6 defines all symbols used in Methodology. Forward process. The noisy latent at timestep t follows t=αt0+σtϵx^t= _tx^0+ _t ε, with ϵ∼(0,) ε (0,I). The flow-matching target velocity is the time derivative =α˙t0+σ˙tϵv= α_tx^0+ σ_t ε. Optimality probability mapping. Given group rewards Rgg=1G\R^g\_g=1^G, we compute normalised advantages R^g=(Rg−μ)/(σ+ε) R^g=(R^g-μ)/(σ+ ), clip to [−A,A][-A,A], and map to rg=R^g/(2A)+0.5∈[0,1]r^g= R^g/(2A)+0.5∈[0,1]. KL regularisation. The penalty ‖θ−ref‖22\|v_θ-v^ref\|_2^2 approximates a KL divergence in velocity space and prevents reward hacking without requiring full distribution estimation. Symbol Description 0x^0 Clean latent patch tx^t Noisy latent at timestep t θv_θ LocDiT velocity field (training policy) oldv^old Frozen sampling policy (EMA-updated) refv^ref Frozen reference policy (for KL) pE_p Multimodal planning embedding i−1m_i-1 Previously generated patch αt,σt _t, _t Flow schedule signal/noise coefficients β NFT guidance strength r Per-sample optimality probability A Advantage clip threshold λKL _KL KL regularisation weight Table 6: Notation used in Methodology Appendix D Instruction-Following Evaluation Protocol In figure 3, Tempo is evaluated at exact (± 10%) and octave-equivalent tolerances alongside mean absolute error in BPM; key is evaluated at exact and loose tolerances, where the loose criterion additionally accepts relative and parallel key matches. Note Name Pitch Class C 0 C# / D♭ 1 D 2 D# / E♭ 3 E 4 F 5 F# / G♭ 6 G 7 G# / A♭ 8 A 9 A# / B♭ 10 B 11 Table 7: Pitch class mapping used for key estimation. Enharmonic equivalents share the same pitch class index. Mode C C# D D# E F F# G G# A A# B Major 1 0 1 0 1 1 0 1 0 1 0 1 Minor 1 0 1 1 0 1 0 1 1 0 1 0 Table 8: Chromagram templates for major and minor modes used in key estimation. Each binary vector encodes the scale degrees present in the corresponding mode, rooted at C. For estimation in other keys, the template is circularly shifted by the target pitch class. Tempo Evaluation We estimate the tempo of each generated clip in beats per minute (BPM) using librosa’s beat tracker. Given a ground-truth tempo b∗b^* and a predicted tempo b b, we evaluate correctness under two criteria. Exact Tempo Accuracy considers a prediction correct if |b^−b∗|/b∗≤0.10| b-b^*|/b^*≤ 0.10, i.e., within a 10% relative tolerance. Octave-Equivalent Tempo Accuracy additionally accepts predictions at half or double the ground-truth tempo, accounting for common octave errors in BPM estimators; a prediction is accepted if any of b b, 2b^2 b, or b^/2 b/2 falls within the 10% tolerance of b∗b^*. We also report Tempo MAE, the mean absolute error in BPM across all samples with valid ground-truth tempo annotations. Metrics Ablation FAD↓ FD↓ KL↓ IS↑ IB↑ Den.↑ Cov.↑ VIBE w/o Soft Rewards 1.6231 10.1979 1.3394 1.6607 ± 0.0388 0.0792 0.1431 0.2191 VIBE w/o Hard Rewards 1.6738 10.2975 1.3617 1.6317 ± 0.0352 0.0796 0.152 0.2132 VIBE 1.5829 10.1282 1.3205 2.3578 ± 0.0798 0.0888 0.6741 0.6095 Table 9: Ablation study on reward design in VIBE. We evaluate the contribution of soft and hard reward components to generation quality and alignment performance. Metrics Model FAD↓ FD↓ KL↓ IS↑ IB↑ Den.↑ Cov.↑ Visual Music Bridge 2.2713 18.8573 1.4181 1.5457 ± 0.0998 0.0163 0.0 0.0825 VIBE (Ours) 1.5829 10.1282 1.3205 2.3578 ± 0.0798 0.0888 0.6741 0.6095 Table 10: Comparison between VIBE and the Visual Music Bridge Technique. We evaluate the impact of the visual bridging strategy on generation quality and video-music alignment metrics. Spectrogram Instruction (abridged) Target Measured Target Measured Tempo Tempo Key Key 120 BPM Chillwave in D Major; lush evolving pads, soft kicks, crisp hi-hats 120.0 119.7 D Major D Major Ambient-electronic, 136.36 BPM in A Major; evolving pads, delicate arpeggios 136.4 144.2 A Major A Major Progressive Trance, 125 BPM in F Major; four-on-the-floor kick, side-chained bass, arpeggiated leads 125.0 125.0 F Major E Major Ethereal ambient-electronic, 150 BPM in B Major; evolving pads, arpeggiated electric guitar 150.0 133.9 B Major B Major Slow atmospheric ambient, ≈ 69.8 BPM in G minor; evolving pads, sustained bass 69.8 133.9 G Minor A Major Table 11: Qualitative case studies showing different failure modes of fine-grained instruction adherence on HarmonySet test videos, with mel spectrograms of the generated clips. Green cells: the generated clip matches the instructed attribute (tempo within ± 10% of target; key requiring identical tonic and mode). Red cells: a miss. The two successful cases span the tolerance band (0.3% and 5.8% tempo error). The last row’s tempo matches only under octave equivalence and is marked as a miss under the exact criterion. Tempo (BPM) and key measured with the estimators of Appendix D. Key Evaluation We estimate the key of each generated clip by computing a chromagram using a constant-Q transform via librosa, averaging chroma energy across frames to obtain a 12-dimensional pitch class profile, and correlating it against major and minor key templates across all 12 pitch classes. The pitch class mapping used for parsing ground-truth key annotations is shown in Table 7, where enharmonic equivalents such as C# and D♭ are treated as identical. The chromagram templates for major and minor modes are shown in Table 8; for a key rooted at pitch class p, the template is circularly shifted by p positions before computing cosine similarity against the predicted chroma profile. The key with the highest similarity is taken as the predicted key. Given a predicted key (kpred,mpred)(k_pred,m_pred) and ground-truth key (k∗,m∗)(k^*,m^*), where k denotes the pitch class and m∈major,minorm∈\major,minor\ denotes the mode, we evaluate under two criteria: • Exact Key Accuracy: the prediction is correct if kpred=k∗k_pred=k^* and mpred=m∗m_pred=m^*. • Loose Key Accuracy: the prediction is accepted under any of three conditions: (i) exact match, (i) relative key match, where a predicted major key accepts a ground-truth minor key a minor third above ((k∗−kpred)mod12=9(k^*-k_pred) 12=9) or vice versa ((k∗−kpred)mod12=3(k^*-k_pred) 12=3), or (i) parallel key match, where kpred=k∗k_pred=k^* but mpred≠m∗m_pred≠ m^*. Both metrics are computed only over samples for which a valid ground-truth key annotation is available in ReelBench. Appendix E Further Ablations and associated details E.1 Importance of Hard and Soft Rewards In each of the RL stages, we use the Hard Verifiable and Soft rewards for penalizing the specific musical attributes not followed from the music prompt into the rendered audio. Table 9 show that better audio quality is observed when both the kinds of rewards are used. E.2 Visuals Music Bridge Wang et al. (2024) RVMBT+V→M=RsoftCM+rtempo+rkey⏟RhardR^T+V$→$M_VMB=R^CM_soft+ r_tempo+r_key_R^hard (15) Each training video is captioned offline by Gemini-2.5-Flash into four diverse music prompts; at each RL step one caption is sampled per candidate and scored by CMI-RM, providing a text-grounded proxy for video-music alignment without invoking the video encoder during training. Appendix F Qualitative Comparisons and Failure Modes Qualitative examples for music generation can be see at our project page. Further, we analyze instruction following ability of VIBE and discuss failure modes in Table 11 Appendix G Human Study Instructions Figure 4 depicts the sample shown to the participants for review. Participants were unpaid volunteers recruited from within our institution aged 18-30. Figure 4: Sample of the study shown to participants for A/B Testing results as shown in Table 5 Model FAD ↓ FD ↓ KL ↓ IS ↑ IB ↑ Density ↑ Coverage ↑ ACE-Step1.5 8.642 31.727 1.667 1.799 ± 0.056 0.083 0.140 0.061 VIBE (Stage 3) (Ours) 3.963 24.465 1.503 1.900 ± 0.079 0.097 0.325 0.094 Table 12: Ablation proving the importance of video conditioning. Comparing text-to-music models on the video-to-music generation task when given video captions as input. This is an essential experiment to understand whether visual signals from the video are truly necessary for multimodal music generation or the context provided by the video can be substituted in text form just as effectively. Our text-to-music (TTM) model outperforms current SOTA TTM model ACE-Step1.5. Figure 5: System prompt and evaluation prompt used to configure Gemini as an Omni-Judge for audio-visual alignment evaluation Lokegaonkar et al. (2026). Axis Definition Rhythmic Sync Whether beat structure, tempo and rhythmic patterns align with the motion dynamics, visual cuts and pacing of the video, i.e. low-level temporal synchronisation between musical rhythm and visual events. Theme Coherence Whether the genre and stylistic character of the music suit the subject matter and setting. The judge also predicts a dominant single-word theme per modality (e.g. nature, urban, sci-fi) and reports whether the two agree. Emotion Alignment Whether the emotional tone of the music matches the visual mood. The judge predicts a dominant emotion per modality (e.g. happy, tense, melancholic, calm) and reports whether they correspond. Cultural Relevance Whether instrumentation, musical style and sonic motifs are culturally appropriate to the setting shown; culturally specific scenes may call for regionally relevant styles or instruments. Temporal Dynamics Whether structural changes in the music—builds, drops, transitions, fades—coincide with salient visual events such as scene changes, camera movement or climactic moments. Instrumentation Fit Whether the particular instruments, synthesisers and timbres suit the visual aesthetic and narrative context of the video. Overall Alignment Holistic integration of the audio and visual streams: how naturally the music complements the video across temporal, thematic, emotional and structural correspondence. Table 13: The seven axes along which Gemini scores audio–visual alignment as defined in Video-Robin Lokegaonkar et al. (2026). Each axis is rated independently on the same scale; Overall Alignment is a separate holistic judgement rather than an aggregate of the preceding six.