Paper deep dive
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/10/2026, 3:23:23 AM
Summary
The paper introduces Syn-Omni, a framework for omnimodal embedding that addresses the limitations of fully shared parameter spaces in existing methods. It proposes Orthogonal Modality-Expert LoRA (OME-LoRA) to separate shared and modality-specific adaptation paths and Progressive Synergy Routing (PSR) to enable controlled cross-modal collaboration. Evaluated on 81 tasks across image, video, audio, and audiovisual modalities, Syn-Omni outperforms baselines like Uni-Omni, Omni-Embed-Nemotron, and e5-omni.
Entities (14)
Relation Signals (15)
Syn-Omni → evaluatedon → Video
confidence 95% · Evaluated across 81 diverse tasks spanning... video... modalities
Syn-Omni → evaluatedon → Image
confidence 95% · Evaluated across 81 diverse tasks spanning image... modalities
Syn-Omni → evaluatedon → Audiovisual
confidence 95% · Evaluated across 81 diverse tasks spanning... audiovisual modalities
Syn-Omni → evaluatedon → Audio
confidence 95% · Evaluated across 81 diverse tasks spanning... audio... modalities
Syn-Omni → usescomponent → OME-LoRA
confidence 95% · we propose Syn-Omni... Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA)
Syn-Omni → usescomponent → PSR
confidence 95% · Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors
Syn-Omni → builton → Qwen-2.5-Omni
confidence 92% · We build our Syn-Omni framework on top of pretrained Qwen-2.5-Omni
OME-LoRA → decomposesinto → Shared LoRA Path
confidence 90% · decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.
Tags
Links
- Source: https://arxiv.org/abs/2610.12256v1
- Canonical: https://arxiv.org/abs/2610.12256v1
Trouble viewing inline? Open PDF directly →
Full Text
103,948 characters extracted from source content.
Expand or collapse full text
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings Youngtaek Oh †thanks: Work done during internship at Sony Group Corporation Affiliation: KAIST Affiliation: Sony Group Corporation Affiliation: youngtaek.oh, junmo.kim@kaist.ac.kr Affiliation: qiyu.wu, hiromi.wakaki, yuhki.mitsufuji@sony.comhttps://github.com/sony/syn-omni Qiyu Wu †thanks: Corresponding Author:˜qiyu.wu@sony.com Affiliation: Sony Group Corporation Affiliation: qiyu.wu, hiromi.wakaki, yuhki.mitsufuji@sony.comhttps://github.com/sony/syn-omni Hiromi Wakaki Affiliation: Sony Group Corporation Affiliation: qiyu.wu, hiromi.wakaki, yuhki.mitsufuji@sony.comhttps://github.com/sony/syn-omni Junmo Kim Affiliation: KAIST Affiliation: youngtaek.oh, junmo.kim@kaist.ac.kr Yuki Mitsufuji Affiliation: Sony Group Corporation Affiliation: qiyu.wu, hiromi.wakaki, yuhki.mitsufuji@sony.comhttps://github.com/sony/syn-omni Abstract Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration. 1 Introduction Omnimodal Large Language Models (Omni-MLLMs) have opened a new frontier toward unified understanding across modalities, including text, images, video, audio, and their composed inputs Xu et al. (2025a). With these versatile foundations, recent studies Xiao et al. (2025); Chen et al. (2026) have successfully repurposed Omni-MLLMs into discriminative embedding models through contrastive fine-tuning Jiang et al. (2025); Meng et al. (2026) with LoRA Hu et al. (2022), offering an efficient and scalable framework for unified embedding spaces across heterogeneous modalities. Figure 1: Omnimodal adaptation strategies. We compare three contrastive embedding setups: (1) modality-specific models, each trained independently on a single modality using a shared LoRA, (2) Uni-Omni (baseline), trained on mixed-modality data using a single shared LoRA, and (3) Syn-Omni (Ours), which combines a shared adaptation path with modality-specialized expert paths. Here, the overall score of the modality-specific setup averages each model across all four modalities, including those it was not trained on. While mixed-modality training enables strong unified representations, Syn-Omni further improves the Uni-Omni baseline across diverse modality scenarios. Despite their strong performance, existing omnimodal embedding methods largely rely on fully shared parameter adaptation over mixed-modality data. While such shared adaptation is effective in learning a unified representation space, it provides limited structural separation between modality-shared and modality-specific representations. Since different modalities exhibit both common semantic structures and distinct modality characteristics, a shared adaptation space may underutilize modality-aware specialization over the course of training. As shown in Fig. 1, different omnimodal adaptation strategies exhibit complementary strengths. Modality-specific models effectively preserve modality-aware characteristics, but their isolated training restricts cross-modal interaction and limits generalization. In contrast, Uni-Omni, a baseline trained on mixed-modality data with a single shared LoRA, learns a unified embedding space that performs well on composed multimodal settings such as audiovisual tasks. However, such fully shared adaptation provides limited structural separation to model modality-specific characteristics, whereas completely isolated experts fail to leverage cross-modal synergy. These observations highlight the need for a structured omnimodal adaptation strategy that can jointly support a unified semantic foundation and modality-aware specialization while encouraging collaborative interactions within an omnimodal embedding space. To this end, we propose Syn-Omni, a framework that structurally separates shared and modality-specialized expert paths while progressively enabling cross-modal collaboration. Our framework consists of two key components: Orthogonal Modality-Expert LoRA (OME-LoRA) and Progressive Synergy Routing (PSR). OME-LoRA separates adaptation into shared and modality-specific LoRA paths to capture both universal semantics and distinct characteristics. Complementing this, PSR gradually enables these isolated modality-experts to interact and establish cross-modal synergy during training. As shown in Fig. 1, Syn-Omni consistently improves upon the unified baseline, showing improved gains on composed multimodal tasks such as audiovisual understanding. We extensively evaluate Syn-Omni across 81 diverse embedding tasks spanning image, video, audio, and audiovisual modalities, covering a broad range of task types including classification, retrieval, and QA. We show that Syn-Omni outperforms existing omnimodal embedding baselines while achieving more balanced performance across these diverse modality settings. These results demonstrate the effectiveness of structured specialization and progressive cross-modal collaboration for unified omnimodal representation learning. We summarize our contributions as follows: • We address the structural limitations of existing fully shared or entirely isolated omnimodal adaptation methods by presenting a unified framework that simultaneously balances robust semantic foundations and modality-aware specialization. • We propose Syn-Omni, which pairs structured shared-expert adaptation in OME-LoRA with progressive cross-modal collaboration in PSR, where experts first establish modality priors and then gradually interact with other modalities. • Syn-Omni is evaluated across 81 tasks in image, video, audio, and audiovisual modalities, with various task types. Results show that Syn-Omni improves baselines across diverse modalities. 2 Related Work Omnimodal Embeddings. Multimodal Large Language Models (MLLMs) have been repurposed into embedding models through contrastive fine-tuning. Early frameworks on vision-language models like E5-V Jiang et al. (2024) and VLM2Vec Jiang et al. (2025) pioneered this transition by applying LoRA Hu et al. (2022) to text-image retrieval. Covering broader modalities, GVE Guo et al. (2025) and VLM2Vec-V2 Meng et al. (2026) extended this paradigm to video, while AuroLA Xu et al. (2026) focused on audio-text retrieval. Furthermore, Omni-Embed-Nemotron Xu et al. (2025b) and WAVE Tang et al. (2026) enable any-to-any retrieval across text, image, audio, and video by leveraging large omnimodal backbones Xu et al. (2025a). To further refine cross-modal alignment, methods such as LCO-Emb Xiao et al. (2025) and e5-omni Chen et al. (2026) introduce explicit calibration strategies, including language-centric refinement and modality-aware temperature scaling. While most existing approaches rely on fully-shared adaptation (e.g., a single LoRA layer) across modalities, we introduce a decoupled adaptation framework in the omnimodal setting. Figure 2: Illustration of the Syn-Omni framework. (a) Overall Framework: Omnimodal inputs xmx_m are encoded by modality-specific encoders and a pretrained LLM backbone with OME-LoRA to produce unified embeddings e(xm)e(x_m). (b) Orthogonal Modality-Expert LoRA (OME-LoRA): Shared LoRA path captures universal semantics, while modality-specific expert LoRAs model modality-aware characteristics. The outputs of modality-experts are combined by the routing weight w. (c) Progressive Synergy Routing (PSR): PSR gradually shifts from initial modality-separate routing to later-stage collaborative routing, enabling progressive cross-modal synergy. Structured Adaptation. Several approaches explicitly separate parameters to encourage specialization. MoELoRA Luo et al. (2024) and MoDULA Ma et al. (2024b) instantiate task-level experts within LoRA, while a separate line imposes orthogonality on the internal subspaces of each adapter Zhang et al. (2023); Li et al. (2025). Beyond architectural separation, LIMoE Mustafa et al. (2022) shows that emergent modality-specific experts require explicit control for stable training. DGL Wei et al. (2025) shows that aggressive fusion can suppress modality-specific learning, while MCA Wu et al. (2025a) reports shortcut behaviors induced by simple contrastive objectives. From an optimization perspective, Uni-X Hao et al. (2026) reveals the limitations of shared transformers across heterogeneous modalities. We adapt these ideas to omnimodal embedding learning, separating experts along the modality axis rather than the task axis and regularizing the shared-expert relation instead of each adapter’s internal subspace. 3 Syn-Omni Framework We propose Syn-Omni, a unified omnimodal embedding framework that moves beyond fully shared adaptation across modalities. As shown in Fig. 2, our approach consists of two main components that jointly realize modality specialization and progressive cross-modal collaboration: Orthogonal Modality-Expert LoRA (OME-LoRA) and Progressive Synergy Routing (PSR). We first introduce our omnimodal embedding formulation Uni-Omni in Sec. 3.1. We then detail the design of OME-LoRA in Sec. 3.2, the PSR mechanism in Sec. 3.3, and the training objectives in Sec. 3.4. 3.1 Omnimodal Embedding Learning We consider a mixed omnimodal training setting, where we construct a heterogeneous training mixture encompassing various modality combinations. Under this setup, we train Uni-Omni, a simple contrastive baseline that adapts a pretrained Omni-MLLM using a single modality-shared LoRA. Omnimodal Representation. We define a base modality set ℳ=T,I,V,AM=\T,I,V,A\, representing text, image, video, and audio, respectively. An input xmx_m is any non-empty subset m⊆ℳm , and we define a modality indicator vmod∈0,1|ℳ|v^mod∈\0,1\^|M| indicating the presence of each modality in xmx_m. To obtain the omnimodal embedding (xm)e(x_m) from a pretrained Omni-MLLM fθf_θ, the input is processed with an instruction prompt, and last-token pooling is applied from the final layer: (xm)=Pool(fθ(xm))e(x_m)=Pool(f_θ(x_m)). Contrastive Optimization via LoRA. The pretrained weights W0∈θW_0∈θ are frozen, while low-rank adapters ϕφ are trained. For an intermediate hidden state h, the adapted output is =W0+ΔWϕ()y=W_0h+ W_φ(h). The model is optimized with an InfoNCE objective to align matched pairs in the unified embedding space. For a batch of N pairs, the loss is: ℒcon=−1N∑i=1Nlogexp(sim(i,i+)/τ)∑j=1Nexp(sim(i,j)/τ),L_con=- 1N _i=1^N (sim(e_i,e_i^+)/τ) _j=1^N (sim(e_i,e_j)/τ), (1) where sim(⋅,⋅)sim(·,·) denotes cosine similarity and τ is a temperature hyperparameter. We further detail our omnimodal training mixture in Sec. A.1. 3.2 Orthogonal Modality-Expert LoRA The training of omnimodal embeddings involves heterogeneous modalities with different characteristics, yet typically relies on a single shared LoRA applied across all modalities. This motivates a more structured design to better organize shared and modality-specific adaptation. To this end, we propose Orthogonal Modality-Expert LoRA (OME-LoRA), which introduces a structured decomposition of shared and modality-expert LoRA paths. Parallel Shared-Expert LoRA. Inspired by the decoupling of universal and domain-specific knowledge in multi-task learning literature (Ma et al., 2024b), we extend this foundation to the omnimodal setting to jointly model shared semantics and modality-specialized characteristics. Specifically, as shown in Fig. 2 (b), a shared LoRA path is consistently activated for all input modalities, while modality-expert LoRA paths are selectively engaged based on the modalities present in xmx_m. For an intermediate hidden state h, the output of the OME-LoRA layer y is computed as follows: =W0+ΔWshr()+∑m∈ℳwmΔWm(),y=W_0h+ W_shr(h)+ _m w_m W_m(h), (2) where ΔWshr W_shr captures modality-shared parameter updates and ΔWm W_m models modality-specific counterparts. The routing weights wmw_m enable adaptive integration of modality-expert outputs. We defer the details of modality-aware specialization in modality-experts and the routing strategy for cross-modal synergy among experts to Sec. 3.3. Decoupling via Orthogonal Penalty. We introduce an orthogonal penalty ℒorthoL_ortho that encourages modality-specific experts to learn complementary weight directions relative to the shared pathway. Given that each LoRA update is decomposed into low-rank matrices ΔW=BA W=BA, we define the vectorized form of the input-side projection matrix as =vec(A)a=vec(A). The penalty is then defined as: ℒortho=1|ℳ|∑m∈ℳ(|~shr⋅m|‖~shr‖‖m‖)2,L_ortho= 1|M| _m ( | a_shr·a_m|\| a_shr\|\|a_m\| )^2, (3) where ℒorthoL_ortho minimizes the cosine similarity between the weight directions of the shared and expert LoRA parameters. Here, we apply the penalty only to A, which determines the input subspace of each path, leaving the output projection B unconstrained. We also treat the shared component as a fixed reference by applying a stop-gradient operator, ~shr=sg[shr] a_shr=sg[a_shr]. This blocks gradient propagation so that the shared path is not affected by the penalty. In this way, the penalty helps reduce redundancy between the shared and expert paths. 3.3 Progressive Synergy Routing The interaction between modality-experts is governed by the routing weights wmw_m in Eq. 2. While hard routing based on the modality indicator vmodv^mod activates only the corresponding experts, it restricts cross-modal knowledge exchange across experts. In contrast, soft routing with a learnable router enables synergy, but it can blunt expert specialization if the experts lack sufficient modality-aware priors early on, as explored in Fig. 5 (left). To facilitate stable collaboration among experts, we propose Progressive Synergy Routing (PSR), allowing experts to first establish modality priors via hard routing and then progressively exchange complementary information across modalities with soft routing, as illustrated in Fig. 2 (c). In addition, the Synergy Margin Loss (SML) is designed to avoid excessive suppression of absent-modality experts, preserving expert interactions necessary for cross-modal synergy. Progressive Blending Mechanism. We first introduce a learnable router fϕrf_ _r that generates soft routing weights R=σ(fϕr(Pool()))v^R=σ(f_ _r(Pool(h))) from the intermediate hidden state h, where fϕrf_ _r is a fully-connected layer with parameters ϕr _r and σ(⋅)σ(·) denotes the sigmoid function. This soft router enables experts from non-input modalities to contribute dynamically to the final outputs. To resolve the trade-off between expert specialization and synergy, we propose to progressively blend the hard and soft routing weights over the training progress t to smoothly shift from isolation to interaction. The blended routing weight wm(t)w_m^(t) is computed as: wm(t)=(1−αt)⋅vmmod+αt⋅vmR,w_m^(t)=(1- _t)· v_m^mod+ _t· v_m^R, (4) where αt∈[0,1] _t∈[0,1] is scheduled along with the training progress ratio t (e.g., current step / total steps): αt=0,0≤t<twarmg(t),twarm≤t<twarm+tblend1,twarm+tblend≤t≤1. _t= cases0,&0≤ t<t_warm\\ g(t),&t_warm≤ t<t_warm+t_blend\\ 1,&t_warm+t_blend≤ t≤ 1. cases (5) Here, g(t)=12[1−cos(πt−twarmtblend)]g(t)= 12[1- (π t-t_warmt_blend)] denotes a cosine-based monotonic transition function from 0 to 1, where twarmt_warm controls how long the experts remain isolated and tblendt_blend sets the length of the transition. The schedule consists of three phases: Warmup (t<twarmt<t_warm) for strict isolation with expert specialization, Blending (twarm≤t<twarm+tblendt_warm≤ t<t_warm+t_blend) for a smooth transition between hard and soft routing, and Synergy (t≥twarm+tblendt≥ t_warm+t_blend) for full dynamic collaboration via soft routing weights Rv^R. This schedule can be viewed as a curriculum Bengio et al. (2009) over routing that eases optimization, where experts are first exposed to their own modality before interacting with others. To empirically capture this transition, Fig. 6 visualizes the evolution of routing weights over the course of training. Figure 3: Conceptual comparison between BCE Loss and Synergy Margin Loss (SML). While BCE strictly suppresses absent modalities to 0, SML maintains a relative margin γ (vpresentR−vabsentR≥γv^R_present-v^R_absent≥γ), which allows absent-modality experts to retain meaningful activations, preserving cross-modal synergy during training. Synergy Margin Loss. To encourage modality-aware specialization in the router, we introduce an auxiliary supervision that guides the soft routing activations Rv^R using the modality indicators vmodv^mod. While Binary Cross-Entropy (BCE) could directly enforce modality-aligned routing, it tends to excessively suppress absent modalities, eliminating potentially useful interactions across experts. To preserve such collaborative capacity while maintaining the dominance of input-relevant modalities, we propose a Synergy Margin Loss ℒSMLL_SML. Let P and A denote the sets of present (vmmod=1v_m^mod=1) and absent (vmmod=0v_m^mod=0) modality indices in the input: ℒSML=1||||∑i∈∑j∈max(0,vjR−viR+γ),L_SML= 1|P||A| _i _j (0,v_j^R-v_i^R+γ), (6) where γ is a margin hyperparameter. As shown in Fig. 3, unlike BCE, the proposed objective enforces relative preference rather than absolute suppression, allowing related experts to retain meaningful activations while preserving specialization. 3.4 Training Strategy We train Syn-Omni in an end-to-end manner by integrating OME-LoRA into the backbone and optimize the model with a joint objective that combines contrastive learning and structural regularization. Architectural Integration. Instead of standard LoRA layers, we deploy OME-LoRA throughout the backbone, where each module consists of a shared adaptation path, modality-experts, and a learnable router. Such learnable parameters are maintained independently at each layer. Joint Optimization. The overall training objective ℒtotalL_total combines the primary contrastive objective with the proposed structural regularization losses: ℒtotal=ℒcon+λorthoℒortho+λSMLℒSML,L_total=L_con+ _orthoL_ortho+ _SMLL_SML, (7) where λortho _ortho and λSML _SML control the strengths of the orthogonal penalty and Synergy Margin Loss, respectively. Crucially, both regularization terms ℒorthoL_ortho and ℒSMLL_SML are computed and averaged across all layers containing OME-LoRA modules. Image (36) Video (23) Audio (12) Audiovisual (10) All (81) 3B Models Omni-Embed-Nemotron Xu et al. (2025b) 44.1 36.5 24.5 28.5 33.4 LCO-Emb Xiao et al. (2025) 58.1 43.7 42.1 35.5 44.8 e5-omni Chen et al. (2026) 64.8 40.6 37.4 31.6 43.6 Uni-Omni (Contrastive baseline) 66.8 40.4 43.1 40.4 47.7 Syn-Omni (Ours) 68.1 40.5 45.6 43.4 49.4 7B Models OmniEmbed-v0.1 Ma et al. (2025a) 44.2 35.0 31.6 29.0 35.0 OmniEmbed-Multivent Zhan et al. (2025) 51.8 40.2 39.5 34.4 41.5 LCO-Emb Xiao et al. (2025) 61.6 45.5 45.2 35.5 46.9 e5-omni Chen et al. (2026) 72.5 44.3 44.3 42.3 50.9 WAVE Tang et al. (2026) 42.7 39.8 33.4 38.1 38.5 Uni-Omni (Contrastive baseline) 71.3 40.2 47.5 43.9 50.7 Syn-Omni (Ours) 71.7 41.4 48.7 44.6 51.6 Table 1: Comparison of omnimodal embedding models using Hit@1 (%). The numbers in parentheses indicate the number of evaluation tasks for each modality and the total. Here, Uni-Omni corresponds to a strong baseline trained with a standard LoRA setting. Across both 3B- and 7B-scale models, Syn-Omni achieves higher overall scores than previous methods, demonstrating well-balanced performance across diverse modalities, while also improving over our baseline. For each model scale, the best results are bolded, and the second-best are underlined. 4 Experiment Training Data. We curate an omnimodal data mixture of 1.12M samples spanning image, video, audio, and audiovisual modalities. The training mixture covers diverse tasks including retrieval, classification, and QA, with various modality compositions ranging from single-modality inputs to composed multimodal scenarios. Detailed dataset types and statistics are provided in Sec. A.1. Model Variants. We build our Syn-Omni framework on top of pretrained Qwen-2.5-Omni Xu et al. (2025a) and train both 3B and 7B scale variants. OME-LoRA modules are inserted into the LLM layers while keeping the remaining backbone parameters frozen. We additionally establish Uni-Omni as a controlled baseline trained with the same data and optimization settings, using only a single shared LoRA and contrastive objective. Implementation Details. Images and videos are processed using the native resolution scaling strategy of Qwen-2.5-VL Bai et al. (2025), while videos are uniformly sampled into 8 frames and audio signals are resampled to 16 kHz. Models are trained for 1 epoch using the AdamW Loshchilov and Hutter (2019) optimizer with a cosine learning rate schedule, a peak learning rate of 10−410^-4, a warmup ratio of 0.04, and a global batch size of 768. We employ GradCache Gao et al. (2021) to maintain a large global batch efficiently. For the contrastive loss, we use a temperature τ of 0.02. Consistent with prior methods Jiang et al. (2025), we pool on the EOS token position followed by L2 normalization to extract the final embeddings. For OME-LoRA, we set the rank of the shared adaptation path to 8 and each modality-expert rank to 4. We use a synergy margin γ of 0.2 and apply the proposed Progressive Synergy Routing (PSR) schedule with warmup and transition ratios of twarm=0.1t_warm=0.1 and tblend=0.5t_blend=0.5, respectively. Additional implementation details and hyperparameter configurations are provided in Sec. B.1. Evaluation Setup. We evaluate our framework on an extensive omnimodal benchmark consisting of 81 tasks spanning image, video, audio, and audiovisual modalities. Each modality group covers multiple embedding-oriented meta-tasks, such as retrieval, classification, and QA. For image and video evaluations, we adopt the MMEB-v1 Jiang et al. (2025) and MMEB-v2 Meng et al. (2026) benchmarks, respectively, while further extending with additional audio and audiovisual benchmarks collected in this work. The complete evaluation dataset details are provided in Sec. A.2. Baselines. We compare Syn-Omni against recent Omni-MLLM-based embedding models, including OmniEmbed-v0.1 Ma et al. (2025a), Omni-Embed-Nemotron Xu et al. (2025b), OmniEmbed-Multivent Zhan et al. (2025), WAVE Tang et al. (2026), LCO-Emb Xiao et al. (2025), and e5-omni Chen et al. (2026). For a fair evaluation, we use officially released checkpoints and perform all evaluations under the unified protocol. Metrics. We use Hit@1 as the evaluation metric, measuring whether the ground-truth target is ranked at the top among candidates. We report both modality-specific average performance and the overall average score from a single run. The overall score is computed by averaging modality-specific scores, where each modality-specific score is obtained by averaging the corresponding meta-task scores. Here, each meta-task score is computed as the average of its associated benchmark scores. Detailed benchmark-level performance breakdowns are provided in Sec. C.4. 4.1 Main Results Overall Performance. In Tab. 1, we compare our framework against recent Omni-MLLM embedding models. While some methods excel in specific modalities, such as LCO-Emb Xiao et al. (2025) on video tasks and e5-omni Chen et al. (2026) on image tasks, their strong performance is less consistent across modalities. In contrast, our contrastive baseline, Uni-Omni, trained with a modality-shared LoRA, already achieves strong performance across all modalities at both the 3B and 7B scales. Building upon this baseline, Syn-Omni reaches the best overall scores of 49.4% and 51.6%, respectively. These improvements over Uni-Omni are statistically significant across the 81 tasks in a paired t-test (p=0.0044p=0.0044 and p=0.0189p=0.0189). Modality-wise Analysis. Breaking down the gains by modality, Syn-Omni improves most on audio and audiovisual tasks (+2.5%+2.5\% and +3.0%+3.0\% at the 3B scale), with more moderate margins on image and video. A similar pattern appears at the 7B scale, where e5-omni leads on image and video while Syn-Omni is stronger on audio and audiovisual tasks and attains a higher overall score. This yields more even coverage across all modalities. OME-LoRA PSR id S-Exp. ℒorthoL_ortho Soft Prog. I V A AV All (a) - - - - 66.8 40.4 43.1 40.4 47.7 (b) ✓ - - - 67.6 39.6 44.9 42.6 48.7 (c) ✓ ✓ - - 67.4 39.9 45.2 41.7 48.6 (d) ✓ ✓ ✓ - 67.3 39.9 45.0 42.1 48.6 (e) ✓ - ✓ ✓ 68.2 40.1 45.2 42.6 49.0 (f) ✓ ✓ ✓ ✓ 68.1 40.5 45.6 43.4 49.4 Table 2: Component analysis of Syn-Omni framework. Starting from the Uni-Omni baseline in (a), each row incorporates individual components of our design. The results show that both OME-LoRA and PSR contribute to improved and balanced performance over the baseline, with the full model achieving the best overall results. 4.2 Ablation Study We conduct a series of ablation experiments to analyze the individual contribution of each proposed component in our Syn-Omni framework. All results are reported using 3B-scale models, including both modality-specific and overall averaged scores. Component Analysis. As shown in Tab. 2, compared to Uni-Omni baseline in (a), introducing a shared-expert structure (S-Exp.) with hard routing in (b) improves performance through structural decomposition. Since hard routing activates experts in isolation and leaves limited interaction for shared-expert decorrelation, the orthogonal penalty alone in (c) yields no additional gain over (b). Soft routing alone in (d) likewise maintains a comparable level, as it lets experts interact before acquiring modality priors, which blunts specialization. When applying Progressive Synergy Routing (PSR) with Synergy Margin Loss (SML) in (e), further improvements are obtained. Under this configuration, the orthogonal penalty provides a complementary gain in (f) (49.0 → 49.4), indicating that reducing redundancy becomes effective once multiple expert outputs are jointly integrated. Beyond these overall numbers, the components introduced on top of the shared-expert structure also shape how experts behave: they yield clearer separation across pathways, progressive participation of absent-modality experts, and routing activations that avoid collapsing to hard assignments, which we further analyze in Sec. 4.3. Structural Efficiency. We explore the impact of different architectural choices under matched parameter budgets. As shown in Tab. 3, simply increasing the rank of a single shared path from 16 to 36 brings only marginal gains, with the rank 24 counterpart in (b) even leading to a slight drop. In contrast, our OME-LoRA decomposes the same total rank (24) into shared and expert paths, which brings improvements at an identical parameter count (44.9M) in (f). Applying PSR on top of this structure in (g) further boosts performance to 49.4%\%. Notably, this surpasses the largest shared-only variant in (c) while using fewer trainable parameters, confirming that the gains stem from structured decomposition rather than increased capacity. id Variant Rank Params. PSR I V A AV All (a) Shr 16 29.9M - 66.8 40.4 43.1 40.4 47.7 (b) Shr 24 44.9M - 67.3 38.7 43.0 41.6 47.6 (c) Shr 36 59.9M - 67.3 40.7 43.8 41.2 48.2 (d) Exp 16 29.9M - 66.1 39.4 42.5 40.1 47.0 (e) Exp 16 33.3M ✓ 65.3 39.5 42.1 40.5 46.9 (f) Shr+Exp 24 44.9M - 67.6 39.6 44.9 42.6 48.7 (g) Shr+Exp 24 48.3M ✓ 68.1 40.5 45.6 43.4 49.4 Table 3: Analysis of the adaptation structure. ‘Shr’ denotes a single shared LoRA path (Uni-Omni) with increasing rank. ‘Exp’ denotes modality-expert paths without a shared LoRA path, and ‘Shr+Exp’ denotes our OME-LoRA architecture. Neither path alone is sufficient: scaling the shared path yields limited gains, and removing it degrades performance further. Routing Strategy (PSR) id Router Progress loss I V A AV All (a) Hard - - 67.4 39.9 45.2 41.7 48.6 (b) Soft - BCE 67.3 39.9 45.0 42.1 48.6 (c) Soft - SML 67.6 40.4 45.1 41.9 48.8 (d) Soft ✓ BCE 67.6 39.6 46.0 41.9 48.8 (e) Soft ✓ SML 68.1 40.5 45.6 43.4 49.4 Table 4: Analysis on Progressive Synergy Routing (PSR) strategy. We evaluate the impact of routing types, progressive blending, and router loss functions in the OME-LoRA layers. We highlight that combining a progressive blending of routing weights with the Synergy Margin Loss (SML) is crucial for both expert specialization and cross-modal collaboration among experts. Role of Shared LoRA. We next evaluate the necessity of the shared LoRA path within the OME-LoRA architecture. As shown in Tab. 3, omitting the shared path leads to a substantial performance drop under both hard routing (f vs. d) and with PSR integrated (g vs. e) cases. More importantly, expert-only adaptation in (d) falls even below the shared-only baseline in (a) at the same parameter count (47.0 vs. 47.7 at 29.9M). This suggests that the shared path provides a unified cross-modal space on which experts build their specialization, and that establishing it matters more than allocating the same capacity to experts alone. Role of Progressive Synergy Routing. We further investigate the impact of our routing strategies, as summarized in Tab. 4. Compared to the hard routing baseline in (a), switching to soft routing with BCE loss in (b) provides limited to no improvement. However, across both standard soft routing (b vs. c) and progressive blending configurations (d vs. e), replacing BCE with the Synergy Margin Loss (SML) consistently improves performance. Our final framework in (e), which integrates both progressive blending and SML, achieves the best overall performance of 49.4%. This demonstrates that the two components, progressive blending with synergy margin loss in PSR, complement each other to deliver the superior results. Figure 4: Sensitivity analysis of hyperparameters on the orthogonal penalty weight λortho _ortho, the synergy margin loss weight λSML _SML, and the margin parameter γ. Red stars indicate the optimal choice in our final model. Figure 5: Pairwise cosine similarity between OME-LoRA pathways. Without progressive blending (left), high cross-path similarities reveal redundancy across different pathways. In contrast, Syn-Omni (right), which differs only in applying progressive blending, produces significantly lower similarities. This indicates effective decoupling and clear specialization across pathways. Sensitivity Analysis. We investigate the sensitivity of three key hyperparameters in Syn-Omni: the orthogonal penalty weight (λortho _ortho), the synergy margin loss weight (λSML _SML), and the margin parameter (γ). As shown in Fig. 4, the best overall score is achieved with λortho=0.20 _ortho=0.20, λSML=0.20 _SML=0.20, and γ=0.2γ=0.2. The performance remains relatively stable around these values, while larger margin values lead to a noticeable performance drop. Figure 6: Routing dynamics of PSR under video-audio inputs. Present-modality experts retain dominant routing weights, while absent-modality experts gradually acquire non-zero activations through progressive blending. This indicates the emergence of synergistic cross-modal interactions along with modality specialization. 4.3 Empirical Analysis To better understand how Syn-Omni structures omnimodal representations, we analyze the behaviors of its two key components, OME-LoRA and PSR, on VALOR-32k Liu et al. (2025) evaluation samples with text, video, and audio inputs. Results are computed from final-layer signals averaged across samples and submodules. Decoupling in OME-LoRA. As shown in Fig. 5, we measure pairwise cosine similarity between the flattened hidden states of the shared pathway and modality-experts using text-video-audio inputs. Without applying progressive blending (left), relying solely on soft routing fails to induce clear separation, evidenced by high cross-path similarities both between the shared pathway and experts, and across experts. In contrast, Syn-Omni (right) produces substantially lower cross-path similarities, indicating clear modality specialization across paths in the OME-LoRA architecture. Routing Dynamics in PSR. We analyze the evolution of PSR routing weights under video-audio inputs to demonstrate the emergence of cross-modal interactions during training in Fig. 6. Under progressive blending, present-modality experts (video and audio) maintain dominant routing weights. In contrast, absent-modality experts (text and image), initially suppressed to zero under hard routing scheme, gradually acquire non-zero weights by introducing soft routing scheme. This behavior demonstrates that PSR enables cross-modal collaboration while preserving modality specialization. We further observe that SML drives this synergistic behavior, whereas BCE loss completely suppresses absent-modality experts, as shown in Fig. 8. 5 Conclusion We propose Syn-Omni, a structured omnimodal adaptation framework that effectively learns a unified representation space. It leverages OME-LoRA to decouple modality-shared and specialized parameters. In addition, Progressive Synergy Routing (PSR) shifts the model from isolated specialization toward dynamic cross-modal collaboration. Across 81 diverse tasks, Syn-Omni outperforms existing baselines, achieving robust and well-balanced performance across image, video, audio, and audiovisual modalities. Limitations Our framework introduces additional inference overhead through dynamic routing in the OME-LoRA layer. While standard LoRA weights can be merged into the frozen backbone after training, our input-dependent routing prevents this for the modality-experts, although the shared pathway remains mergeable. Specifically, the increased overhead due to the expert pathways active at inference amounts to 17–18%\% higher latency with less than 1%\% additional GPU memory, as reported in Tab. 7. This cost is most visible in latency-sensitive settings such as real-time query encoding. Ethical Considerations Our framework may inherit systemic biases from large-scale training data, leading to skewed cross-modal retrieval behaviors. Additionally, such unified omnimodal embeddings could be misused for unauthorized surveillance or identity linkage, necessitating strict deployment oversight. References Agostinelli et al. (2023) A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank Musiclm: generating music from text. arXiv preprint arXiv:2301.11325. Cited by: Table 10, Table 14, Table 14, 11st item. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, J. Lin, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §B.1, §4. Barbu et al. (2019) A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz ObjectNet: a large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, Vol. 32, p. 9453–9463. Cited by: Table 12. Bengio et al. (2009) Y. Bengio, J. Louradour, R. Collobert, and J. Weston Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, p. 41–48. Cited by: §3.3. Bolya et al. (2025) D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Bangalath, J. Wang, M. Monteiro, H. Xu, S. Dong, N. Ravi, S. Li, P. Dollar, and C. Feichtenhofer Perception encoder: the best visual embeddings are not at the output of the network. In Advances in Neural Information Processing Systems, Vol. 38, p. 60884–60937. Cited by: Table 9, 3rd item. Caba Heilbron et al. (2015) F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles Activitynet: a large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, p. 961–970. Cited by: 5th item. Carreira et al. (2019) J. Carreira, E. Noland, C. Hillier, and A. Zisserman A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987. Cited by: Table 13. Chang et al. (2022) Y. Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y. Bisk Webqa: multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16495–16504. Cited by: Table 12, Table 8. Chen and Dolan (2011) D. Chen and W. Dolan Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, D. Lin, Y. Matsumoto, and R. Mihalcea (Eds.), Portland, Oregon, USA, p. 190–200. External Links: Link Cited by: Table 13, Table 13, Table 9, 4th item. Chen et al. (2026) H. Chen, S. Gao, R. Timofte, T. Sakai, and Z. Dou E5-omni: explicit cross-modal alignment for omni-modal embeddings. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, p. 19430–19443. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: §B.1, 35th item, §1, §2, Table 1, Table 1, §4.1, §4. Chen et al. (2023) S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, and J. Liu VAST: a vision-audio-subtitle-text omni-modality foundation model and dataset. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 11, 15th item. Dao (2024) T. Dao FlashAttention-2: faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §B.1, 39th item. Das et al. (2017) A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 326–335. Cited by: Table 12, Table 8. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: Table 12, Table 8. Deng et al. (2024) Z. Deng, Y. Ma, Y. Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos MusiLingo: bridging music and text with pre-trained language models for music captioning and query response. In Findings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 3643–3655. External Links: Link, Document Cited by: Table 10, 13rd item. Drossos et al. (2020) K. Drossos, S. Lipping, and T. Virtanen Clotho: an audio captioning dataset. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 736–740. Cited by: Table 14, Table 14, 25th item. Engel et al. (2017) J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan Neural audio synthesis of musical notes with wavenet autoencoders. In International conference on machine learning, p. 1068–1077. Cited by: Table 14, 23rd item. Everingham et al. (2015) M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman The pascal visual object classes challenge: a retrospective. International journal of computer vision 111 (1), p. 98–136. Cited by: Table 12, Table 8. Fu et al. (2025) C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, P. Chen, Y. Li, S. Lin, S. Zhao, K. Li, T. Xu, X. Zheng, E. Chen, C. Shan, R. He, and X. Sun Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24108–24118. Cited by: Table 13. Fu et al. (2023) S. Fu, N. Y. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola DreamSim: learning new dimensions of human visual similarity using synthetic data. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 12, Table 8. Gao et al. (2017) J. Gao, C. Sun, Z. Yang, and R. Nevatia Tall: temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, p. 5267–5275. Cited by: Table 13. Gao et al. (2021) L. Gao, Y. Zhang, J. Han, and J. Callan Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), A. Rogers, I. Calixto, I. Vulić, N. Saphra, N. Kassner, O. Camburu, T. Bansal, and V. Shwartz (Eds.), Online, p. 316–321. External Links: Link, Document Cited by: §B.1, 38th item, §4. Gemmeke et al. (2017) J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter Audio set: an ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), p. 776–780. Cited by: Table 10, Table 11, 10th item, 14th item. Goyal et al. (2017) R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Cited by: Table 13. Guo et al. (2025) Z. Guo, M. Li, Y. Zhang, D. Long, P. Xie, and X. Chu Towards universal video retrieval: generalizing video embedding via synthesized multimodal pyramid curriculum. arXiv preprint arXiv:2510.27571. Cited by: §2. Gurari et al. (2018) D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham Vizwiz grand challenge: answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3608–3617. Cited by: Table 12. Han et al. (2025) M. Han, L. Yang, X. Chang, L. Yao, and H. Wang Shot2story: a new benchmark for comprehensive understanding of multi-shot videos. In International Conference on Learning Representations, Cited by: Table 11, 17th item. Hao et al. (2026) J. Hao, H. Liu, X. Xiao, Q. Huang, and J. Yu Uni-x: mitigating modality conflict with a two-end-separated architecture for unified multimodal models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Hendricks et al. (2017) L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, p. 5803–5812. Cited by: Table 13, Table 13, Table 9, 6th item. Hendrycks et al. (2021a) D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer The many faces of robustness: a critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, p. 8340–8349. Cited by: Table 12. Hendrycks et al. (2021b) D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15262–15271. Cited by: Table 12. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Hu et al. (2023) H. Hu, Y. Luan, Y. Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M. Chang Open-domain visual entity recognition: towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12065–12075. Cited by: Table 12. Hudson and Manning (2019) D. A. Hudson and C. D. Manning Gqa: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6700–6709. Cited by: Table 12. Jiang et al. (2024) T. Jiang, M. Song, Z. Zhang, H. Huang, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang E5-v: universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580. Cited by: §2. Jiang et al. (2025) Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y. Zhou, and W. Chen VLM2vec: training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §A.2, Table 8, 1st item, 19th item, §1, §2, §4, §4. Kazemzadeh et al. (2014) S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg ReferItGame: referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans (Eds.), Doha, Qatar, p. 787–798. External Links: Link, Document Cited by: Table 12, Table 12. Kiela et al. (2020) D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine The hateful memes challenge: detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems, Vol. 33, p. 2611–2624. Cited by: Table 12, Table 8. Kim et al. (2019) C. D. Kim, B. Kim, H. Lee, and G. Kim AudioCaps: generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 119–132. External Links: Link, Document Cited by: Table 10, Table 14, Table 14, 24th item, 8th item. Kim et al. (2025) S. Kim, H. Oh, J. Lee, A. Senocak, J. S. Chung, and T. Oh Avhbench: a cross-modal hallucination benchmark for audio-visual large language models. In International Conference on Learning Representations, External Links: Link Cited by: Table 15, Table 15, Table 15, 30th item. Koepke et al. (2023) A. S. Koepke, A. Oncescu, J. F. Henriques, Z. Akata, and S. Albanie Audio retrieval with natural language queries: a benchmark study. IEEE Transactions on Multimedia 25, p. 2675–2685. External Links: Document Cited by: Table 14, 26th item. Krishna et al. (2017) R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, p. 706–715. Cited by: Table 9, 5th item. Kuehne et al. (2014) H. Kuehne, A. Arslan, and T. Serre The language of actions: recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 780–787. Cited by: Table 13. Kuehne et al. (2011) H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre HMDB: a large video database for human motion recognition. In 2011 International conference on computer vision, p. 2556–2563. Cited by: Table 13. Lei et al. (2021) J. Lei, T. L. Berg, and M. Bansal Detecting moments and highlights in videos via natural language queries. In Advances in Neural Information Processing Systems, Vol. 34, p. 11846–11858. Cited by: Table 13. Li et al. (2024) K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, L. Wang, and Y. Qiao MVBench: a comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 22195–22206. Cited by: Table 13. Li et al. (2025) Z. Li, S. Sajadmanesh, J. Li, and L. Lyu StelLA: subspace learning in low-rank adaptation using stiefel manifold. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: Table 12, Table 12, Table 12, Table 8, Table 8. Liu et al. (2021a) F. Liu, Y. Wang, T. Wang, and V. Ordonez Visual news: benchmark and challenges in news image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 6761–6771. External Links: Link, Document Cited by: Table 12, Table 12, Table 8. Liu et al. (2025) J. Liu, S. Chen, X. He, L. Guo, X. Zhu, W. Wang, and J. Tang VALOR: vision-audio-language omni-perception pretraining model and dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (2), p. 708–724. External Links: Document Cited by: §C.1, Table 15, Table 15, Table 15, Table 15, 29th item, §4.3. Liu et al. (2023) S. Liu, W. Feng, T. Fu, W. Chen, and W. Wang EDIS: entity-driven image search over multimodal web content. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 4877–4894. External Links: Link, Document Cited by: Table 12. Liu et al. (2021b) Z. Liu, C. Rodriguez-Opazo, D. Teney, and S. Gould Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2125–2134. Cited by: Table 12, Table 8. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §4. Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems, Vol. 35, p. 2507–2521. Cited by: Table 12. Luo et al. (2024) T. Luo, J. Lei, F. Lei, W. Liu, S. He, J. Zhao, and K. Liu Moelora: contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint arXiv:2402.12851. Cited by: §2. Ma et al. (2025a) X. Ma, L. Gao, S. Zhuang, J. S. Zhan, J. Callan, and J. Lin Tevatron 2.0: unified document retrieval toolkit across scale, language, and modality. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 4061–4065. Cited by: 36th item, 41st item, Table 1, §4. Ma et al. (2024a) X. Ma, S. Lin, M. Li, W. Chen, and J. Lin Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 6492–6505. External Links: Link, Document Cited by: Table 12. Ma et al. (2024b) Y. Ma, Z. Liang, H. Dai, B. Chen, D. Gao, Z. Ran, W. Zihan, L. Jin, W. Jiang, G. Zhang, X. Cai, and L. Yang MoDULA: mixture of domain-specific and universal LoRA for multi-task learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 2758–2770. External Links: Link, Document Cited by: §2, §3.2. Ma et al. (2025b) Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y. Chao, R. Xu, W. Chen, Y. Chen, Z. Chen, J. Cong, K. Li, K. Li, S. Li, X. Li, X. Li, Z. Lian, Y. Liang, M. Liu, Z. Niu, T. Wang, Y. Wang, Y. Wang, Y. Wu, G. Yang, J. Yu, R. Yuan, Z. Zheng, Z. Zhou, H. Zhu, W. Xue, E. Benetos, K. Yu, E. Chng, and X. Chen MMAR: a challenging benchmark for deep reasoning in speech, audio, music, and their mix. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: Table 14, 28th item. Mangalam et al. (2023) K. Mangalam, R. Akshulakov, and J. Malik EgoSchema: a diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems, Vol. 36, p. 46212–46244. Cited by: Table 13. Marino et al. (2019) K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi Ok-vqa: a visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, p. 3195–3204. Cited by: Table 12, Table 8. Masry et al. (2022) A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 2263–2279. External Links: Link, Document Cited by: Table 12, Table 8. Mathew et al. (2022) M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1697–1706. Cited by: Table 12, Table 8. Mathew et al. (2021) M. Mathew, D. Karatzas, and C. Jawahar Docvqa: a dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 2200–2209. Cited by: Table 12, Table 8. Mei et al. (2024) X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang Wavcaps: a chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing 32, p. 3339–3354. Cited by: Table 10, 9th item. Melechovsky et al. (2024) J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria Mustango: toward controllable text-to-music generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 8293–8316. External Links: Link, Document Cited by: Table 10, 12nd item. Meng et al. (2026) R. Meng, Z. Jiang, Y. Liu, M. Su, X. Yang, Y. Fu, C. Qin, R. Thirukovalluru, X. Zhang, Z. Chen, R. Xu, C. Xiong, Y. Zhou, W. Chen, and S. Yavuz VLM2vec-v2: advancing multimodal embedding for videos, images, and visual documents. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §A.2, §B.1, §B.2, 20th item, §1, §2, §4. Mustafa et al. (2022) B. Mustafa, C. Riquelme, J. Puigcerver, R. Jenatton, and N. Houlsby Multimodal contrastive learning with limoe: the language-image mixture of experts. In Advances in Neural Information Processing Systems, Vol. 35, p. 9564–9576. Cited by: §2. Piczak (2015) K. J. Piczak ESC: dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, p. 1015–1018. Cited by: Table 14, 21st item. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139, p. 8748–8763. Cited by: Table 12. Sakshi et al. (2025) S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha MMAU: a massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Table 14, 27th item. Schwenk et al. (2022) D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi A-okvqa: a benchmark for visual question answering using world knowledge. In European conference on computer vision, p. 146–162. Cited by: Table 12, Table 8. Singh et al. (2019) A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 8317–8326. Cited by: Table 12. Soomro et al. (2012) K. Soomro, A. R. Zamir, and M. Shah Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Cited by: Table 13. Tang et al. (2026) C. Tang, Q. Xiao, K. Mei, T. Wang, F. Rao, and C. Zhang WAVE: learning unified & versatile audio-visual embeddings with multimodal LLM. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: 37th item, §2, Table 1, §4. Tian et al. (2018) Y. Tian, J. Shi, B. Li, Z. Duan, and C. Xu Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), p. 247–263. Cited by: Table 15, Table 15, 26th item. Tzanetakis and Cook (2002) G. Tzanetakis and P. Cook Musical genre classification of audio signals. IEEE Transactions on speech and audio processing 10 (5), p. 293–302. Cited by: Table 14, 22nd item. Wang et al. (2019) X. Wang, J. Wu, J. Chen, L. Li, Y. Wang, and W. Y. Wang Vatex: a large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, p. 4581–4591. Cited by: Table 13, Table 13. Wang et al. (2024) Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng, Z. Wang, Y. Shi, T. Jiang, S. Li, J. Xu, H. Zhang, Y. Huang, Y. Qiao, Y. Wang, and L. Wang InternVideo2: scaling foundation models for multimodal video understanding. In European conference on computer vision, p. 396–416. Cited by: Table 11, 16th item. Wang et al. (2022) Z. Wang, X. Shan, X. Zhang, and J. Yang N24News: a new dataset for multimodal news classification. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis (Eds.), Marseille, France, p. 6768–6775. External Links: Link Cited by: Table 12, Table 8. Wei et al. (2025) S. Wei, C. Luo, and Y. Luo Boosting multimodal learning via disentangled gradient learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 22879–22888. Cited by: §2. Wu et al. (2021) H. Wu, Y. Gao, X. Guo, Z. Al-Halah, S. Rennie, K. Grauman, and R. Feris Fashion iq: a new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, p. 11307–11317. Cited by: Table 12. Wu et al. (2025a) Q. Wu, S. Cui, S. Hayakawa, W. Wang, H. Wakaki, and Y. Mitsufuji MCA: modality composition awareness for robust composed multimodal retrieval. arXiv preprint arXiv:2510.15543. Cited by: §2. Wu et al. (2025b) Y. Wu, Z. Qi, Y. Wu, J. Sun, Y. Wang, and S. Wang Learning fine-grained representations through textual token disentanglement in composed video retrieval. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Table 9, 7th item. Xiao et al. (2025) C. Xiao, H. P. Chan, H. Zhang, W. Xu, M. Aljunied, and Y. Rong Scaling language-centric omnimodal representation learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: 34th item, §1, §2, Table 1, Table 1, §4.1, §4. Xiao et al. (2010) J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba Sun database: large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, p. 3485–3492. Cited by: Table 12, Table 8. Xiao et al. (2021) J. Xiao, X. Shang, A. Yao, and T. Chua NExT-qa: next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9777–9786. Cited by: Table 13. Xu et al. (2026) J. Xu, C. Thomé, D. Horak, W. Xie, and A. Zisserman Scaling audio-text retrieval with multimodal large language models. arXiv preprint arXiv:2602.18010. Cited by: §2. Xu et al. (2025a) J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215. Cited by: §B.1, §B.1, 32nd item, §1, §2, §4. Xu et al. (2016) J. Xu, T. Mei, T. Yao, and Y. Rui Msr-vtt: a large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 5288–5296. Cited by: Table 13, Table 13, Table 9, 4th item. Xu et al. (2025b) M. Xu, W. Zhou, Y. Babakhin, G. Moreira, R. Ak, R. Osmulski, B. Liu, E. Oldridge, and B. Schifferer Omni-embed-nemotron: a unified multimodal retrieval model for text, image, audio, and video. arXiv preprint arXiv:2510.03458. Cited by: 33rd item, §2, Table 1, §4. Yang et al. (2022) P. Yang, X. Wang, X. Duan, H. Chen, R. Hou, C. Jin, and W. Zhu Avqa: a dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, p. 3480–3491. Cited by: 18th item. Ye et al. (2024) Q. Ye, Z. Yu, R. Shao, X. Xie, P. Torr, and X. Cao Cat: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. In European Conference on Computer Vision, p. 146–164. Cited by: Table 11, 18th item. Yu et al. (2019) Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao Activitynet-qa: a dataset for understanding complex web videos via question answering. In Proceedings of the AAAI conference on artificial intelligence, p. 9127–9134. Cited by: Table 13. Yuan et al. (2025) H. Yuan, J. Ni, Z. Liu, Y. Wang, J. Zhou, Z. Liang, B. Zhao, Z. Cao, J. Wen, and Z. Dou MomentSeeker: a task-oriented benchmark for long-video moment retrieval. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link Cited by: Table 13. Zhan et al. (2025) J. S. Zhan, C. Zhang, S. Zhuang, X. Ma, and J. Lin MAGMaR shared task system description: video retrieval with omniembed. arXiv preprint arXiv:2506.09409. Cited by: 36th item, Table 1, §4. Zhang et al. (2023) Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, Cited by: §2. Zhang et al. (2025) R. Zhang, L. Gui, Z. Sun, Y. Feng, K. Xu, Y. Zhang, D. Fu, C. Li, A. G. Hauptmann, Y. Bisk, and Y. Yang Direct preference optimization of video large multimodal models from language model reward. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 694–717. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: Table 9, Table 9, 2nd item. Zhou et al. (2018a) B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba Places: a 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (6), p. 1452–1464. External Links: Document Cited by: Table 12. Zhou et al. (2018b) L. Zhou, C. Xu, and J. Corso Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, Cited by: Table 13, Table 13. Zhou et al. (2025) Z. Zhou, R. Wang, Z. Wu, and Y. Jiang Daily-omni: towards audio-visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862. Cited by: Table 15, 31st item. Zhu et al. (2016) Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei Visual7w: grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4995–5004. Cited by: Table 12, Table 12, Table 8. Contents 1 Introduction 2 Related Work 3 Syn-Omni Framework 3.1 Omnimodal Embedding Learning 3.2 Orthogonal Modality-Expert LoRA 3.3 Progressive Synergy Routing 3.4 Training Strategy 4 Experiment 4.1 Main Results 4.2 Ablation Study 4.3 Empirical Analysis 5 Conclusion References A Additional Details A.1 Training Datasets A.2 Evaluation Datasets B Detailed Experimental Settings B.1 Training Details & Hyperparameters B.2 Evaluation Pipeline C Detailed Results C.1 Routing Behavior across Layers C.2 Modality-Specific Models C.3 Runtime Analysis C.4 Per-dataset Evaluation D AI Tool Usage E Artifact Appendix A Additional Details A.1 Training Datasets To facilitate omnimodal learning, we construct a comprehensive training mixture categorized into four modality groups: image, video, audio, and audiovisual. Beyond single modalities, this mixture features composed modality queries and/or targets, allowing the model to learn versatile alignment spaces. The distribution and coverage of sample counts for each query-to-target modality composition are visualized in Fig. 7, while detailed dataset-level specifications for each modality group are provided in Tabs. 8, 9, 10 and 11, respectively. For each dataset, if the original dataset size exceeded our specified target size, we randomly sub-sampled it to match that specified size. As shown in Fig. 1, while the modality-specific models were trained independently only on the datasets corresponding to their respective modality groups, our Uni-Omni and Syn-Omni models were jointly trained on the entire mixture encompassing all modalities simultaneously. Figure 7: Distribution of sample counts for each query-to-target modality composition in our training mixture. For each composition, ‘T’, ‘I’, ‘V’, and ‘A’ correspond to text, image, video, and audio modality, respectively. We also specify the total sample size per modality group. A.2 Evaluation Datasets To construct our evaluation suite across four distinct modalities: image, video, audio, and audiovisual, we leverage and extend existing evaluation frameworks. Specifically, for the image and video tasks, we utilize the evaluation suites from MMEB-v1 Jiang et al. (2025) and MMEB-v2 Meng et al. (2026), respectively. While MMEB-v2 natively includes text-to-video retrieval, it lacks the corresponding video-to-text direction. Therefore, we integrated video-to-text retrieval to ensure a bidirectional evaluation. Furthermore, we expand the benchmark’s scope by incorporating additional audio and audiovisual tasks. The detailed statistics and configurations for each modality are provided from Tab. 12 to Tab. 15, each specifying the task type, query-to-target composition, and data sizes. In these tables, the corpus size is defined differently based on the task type: it denotes the number of classes for classification (CLS) tasks, the number of choices for question answering (QA) tasks, and the actual database size for retrieval (RET) tasks. Our data collection includes a wide variety of common-domain settings for each modality, such as integrating both generic audio and music domains. English is the primary language across all multi-modal tasks. Appendix B Detailed Experimental Settings B.1 Training Details & Hyperparameters Model Backbone. We select the pretrained Qwen-2.5-Omni Xu et al. (2025a) as the backbone. During training, the visual and audio encoders are frozen, and LoRA parameters are inserted into the LLM layers. The LoRA target modules are q_proj, k_proj, v_proj, o_proj, down_proj, up_proj, and gate_proj, which are applied to each of the 36 LLM layers of the Qwen-2.5-Omni backbone. Input Processing. For the input processing, we adopt the native configuration of Qwen-2.5-Omni Xu et al. (2025a) unless specified otherwise. Specifically, images and videos are processed using the native resolution scaling strategy of Qwen-2.5-VL Bai et al. (2025). The minimum and maximum image resolutions are set to 50,176 pixels (64×28×2864× 28× 28) and 784,000 pixels (1000×28×281000× 28× 28), respectively. For video inputs, videos are uniformly sampled into 8 frames, and the per-frame resolution is bounded between a minimum of 100,352 pixels (128×28×28128× 28× 28) and a maximum of 156,800 pixels (200×28×28200× 28× 28). Audio signals are sampled at 16,000 Hz. The textual inputs are truncated to a maximum of 512 text tokens (without chat templates), and consistent with prior approaches Meng et al. (2026); Chen et al. (2026), left-padding is applied during batching. LoRA Configurations. For the modality-specific models and the standard LoRA baseline Uni-Omni mentioned in Fig. 1, the LoRA rank (r) is 16. For Syn-Omni with OME-LoRA, the rank of the shared adaptation path is 8, and each modality-expert rank is 4, making the total effective rank 24. Across all configurations, the LoRA scaling factor α is set to two times the rank. LoRA dropout is applied with probability 0.1. The detailed parameter counts for both the Uni-Omni baseline and Syn-Omni variants under these LoRA configurations are summarized in Tab. 5. Notably, the modality router introduces minimal parameter overhead, less than 8% of the total trainable parameters. Uni-Omni Syn-Omni 3B 7B 3B 7B Backbone Qwen-2.5-Omni LoRA rank (Shared) 16 8 LoRA rank (Expert) 0 4×44× 4 LoRA rank (Total) 16 24 LoRA params. 29.9 M 40.4 M 44.9 M 60.6 M Router params. 0.0 M 0.0 M 3.4 M 4.5 M Total trainable params. 29.9 M 40.4 M 48.3 M 65.1 M Table 5: Summary of LoRA configuration and trainable parameters for Uni-Omni and Syn-Omni model variants. Optimization. All models are trained for one epoch using the Hugging Face Trainer with FlashAttention-2 Dao (2024) in BFloat16 mixed-precision. We use the AdamW optimizer with β1=0.9 _1=0.9, β2=0.999 _2=0.999, and ϵ=10−8ε=10^-8. We employ a cosine learning rate schedule with a peak learning rate of 10−410^-4, a warmup ratio of 0.04, a weight decay of 0.01, and a maximum gradient norm of 1.0. The global batch size is set to 768. We employ GradCache Gao et al. (2021) to increase the global batch size, where the chunk size is set to 1 for the 3B models and 4 for the 7B models. We do not employ hard negative examples. The 3B models are trained using 8× NVIDIA A100 (40GB) GPUs, which takes approximately 5 days for Syn-Omni, and the 7B models are trained using 8× NVIDIA H200 (141GB) GPUs, taking approximately 40 hours for Syn-Omni. Training Objective. The baseline models and modality-specific models are optimized using the contrastive loss. Syn-Omni is optimized using the contrastive loss combined with the orthogonal penalty and the synergy margin loss as proposed in the main paper. The contrastive loss uses a temperature parameter τ of 0.02. For the orthogonal penalty, the loss weight is set to 0.2. The Progressive Synergy Routing (PSR) schedule uses warmup and transition ratios of twarm=0.1t_warm=0.1 and tblend=0.5t_blend=0.5. For the synergy margin loss, the margin γ is 0.2 and the loss weight is set to 0.2. B.2 Evaluation Pipeline For a fair evaluation, previous methods are evaluated under the unified setting using the official checkpoints. We extend the VLM2Vec-V2 codebase used for evaluation Meng et al. (2026), which enables the integrated evaluation of diverse modalities and task types while facilitating seamless modality expansion. While this evaluation framework originally supports image and video inputs, we extend it to encompass audio and audiovisual modalities also, thereby enabling an any-to-any query-to-target retrieval setting. The input processing for text, image, video, and audio modalities remains identical to those used during the training phase. For inference, standard LoRA adapter parameters are merged into the base pretrained weights, and such LoRA layers are not utilized any further. In contrast, for Syn-Omni, only the shared adaptation path is merged into the backbone weights and removed, whereas the expert paths are explicitly maintained to support dynamic routing during inference, where the soft routing weights generated by the router are utilized. Figure 8: Visualization of layer-wise soft routing weights under different input modalities. The baseline (BCE loss) exhibits a collapsed hard routing behavior with nearly binary activations, whereas ours (synergy margin loss) dynamically allocates routing weights even to absent-modality experts to encourage cross-modal synergy. Appendix C Detailed Results C.1 Routing Behavior across Layers Extending the analysis in Fig. 6, we investigate how different router loss formulations affect routing dynamics at the layer level. While the main paper tracks routing weights across training steps, this section visualizes the layer-wise soft routing activations of the converged model using audiovisual, video-only, and audio-only inputs from VALOR-32k Liu et al. (2025). Under this configuration, Fig. 8 compares the binary cross-entropy (BCE) loss against our synergy margin loss across the entire layer hierarchy. As shown in the figure, BCE loss forces the router into strict modality classification based on the input type. Except for the first layer, this results in routing activations approaching 1 for present modalities and 0 for absent ones, collapsing into a deterministic hard routing behavior despite using a soft routing formulation. In contrast, our synergy margin loss maintains dominant activations for the input modalities while preserving meaningful activations even for absent modalities across all layers. This collaborative activation behavior across the layer hierarchy successfully facilitates cross-modal synergy, aligning with our architectural objectives. C.2 Modality-Specific Models We present the extended evaluation of the modality-specific models introduced in Fig. 1, reporting their performance across all modalities in Tab. 6. Models trained exclusively on a single modality demonstrate strong performance within their respective modality. For instance, the audio-specific model performs well on audio tasks. A notable exception is observed in the audiovisual-specific model. This indicates that exposure to individual image, video, and audio data within a unified training scheme is helpful for audiovisual tasks. I V A AV All I-specific 68.0 43.0 35.4 33.3 44.9 V-specific 48.1 41.1 34.9 35.9 40.0 A-specific 40.3 36.5 43.5 30.9 37.8 AV-specific 38.0 32.4 34.6 37.1 35.5 Uni-Omni (3B) 66.8 40.4 43.1 40.4 47.7 Syn-Omni (3B) 68.1 40.5 45.6 43.4 49.4 Table 6: Comprehensive evaluation of modality-specific models on all modalities. The results demonstrate intrinsic cross-modal transferability and highlight the necessity of unified multimodal training for cross-modal synergy. Here, I, V, A, and AV denote image, video, audio, and audiovisual modalities, respectively. Model Latency (ms) Peak GPU Mem. (MB) Uni-Omni (3B) 300.3 ± 25 9612 Syn-Omni (3B) 354.6 ± 20 9676 Uni-Omni (7B) 377.1 ± 26 17672 Syn-Omni (7B) 443.5 ± 30 17765 Table 7: Runtime analysis on Uni-Omni and our Syn-Omni model variants. We report mean latency (ms) and peak GPU memory usage (MB), testing on VALOR-32k audiovisual input samples. Dataset Composition Size ImageNet 1K Deng et al. (2009) I → T 15,000 N24News Wang et al. (2022) TI → T 15,000 HatefulMemes Kiela et al. (2020) I → T 8,500 VOC2007 Everingham et al. (2015) I → T 7,844 SUN397 Xiao et al. (2010) I → T 15,000 OK-VQA Marino et al. (2019) TI → T 9,007 A-OKVQA Schwenk et al. (2022) TI → T 15,000 DocVQA Mathew et al. (2021) TI → T 15,000 InfographicsVQA Mathew et al. (2022) TI → T 15,000 ChartQA Masry et al. (2022) TI → T 15,000 Visual7W Zhu et al. (2016) TI → T 20,000 VisDial Das et al. (2017) T → I 20,000 CIRR Liu et al. (2021b) TI → I 20,000 VisualNews Liu et al. (2021a) T → I 20,000 I → T 20,000 MSCOCO-Retrieval Lin et al. (2014) T → I 20,000 I → T 20,000 MSCOCO-Grounding Lin et al. (2014) TI → I 20,000 NIGHTS Fu et al. (2023) I → I 15,941 WebQA Chang et al. (2022) T → IT 17,166 Sum 323,458 Table 8: An overview of image-text datasets in the omnimodal training dataset. Each dataset is sourced from the MMEB-v1 Jiang et al. (2025), where either the full original set or a randomly sampled subset was utilized. Dataset Composition Size LLaVAHound Retrieval Zhang et al. (2025) T → V 30,000 V → T 30,000 LLaVAHound QA Zhang et al. (2025) TV → T 30,000 PE-Video Bolya et al. (2025) V → T 40,000 T → V 40,000 MSVD Chen and Dolan (2011) V → T 1,200 T → V 1,200 MSR-VTT Xu et al. (2016) V → T 9,000 T → V 9,000 ActivityNet Krishna et al. (2017) V → T 10,008 T → V 10,008 DiDeMo Hendricks et al. (2017) V → T 8,395 T → V 8,395 FineCVR Wu et al. (2025b) TV → V 40,000 Sum 267,206 Table 9: An overview of video-centric datasets included in the omnimodal training dataset. Dataset Composition Size AudioCaps Kim et al. (2019) A → T 45,000 T → A 45,000 WavCaps Mei et al. (2024) A → T 45,000 T → A 45,000 AudioSet-SL Gemmeke et al. (2017) A → T 20,000 MusicCaps Agostinelli et al. (2023) A → T 2,580 T → A 2,580 MusicBench Melechovsky et al. (2024) A → T 20,000 T → A 20,000 MusicInstruct Deng et al. (2024) TA → T 20,000 Sum 265,160 Table 10: An overview of audio-centric datasets included in the omnimodal training dataset. Dataset Composition Size AudioSet Gemmeke et al. (2017) V → A 30,000 A → V 30,000 VAST Chen et al. (2023) VA → T 30,000 T → VA 30,000 InternVideo2 Wang et al. (2024) VA → T 30,000 T → VA 30,000 Shot2Story Han et al. (2025) VA → T 30,000 T → VA 30,000 AVInstruct Ye et al. (2024) TVA → T 19,861 Sum 259,861 Table 11: An overview of audiovisual-centric datasets included in the omnimodal training dataset. Task Dataset Composition Query Size Corpus Size I-CLS ImageNet-1K Deng et al. (2009) I2T 1000 1000 N24News Wang et al. (2022) TI2T 1000 24 HatefulMemes Kiela et al. (2020) I2T 1000 2 VOC2007 Everingham et al. (2015) I2T 1000 20 SUN397 Xiao et al. (2010) I2T 1000 397 Place365 Zhou et al. (2018a) I2T 1000 365 ImageNet-A Hendrycks et al. (2021b) I2T 1000 1000 ImageNet-R Hendrycks et al. (2021a) I2T 1000 200 ObjectNet Barbu et al. (2019) I2T 1000 113 Country211 Radford et al. (2021) I2T 1000 211 I-QA OK-VQA Marino et al. (2019) TI2T 1000 1000 A-OKVQA Schwenk et al. (2022) TI2T 1000 894 DocVQA Mathew et al. (2021) TI2T 1000 1000 InfographicsVQA Mathew et al. (2022) TI2T 1000 1000 ChartQA Masry et al. (2022) TI2T 1000 1000 Visual7W Zhu et al. (2016) TI2T 1000 1000 ScienceQA Lu et al. (2022) TI2T 1000 736 VizWiz Gurari et al. (2018) TI2T 1000 1000 GQA Hudson and Manning (2019) TI2T 1000 1000 TextVQA Singh et al. (2019) TI2T 1000 1000 IT RET MSCOCO_i2t Lin et al. (2014) I2T 1000 1000 VisualNews_i2t Liu et al. (2021a) I2T 1000 1000 VisDial Das et al. (2017) T2I 1000 1000 MSCOCO_t2i Lin et al. (2014) T2I 1000 1000 VisualNews_t2i Liu et al. (2021a) T2I 1000 1000 WebQA Chang et al. (2022) T2TI 1000 1000 EDIS Liu et al. (2023) T2TI 1000 1000 Wiki-S-NQ Ma et al. (2024a) T2I 1000 1000 CIRR Liu et al. (2021b) TI2I 1000 1000 NIGHTS Fu et al. (2023) I2I 1000 1000 OVEN Hu et al. (2023) TI2TI 1000 1000 FashionIQ Wu et al. (2021) TI2I 1000 1000 VG MSCOCO Lin et al. (2014) TI2I 1000 1000 RefCOCO Kazemzadeh et al. (2014) TI2I 1000 1000 RefCOCO-Matching Kazemzadeh et al. (2014) TI2TI 1000 1000 Visual7W-Pointing Zhu et al. (2016) TI2I 1000 1000 Table 12: Evaluation datasets for image modality, including task type and query-to-target composition. Task Dataset Composition Query Size Corpus Size V-CLS SmthSmthV2 Goyal et al. (2017) V2T 1000 174 HMDB51 Kuehne et al. (2011) V2T 1000 51 UCF101 Soomro et al. (2012) V2T 1000 101 K700 Carreira et al. (2019) V2T 1000 700 Breakfast Kuehne et al. (2014) V2T 433 10 T2V RET MSR-VTT Xu et al. (2016) T2V 1000 1000 MSVD Chen and Dolan (2011) T2V 670 670 DiDeMo Hendricks et al. (2017) T2V 1004 1004 YouCook2 Zhou et al. (2018b) T2V 3179 3179 VATEX Wang et al. (2019) T2V 4478 4478 V2T RET MSR-VTT Xu et al. (2016) V2T 1000 995 MSVD Chen and Dolan (2011) V2T 670 660 DiDeMo Hendricks et al. (2017) V2T 1004 1004 YouCook2 Zhou et al. (2018b) V2T 3179 3105 VATEX Wang et al. (2019) V2T 4478 4478 M-RET QVHighlight Lei et al. (2021) TV2V 1083 10 Charades-STA Gao et al. (2017) TV2V 727 10 MomentSeeker Yuan et al. (2025) TV2V 1602 10 V-VQA Video-MME Fu et al. (2025) TV2T 2700 4 NExTQA Xiao et al. (2021) TV2T 8564 5 EgoSchema Mangalam et al. (2023) TV2T 500 5 MVBench Li et al. (2024) TV2T 4000 5 ActivityNetQA Yu et al. (2019) TV2T 1000 2 Table 13: Evaluation datasets for video modality, including task type and query-to-target composition. Task Dataset Composition Query Size Corpus Size A-CLS ESC50 Piczak (2015) A2T 1237 50 GTZAN Tzanetakis and Cook (2002) A2T 290 10 NSynth Engel et al. (2017) A2T 4096 10 T2A RET AudioCaps Kim et al. (2019) T2A 4411 883 Clotho Drossos et al. (2020) T2A 5225 1045 MusicCaps Agostinelli et al. (2023) T2A 2772 2772 SoundDescs Koepke et al. (2023) T2A 1000 1000 A2T RET AudioCaps Kim et al. (2019) A2T 883 4201 Clotho Drossos et al. (2020) A2T 1045 5225 MusicCaps Agostinelli et al. (2023) A2T 2772 2772 A-QA MMAU Sakshi et al. (2025) TA2T 1000 4 MMAR Ma et al. (2025b) TA2T 995 4 Table 14: Evaluation datasets for audio modality, including task type and query-to-target composition. Task Dataset Composition Query Size Corpus Size A2V RET AVE Tian et al. (2018) A2V 402 402 VALOR32k Liu et al. (2025) A2V 3239 3239 V2A RET AVE Tian et al. (2018) V2A 402 402 VALOR32k Liu et al. (2025) V2A 3239 3239 T2VA RET AVHBench Kim et al. (2025) T2VA 1105 1105 VALOR32k Liu et al. (2025) T2VA 3239 3239 VA2T RET AVHBench Kim et al. (2025) VA2T 1105 1105 VALOR32k Liu et al. (2025) VA2T 3239 3239 AV-QA AVHBenchQA Kim et al. (2025) TVA2T 5293 2 DailyOmni Zhou et al. (2025) TVA2T 1196 4 Table 15: Evaluation datasets for audiovisual modality, including task type and query-to-target composition. Intriguingly, the image-specific model exhibits solid performance on video tasks, while the video-specific model retains reasonable results on both image and audiovisual tasks. These trends indicate that a unified training setting, where a model is exposed to diverse modalities, can induce cross-modal synergy. This serves as the primary motivation behind the design of our Orthogonal Modality-Expert LoRA (OME-LoRA) paired with the Progressive Synergy Routing (PSR) mechanism. While the Uni-Omni baseline exhibits performance comparable to or slightly lower than that of the respective modality-specific models in their native modalities, our Syn-Omni framework consistently delivers improved performance, demonstrating the effectiveness of cross-modal synergy. C.3 Runtime Analysis To evaluate inference-time efficiency, we benchmark Uni-Omni and Syn-Omni on the VALOR-32k audiovisual input using a single NVIDIA RTX A6000 GPU (batch size 1). As shown in Tab. 7, Syn-Omni introduces a slight increase in both latency and memory across 3B and 7B scales. Specifically, for the 3B variant, the mean latency increases from 300.3 ms to 354.6 ms, while the peak GPU memory changes from 9612 MB to 9676 MB. This overhead is due to architectural differences at inference time. In Uni-Omni, standard LoRA adapters are fully merged into the backbone weights, maintaining the original backbone’s inference path. In contrast, while Syn-Omni merges its shared adaptation path, it explicitly retains the four modality-specific expert paths and router parameters to support input-dependent routing. C.4 Per-dataset Evaluation We present the comprehensive benchmark-level results across all evaluated modalities and tasks. Individual dataset performances for image, video, audio, and audiovisual tasks are detailed in Tab. 16, Tab. 17, Tab. 18, and Tab. 19, respectively. Nemotron (3B) LCO-Emb (3B) e5-omni (3B) Uni-Omni (3B) Syn-Omni (3B) OmniEmbed (7B) Multivent (7B) LCO Emb (7B) e5-omni (7B) WAVE (7B) Uni-Omni (7B) Syn-Omni (7B) Average Overall (36) 44.1 58.1 64.8 66.8 68.1 44.2 51.8 61.6 72.5 42.7 71.3 71.7 I-CLS (10) 47.9 57.3 60.7 64.8 65.4 44.5 56.5 59.5 67.6 49.0 66.1 67.5 I-QA (10) 20.1 58.2 61.3 63.1 62.7 22.5 29.6 62.7 69.9 25.8 67.8 66.8 I-RET (12) 58.7 55.4 68.3 66.3 66.3 49.5 61.1 58.9 71.9 44.1 68.3 69.1 VG (4) 49.5 61.5 69.1 72.8 77.8 60.5 60.0 65.1 80.9 51.9 83.2 83.6 Image Classification (I-CLS) ImageNet-1K 60.6 65.5 72.9 74.2 75.0 56.4 68.6 72.0 79.6 47.8 76.5 77.1 N24News 36.9 50.0 73.1 73.4 77.6 42.5 42.3 45.6 77.9 43.3 76.5 80.2 HatefulMemes 48.1 56.8 57.3 67.5 72.3 52.5 51.2 66.4 69.0 51.2 71.2 75.7 VOC2007 53.0 66.4 71.8 87.5 88.2 75.1 68.3 54.4 83.3 77.2 87.9 89.3 SUN397 54.8 73.8 71.2 78.0 76.1 46.1 61.6 73.1 75.0 62.7 76.6 77.2 Place365 31.6 41.0 40.4 43.4 44.9 16.2 37.1 38.6 44.7 33.7 44.9 47.1 ImageNet-A 41.6 45.4 43.3 53.4 51.8 29.9 52.4 53.0 60.4 24.5 57.6 58.0 ImageNet-R 82.8 72.0 85.6 85.1 82.4 62.7 87.7 83.0 86.7 68.0 86.5 84.6 ObjectNet 46.2 73.2 65.5 55.5 55.7 37.8 62.9 75.4 68.2 60.9 52.3 53.3 Country211 23.6 29.3 26.0 29.8 29.8 25.7 32.5 33.5 30.8 20.7 30.8 32.3 Image QA (I-QA) OK-VQA 17.4 57.8 62.7 65.2 66.8 24.2 29.3 62.0 70.4 32.7 70.5 70.4 A-OKVQA 12.5 48.1 52.2 55.4 53.6 15.6 20.0 53.9 59.6 25.0 57.8 59.3 DocVQA 17.5 84.1 81.6 91.3 92.6 22.8 25.8 89.6 93.3 18.6 93.4 92.3 InfographicsVQA 8.5 58.6 49.9 68.2 63.5 11.7 12.4 66.6 68.9 17.0 74.7 72.9 ChartQA 13.3 40.1 47.7 58.7 58.5 12.4 14.0 46.9 65.7 12.5 64.5 63.1 Visual7W 7.4 48.5 55.2 49.5 48.1 4.8 11.9 47.9 63.7 18.7 53.3 50.8 ScienceQA 25.3 44.4 48.7 43.2 43.9 25.5 33.2 52.2 56.9 21.8 54.4 52.6 VizWiz 34.8 54.6 52.5 52.3 51.0 26.0 41.8 54.1 55.9 31.2 54.8 52.1 GQA 28.5 65.4 79.1 67.4 67.9 37.1 48.8 69.7 78.4 50.2 70.1 70.0 TextVQA 36.1 80.6 83.0 79.5 81.2 44.9 58.5 84.3 85.9 29.9 84.0 84.0 Image-Text Retrieval (I-RET) VisDial 52.8 50.5 77.8 71.0 73.9 50.6 56.2 54.7 80.8 19.7 74.8 76.1 CIRR 13.2 42.9 41.1 47.8 48.8 14.4 16.2 47.1 49.3 31.4 48.0 52.5 VisualNews_t2i 60.1 54.9 70.1 70.7 72.6 37.7 63.3 61.3 74.5 45.3 74.3 75.2 VisualNews_i2t 58.4 59.4 76.4 73.4 75.1 43.8 64.2 61.1 82.3 36.3 75.7 77.3 MSCOCO_t2i 59.9 59.2 74.3 65.6 67.1 51.2 63.6 65.2 76.2 63.5 70.7 70.7 MSCOCO_i2t 56.2 59.7 71.7 67.2 67.4 40.7 59.0 62.0 74.1 55.1 70.1 69.7 NIGHTS 64.6 56.7 66.3 68.5 67.3 59.2 63.7 61.3 64.8 58.2 67.2 67.6 WebQA 90.2 84.7 90.5 89.1 88.9 77.1 90.9 84.0 88.6 51.5 90.3 90.1 FashionIQ 8.6 14.9 16.5 19.9 19.1 8.3 11.4 19.3 19.4 6.5 18.8 22.9 Wiki-S-NQ 87.0 52.6 74.2 69.1 62.3 84.2 86.6 55.0 83.3 51.1 64.4 63.6 OVEN 72.7 75.2 75.8 74.0 74.1 65.2 74.6 77.2 79.1 57.2 77.3 76.9 EDIS 81.2 53.8 85.3 79.9 79.2 61.7 83.0 58.4 90.0 53.1 88.3 86.9 Visual Grounding (VG) MSCOCO 35.1 45.5 50.9 67.9 68.0 40.0 42.4 44.2 67.2 25.1 74.1 76.1 RefCOCO 55.1 68.8 76.6 75.3 84.8 67.8 65.9 73.5 83.7 49.7 87.3 91.0 RefCOCO-Matching 65.3 75.7 79.2 80.3 85.6 74.6 69.8 82.6 91.6 83.1 91.6 90.6 Visual7W-Pointing 42.7 55.8 69.6 67.8 72.9 59.6 62.0 60.2 81.1 49.7 79.6 76.6 Table 16: Individual performances on image modality tasks. The method order is identical to that in Tab. 1. Nemotron (3B) LCO-Emb (3B) e5-omni (3B) Uni-Omni (3B) Syn-Omni (3B) OmniEmbed (7B) Multivent (7B) LCO Emb (7B) e5-omni (7B) WAVE (7B) Uni-Omni (7B) Syn-Omni (7B) Average Overall (23) 36.5 43.7 40.6 40.4 40.5 35.0 40.2 45.5 44.3 39.8 40.2 41.4 V-CLS (5) 43.1 44.6 37.6 47.6 44.7 36.4 50.7 47.6 49.4 48.9 49.5 47.0 T2V RET (5) 34.8 34.2 38.9 27.9 29.3 33.7 35.9 36.9 36.1 30.0 25.0 28.5 V2T RET (5) 32.0 34.3 33.2 38.3 39.2 27.9 34.7 35.9 40.8 35.1 40.5 41.6 M-RET (3) 25.6 47.2 41.0 36.9 37.5 28.7 28.1 47.1 34.1 40.1 32.4 36.1 V-QA (5) 46.9 58.1 52.0 51.1 51.7 48.3 51.8 60.2 61.1 44.9 53.5 53.8 Video Classification (V-CLS) K700 38.5 42.6 42.4 46.3 43.9 31.5 53.3 43.2 50.1 51.4 52.0 48.0 SmthSmthV2 40.0 44.1 35.2 45.9 45.9 34.6 54.8 46.5 46.1 47.2 48.8 51.1 HMDB51 48.0 53.2 34.1 52.1 47.3 26.2 53.3 57.7 56.3 53.4 54.7 44.2 UCF101 62.7 64.2 51.9 71.5 63.1 62.2 68.2 67.8 69.5 72.0 66.6 68.3 Breakfast 26.3 18.7 24.5 22.2 23.1 27.3 24.0 22.6 24.9 20.3 25.4 23.6 Text → Video Retrieval (T2V RET) DiDeMo 36.0 33.1 41.6 27.4 27.4 27.6 34.7 34.7 35.7 25.6 23.1 27.3 MSR-VTT 37.8 37.4 40.0 39.6 43.1 40.2 40.5 41.2 42.1 34.6 37.6 43.9 MSVD 55.8 58.4 62.1 43.3 46.7 56.4 56.9 60.0 56.7 50.9 39.9 45.1 VATEX 28.8 29.1 34.0 25.5 24.3 28.1 30.1 32.3 30.8 29.4 23.0 21.3 YouCook2 15.5 13.0 16.8 3.7 5.0 16.0 17.4 16.1 15.3 9.7 1.7 4.7 Video → Text Retrieval (V2T RET) DiDeMo 32.0 32.4 31.9 42.5 42.0 27.7 35.7 33.6 43.0 32.7 44.3 46.4 MSR-VTT 34.3 38.8 37.6 42.4 45.8 33.2 37.3 37.9 44.2 39.0 47.0 48.1 MSVD 53.6 56.9 57.6 60.7 61.5 49.7 56.9 60.1 63.1 58.4 62.5 63.7 VATEX 25.6 28.3 28.4 29.9 31.4 21.5 28.5 30.0 37.2 30.8 31.6 32.0 YouCook2 14.4 15.4 10.6 15.8 15.3 7.4 15.1 17.6 16.4 14.7 17.3 18.0 Moment Retrieval (M-RET) QVHighlight 26.9 70.3 60.4 51.9 53.4 29.9 30.0 69.2 44.1 56.0 41.5 50.9 Charades-STA 11.6 25.2 19.5 18.0 16.6 14.3 13.1 24.3 16.9 26.4 16.8 14.9 MomentSeeker 38.3 46.1 43.1 40.8 42.6 41.9 41.2 47.8 41.4 38.0 39.0 42.7 Video QA (V-QA) MVBench 42.3 58.0 51.5 50.9 53.1 43.6 47.2 60.9 60.1 39.3 54.7 54.5 Video-MME 40.1 46.1 43.0 41.3 41.6 38.4 40.9 49.7 49.0 32.1 43.6 44.0 NExTQA 43.9 67.4 58.9 55.5 59.4 45.9 48.1 70.3 71.1 38.4 60.8 60.0 EgoSchema 46.6 43.8 45.6 34.8 30.6 46.4 51.4 42.8 51.4 48.0 34.2 35.4 ActivityNetQA 61.6 75.1 61.1 72.9 73.9 67.3 71.5 77.3 74.1 66.5 74.2 75.1 Table 17: Individual performances on video modality tasks. The method order is identical to that in Tab. 1. Nemotron (3B) LCO-Emb (3B) e5-omni (3B) Uni-Omni (3B) Syn-Omni (3B) OmniEmbed (7B) Multivent (7B) LCO Emb (7B) e5-omni (7B) WAVE (7B) Uni-Omni (7B) Syn-Omni (7B) Average Overall (12) 24.5 42.1 37.4 43.1 45.6 31.6 39.5 45.2 44.3 33.4 47.5 48.7 A-CLS (3) 40.7 67.5 51.5 59.5 62.0 36.0 52.9 71.1 62.8 56.3 67.4 67.2 T2A RET (4) 6.4 19.7 26.0 28.7 30.8 25.5 28.6 24.5 30.2 22.5 32.5 34.9 A2T RET (3) 6.8 17.3 14.2 30.0 32.4 17.2 21.0 18.7 20.7 17.7 32.5 34.0 A-QA (2) 44.0 63.8 57.9 54.4 57.0 47.8 55.4 66.4 63.7 37.0 57.6 58.5 Audio Classification (A-CLS) ESC50 73.6 77.8 78.2 86.8 87.0 51.5 77.8 78.8 81.9 82.8 88.0 88.5 GTZAN 28.6 74.8 44.1 52.1 57.9 34.5 51.4 82.8 61.4 57.9 66.9 63.4 NSynth 20.0 50.0 32.1 39.6 41.0 22.0 29.5 51.6 45.0 28.1 47.3 49.7 Text → Audio Retrieval (T2A RET) AudioCaps 9.4 23.3 33.9 39.9 43.2 39.0 33.5 26.9 38.6 32.1 41.2 44.9 Clotho 7.2 17.2 21.6 20.4 20.6 21.8 21.5 19.8 22.7 19.1 21.3 21.8 MusicCaps 3.5 8.0 10.5 19.8 23.4 11.8 13.2 11.9 14.8 8.7 20.6 22.6 SoundDescs 5.3 30.4 38.1 34.6 36.1 29.4 46.2 39.3 44.5 29.9 46.9 50.5 Audio → Text Retrieval (A2T RET) AudioCaps 10.6 26.0 26.0 49.4 52.8 33.7 34.8 30.0 36.8 26.6 54.2 57.6 Clotho 7.4 17.4 13.3 23.4 23.6 15.2 19.4 18.0 17.1 19.1 26.9 26.5 MusicCaps 2.3 8.3 3.3 17.2 20.7 2.7 8.9 8.2 8.0 7.3 16.4 18.0 Audio QA (A-QA) MMAU 50.7 70.5 63.9 59.9 62.1 50.0 58.7 74.9 70.5 36.1 64.8 61.8 MMAR 37.4 57.1 52.0 48.8 52.0 45.5 52.1 58.0 56.9 38.0 50.5 55.3 Table 18: Individual performances on audio modality tasks. The method order is identical to that in Tab. 1. Nemotron (3B) LCO-Emb (3B) e5-omni (3B) Uni-Omni (3B) Syn-Omni (3B) OmniEmbed (7B) Multivent (7B) LCO Emb (7B) e5-omni (7B) WAVE (7B) Uni-Omni (7B) Syn-Omni (7B) Average Overall (10) 28.5 35.5 31.6 40.4 43.4 29.0 34.4 35.5 42.3 38.1 43.9 44.6 A2V RET (2) 5.2 11.1 7.0 17.4 19.1 5.2 11.4 12.6 7.2 8.8 22.0 23.1 V2A RET (2) 7.3 10.5 9.2 14.3 17.9 13.5 16.1 13.8 14.8 19.9 20.1 18.9 T2VA RET (2) 46.9 51.1 64.4 63.3 64.2 57.2 48.9 51.7 65.7 52.6 64.7 64.9 VA2T RET (2) 41.4 46.6 30.5 62.5 63.0 27.3 49.1 46.3 62.4 56.9 64.3 63.9 AV-QA (2) 41.5 58.4 46.9 44.8 52.6 41.8 46.7 53.3 61.6 52.4 48.5 52.0 Audio → Video Retrieval (A2V RET) AVE 6.7 12.9 10.2 18.2 19.4 8.2 11.2 15.2 9.5 10.0 23.4 23.9 VALOR32k 3.7 9.2 3.9 16.6 18.8 2.2 11.6 9.9 5.0 7.7 20.6 22.2 Video → Audio Retrieval (V2A RET) AVE 10.9 12.7 10.9 12.9 19.9 16.4 17.4 14.7 15.9 22.1 20.6 15.9 VALOR32k 3.7 8.4 7.5 15.6 16.0 10.5 14.8 13.0 13.6 17.7 19.6 21.9 Text → Audiovisual Retrieval (T2VA RET) AVHBench 56.2 60.6 75.0 74.9 75.7 67.3 57.6 61.1 76.5 62.4 75.4 76.0 VALOR32k 37.6 41.6 53.8 51.6 52.7 47.1 40.1 42.4 54.9 42.7 54.0 53.9 Audiovisual → Text Retrieval (VA2T RET) AVHBench 50.2 53.2 23.6 73.5 74.8 36.6 58.6 54.6 71.1 67.1 74.8 75.4 VALOR32k 32.6 39.9 37.4 51.6 51.3 18.1 39.6 38.0 53.8 46.8 53.8 52.4 Audiovisual QA (AV-QA) AVHBenchQA 53.4 69.8 55.8 63.8 65.5 52.2 60.7 60.5 79.5 76.2 60.2 66.7 DailyOmni 29.5 47.0 38.1 25.8 39.7 31.4 32.6 46.2 43.8 28.6 36.8 37.2 Table 19: Individual performances on audiovisual modality tasks. The method order is identical to that in Tab. 1. Appendix D AI Tool Usage We utilized ChatGPT to improve the writing fluency and grammar of the manuscript. The development of the core ideas, technical methodologies, experimental designs, and analyses remains the original work of the authors. Appendix E Artifact We meticulously document the licenses, access URLs, and intended uses of the artifacts utilized in this study. All datasets, models, and software artifacts were used in accordance with their original licenses and intended research usage conditions when specified. For redistributed repositories or mirrors, we additionally verified the original artifact licenses when available. All derivatives or model checkpoints in our work are targeted for non-commercial research purposes. • MMEB v1 Training Set Jiang et al. (2025): Sourced from the official HuggingFace repository11 1 https://huggingface.co/datasets/TIGER-Lab/MMEB-train under the Apache-2.0 license. • LLaVAHound Zhang et al. (2025): Sourced from the official HuggingFace repository22 2 https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction under the Apache-2.0 license. • PE-Video Bolya et al. (2025): Sourced from the official HuggingFace repository33 3 https://huggingface.co/datasets/facebook/PE-Video under the C-BY-NC-4.0 license for non-commercial academic research. • MSVD Chen and Dolan (2011) & MSR-VTT Xu et al. (2016): Sourced from a public redistributed HuggingFace repository44 4 https://huggingface.co/datasets/friedrichor/MSVD55 5 https://huggingface.co/datasets/friedrichor/MSR-VTT while original licenses unspecified. • ActivityNet Captions Caba Heilbron et al. (2015); Krishna et al. (2017): Sourced from a redistributed HuggingFace repository66 6 https://huggingface.co/datasets/friedrichor/ActivityNet_Captions under the Apache License 2.0. • DiDeMo Hendricks et al. (2017): Sourced from a redistributed HuggingFace repository77 7 https://huggingface.co/datasets/friedrichor/DiDeMo under the BSD 2-Clause license. • FineCVR Wu et al. (2025b): Sourced from the official GitHub repository88 8 https://github.com/May2333/FDCA, while license information is unknown. • AudioCaps Kim et al. (2019): Sourced from a redistributed HuggingFace mirror99 9 https://huggingface.co/datasets/CLAPv2/audiocaps under the MIT license. • WavCaps Mei et al. (2024): Sourced from the official HuggingFace repository1010 10 https://huggingface.co/datasets/cvssp/WavCaps under the C-BY-4.0 license. • AudioSet Strong Gemmeke et al. (2017): Sourced from a redistributed HuggingFace mirror1111 11 https://huggingface.co/datasets/CLAPv2/audioset_strong under the C BY 4.0 specification. • MusicCaps Agostinelli et al. (2023): Sourced from a redistributed HuggingFace mirror1212 12 https://huggingface.co/datasets/CLAPv2/MusicCaps under the C-BY-SA-4.0 framework. • MusicBench Melechovsky et al. (2024): Sourced from the official HuggingFace repository1313 13 https://huggingface.co/datasets/amaai-lab/MusicBench under the C-BY-SA-3.0 license. • MusicInstruct Deng et al. (2024): Sourced from the official HuggingFace repository1414 14 https://huggingface.co/datasets/m-a-p/Music-Instruct under the C-BY-NC-4.0 license. • AudioSet Gemmeke et al. (2017): Sourced from the official page1515 15 https://research.google.com/audioset/ under the C BY 4.0 license. • VAST Chen et al. (2023): Sourced from the official GitHub repository1616 16 https://github.com/CASIA-IVA-Lab/VAST under the MIT license. • InternVideo2 Wang et al. (2024): Sourced from the official HuggingFace repository1717 17 https://huggingface.co/datasets/OpenGVLab/InternVideo2_Vid_Text under the C-BY-NC-SA-4.0 license. • Shot2Story Han et al. (2025): Sourced from the official HuggingFace repository1818 18 https://huggingface.co/ByteDance/shot2story under the C-BY-NC-SA-4.0 license, restricted to non-commercial academic usage. • AVInstruct Ye et al. (2024): Sourced from the official GitHub repository1919 19 https://github.com/rikeilong/Bay-CAT under the Apache-2.0 license, utilizing video sources derived from AVQA Yang et al. (2022). • MMEB v1 Evaluation Set Jiang et al. (2025): Sourced from the official HuggingFace repository2020 20 https://huggingface.co/datasets/TIGER-Lab/MMEB-eval under the Apache-2.0 license. • MMEB v2 Evaluation Set Meng et al. (2026): Sourced from the official HuggingFace repository2121 21 https://huggingface.co/datasets/TIGER-Lab/MMEB-V2 under the Apache-2.0 protocol. • ESC-50 Piczak (2015): Sourced from a redistributed HuggingFace mirror2222 22 https://huggingface.co/datasets/CLAPv2/esc50_no_overlap under the C-BY-NC license. • GTZAN Tzanetakis and Cook (2002): Sourced from a redistributed HuggingFace mirror2323 23 https://huggingface.co/datasets/m-a-p/GTZAN. • NSynth Engel et al. (2017): Sourced from a redistributed HuggingFace mirror2424 24 https://huggingface.co/datasets/TwinkStart/Nsynth under the C BY 4.0 license. • AudioCaps (for Eval) Kim et al. (2019): Sourced from a redistributed HuggingFace mirror2525 25 https://huggingface.co/datasets/OpenSound/AudioCaps under the MIT license. • Clotho Drossos et al. (2020): Sourced from the official archive2626 26 https://zenodo.org/records/4783391 under its custom academic non-commercial research license. • SoundDescs Koepke et al. (2023) & AVE Tian et al. (2018): Sourced from a redistributed HuggingFace repository2727 27 https://huggingface.co/datasets/VLM2Vec/MMEB-V3. SoundDescs follows the Apache License 2.0, while the license information for AVE was not explicitly provided in the original repository. • MMAU Sakshi et al. (2025): Sourced from a redistributed HuggingFace repository2828 28 https://huggingface.co/datasets/gamma-lab-umd/MMAU-test-mini under the C-BY-SA-4.0 standards. • MMAR Ma et al. (2025b): Sourced from the official HuggingFace repository2929 29 https://huggingface.co/datasets/BoJack/MMAR under the C-BY-NC-4.0 license. • VALOR 32k Liu et al. (2025): Sourced from the official GitHub repository3030 30 https://github.com/CASIA-IVA-Lab/VALOR under the MIT framework. • AVHBench Kim et al. (2025): Sourced from the official GitHub repository3131 31 https://github.com/kaist-ami/AVHBench. • DailyOmni Zhou et al. (2025): Sourced from a redistributed HuggingFace repository3232 32 https://huggingface.co/datasets/mteb/Daily-Omni under the GPL-3.0 license. • Qwen 2.5 Omni Xu et al. (2025a): Sourced from the official HuggingFace repository3333 33 https://huggingface.co/Qwen/Qwen2.5-Omni-7B under the Apache-2.0 license. • Omni-Embed-Nemotron Xu et al. (2025b): Sourced from the official HuggingFace repository3434 34 https://huggingface.co/nvidia/omni-embed-nemotron-3b under NVIDIA’s custom non-commercial scientific license (customized-nscl-v1) for non-commercial research and scientific evaluation. • LCO-Emb Xiao et al. (2025): Sourced from the official HuggingFace repository3535 35 https://huggingface.co/LCO-Embedding under the Apache-2.0 policy. • e5-omni Chen et al. (2026): Sourced from the official HuggingFace repository3636 36 https://huggingface.co/collections/Haon-Chen/e5-omni under the MIT license. • OmniEmbed-v0.1 & OmniEmbed-Multivent Ma et al. (2025a); Zhan et al. (2025): Sourced from the official HuggingFace repository3737 37 https://huggingface.co/Tevatron/OmniEmbed-v0.13838 38 https://huggingface.co/Tevatron/OmniEmbed-v0.1-multivent under the MIT license. • WAVE Tang et al. (2026): Sourced from the official GitHub repository3939 39 https://github.com/TCL606/WAVE under the Apache License 2.0. • GradCache Gao et al. (2021): Sourced from the official GitHub repository4040 40 https://github.com/luyug/GradCache under the Apache-2.0 license. • Flash-Attention-2 Dao (2024): Sourced from the official GitHub repository4141 41 https://github.com/dao-ailab/flash-attention under the BSD-3-Clause license. • qwen-omni-utils: Sourced from the official GitHub repository4242 42 https://github.com/qwenlm/qwen2.5-omni under the Apache-2.0 license. • Tevatron 2.0 Ma et al. (2025a): Sourced from the official GitHub repository4343 43 https://github.com/texttron/tevatron under the Apache-2.0 protocol. All datasets were obtained from public benchmarks and repositories, and we relied on the original providers’ release procedures for potentially identifiable or offensive content.