Paper deep dive

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs

Hoigi Seo, Byung Hyun Lee, Minjun Kim, Dohyun Mah, Jongho Lee, Se Young Chun

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 158

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-modal large language models (MLLMs) achieve strong modality understanding by pairing a large language model (LLM) with an encoder for a target modality such as vision, video, or audio. However, improving an MLLM's capability for a given modality typically requires additional training on large modality-specific datasets, incurring substantial data collection and compute costs. Model merging offers an alternative, but it is often infeasible for data-scarce, large per-sample size, or domain-specific modalities (\textit{e.g.}, audio and video), where same-modality model variants are rarely available. In this work, we characterize an intriguing asymmetric phenomenon: merging a well-aligned, data-rich source-modality MLLM into a data-scarce target-modality MLLM substantially improves the target on its own benchmarks. Our theoretical and empirical analyses show that this gain stems from enhanced alignment between modality-specific and textual tokens, induced by the stronger donor modality. Specifically, we derive a mutual-information lower bound that is monotonic in alignment-related quantities and strongly correlated with downstream MLLM performance. Building on this principle, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers textual alignment from a strong, well-aligned source (donor) modality to a weak target (recipient) modality, boosting target-modality performance without further fine-tuning. We further show that the alignment-enhancing objective admits a closed-form weight-space solution computed from only a small calibration set. DCAT outperforms existing model-merging methods, offering an efficient path toward cross-modal alignment transfer. Project page with code is available at \url{this https URL}

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

154,451 characters extracted from source content.

Expand or collapse full text

Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs Hoigi Seo 1∗ Byung Hyun Lee 1∗ Minjun Kim 1∗ Dohyun Mah 1 Jongho Lee 1,2 Se Young Chun 1,2,3† 1 Dept. of ECE 2 INMC & AIIS 3 IPAI, Seoul National University, Republic of Korea seohoiki3215, ldlqudgus756, minjoony, dohyun0505 jonghoyi, sychun@snu.ac.kr Abstract Multi-modal large language models (MLLMs) achieve strong modality understand- ing by pairing a large language model (LLM) with an encoder for a target modality such as vision, video, or audio. However, improving an MLLM’s capability for a given modality typically requires additional training on large modality-specific datasets, incurring substantial data collection and compute costs. Model merging offers an alternative, but it is often infeasible for data-scarce, large per-sample size, or domain-specific modalities (e.g., audio and video), where same-modality model variants are rarely available. In this work, we characterize an intriguing asymmetric phenomenon: merging a well-aligned, data-rich source-modality MLLM into a data-scarce target-modality MLLM substantially improves the target on its own benchmarks. Our theoretical and empirical analyses show that this gain stems from enhanced alignment between modality-specific and textual tokens, induced by the stronger donor modality. Specifically, we derive a mutual-information lower bound that is monotonic in alignment-related quantities and strongly correlated with downstream MLLM performance. Building on this principle, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers textual alignment from a strong, well-aligned source (donor) modality to a weak target (recipient) modality, boosting target-modality performance without further fine-tuning. We further show that the alignment-enhancing objective ad- mits a closed-form weight-space solution computed from only a small calibration set. DCAT outperforms existing model-merging methods, offering an efficient path toward cross-modal alignment transfer. Project page with code is available at https://seohoiki3215.github.io/DCAT_project_page 1 Introduction Large language models (LLMs) [3,41,54,67,70] achieve strong performance across textual tasks through scaling in data and model size. Most existing multi-modal LLMs (MLLMs) [12,42,81] have advanced visual reasoning, video understanding, or audio recognition by pairing a strong LLM backbone with a single modality encoder such as vision [52,68,88], video [35,39,46] or audio [53,59,62] encoders, and a projector [36,42] that aligns the modality with the textual space. However, enhancing the target-modality capability of an MLLM typically requires additional training on large modality-specific data, incurring substantial data collection and compute costs. Model merging [9,17,28,48,75,82] offers a promising alternative for improving MLLMs by com- bining same-architecture models with little data. For vision MLLMs, multiple same-backbone variants and abundant modality data are often available and can be merged to enhance visual capabilities. * equal contribution,† corresponding author. Preprint. arXiv:2610.04580v1 [cs.AI] 3 Oct 2026 However, such data abundance and same-modality variants are much scarcer for other domain-specific modalities, where training multiple variants is costly or infeasible. While existing studies have mostly focused on merging within a single modality, merging different-modality MLLMs remains largely unexplored. We argue that MLLMs trained for different modalities may still share the same base LLM, making cross-modality weight merging possible. Thus, we ask a more targeted question: can a source-modality MLLM (donor) transfer useful alignment to a target-modality MLLM (recipient), thereby improving the recipient’s target-modality performance without additional training? Audio MLLMVideo MLLM Weak performance Vision MLLM Strong performance Audio MLLMVideo MLLM Alignment Increased DCAT Figure 1: Conceptual diagram of DCAT. DCAT transfers alignment from a strong donor modality model to a rel- atively weak recipient modality model, thereby improving the recipient model’s modality-specific performance. In this paper, we identify and study an intriguing and re- curring phenomenon: merging a donor and a recipient via existing merging methods can yield substantial perfor- mance gains over the original recipient on its associated benchmark. We investigate this phenomenon along two complementary axes. On the theoretical side, we analyze it through the lens of mutual information (MI) [31,61]. Under a mild assumption, we prove that a lower bound on the MI between text and the other modality is mono- tonic in the product of two quantities: the Subspace Energy Overlap (SEO), which measures the extent to which each modality’s token energy lies within the text-aligned sub- space, and the Spectral Diversity (SD) [56], which mea- sures how uniformly that energy is distributed within this subspace. We refer to this product as the Alignment score. On the empirical side, across hundreds of model check- points spanning multiple architectures, we conduct a series of statistically tested analyses and find that the Alignment score is strongly and significantly correlated with MLLM performance, and that merged models exhibiting perfor- mance gains also show a statistically significant increase in the Alignment. Taken together, these theoretical and empir- ical results support our MI-lower-bound-based Alignment score, as a plausible explanatory mechanism for the observed benefits of model merging. Building on these analyses, we propose Directional Cross-modal Alignment Transfer (DCAT), a novel framework that transfers alignment from a high-alignment donor MLLM to a recipient MLLM using only a few hundred calibration samples, thereby improving the recipient’s target- modality performance. To our knowledge, we are the first to formulate and analyze this phenomenon; accordingly, the most natural baselines for DCAT are the existing model-merging methods through which the phenomenon is initially observed. As illustrated in Fig. 1, we instantiate this directional transfer by using vision MLLMs, which generally exhibit high alignment, as donors and audio or video MLLMs as recipients. Across a diverse suite of benchmarks, including MMAU [69], AIR- Bench [84], Video-MME [16], and MVBench [37], DCAT achieves 29.55% improvement for audio tasks and 24.92% for video tasks on average over the original recipient models. Our contributions can be summarized as follows: • We investigate an asymmetric phenomenon: model merging between MLLMs trained on different modalities improves target-modality performance. We introduce a theory-motivated alignment metric that explains this phenomenon, supported by strong empirical analysis. •Building on this analysis, we propose Directional Cross-modal Alignment Transfer (DCAT), a method that leverages a small amount of data to perform a closed-form weight update, thereby transferring alignment from a well-aligned modality to a relatively weaker one. •We conduct extensive experiments across multiple MLLMs on various benchmarks. DCAT consistently outperforms existing model merging methods and achieves average performance gains of 29.55% over the base model on audio tasks and 24.92% on video tasks. 2 Related Work Multi-modal large language models. Recent advances in LLMs have spurred extensive work on MLLMs, which attach modality-specific encoders and projectors to an LLM to generate text 2 Table 1: Quantitative comparison between modality-specific models and merged models. We report (Left) the MMAU [69] performance obtained by merging an audio-only model with an audio- vision merged model (A+V), and (Right) the Video-MME [16] performance obtained by merging a video-only model with a video-vision merged model (Vd+V). Results are reported for Task Arithmetic (TA), ISO-C (ISO), TSV, and OptMerge (Opt.). In all cases, merging is applied only to the LLM and the Alignment score was measured at the 21st layer of the Vicuna model. We set γ = 0.8. Music Sound Speech Overall Align γ Audio only 54.79 60.6631.2348.90 0.0091 A+V (TA)55.69 64.5639.3453.20 0.0103 A+V (ISO) 58.68 66.6734.8353.40 0.0950 A+V (TSV) 55.99 66.6735.4452.70 0.0100 A+V (Opt.) 59.58 64.2635.4453.10 0.0099 Short Medium Long Overall Align γ Video only39.8934.2230.89 35.00 0.0248 Vd+V (TA)44.6734.7831.89 37.11 0.0287 Vd+V (ISO) 43.6738.2233.44 38.44 0.0269 Vd+V (TSV) 46.4436.6734.00 39.04 0.0278 Vd+V (Opt.) 44.3337.0033.56 38.30 0.0267 from inputs such as vision [2,4,7,34,35,39,42,46,60,73,93] and audio [11,12,23,24,26,80]. Training these models is compute-intensive, often requiring optimization of the LLM together with newly added components. Moreover, modality-specific data, such as video, requires far more storage than text and is costly or difficult to collect, making modality-specific performance gains expensive. These challenges are especially severe for rare or high-volume modalities with limited supervision, motivating training-free and data-efficient alternatives for improving MLLM capabilities. Model merging.As neural networks scale, deploying separate task specialists becomes prohibitive, motivating model merging as a training-free way to consolidate independently fine-tuned models. TA [28] enables model editing via task vectors, defined as weight differences between fine-tuned and pretrained models. Subsequent methods mitigate interference among task-specific updates: TIES [82] trims redundant changes and resolves sign conflicts; ISO [48] separates task-specific subspaces; TSV [17] decomposes updates into singular directions for low-dimensional merging; and WUDI [9] identifies interfering vectors for robust merging. OptMerge [75] extends this paradigm across heteroge- neous tasks and modalities. However, existing studies have not systematically examined cross-modal merging from a directional transfer perspective, where a source-modality MLLM is used to improve a designated target-modality MLLM on its own modality-specific benchmarks. 3 Analysis 3.1 Model merging increases modality performance While task-wise model merging, which combines LLMs fine-tuned on different downstream tasks into one model, has been extensively studied, comparatively little attention has been paid to merging MLLMs trained for different input modalities. Tab. 1 reports MMAU [69] scores for vision–audio MLLM merges and Video-MME [16] scores for vision–video MLLM merges. All checkpoints are from OptMerge [75] which are Vicuna [10]-based. Despite modality heterogeneity, merging with a vision MLLM improves audio and video benchmark performance by up to 18.45% over the original modality-specific MLLM. This raises a natural question: what drives this striking performance gain? 3.2 On the lower bound of Mutual Information We interpret this phenomenon through the lens of the mutual information (MI) between text and the other modality since it provides a principled way to quantify how much modality-specific information is accessible from text representations. As a first step toward making this perspective concrete, we introduce a quantity that we call the Subspace Energy Overlap (SEO), defined as follows. Definition 1 (Subspace Energy Overlap). LetY text ∈ R N t ×d denote the text-token activation matrix, and letU k ∈ R k×d be the top-kright singular vectors of the centered text activations. For modality-token activations Y mod =y i N m i=1 , the Subspace Energy Overlap (SEO) is defined as SEO = ∥Y mod U ⊤ k ∥ 2 F ∥Y mod ∥ 2 F .(1) 3 Audio-Vision merged Vision benchmarkAudio benchmark Video-Vision merged Vision benchmarkVideo benchmark Figure 2: Empirical analysis of our alignment metric and performance. We analyze the correlation between the Alignment score and modality performance across a total ofN = 114model checkpoints. We use MMAU [69] for audio, Video-MME [16] for video, and TextVQA [63] for vision performance. Checkpoints are obtained from a range of merging methods [9,17,28,48,75,82], combined with several variations of the merged weights. The red dotted line refers to the performance of original model. For all experiments, we chosek = 32andγ = 0.8, and measure performance on 300 questions per benchmark across categories. Text embeddings are extracted from the questions. SEO quantifies the fraction of modality-token energy captured by the principal text subspace, with higher values indicating that modality representations are more concentrated within this subspace. Next, we introduce a second quantity, which we call the Spectral Diversity (SD), defined as follows. Definition 2 (Spectral Diversity). LetZ = Y mod U ⊤ k ∈ R N m ×k andΣ Z be the empirical covariance matrix of the projected representations. Denote the eigenvalues ofΣ Z byλ j k j=1 , and define p j = λ j P k ℓ=1 λ ℓ . The Spectral Diversity (SD) is defined as SD = 1 k exp   − k X j=1 p j logp j   .(2) Intuitively,k· SDcorresponds to the effective rank [56] in text-subspace for the projected modality representation. Thus, the effective-rank view makes SD directly interpretable in the text-subspace: higher SD indicates that modality information is broadly distributed within that subspace. Then, the following theorem lower-bounds the mutual information between modality tokens and text. Theorem 1 (Weighted Alignment-MI Lower Bound). Letz = yU ⊤ k whereU k contains the top- kright singular vectors of the centered text activation matrix. Suppose thatz ∼ N (μ Z , Σ Z ), z|x∼N (μ z (x), Σ z|x ), and p j ≥ δ k for some sufficiently large δ ∈ (0, 1]. Then, for any γ ∈ (0, 1): I(y;x) ≥ κ(γ)· log SEO γ · SD 1−γ + C δ , κ(γ) = k 2 min(γ, 1− γ) (3) with C δ = k 2 log(tr(Σ y )/k)− 1 2 log det( ̄ Σ z|x ) + k 2 logδ. The proof of Theorem 1 is in Sec. B.1 and we empirically verify thatδis not vanishingly small in Sec. B.5. Although the lower bound in this theorem is maximized atγ = 0.5, the most suitable value of γ may vary across models and modalities; we therefore consider a more flexible, looser bound. It is widely recognized that increasing the MI between modalities is essential for improving perfor- mance in multi-modal models [1,51,65]. This implies that maximizing the lower bound established in Theorem 1 provides a principled means of enhancing performance. Motivated by the bound in Eq. 3, we define the Alignment score as a weighted geometric mean of SEO and SD: Align γ := SEO γ · SD 1−γ .(4) That is, it measures how much modality-token energy enters the text principal subspace (SEO) and how broadly that energy is distributed across the effective directions of the text subspace (SD). In the following section, we leverage the proposedAlign γ to analyze the effect of various weight choices and present a statistically grounded empirical study examining whether the performance gains from model merging can be attributed to an increase in the MI lower bound for each modality. 4 3.3 Empirical analysis on Alignment score We measure theAlign γ for each model before and after merging, with the Align. column of Tab. 1. The alignment increases significantly under model merging (p≪ 10 −10 , Wilcoxon signed-rank test), suggesting that merging may improve performance by raising the MI lower bound. To test whether this relationship is global, we evaluate 114 Vicuna-based MLLM checkpoints generated by multiple merging methods, low-rank approximations of weight differences, and module- restricted merging (e.g., attention or FFN). For each checkpoint, we plot modality-wise performance againstAlign γ at the 21st layer (Fig. 2), where Pearson (r) and Spearman (ρ) coefficients show significant (p < 10 −3 ) strong positive correlations across all modalities. Audio and video models improve clearly, while the vision model shows only marginal improvement or degradation; thus, we refer to relatively degraded models as donor models and improved models as recipient models. A+V àVVd+VàVdA+V àAVd+VàVd Layer-wise Pearson correlation Figure 3: Layer-wise Pearson correlation be- tween Alignment score and performance across MLLM combinations and modalities. We mea- sure alignment in a layer-wise manner for each modality across multiple modality pairs. A con- sistently strong positive correlation between the Alignment score and performance emerges from the middle layers onward. We selected the 21st layer, the mid-to-late layer, to see this trend. In Fig. 3, we plot the Pear- son coefficient betweenAlign γ and performance at each layer. Beyond the 19th layer, we con- sistently observe a high correlation across all modality merge pairs and performance. This finding is consistent with prior works [22,29, 60,64,71,79,85,86] showing that MLLMs ex- hibit high, meaningful attention from the middle layers onward, which has been interpreted as the point where alignment between text and the corresponding modality begins. In other words, the phenomenon captured here parallels obser- vations made in earlier studies using different metrics, suggesting that ourAlign γ metric not only functions as a lower bound on MI but also quantifies the alignment between textual and non-textual modalities. To ensure robustness, we adopt the 21st layer in our analysis, as it exhibits the lowest standard deviation. Experimental de- tails and broader analysis of alignment across different models and benchmarks are provided in the Appendix Sec. D and Sec. E. 4 Method Building on these analyses, we propose Directional Cross-modal Alignment Transfer (DCAT), which transfers donor alignment to recipient models via a closed-form solution using minimal calibration inputs. DCAT exploits cross-modal merging benefits by targeting the alignment-related terms in Eq. (3). We further incorporate C δ to preserve the overall modality activation scale. Notation. Specifically, letθ 0 be the pretrained base weight (e.g., base LLM weight) for a given layer, and letτ r = θ r − θ 0 andτ d = θ d − θ 0 denote the recipient and donor task vectors. We denote recipient-modality calibration inputs and text calibration inputs asX mod andX text , respectively. For a candidate task vectorτ, the corresponding modality-token and text-token activations are given by X mod (θ 0 + τ ) and X text (θ 0 + τ ). Then recipient activations are computed as: Y text = X text (θ 0 + τ r ), Y mod = X mod (θ 0 + τ r ).(5) We also define the orthogonal projector ofU k and its orthogonal complement projector asP k (= U ⊤ k U k )andP ⊥ k (= I − P k ), respectively. Here,U k represents the principal text subspace, soP k and P ⊥ k decompose an activation into the text-subspace component and its residual component. Cross-modal alignment improvement.Motivated by the former analysis, DCAT first introduces a donor-guided cross-modal alignment term on text inputs: L cross (τ ) = β·∥X text (τ − τ r )∥ 2 F + (1− β)·∥X text (τ − τ d )∥ 2 F .(6) 5 The recipient anchor preserves the recipient model’s original behavior and prevents the update from overwriting modality-specific capabilities. The donor anchor is complementary: from the perspective of linear mode connectivity [15,77], donor weights on text inputs define directions along which DCAT can improve modality-text alignment while remaining in a text-compatible region of weight space. Thus, the donor-side term does not merely pull the solution toward the donor model; it transfers donor-induced cross-modal alignment directions through text space, allowing the recipient model to benefit from the donor while suppressing unexpected drift of C δ . Activation-space target construction. Beyond donor-guided cross-modal alignment transfer, DCAT further improves the recipient model’s self-alignment by constructing an activation-space target for the recipient modality. SinceU k defines the principal text subspace, we project the recipient representation asX mod (θ 0 + τ r )U T k . We compute its SVD asA diag(σ 1 ,...,σ k )B ⊤ and define ρ = ∥X mod (θ 0 + τ r )∥ F / √ k, which preserves the total output energy across allktext-subspace directions. DCAT then reshapes the original spectrum into a more balanced projected target T α : T α = c α · A diag σ α j ρ 1−α k j=1 B ⊤ U k , c α = ∥X mod (θ 0 + τ r )∥ F P k j=1 σ 2α j ρ 2(1−α) 1/2 .(7) Note that T α reflects favorable properties for increasing the lower bound in Theorem 1 such that: T α P ⊥ k = 0, ∥T α ∥ F =∥X mod (θ 0 + τ r )∥ F .(8) Derivation for Eq. 8 is provided in Appendix Sec. B.2. The first property makesT α contained in the principal text subspace, while the second prevents the target from changing the scale of the recipient activations. Thus,T α improves self-alignment by pulling the recipient modality toward its own text subspace while preserving the recipient’s activation energy, which prevents increasing text-subspace overlap by merely shrinking the modality activation, undesirable for the trace component in C δ . The role ofαis to control the trade-off between preserving the recipient representation within the text subspace and increasing spectral diversity, while the target remains norm-preserving for allα. Specifically, keeping the original singular valuesσ j would preserve the recipient geometry but may leave the projected representation spectrally concentrated. On the other hand, replacing them with a fully isotropic spectrum would maximize diversity but may distort the recipient representation. Unified self-alignment matching.Given the targetT α , DCAT improves self-alignment by matching the recipient-modality activations to this target: L self (τ ) =∥X mod (θ 0 + τ )P ⊥ k ∥ 2 F | z Encouraging SEO +∥X mod (θ 0 + τ )P k − T α ∥ 2 F | z Encouraging SD =∥X mod (θ 0 + τ )− T α ∥ 2 F ,(9) where the equality follows from the orthogonality ofP k (derived in Appendix Sec. B.3). Note that the first term reduces activation energy outside the principal text subspace, while the second shapes the projected representation within that subspace. Therefore, once the targetT α is constructed inside the text subspace with a balanced spectrum,L self (τ ) simultaneously encourages SEO and SD. Closed-form DCAT update. Combining the cross-modal alignment improvement termL cross (τ ) and the self-alignment matching termL self (τ ), the final objective is: min τ L self (τ ) + η·L cross (τ ).(10) That is,L cross (τ )transfers donor-induced cross-modal alignment directions while preserving the recipient model’s original text behavior, andL self (τ )further reshapes the recipient-modality activa- tions toward the norm-preserving targetT α . The donor enters through the cross-modal alignment improvement term, which biases the closed-form solution toward donor-compatible text-aligned directions without overwriting the recipient modality representation. Since Eq. 10 is quadratic inτ, DCAT admits a closed-form layer-wise solution without gradient-based optimization (see Appendix Sec. B.4). To avoid mismatch from applying all layer-wise solutions at once, sinceT α is constructed using pre-update recipient activations, DCAT progressively merges each layer’s solution and recomputes calibration activations for the next layer using the partially updated model. This reduces accumulated activation drift while preserving closed-form efficiency. 6 Schedulingα. The degree of alignment can vary across layers, as well as across the linear pro- jections within each layer. Accordingly,αshould adaptively balance SEO and SD by adjusting the spectrum of the targetT α inside the principal text subspace. To this end, we apply a simple yet effective adaptive schedule for α as follows: α = Sigmoid log SEO r log SD r + ε ,(11) whereε > 0is for stability. This schedule makes the target more conservative when the recipient has weak SEO, and more isotropic when the projected representation is spectrally concentrated. Ac- cordingly, largerαpreserves more of the recipient’s projected spectrum, while smallerαencourages SD. To support this formulation, we analyze the effective rank of textual and modality-specific token representations across models and modalities; details are provided in Appendix Sec. E. 5 Experiment 5.1 Experimental settings Baselines. Although first observed in model merging, DCAT has a distinct goal: transferring stronger donor alignment to a recipient rather than equipping one MLLM with multiple tasks or modalities. Because both settings combine models at the weight level, we compare DCAT with repre- sentative merging baselines: Task Arithmetic [28] (TA), TIES-merging [82] (TIES), ISO-C [48] (ISO), Task Singular Vector [17] (TSV), WUDI [9], and OptMerge [75]. Experiments cover Audio+Vision and Video+Vision settings on two widely used backbones, Vicuna [10] and LLaMA 3.1 [19]. For Vicuna, we use the weights released by OptMerge. For LLaMA 3.1, we use in-the-wild models that share the same architecture but are independently trained with different recipes, demonstrating DCAT’s robustness and practical utility. Full baseline details are in the Appendix Sec. D.2. Benchmarks. We extensively compare DCAT’s alignment transfer performance across audio and video benchmarks. For audio, we evaluate on MMAU [69] (1K samples), AIR-Bench Foun- dation [84] (25K samples), MuCho-music [74] (1.2K samples), Clotho-AQA [40] (1.4K samples), and VocalSound [18] (3.6K samples). For video, we evaluate on Video-MME [16] (2.7K samples), TempCompass [44] (5.5K samples), MVBench [37] (4K samples), and MLVU [91] (2.2K samples). Subcategory-level results are provided in the Appendix Sec. F.2. Implementation details. DCAT progressively merges models layer by layer from the first to the last layer. At each layer, our closed-form update is applied to all linear projections, aligning internal representations across computational modules. For calibration, we use AVQA [83], which supports both audio and video: audio tracks calibrate audio MLLMs, and video tracks calibrate video MLLMs, using 256 samples each. Remaining hyperparameters, including the text subspace dimensionk, cross-modal alignment weight η, and anchor parameter β, are detailed in Appendix C. 5.2 Quantitative comparison To evaluate how effectively DCAT boosts modality performance, we compare it against various model merging methods. The Audio+Vision results are shown in the top of Tab. 2, and the Video+Vision results in the bottom. For the LLaMA-based model, the default merging coefficientλ = 1.0used by most methods consistently degrades performance; for a fair comparison, we therefore additionally report each method’s performance atλ = 0.2, which yields its best results. On audio, DCAT improves over the baseline by 7.62% on comprehensive benchmarks such as MMAU, and by up to 100.92% on MuCho-Music. On video, DCAT delivers gains of 16.37% on Video-MME and 23.83% on MVBench. Together with our analysis, these results demonstrate DCAT’s effectiveness. 5.3 Does DCAT increase SEO and SD? Yes. We theoretically showed that increasing SEO and SD can raise a lower bound on the MI, and on this basis we proposed DCAT. To verify that DCAT indeed increases SEO and SD in practice, we measured both quantities at every layer of each Vicuna-based model under the Audio+Vision and Video+Vision settings. In addition, to check whether the first term ofC δ in Theorem 1, k 2 log(tr(Σ y /k)), is preserved, we measured the token norm ratio∥Y aft. mod ∥ F /∥Y bef. mod ∥ F between before and after applying 7 Table 2: Quantitative comparison of DCAT. We evaluate DCAT against various model merging methods across benchmarks, using a vision model as donor and an audio/video model as recipient. (Top) In Audio+Vision, DCAT transfers vision-model alignment to the audio model, achieving gains up to 100.92% (MuCho-Music) over the original. (Bottom) Video+Vision shows a similar trend, with improvements up to 53.88% (MLVU). ( ∗ Since our LLaMA 3.1-based audio model does not tune the language backbone, several merging methods reduce to TA, yielding identical outputs.) MMAUAIR-Bench MuChoClothoVocalAverage MusicSoundSpeechOverallMusicSoundSpeechOverall Vicuna-based Original54.7960.6631.2348.9038.4446.7644.6243.1726.9661.2348.9845.85 TA [28]55.6964.5639.3453.2045.6457.4847.0349.0950.0481.0044.8155.63 TIES [82]54.4962.4636.0451.0044.8655.9247.5148.6838.4274.9750.9352.80 ISO [48]58.6866.6734.8353.4045.8260.7746.7249.7952.8278.5744.3155.78 TSV [17]55.9966.6735.4452.7047.0358.5148.6950.5150.8081.2147.5456.55 WUDI [9]55.9965.1739.9453.7045.8458.1646.8549.2349.4580.7246.9856.02 OptMerge [75]59.5864.2635.4453.1047.0258.2448.8350.5050.2180.2449.0756.62 DCAT (Ours)60.2269.0641.7456.2048.4659.5149.3652.4454.1782.9651.2559.40 LLaMA 3.1-based Original52.4057.6656.4655.5033.5336.3553.0042.8652.5755.4831.0247.49 λ = 1 . 0 TA ∗ 48.5054.6554.6552.6035.5432.9746.2339.6849.6257.0727.5745.31 TIES52.1056.1659.7656.0037.4336.1149.3942.4354.9357.5630.4948.28 ISO55.3957.9659.4657.6036.1534.6947.0140.6355.9458.6729.3548.44 TSV ∗ 48.5054.6554.6552.6035.5432.9746.2339.6849.6257.0727.5745.31 WUDI49.1055.5656.1653.6035.8732.5345.9739.5649.9657.2826.9045.46 OptMerge ∗ 48.5054.6554.6552.6035.5432.9746.2339.6849.6257.0727.5745.31 λ = 0 . 2 TA ∗ 55.9958.5657.0657.2035.7335.5751.7542.8154.5157.5631.2248.66 TIES 57.4958.2657.0657.6035.2636.0153.1043.3754.5956.6630.5848.56 ISO52.9958.5656.7656.1034.5536.4353.6743.5053.2457.4930.1948.10 TSV ∗ 55.9958.5657.0657.2035.7335.5751.7542.8154.5157.5631.2248.66 WUDI54.4957.0655.5655.7035.4935.4552.1442.8852.7457.4929.9147.74 OptMerge ∗ 55.9958.5657.0657.2035.7335.5751.7542.8154.5157.5631.2248.66 DCAT (Ours)58.5659.1659.7659.1637.9637.2854.2544.9358.3061.5631.9451.18 Video-MMETempCompass MVBenchMLVUAverage ShortMediumLongOverallActionAttr.Dir.OrderSpeedOverall Vicuna-based Original39.8934.2230.8935.0078.4854.0048.5347.5645.8555.4638.4832.5940.38 TA [28]44.6734.7831.8937.1178.1653.1346.8949.0946.9955.3843.6844.8445.25 TIES [82]31.4427.7825.6728.3076.5350.7846.4547.6648.8254.6232.7833.1537.21 ISO [48]43.6738.2233.4438.4479.3853.8146.4546.4447.7755.3843.0045.3145.53 TSV [17]46.4436.6734.0039.0482.7255.7647.7549.2947.6957.2444.5845.1846.51 WUDI [9]44.5634.2230.8936.5676.9453.7146.1148.6846.7254.9343.3044.3844.79 OptMerge [75]44.3337.0033.5638.3080.6855.5747.4047.2648.3856.4744.8343.2945.72 DCAT (Ours)47.9640.4636.1741.5383.8058.4250.9850.8353.5259.7147.6546.3448.81 LLaMA 3.1-based Original47.3335.4431.7838.1979.8744.5345.3342.2840.4451.2841.0031.7940.57 λ = 1 . 0 TA48.2236.3332.1138.8977.8349.5143.1745.6346.2953.1143.8845.2245.28 TIES 50.7838.6735.0041.4874.7444.6341.7037.4034.0647.2246.8847.1245.68 ISO50.6738.6733.3340.8944.0132.0326.5619.4120.1728.8546.0345.2840.26 TSV51.7839.1134.6741.8583.2952.8346.6347.2650.3156.7747.0048.0848.43 WUDI50.7836.6733.0040.1519.4811.9110.733.865.2410.5345.3547.5135.89 OptMerge52.4439.8936.2242.855.794.004.331.931.573.5947.2349.2435.73 λ = 0 . 2 TA52.5638.1138.1142.9363.4938.0936.3335.8739.0443.1543.1840.9442.55 TIES46.5637.3335.7839.8965.5337.7937.2036.1844.5444.9245.4338.5842.21 ISO45.6736.4436.4439.5283.2152.3447.2346.8552.5857.1944.8044.4846.50 TSV49.2237.6737.4441.4479.0547.5643.4343.9049.1753.3645.6540.4245.22 WUDI51.5637.0038.3342.3085.7453.5247.7548.7852.6658.4444.5543.8147.28 OptMerge47.7837.7836.2240.5983.9451.8646.7147.0553.1057.3046.4841.8346.55 DCAT (Ours)53.6740.3339.3344.4486.3755.6450.4750.1956.0259.8851.1850.1551.41 DCAT, while the second termdet( ̄ Σ z|x )is intractable to estimate directly. The results, presented in Fig. 4, show that DCAT yields a significant increase in both SEO and SD while keeping the norm ratio essentially unchanged, suggesting that DCAT successfully raises the lower bound on the MI. 5.4 Ablation study To assess the contribution of each component of DCAT, we conduct an ablation study summarized in Tab. 3. Experiments use Vicuna-based models: the audio+vision setting employs the full MMAU benchmark, while the video+vision setting uses 900 randomly sampled examples from Video-MME. A2–A5 ablate the components of DCAT’s objective and the adaptiveαschedule and the results show that each component contributes meaningfully to performance and confirm both the validity of design choices. Further ablation study including hyperparameter sweep, calibration dataset size and projection dimension k is provided in Appendix Sec. F.8. 8 Layer-wise SEO ComparisonLayer-wise SD Comparison Layer-wise SEO Comparison Layer-wise SD Comparison Audio + Vision Video + Vision Layer-wise Relative Norm comparison Layer-wise Relative Norm comparison Figure 4: SEO, SD and token norm ratio before and after applying DCAT. We compare SEO, SD, and the norm ratio of modality tokens before and after applying DCAT. As depicted in first and second column, SEO and SD increase to a statistically significant degree (p < 0.05), while the relative norm remains close to 1.0, confirming that our DCAT formulation behaves as intended. Table 3: Ablation study on DCAT. We ablate the individual components of DCAT. (A2–A5) isolate the contribution of each term in the DCAT objective by either dropping a term or fixing theα scheduler, while (A6–A9) examine the sensitivity of DCAT to its key hyperparameters. IDStudy subject MMAUVideo-MME MusicSoundSpeechOverallShortMediumLongOverall A0Original54.7960.6631.2348.9038.3333.6730.6734.22 A1Base (DCAT)60.2269.0641.7456.2046.3339.6734.3340.11 A2w/o SEO encouraging term inL self (τ )57.4966.9734.8353.1045.3334.0030.3336.56 A3w/o SD encouraging term inL self (τ )55.0964.2634.5351.3035.6725.0025.0028.56 A4w/oL cross (τ )25.4527.6324.6225.903.334.672.673.56 A5Fixed α scheduling55.9961.8636.3451.4036.6731.3327.6731.89 6 Discussion As shown in Fig. 2, we consistently observe that vision models suffer a performance drop when merging is applied, and this trend holds across multiple models and architectures. Accordingly, most of our experiments employ a vision model as the donor and MLLMs of other modalities as recipients for alignment transfer. We offer the following conjecture as to why the donor has consistently been a vision model: because visual data is easier to obtain and far more abundant than audio or video data, existing vision MLLMs are likely to be more thoroughly aligned than their counterparts in other modalities. Under this hypothesis, DCAT is expected to leverage models trained on common modalities to enhance the performance of MLLMs built for relatively rare and data-scarce modalities. 7 Conclusion In this paper, we report the phenomenon that inter-modality model merging can improve performance, and we interpret this effect from a Mutual Information (MI) perspective by deriving a lower bound on the MI and proposing a monotonic Alignment metric. An extensive empirical and statistical analysis demonstrates that this Alignment exhibits a strong correlation with downstream performance. Building on these theoretical and empirical findings, we propose Directional Cross-Modal Alignment Transfer (DCAT), which transfers alignment from a donor modality model to a recipient modality model in closed form. Across multiple audio and video MLLMs, we validate DCAT on a wide range of benchmarks, achieving average relative improvements of 29.55% on audio and 24.92% on video over the original models, thereby substantiating the soundness and the effectiveness of DCAT. 9 Acknowledgment This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government(MSIT) [No.RS-2021-I211343, Artificial Intelligence Graduate School Program (Seoul National University), No.RS-2025-02314125, Effective Human-Machine Teaming With Multimodal Hazy Oracle Models], the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2025-02263628), Electron- ics and Telecommunications Research Institute (ETRI) grant funded by the Korean government [26CR1100, Development of molecular endoscopy for real-time precision tumor analysis], and Korea Health Industry Development Institute(RS-2024-00439677). Also, the authors acknowledged the financial support from the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University. References [1] Antonio Almudévar, José Miguel Hernández-Lobato, Sameer Khurana, Ricard Marxer, and Alfonso Ortega. Aligning multimodal representations through an information bottleneck. ICML, 2025. [2]Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. [3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [4] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [5]Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pages 7432–7439, 2020. [6]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165, 2020. [7]Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. [8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056–27087, 2024. [9]Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end it: Guiding data-free model merging via task vectors. ICML, 2025. [10]Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023. [11]Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024. [12] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. [13]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, pages 2924–2936, 2019. [14]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. 10 [15]Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020. [16]Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, pages 24108–24118, 2025. [17]Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In CVPR, pages 18695–18705, 2025. [18]Yuan Gong, Jin Yu, and James Glass. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155, 2022. [19]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [20]Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018. [21]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [22]Ming-Hao Hsu, Xueyao Zhang, Xiaohai Tian, Jun Zhang, and Zhizheng Wu. Anatomy of the modality gap: Dissecting the internal states of end-to-end speech llms. arXiv preprint arXiv:2603.01502, 2026. [23]Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, et al. Wavllm: Towards robust and adaptive speech large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4552–4572, 2024. [24]Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946, 2025. [25] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019. [26]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [27]Dang Huu-Tien, Takeshi Suzuki, Le-Minh Nguyen, et al. Regmean++: Enhancing effectiveness and generalization of regression mean for model merging. TMLR, 2025. [28]Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR, 2023. [29]Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In CVPR, pages 25004–25014, 2025. [30]Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. ICLR, 2023. [31] J Kreer. A question of terminology. IRE Transactions on Information Theory, 3(3):208–208, 1957. [32]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, pages 785–794, 2017. [33]Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, March 2024. Bo Li, Peiyuan Zhang, Kaichen Zhang, and Fanyi Pu contributed equally. [34] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. TMLR, 2025. 11 [35]Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. [36]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. [37]Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, pages 22195–22206, 2024. [38]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pages 292–305, 2023. [39] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In EMNLP, pages 5971–5984, 2024. [40]Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140–1144. IEEE, 2022. [41] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [42]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36:34892–34916, 2023. [43]Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233. Springer, 2024. [44]Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? In ACL, pages 8731–8772, 2024. [45]Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328, 2025. [46]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACL, pages 12585–12602, 2024. [47]Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS, 36:46212–46244, 2023. [48]Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D Bagdanov, and Joost Van De Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. ICML, 2025. [49] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, pages 2381–2391, 2018. [50]Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. [51] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. [53]Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023. 12 [54]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [55]Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95, 2011. [56]Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE, 2007. [57]Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [58]Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In EMNLP-IJCNLP, pages 4463–4473, 2019. [59] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862, 2019. [60]Hoigi Seo, Dong Un Kang, Hyunjin Cho, Joohoon Lee, and Se Young Chun. On epistemic uncertainty of visual tokens for object hallucinations in large vision-language models. NeurIPS, 2025. [61] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. [62]Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, et al. Sam audio: Segment anything in audio. arXiv preprint arXiv:2512.18099, 2025. [63]Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. [64]Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. ICML, 2025. [65]Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai. Towards all-in-one pre-training via maximizing multi-modal mutual information. In CVPR, pages 15888–15899, 2023. [66]Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACL, pages 4149–4158, 2019. [67]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [68]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. [69]Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. ICLR, 2025. [70]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017. [71]Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda. How visual representa- tions map to language feature space in multimodal llms. arXiv preprint arXiv:2506.11976, 2025. [72]Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. [73]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 13 [74]Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, György Fazekas, and Dmitry Bogdanov. Muchomusic: Evaluating music understanding in multimodal audio-language models. In ISMIR, 2024. [75]Yongxian Wei, Runxi Cheng, Weike Jin, Enneng Yang, Li Shen, Lu Hou, Sinan Du, Chun Yuan, Xiaochun Cao, and Dacheng Tao. Optmerge: Unifying multimodal llm capabilities and modalities via model merging. arXiv preprint arXiv:2505.19892, 2025. [76]Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, 2017. [77]Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo.π-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. In ICML, 2023. [78]Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurIPS, 37:28828–28857, 2024. [79]Hongxuan Wu, Yukun Zhang, and Xueqing Zhou. How vision becomes language: A layer-wise information- theoretic analysis of multimodal reasoning. arXiv preprint arXiv:2602.15580, 2026. [80]Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025. [81]Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empower- ing large language models to understand point clouds. In ECCV, pages 131–147. Springer, 2024. [82]Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. NeurIPS, 36:7093–7115, 2023. [83]Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In ACMMM, 2022. [84]Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729, 2024. [85] Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. In CVPR, pages 24972–24982, 2025. [86]Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sangbeom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, et al. Visual representation alignment for multimodal large language models. arXiv preprint arXiv:2509.07979, 2025. [87]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In ACL, pages 4791–4800, 2019. [88] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. [89]Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. [90] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. [91]Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. [92]Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment. arXiv preprint arXiv:2310.01852, 2023. [93] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. ICLR, 2024. 14 A Code We provide the code implementation of DCAT for reproducibility. Please refer to this link for environment setup instructions and execution guidelines. B Proof B.1 Proof of Theorem 1 We first derive two auxiliary lemmas that connect the definitions of SEO and SD to the covariance quantities used in the information-theoretic lower bound. Lemma 1 (SEO bounded by projected covariance ratio). LetY mod ∈ R N m ×d be the modality-token activation matrix, and letU k ∈ R k×d contain the top-kright singular vectors of the centered text-token activations. Define μ y := 1 N m Y ⊤ mod 1∈ R d , e Y mod := Y mod − 1μ ⊤ y .(A1) Assume that the mean activation is orthogonal to the text principal subspace 1 : μ y U ⊤ k = 0.(A2) Define Z := Y mod U ⊤ k , e Z := e Y mod U ⊤ k ,(A3) and define the unbiased covariance matrices Σ y := 1 N m − 1 e Y ⊤ mod e Y mod ,Σ Z := 1 N m − 1 e Z ⊤ e Z.(A4) The Subspace Energy Overlap is defined using the uncentered activations: SEO = ∥Y mod U ⊤ k ∥ 2 F ∥Y mod ∥ 2 F .(A5) If tr(Σ y ) > 0, then SEO≤ tr(Σ Z ) tr(Σ y ) .(A6) Proof.We decompose the modality activations into their centered component and mean component: Y mod = e Y mod + 1μ ⊤ y .(A7) Since 1 ⊤ e Y mod = 0, these two components are orthogonal in Frobenius inner product. Therefore, ∥Y mod ∥ 2 F =∥ e Y mod + 1μ ⊤ y ∥ 2 F =∥ e Y mod ∥ 2 F +∥1μ ⊤ y ∥ 2 F = (N m − 1) tr(Σ y ) + N m ∥μ y ∥ 2 2 . (A8) Next, consider the projected activations. By the assumption μ y U ⊤ k = 0, we have 1μ ⊤ y U ⊤ k = 0.(A9) Hence, ∥Y mod U ⊤ k ∥ 2 F =∥ e Y mod U ⊤ k + 1μ ⊤ y U ⊤ k ∥ 2 F =∥ e Y mod U ⊤ k ∥ 2 F =∥ e Z∥ 2 F = (N m − 1) tr(Σ Z ). (A10) 1 Our empirical validation on LLaMA-3.1 indicates∥μ y U ⊤ k ∥ 2 /∥μ y ∥ 2 is0.09726± 0.02166which is sufficiently small. 15 Substituting Eq. (A8) and Eq. (A10) into the definition of SEO gives SEO = ∥Y mod U ⊤ k ∥ 2 F ∥Y mod ∥ 2 F = (N m − 1) tr(Σ Z ) (N m − 1) tr(Σ y ) + N m ∥μ y ∥ 2 2 . (A11) Since N m ∥μ y ∥ 2 2 ≥ 0, we have (N m − 1) tr(Σ y ) + N m ∥μ y ∥ 2 2 ≥ (N m − 1) tr(Σ y ).(A12) Therefore, SEO = (N m − 1) tr(Σ Z ) (N m − 1) tr(Σ y ) + N m ∥μ y ∥ 2 2 ≤ (N m − 1) tr(Σ Z ) (N m − 1) tr(Σ y ) = tr(Σ Z ) tr(Σ y ) . (A13) Lemma 2 (SD in terms of determinant and trace ofΣ Z ). LetΣ Z ≻ 0be the empirical covariance matrix of the projected modality-token activationsZ = Y mod U ⊤ k , and letλ 1 ,...,λ k > 0denote its eigenvalues. Define p j = λ j P k ℓ=1 λ ℓ , j = 1,...,k.(A14) The Spectral Diversity is SD = 1 k exp   − k X j=1 p j logp j   .(A15) Define the determinant-normalized spectral volume as SD det := det(Σ Z ) 1/k tr(Σ Z )/k .(A16) Assume that the projected spectrum satisfies the condition p j ≥ δ k , ∀j ∈1,...,k,(A17) for some sufficiently large δ ∈ (0, 1]. Then SD det ≥ δ SD.(A18) Equivalently, log SD det ≥ log SD + logδ.(A19) Proof. First, by the entropy upper bound of a discrete distribution over k outcomes, − k X j=1 p j logp j ≤ logk.(A20) Therefore, from Eq. (A15), 0 < SD≤ 1.(A21) Next, using P j λ j = tr(Σ Z ) and p j = λ j / P k ℓ=1 λ ℓ , we have λ j = p j tr(Σ Z ).(A22) 16 Thus, SD det = det(Σ Z ) 1/k tr(Σ Z )/k = Q k j=1 λ j 1/k tr(Σ Z )/k = Q k j=1 p j tr(Σ Z ) 1/k tr(Σ Z )/k = k   k Y j=1 p j   1/k =   k Y j=1 kp j   1/k . (A23) By the condition in Eq. (A17), we have kp j ≥ δ for all j. Hence, SD det =   k Y j=1 kp j   1/k ≥ δ.(A24) Since SD≤ 1, it follows that SD det ≥ δ ≥ δ SD.(A25) Taking logarithms gives log SD det ≥ log SD + logδ.(A26) We empirically validate the use ofδin Sec. B.5. Based on the results of Lemma 1 and Lemma 2, we prove the main theorem. Theorem 1 (Weighted Alignment-MI Lower Bound). Letz = yU ⊤ k , whereU k ∈ R k×d contains the top-kright singular vectors of the centered text activation matrix. LetΣ y = Cov(y)and Σ Z = Cov(z). For each text-token representation x, let Σ z|x := Cov(z|x), ̄ Σ z|x := E x Σ z|x .(A27) With assumption in Lemma 2, we also supposed that Σ Z ≻ 0, ̄ Σ z|x ≻ 0, and: (A1) z ∼N (μ Z , Σ Z ); (A2) z|x∼N (μ z (x), Σ z|x ); Then, for any γ ∈ (0, 1), I(y;x)≥ κ(γ) log SEO γ SD 1−γ + C δ , κ(γ) = k 2 min(γ, 1− γ) ,(A28) where C δ = k 2 log tr(Σ y ) k − 1 2 log det( ̄ Σ z|x ) + k 2 logδ.(A29) Proof. Since z = yU ⊤ k is a deterministic function of y, the Markov chain x←→ y −→ z(A30) holds. By the data processing inequality, I(y;x)≥ I(z;x).(A31) 17 It therefore suffices to lower-bound I(z;x). By the definition of mutual information, I(z;x) = h(z)− h(z|x).(A32) Since z ∼N (μ Z , Σ Z ) and Σ Z ≻ 0, its marginal entropy satisfies: h(z) = 1 2 log det(2πe Σ Z ).(A33) Under the conditional Gaussian assumption z|x∼N (μ z (x), Σ z|x ), we have h(z|x) = E x 1 2 log det(2πe Σ z|x ) .(A34) Since log det(·) is concave on the positive definite cone, Jensen’s inequality gives E x log det(Σ z|x ) ≤ log det E x [Σ z|x ] = log det( ̄ Σ z|x ).(A35) Therefore, h(z|x)≤ 1 2 log det(2πe ̄ Σ z|x ).(A36) Combining the preceding displays yields I(z;x) = h(z)− h(z|x) ≥ 1 2 log det(2πe Σ Z )− 1 2 log det(2πe ̄ Σ z|x ) = 1 2 log det(Σ Z )− 1 2 log det( ̄ Σ z|x ). (A37) We now express the determinant term using SEO and SD. By the definition of SD det , det(Σ Z ) 1/k = tr(Σ Z ) k SD det .(A38) By Lemma 1, tr(Σ Z ) k ≥ tr(Σ y ) k SEO.(A39) Hence, log det(Σ Z ) = k log det(Σ Z ) 1/k = k log tr(Σ Z ) k SD det ≥ k log tr(Σ y ) k SEO· SD det . (A40) By Lemma 2, the condition in Eq. A17 implies log SD det ≥ log SD + logδ.(A41) Therefore, log det(Σ Z )≥ k log tr(Σ y ) k + k log SEO + k log SD + k logδ.(A42) Substituting this into the mutual-information lower bound gives I(z;x)≥ k 2 log SEO + k 2 log SD + k 2 log tr(Σ y ) k − 1 2 log det( ̄ Σ z|x ) + k 2 logδ = k 2 (log SEO + log SD) + C δ . (A43) Finally, we convert the unweighted log-product into the weighted alignment score. Let a := log SEO, b := log SD.(A44) 18 Since 0 < SEO≤ 1 and 0 < SD≤ 1, we have a≤ 0 and b≤ 0. For any γ ∈ (0, 1), let m γ := min(γ, 1− γ).(A45) Since γ ≥ m γ and 1− γ ≥ m γ , and since a,b≤ 0, we have γa≤ m γ a,(1− γ)b≤ m γ b.(A46) Adding both inequalities gives γa + (1− γ)b≤ m γ (a + b).(A47) Equivalently, a + b≥ 1 m γ (γa + (1− γ)b).(A48) Since γa + (1− γ)b = log SEO γ SD 1−γ ,(A49) we obtain log SEO + log SD≥ 1 min(γ, 1− γ) log SEO γ SD 1−γ .(A50) Consequently, I(z;x)≥ k 2 min(γ, 1− γ) log SEO γ SD 1−γ + C δ . (A51) Since I(y;x)≥ I(z;x), we conclude that I(y;x)≥ k 2 min(γ, 1− γ) log SEO γ SD 1−γ + C δ .(A52) This completes the proof. B.2 Derivation of Eq. 8 In the main text, for the top-kright singular vectorsU k of the text activations, we defined the corresponding orthogonal projector and its orthogonal complement asP k (= U ⊤ k U k )andP ⊥ k (= I− P k ), respectively. We further defined the uniform singular value that preserves the total output energy: ρ = ∥X mod (θ 0 + τ r )∥ F √ k . In addition, we used the following target for the modality-token activations after the weight update: T α = c α · A diag σ α j ρ 1−α k j=1 B ⊤ U k , c α = ∥X mod (θ 0 + τ r )∥ F P k j=1 σ 2α j ρ 2(1−α) 1/2 We now derive the following two properties of T α : T α P ⊥ k = 0, ∥T α ∥ F =∥X mod (θ 0 + τ r )∥ F . For the first one, T α can be written in the form T α = QU k for some matrix Q. Hence, we obtain: T α P ⊥ k = QU k (I − P k ) = QU k (I − U ⊤ k U k ) = QU k − QU k U ⊤ k U k = QU k − QU k = 0, since U k U ⊤ k = I k . For the second one, we denote D α as: D α := diag σ α j ρ 1−α k j=1 . 19 Then, starting from the definition of T α : ∥T α ∥ 2 F = c 2 α · AD α B ⊤ U k 2 F . Expanding the Frobenius norm via∥M∥ 2 F = tr(M ⊤ M ): AD α B ⊤ U k 2 F = tr U ⊤ k B D α A ⊤ A | z =I k D α B ⊤ U k = tr U ⊤ k B D 2 α B ⊤ U k .(A53) Applying the cyclic property of the trace, tr(PQRS) = tr(SPQR): tr U ⊤ k B D 2 α B ⊤ U k = tr U k U ⊤ k | z =I k B D 2 α B ⊤ = tr B D 2 α B ⊤ .(A54) Since B is orthogonal, tr B D 2 α B ⊤ = tr(D 2 α ) = k X j=1 σ 2α j ρ 2(1−α) .(A55) Substituting (A53)–(A55) back and inserting the definition of c α : ∥T α ∥ 2 F = c 2 α · k X j=1 σ 2α j ρ 2(1−α) = ∥X mod (θ 0 + τ r )∥ 2 F k X j=1 σ 2α j ρ 2(1−α) · k X j=1 σ 2α j ρ 2(1−α) =∥X mod (θ 0 + τ r )∥ 2 F . B.3 Derivation of Eq. 9 In Eq. 9 of the main text, we adopted the following objective to enhance alignment: L self (τ ) =∥X mod (θ 0 + τ )P ⊥ k ∥ 2 F | z Encouraging SEO +∥X mod (θ 0 + τ )P k − T α ∥ 2 F | z Encouraging SD =∥X mod (θ 0 + τ )− T α ∥ 2 F . Then, the second equality can be derived as follows. From the previous derivation,T α satisfies T α P ⊥ k = 0, which is equivalent to: T α P k = T α .(A56) For notational convenience, let us writeY := X mod (θ 0 +τ ). SinceP k +P ⊥ k = I, we can decompose Y and T α as: Y = Y P k + Y P ⊥ k , T α = T α P k + T α P ⊥ k = T α + 0 = T α , where the second equality uses (A56). Therefore: Y − T α = (Y P k − T α ) + Y P ⊥ k .(A57) Expanding the squared Frobenius norm of (A57): ∥Y − T α ∥ 2 F =∥Y P k − T α ∥ 2 F +∥Y P ⊥ k ∥ 2 F + 2 tr (Y P k − T α ) ⊤ Y P ⊥ k . 20 It remains to show that the cross term vanishes. We split it into two parts: tr (Y P k − T α ) ⊤ Y P ⊥ k = tr P k Y ⊤ Y P ⊥ k | z (I) − tr T ⊤ α Y P ⊥ k | z (I) .(A58) For the term (I), the cyclic property of the trace gives us: tr P k Y ⊤ Y P ⊥ k = tr P ⊥ k P k | z = 0 Y ⊤ Y = 0,(A59) where we usedP ⊥ k P k = 0. For the term (I), sinceT α = QU k by its definition, we haveT ⊤ α = U ⊤ k Q ⊤ . By the cyclic property of the trace: tr T ⊤ α Y P ⊥ k = tr U ⊤ k Q ⊤ Y P ⊥ k = tr P ⊥ k U ⊤ k | z = 0 Q ⊤ Y = 0,(A60) where we used P ⊥ k U ⊤ k = (I − U ⊤ k U k )U ⊤ k = U ⊤ k − U ⊤ k U k U ⊤ k | z =I k = U ⊤ k − U ⊤ k = 0. B.4 Derivation of closed form for DCAT Proposition 1. Let X m := X mod , X t := X text for brevity. Assume that X ⊤ m X m + ηX ⊤ t X t ≻ 0.(A61) Then, for a given layer with pretrained weight θ 0 ∈ R d in ×d out , the DCAT objective is defined as: min τ L self (τ ) + η·L cross (τ ),(A62) with L self (τ ) =∥X m (θ 0 + τ )− T α ∥ 2 F , L cross (τ ) = β∥X t (τ − τ r )∥ 2 F + (1− β)∥X t (τ − τ d )∥ 2 F . Then, the objective (A62) has a unique minimizer as: τ ⋆ = X ⊤ m X m + η X ⊤ t X t −1 h X ⊤ m T α − X m θ 0 + η X ⊤ t X t βτ r + (1− β)τ d i .(A63) Proof. The total objectivef (τ ) :=L self (τ ) +ηL cross (τ )is a quadratic function ofτ, so we proceed by computing its gradient and setting it to zero. First,L self (τ ) can be represented as: L self (τ ) =∥X m (θ 0 + τ )− T α ∥ 2 F = tr τ ⊤ X ⊤ m X m τ + 2 tr τ ⊤ X ⊤ m (X m θ 0 − T α ) + const. Using the identities∇ τ tr(τ ⊤ Mτ ) = (M +M ⊤ )τ = 2Mτfor symmetricM, and∇ τ tr(τ ⊤ C) = C, we obtain ∇ τ L self (τ ) = 2X ⊤ m X m τ + 2X ⊤ m (X m θ 0 − T α ) = 2X ⊤ m X m (θ 0 + τ )− T α .(A64) Similarly, the gradient ofL cross can be represented as: ∇ τ L cross (τ ) = 2β X ⊤ t X t (τ − τ r ) + 2(1− β)X ⊤ t X t (τ − τ d ) = 2X ⊤ t X t τ − 2X ⊤ t X t βτ r + (1− β)τ d .(A65) Then,∇ τ L self + η∇ τ L cross from (A64) and (A65) is represented as: X ⊤ m X m τ + X ⊤ m (X m θ 0 − T α ) + η X ⊤ t X t τ − η X ⊤ t X t βτ r + (1− β)τ d = 0.(A66) Rearranging Eq. A66 gives the linear system: X ⊤ m X m + η X ⊤ t X t | z =:A τ = X ⊤ m (T α − X m θ 0 ) + η X ⊤ t X t βτ r + (1− β)τ d |z =:B . Since we stack sufficiently large tokens to constructX m andX t , we assumeX ⊤ m X m +η X ⊤ t X t ≻ 0 , the closed-form solution for τ is: τ ⋆ = X ⊤ m X m + η X ⊤ t X t −1 h X ⊤ m T α − X m θ 0 + η X ⊤ t X t βτ r + (1− β)τ d i . 21 Table A1: Empirical validation onSD,SD det , andδ eff . To assess the tightness of the bound, we compute the entropy-based spectral diversitySD, the proof-side spectral diversitySD det , and spectral conversion factorδ eff for Vicuna-based models in Fig. 2. The results show thatδ eff remains close to one, indicating that the entropy-based effective-rank SD introduces only a mild constant-level relaxation in the MI lower bound. Audio+VisionVideo+Vision SDSD det δ eff ≤SDSD det δ eff ≤ 0.2839±0.02730.2752±0.02280.9704±0.02240.2015±0.01780.2226±0.01971.1046±0.0140 B.5 Empirical validation of δ The proof of Theorem 1 involves a spectral non-degeneracy factor that connects the proof-side spectral termSD det with the entropy-based effective-rank measureSDused in the main text. In particular, the relevant condition can be expressed as the existence of a constant δ > 0 such that SD det ≥ δ SD.(A67) Equivalently, the largest empirical value supported by a given spectrum is δ eff := SD det SD .(A68) Importantly,δ eff is a spectral conversion factor between two spectral-spread measures, not a per- eigenvalue probability lower bound of the formp j ≥ δ/k. Therefore,δ eff is not constrained to be at most one, and values larger than one do not contradict the probability constraint P j p j = 1. For empirical validation, we compute bothSDandSD det using the features examined in our analysis, and report their ratioδ eff in Table A1. As shown in the table,δ eff remains close to one in both settings: 0.9704for Audio+Vision and1.1046for Video+Vision. These results indicate that the conversion fromSD det to the entropy-basedSDincurs only a mild constant-level relaxation in the MI lower bound. Thus, although the main text adopts the more interpretable entropy-based effective-rank measureSD, the empirical gap betweenSDandSD det is small in our analyzed model families. This supports the use of SD as a practical and meaningful spectral diversity metric for the proposed Alignment score. C Implementation Details C.1 Models and computation For the Vicuna, the setting allows a controlled comparison because the three MLLMs are trained from the same base under the same recipe except for the modality in OptMerge [75]. To complement this controlled setting, we additionally evaluate DCAT on independently trained models that nevertheless share the LLaMA-3.1-8B-Instruct backbones for audio, video, and vision modalities. For the two backbones, the vision MLLM is used as the donor in every experiment, since vision MLLMs are the modality on which standard merging tends to leave performance unchanged or slightly degraded while consistently lifting the other modalities. All experiments run on a single NVIDIA L40S 48 GB GPU and the DCAT solver over all projections completes in about 25-30 minutes per setting. C.2 Calibration and alignment-target data For all four settings, calibration data is drawn from the publicly available AVQA dataset [83] since the evaluation benchmarks (MMAU, Video-MME, etc.) are entirely disjoint from AVQA so that calibration and evaluation never share data. AVQA provides paired audio and video streams aligned to the same set of question/answer prompts, which lets us instantiate DCAT in a uniform way across audio and video recipients without changing the underlying source ofX text . For audio recipients, we feed the audio track in AVQA to the recipient MLLM as the modality input; for video recipients, we instead feed the video clip of the same sample with the same prompt used for the audio track. To solve the objective in Eq. 10, we draw small set of AVQA dataset (256 samples for each modality). 22 Table A2: Full implementation configuration of DCAT for the four headline settings. A blank entry under Evaluation seed indicates that the benchmark is small enough to be evaluated in full and therefore does not require subsampling. ComponentVicuna+AudioVicuna+VideoLLaMA+AudioLLaMA+Video BackboneVicuna-7B-v1.5Vicuna-7B-v1.5LLaMA-3.1-8B-InstructLLaMA-3.1-8B-Instruct Calibration sourceAVQA (audio track)AVQA (video track)AVQA (audio track)AVQA (video track) # calibration samples 256256256256 k (text subspace dim)128128256256 β 0.300.300.950.95 η1×10 5 1×10 5 1×10 6 1×10 6 C.3 Progressive, dependency-aware merging Although the closed-form update in Eq. 10 could in principle be solved for all projections in parallel, we found it preferable to apply the update progressively along two axes—across layers and across projection groups within each layer. Concretely, we process layers sequentially, and within each layer we partition the seven projections into four dependency-aware groups that mirror the activation-flow structure of a transformer block: query,key,value and out projections in the attention layers, up,gate and down projections in the attention projections and the feed-forward networks. The four groups are merged in the listed order, and after every group merge we re-collectX mod and X text at the relevant point so that the next group is solved against activations that already reflect the previous update. In this way, errors that would arise from treating projections as independent are absorbed iteratively, and the alignment accumulated across layers is corrected as it propagates. C.4 Hyperparameter values In DCAT we should control three hyperparameters: the text-subspace dimensionk, the recipient/donor balanceβin the cross-modal alignment term, and the cross-modal alignment weightη. For example, the text-subspace dimensionkis shared within a backbone family (k = 128for Vicuna andk = 256 for LLaMA-3.1-8B-Instruct), reflecting that hidden states of LLaMA-3.1-8B-Instruct are wider and require a larger effective rank to span the principal text directions. The complete configuration is reported in Table A2. C.5 Closed-form solve and engineering notes The DCAT objective in Eq. 10 is quadratic in τ and admits a layer-wise SPD system Aτ = B with A = X ⊤ mod X mod + η X ⊤ text X text . We solve this system in float64 using a Cholesky-based linear solver, with a small ridgeε solve = 10 −6 tr(A)/dadded on the diagonal for numerical stability. Activation matrices used by the solver are collected in float32, and the resulting weight delta is applied to the recipient checkpoint in its native bfloat16 precision. No gradient-based optimization is involved at any point. D Experimental Details D.1 Analysis details Alignment-accuracy Correlation Analysis. To examine whether the proposed alignment metric is correlated with model performance, we conduct a correlation analysis between alignment and benchmark accuracy using both Pearson and Spearman correlation coefficients. The analysis is performed on Audio+Vision and Video+Vision settings, under Vicuna [10] and LLaMA 3.1 [19] backbones. For model merging, we include Task Arithmetic [28] (TA), TIES-merging [82] (TIES), ISO-C [48] (ISO), Task Singular Vector [17] (TSV), WUDI [9], and OptMerge [75]. For each merging method, we first construct a fully merged model by merging the Vision model with either the Audio or Video model. Because correlation analysis requires multiple evaluation points rather than a single merged checkpoint, we further generate additional model variants by 23 applying SVD-based low-rank decomposition to the fully merged network parameters. Specifically, starting from each full-merge checkpoint, we decompose the merged weights and reconstruct a series of low-rank approximations with ranks 1, 2, 4, 8, 16, 32, 64, and 128. We then evaluate both the full-merge model and these SVD-derived variants, and compute Pearson and Spearman correlations between their Alignment scores and benchmark accuracies. This allows us to analyze how changes in the merged parameter space are reflected in both alignment and task performance. The evaluation is conducted separately on the benchmark corresponding to each constituent modality. For example, in the Audio+Vision setting, we measure performance on both the audio benchmark and the vision benchmark. We use TextVQA [63] for vision, MMAU [69] for audio, and Video-MME [16] for video. In addition, we examine a broader set of benchmarks, whose results are reported in Sec. E.1 For each benchmark, we evaluate 300 samples that are equally sampled across categories to ensure balanced coverage. For video evaluation, we use tokens from only the first, middle, and last frames of each sample for computational efficiency, instead of processing all frames. To provide a reference for the effect of merging itself, we additionally evaluate the original modality- specific models, i.e., the models trained on a single modality without merging, on the corresponding benchmarks. These reference results allow us to determine whether merging improves or degrades performance relative to the original specialized models. For experiments, the proposed alignment metric is computed between modality tokens and text tokens at each backbone layer withk = 32andγ = 0.8, and the alignment values are averaged over all samples in the benchmark dataset to obtain a layer-wise alignment profile. Since alignment varies substantially across layers, we analyze the results in a layer-wise manner, as shown in Fig. 3. Consistent with prior observations that cross-modal alignment tends to be strongest in middle layers, we select the 21st layer as a representative layer and report the linear regression plots and correlation values based on this layer in Fig. 2. D.2 Comparison details D.2.1 Baselines Task Arithmetic. Task Arithmetic represents the effect of task-specific fine-tuning as a task vector in weight space, defined by the difference between the fine-tuned model and the pretrained initialization [28]. Given a pretrained model with parametersθ 0 and modality- or task-specific models θ i N i=1 fine-tuned from the same initialization, each task vector is computed asτ i = θ i − θ 0 . Model merging is then performed by adding a scaled sum of task vectors back to the pretrained weights: θ TA = θ 0 + λ N X i=1 (θ i − θ 0 ),(A69) whereλcontrols the strength of the merged update. The original work shows that such vectors can be added or negated to compose or suppress task behaviors, enabling training-free model editing and multi-task merging at the weight level. TIES-Merging.TIES-Merging addresses parameter interference that arises when merging multiple task-specific models fine-tuned from the same pretrained initialization [82]. Given task vectors τ i = θ i − θ 0 , TIES first trims small-magnitude updates by resetting them to zero, thereby removing redundant parameter changes. It then elects a shared sign for each parameter dimension and merges only the task-vector entries whose signs agree with the elected sign. A simplified form of the resulting merge can be written as θ TIES = θ 0 + λm, m j = 1 |A j | X i∈A j ̃τ i,j , A j = ( i : sign( ̃τ i,j ) = sign N X k=1 ̃τ k,j !) , (A70) where ̃τ i denotes the trimmed task vector andλcontrols the merge strength. By resolving sign disagreement before averaging, TIES-Merging reduces destructive interference and improves training- free multi-task model merging compared to direct weight averaging or task-vector addition. Isotropic Merging (ISO-C). Isotropic Merging analyzes model merging from the perspective of the singular subspaces of task-specific weight updates [48]. Given layer-wise task matrices 24 ∆ i = W i − W 0 , it first constructs a merged task matrix∆ = P N i=1 ∆ i and decomposes it as ∆ = U diag(σ)· V ⊤ . Instead of preserving the skewed singular value spectrum of the summed update, Iso-C replaces the singular values with a uniform scale, thereby balancing the contribution of different singular directions. Iso-CTS further augments this common subspace with task-specific residual directions, which helps retain underrepresented task information when merging many diverse models. The resulting merge can be written as W Iso = W 0 + λU CTS ( ̄σI)V ⊤ CTS , ̄σ = 1 r r X j=1 σ j ,(A71) whereU CTS andV CTS denote the orthogonalized bases obtained from common and task-specific subspaces, and λ controls the merge strength. Task Singular Vectors. Task Singular Vectors (TSV) analyzes task vectors at the layer level by preserving each task-specific update as a matrix rather than flattening the entire network into a single vector [17]. Given a pretrained layerW ℓ 0 and task-specific layersW ℓ i N i=1 , TSV first decomposes each layer task matrix∆ ℓ i = W ℓ i −W ℓ 0 using SVD and retains only the top-rsingular components. The method observes that overlap between singular vectors from different tasks induces task interference, and therefore TSV-Merge reduces such interference by decorrelating the concatenated left and right singular-vector bases through whitening or an equivalent orthogonal Procrustes step. A simplified layer-wise merge can be written as W ℓ TSV = W ℓ 0 + λ ˆ U ℓ ̄ Σ ℓ ( ˆ V ℓ ) ⊤ .(A72) where ˆ U ℓ and ˆ V ℓ denote the decorrelated versions of ̄ U ℓ and ̄ V ℓ , and λ controls the merge strength. By combining low-rank approximation with singular-vector decorrelation, TSV-Merge aims to reduce destructive interference among task-specific updates while keeping the merge training-free. WUDI-Merging. WUDI-Merging is a data-free model merging method that reduces interference among task-specific experts by exploiting the observation that linear-layer task vectors approximately span the corresponding input subspace [9]. Given a pretrained modelθand task vectorsτ i = θ i − θ, WUDI focuses on the linear layers and directly optimizes the merged task vectorτ m,l to stay compatible with each task vectorτ i,l . For each linear layerl, the optimization objective is written as L l (τ m,l ) = X i 1 ∥τ i,l ∥ 2 F (τ m,l − τ i,l )τ ⊤ i,l 2 F , θ m = θ + τ m . (A73) The merged task vectors are optimized independently for each linear layer and then assembled into the final merged model. By using task vectors themselves as guidance signals, WUDI-Merging mitigates destructive interference without requiring additional training data, test-time samples, or extra task-specific modules. OptMerge. OptMerge studies model merging for MLLMs and proposes a data-free optimization method for combining task- or modality-specific expert models [75]. Given task vectorsτ i = θ i − θ 0 , OptMerge observes that directly optimizing the merged task vector can be unstable due to redundant and noisy components in task-specific updates. For each linear layerl, it first applies a low-rank approximation to the centered task matrix,τ i,l − ̄τ l ≈ U i,k Σ i,k V ⊤ i,k , and uses the denoised update ̃τ i,l = U i,k Σ i,k V ⊤ i,k + ̄τ l for optimization. The layer-wise merged vector is obtained by minimizing task-vector interactions: τ ∗ m,l = arg min τ m,l N X i=1 (τ m,l − ̃τ i,l )S ⊤ i,l 2 F ∥τ i,l ∥ 2 F , S i,l = Σ i,k V ⊤ i,k , θ m = θ 0 + τ m . (A74) For LoRA fine-tuned models, OptMerge further uses direct truncated SVD, mean initialization, and SGD-based optimization to avoid excessive growth of the merged vector norm. D.2.2 Base models Vicuna-based MLLMs. We use the Vicuna-based modality-specific MLLMs released by Opt- Merge [75]. All models share Vicuna-7B-v1.5 as the language backbone, while employing modality- specific encoders and connectors: the vision-language model uses CLIP-ViT-L-336px [52] with an 25 MLP projector, the audio-language model uses BEATs-Iter3+ [45] with a Q-Former [36] connec- tor, and the video-language model uses LanguageBind [92] with an MLP projector. These models are trained on bi-modal modality-text data for vision, audio, and video, respectively, following a two-stage procedure consisting of connector alignment and subsequent fine-tuning. Following the original modality-merging setup, we preserve each modality-specific encoder and connector, and apply merging only to the shared language-model parameters. This setup allows us to evaluate whether cross-modality merging can transfer alignment across different modality-specific MLLMs while keeping the underlying language backbone fixed. LLaMA 3.1-based MLLMs.We further evaluate DCAT on LLaMA 3.1-based MLLMs collected from publicly available in-the-wild checkpoints. Specifically, we use LLaVA-LLaMA3.1-8B 2 as the vision-language model, Ultravox-v0.4.1-LLaMA-3.1-8B 3 as the audio-language model, and LLaVA-Video-LLaMA-3.1-8B 4 as the video-language model. Unlike the Vicuna-based models, these checkpoints are independently developed and trained with different data mixtures, training recipes, and modality-specific components, while sharing the same LLaMA 3.1-8B language-model architecture. For each model, we preserve its original modality encoder and connector, and apply merging only to the language-model parameters that are architecturally compatible across models. This setting provides a more practical and less controlled testbed for evaluating whether DCAT can robustly transfer cross-modal alignment across heterogeneous real-world MLLMs. Considered, but not experimented models.Our merging setup assumes that the modality-specific MLLMs share the same language-model architecture and parameter shapes, since both DCAT and the baseline merging methods operate directly at the weight level. We also considered other in- the-wild MLLMs that appear to share the same backbone family, such as Qwen-Audio [12] and LLaVA-Next [35] with Qwen-1.5 [3] backbones. However, these models often differ in tokenizer size, input embedding size, and LM head size, which prevents direct weight-space merging across all compared methods without additional architectural modification or retraining. Similarly, several recent public MLLMs, including Idefics3-8B-Llama3 5 and Llama-3.1-Nemotron-Nano-VL-8B-V1 6 , were considered but excluded because their backbone implementations or parameter shapes are not fully compatible with the other modality-specific models. We did not introduce additional training or ad-hoc resizing procedures to force compatibility, as this would confound the comparison among training-free merging methods. To the best of our knowledge, our experiments cover the publicly available in-the-wild MLLMs that satisfy the required architectural compatibility conditions; we provide a more detailed discussion of this limitation in Sec. G. D.2.3 Benchmarks To compare the overall performance across various tasks, we evaluated the baseline and merged models on multiple audio and video question-answering benchmarks. For audio, we used MMAU [69], AIR-Bench-Foundation [84], MuChoMusic [74], Clotho-AQA [40], and VocalSound [18]. For video, we used Video-MME [16], TempCompass [44], MVBench [37], MLVU [91], EgoSchema [47], and LongVideoBench [78]. MMAU and AIR-Bench-Foundation consist of 1,000 and 24,683 multiple-choice questions (MCQs), respectively, paired with audio clips covering natural sounds, speech, and music. MuChoMusic contains 1,187 MCQs designed to evaluate the music understanding capability of models. For Clotho- AQA, we used a subset consisting of 1,442 binary QA questions related to audio. VocalSound contains 3,591 questions for classifying six types of human vocal sounds. We used a custom prompt to prevent the model from generating answers outside the given choices. Video-MME contains 2,700 MCQs based on 900 videos with diverse durations and categories. For TempCompass, we used the multiple-choice, caption-matching, and yes/no subsets, which contain 1,580, 1,503, and 2,453 questions, respectively. These subsets evaluate the temporal understanding capability of models across five categories: action, attribute change, direction, event order, and speed. MVBench consists of 20 diverse question sets for evaluating temporal perception, with 200 questions 2 https://huggingface.co/modelscope/llava-llama3.1-8b 3 https://huggingface.co/fixie-ai/ultravox-v0_4_1-llama-3_1-8b 4 https://huggingface.co/weizhiwang/LLaVA-Video-Llama-3.1-8B 5 https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3 6 https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-VL-8B-V1 26 in each set. MLVU consists of 2,174 MCQs that evaluate long-video understanding ability across various genres. For EgoSchema, we used the subset containing 500 long-form video MCQ pairs. LongVideoBench consists of 1,337 MCQs designed to evaluate long-context video understanding. Overall results for these benchmarks, except EgoSchema and LongVideoBench, are reported in Table 2, and detailed results for specific categories in MVBench and TempCompass are provided in Appendix F.2. Additionally, we used GQA [25], MMBench [43], MMStar [8], POPE [38], and TextVQA [63] to evaluate the vision capabilities of the models. The GQA testdev-balanced split contains 12,578 real-world visual question-answering examples. MMBench (Dev_EN) and MMStar contain 1,164 and 1,500 questions, respectively, for evaluating various capabilities of vision-language models. POPE consists of 9,000 binary QA pairs designed to evaluate object hallucination. TextVQA (Val) consists of 5,000 questions for evaluating a model’s ability to read text in images. Quantitative results for the vision benchmarks are provided in Appendix F.3. Each benchmark was evaluated using the lmms-eval [33,89] toolkit. However, some prompts or evaluation methods were customized for our evaluation. Example prompts and evaluation methods are summarized in Table A13. E Additional Analysis E.1 Additional correlation analysis To further assess the generality of the relationship between the proposed Alignment score and modality performance, we extend the correlation analysis to a broader set of benchmarks. For vision, we additionally consider GQA [25], OCR-VQA [50], MathVision [72], and VizWiz [20]; for audio, we use VocalSound [18], MuCho-music [74], Clotho-AQA [40]; and for video, we use TempCompass [44], MVBench [37], and MLVU [91]. The experimental protocol follows Sec. D.1. In particular, we perform the analysis for both Audio+Vision and Video+Vision settings on both Vicuna and LLaMA 3.1 backbones. For each merged setting, we evaluate the benchmarks corresponding to each constituent modality. As in the main analysis, we examine the relationship between Alignment score and modality performance both layer-wise and at the representative middle layer (21st layer). The layer-wise results are shown in Fig. A1- A2, and the corresponding 21st-layer regression and correlation results are presented in Fig. A3-A4. Across all additional benchmarks and backbone settings, we observe a consistent trend. The strongest positive correlation between the Alignment score and performance appears in the middle layers, in agreement with the observations from the main analysis. Moreover, both Pearson and Spearman correlation coefficients between alignment and accuracy are positive across all cases in the 21st layer, indicating that higher alignment is consistently associated with better benchmark accuracy. These results further support that the proposed Alignment score is not specific to a particular benchmark or backbone, but instead provides a robust indicator of model quality across diverse evaluation settings. 27 MMAU MMAU Audio-Vision merged GQAMathVision OCRVQAVizWiz MVBenchMLVUTempCompass Vision benchmark MuChoClotho GQAOCRVQA Video-Vision merged Audio benchmark Vision benchmark Video benchmark Figure A1: Layer-wise Pearson correlation between Alignment score and modality performance on Vicuna backbone. We measure the Alignment score in a layer-wise manner for vision, audio, and video benchmark on Audio+Vision and Video+Vision Vicuna backbone. Across all cases, the correlation between Alignment score and accuracy is strongest in the middle layers. 28 MMAU Video-MME Vo c a lMuCho Video-Vision merged TempCompass Audio-Vision merged TextVQAOCR-VQA MathVision Vizwiz Vision benchmark Audio benchmark Audio-Vision merged Audio benchmark Figure A2: Layer-wise Pearson correlation between Alignment score and modality performance on LLaMA-3.1. We measure the Alignment score in a layer-wise manner for vision, audio, and video on Audio+Vision and Video+Vision LLaMa backbone. Across all cases, the correlation between the Alignment score and accuracy is strongest in the middle layers. 29 Audio-Vision merged Vision benchmarkVision benchmark Video-Vision merged Vision benchmarkVision benchmark Video-Vision mergedAudio-Vision merged Vision benchmarkVision benchmarkAudio benchmarkAudio benchmark Video benchmarkVideo benchmarkVideo benchmark Figure A3: Correlation between Alignment score and modality performance on the 21st layer of Vicuna backbone. We analyze the correlation between the Alignment score and modality per- formance across checkpoints obtained from merging methods [9,17,28,48,75,82], and their SVD-derived variants. The red dotted line refers to the performance of original model. Across all results, the Alignment score shows a positive correlation with modality performance under both Pearson and Spearman measures. 30 Audio-Vision merged Audio benchmarkAudio benchmark Video-Vision merged Video benchmark Audio benchmark Video benchmark Audio-Vision merged Vision benchmarkVision benchmarkVision benchmarkVision benchmark Figure A4: Correlation between Alignment score and modality performance on the 21st layer of LLaMA-3.1 backbone. We analyze the correlation between the Alignment score and modality performance across checkpoints obtained from merging methods [9,17,28,48,75,82], and their SVD-derived variants. The red dotted line refers to the performance of original model. Across all results, the Alignment score shows a positive correlation with modality performance under both Pearson and Spearman measures. 31 E.2 Additional layer-wise SEO and SD analysis To further examine how DCAT affects the internal structure of the merged model, we conduct an additional layer-wise analysis of SEO and SD at the level of individual transformer components. Specifically, for the Vicuna backbone, we compare SEO and SD before and after applying DCAT for each major submodule in both the MLP and attention blocks, including the MLP projections (gate, down, andup) and the attention projections (q,k,v, ando). This analysis is performed on the merged Vicuna models in the Audio+Vision and Video+Vision settings. The results are shown in Fig. A5 and Fig. A6. Across all examined components and across both merged models, applying DCAT consistently increases both SEO and SD. This trend is observed not only at the model level but also throughout the individual layers and their constituent projections, indicating that the effect of DCAT is broadly distributed across the transformer rather than being confined to a small subset of parameters. These results provide additional empirical support for our theoretical motivation, showing that DCAT systematically encourages the structural properties associated with a higher mutual-information lower bound throughout the network. 32 Layer-wise SEO ComparisonLayer-wise SD Comparison BeforeAfter 푾 푫풐풘풏 푾 푮풂풕풆 푾 푼풑 푾 풌 푾 푸 푾 푽 푾 푶 푾 푫풐풘풏 푾 푮풂풕풆 푾 푼풑 푾 풌 푾 푸 푾 푽 푾 푶 Figure A5: Layer-wise SEO and SD of individual transformer submodules before and after applying DCAT on the Audio+Vision Vicuna. We compare SEO and SD for the MLP projections (W gate ,W down , andW up ) and the attention projections (W k ,W q ,W v , andW o ) across all Vicuna layers. In all examined components, applying DCAT consistently increases both SEO and SD. 33 Layer-wise SEO ComparisonLayer-wise SD Comparison BeforeAfter 푾 푫풐풘풏 푾 푮풂풕풆 푾 푼풑 푾 풌 푾 푸 푾 푽 푾 푶 푾 푫풐풘풏 푾 푮풂풕풆 푾 푼풑 푾 풌 푾 푸 푾 푽 푾 푶 Figure A6: Layer-wise SEO and SD of individual transformer submodules before and after applying DCAT on the Video+Vision Vicuna. We compare SEO and SD for the MLP projections (W gate ,W down , andW up ) and the attention projections (W k ,W q ,W v , andW o ) across all Vicuna layers. In all examined components, applying DCAT consistently increases both SEO and SD. 34 051015202530 Layer 10 0 10 1 10 2 Effective rank Vicuna: Audio + Vision → Audio 051015202530 Layer 10 0 10 1 10 2 Effective rank Vicuna: Video + Vision → Video 051015202530 Layer 10 1 10 2 Effective rank LLaMA 3.1: Audio + Vision → Audio 051015202530 Layer 10 0 10 1 10 2 Effective rank LLaMA 3.1: Video + Vision → Video 0.4 0.6 0.8 1.0 α 0.4 0.6 0.8 1.0 α 0.4 0.6 0.8 1.0 α 0.4 0.6 0.8 1.0 α Eff. rank (text)Eff. rank (modality)α Figure A7: Layer-wise effective rank and scheduling ofα. DCAT schedulesαinversely with the text effective rank, as small text effective rank with largeαcould introduce noisy alignment between text and modality-token activations. E.3 Rationale of scheduling α DCAT adjustsαbased on the relative changes in SEO and SD, as described in Eq. 11. This design is motivated by the layer-wise structure of the text subspace. As shown in Fig. A7, the effective rank of text-token activations generally decreases in the middle layers, although the degree of this trend varies across architectures and modalities. When the text subspace is effectively low-rank, aggressively increasing SD may spread modality information over weak or noisy text-space directions, leading to noisy alignment rather than reliable modality-text alignment. To mitigate this risk, DCAT uses the relative behavior of SEO and SD to adaptively scheduleα, making the target more conservative when the reliable text subspace is limited. Empirically, this schedule induces an inverse relationship between the text effective rank andα. In layers where the text effective rank is small,αbecomes larger, encouraging DCAT to emphasize SEO and preserve the recipient’s projected spectrum rather than over-promoting SD. Conversely, when the text effective rank is larger,αdecreases and allows stronger encouragement of SD. Although the LLaMA 3.1-based audio model shows a relatively different effective-rank pattern compared with the other settings, the scheduledαstill follows the intended inverse trend with respect to the text effective rank. F Additional Results F.1 Statistical significance of presented results To assess whether the quantitative improvements reported in the main text are statistically significant, we additionally compute the performance gain relative to the original modality-specific model and the correspondingp-value for each benchmark in the Audio+Vision and Video+Vision Vicuna settings. Specifically, for each merging method ([9,17,28,48,75,82]), including ours, we report the performance difference in percentage points (∆, %p) with respect to the original model, together with its statistical significance. We evaluate statistical significance using McNemar’s test, which is appropriate in our setting because all compared models are evaluated on the same paired samples from each benchmark. Since each sample yields a matched pair of outcomes between two models (correct or incorrect), McNemar’s test provides a natural way to determine whether the observed difference in accuracy is statistically meaningful beyond aggregate score differences. 35 Table A3: p-value of DCAT on audio+vision MMAUAIR-BenchMuChoClothoVocal ∆ (%p) p-value∆ (%p) p-value∆ (%p) p-value∆ (%p) p-value∆ (%p) p-value Vicuna-based TA+4.301.71e-02+5.913.62e-61+23.088.57e-29+19.764.14e-36-4.181.75e-07 TIES+2.101.78e-01+5.503.74e-78+11.467.79e-17+13.731.83e-24+1.951.55e-02 ISO+4.505.57e-03+6.623.53e-88+25.861.64e-41+17.342.99e-33-4.682.86e-13 TSV +3.802.80e-02+7.343.15e-112+23.841.65e-33+19.974.21e-37-1.454.50e-02 WUDI+4.806.74e-03+6.065.07e-64+22.491.35e-27+19.491.09e-34-2.019.10e-03 OptMerge+4.207.74e-03+7.336.15e-127+23.255.65e-37+19.001.97e-36+0.089.39e-01 DCAT (Ours)+7.304.12e-04+9.272.42e-116+27.217.82e-39+21.732.27e-43+2.275.83e-03 LLaMA 3.1-based TA ∗ (λ = 1.0)-2.908.41e-02-3.182.78e-17-2.951.01e-01+1.602.62e-01-3.453.47e-04 TIES (λ = 1.0)+0.507.85e-01-0.432.17e-01+2.361.67e-01+2.085.99e-02-0.534.11e-01 ISO (λ = 1.0)+2.101.52e-01-2.235.00e-11+3.374.42e-02+3.191.48e-03-1.671.26e-06 TSV ∗ (λ = 1.0)-2.908.41e-02-3.182.78e-17-2.951.01e-01+1.602.62e-01-3.453.47e-04 WUDI (λ = 1.0) -1.902.69e-01-3.302.47e-18-2.611.43e-01+1.802.12e-01-4.121.80e-05 OptMerge ∗ (λ = 1.0)-2.908.41e-02-3.182.78e-17-2.951.01e-01+1.602.62e-01-3.453.47e-04 TA ∗ (λ = 0.2)+1.702.21e-01-0.048.92e-01+1.942.17e-01+2.087.85e-03+0.196.84e-01 TIES (λ = 0.2)+2.101.01e-01+0.516.96e-02+2.021.17e-01+1.187.48e-02-0.451.09e-01 ISO (λ = 0.2)+0.606.75e-01+0.641.54e-02+0.674.75e-01+2.019.14e-04-0.842.88e-03 TSV ∗ (λ = 0.2)+1.702.21e-01-0.048.92e-01+1.942.17e-01+2.087.85e-03+0.196.84e-01 WUDI (λ = 0.2)+0.209.39e-01+0.029.57e-01+0.179.50e-01+2.011.49e-02-1.113.11e-02 OptMerge ∗ (λ = 0.2)+1.702.21e-01-0.048.92e-01+1.942.17e-01+2.087.85e-03+0.196.84e-01 DCAT (Ours)+3.665.20e-03+2.073.41e-14+5.731.73e-05+6.082.34e-13+0.925.33e-03 Table A4: p-value of DCAT on video+vision Video-MMETempCompassMVBenchMLVU ∆ (%p) p-value∆ (%p) p-value∆ (%p) p-value∆ (%p) p-value Vicuna-based TA+2.117.31e-02-0.079.47e-01+5.201.30e-07+12.373.29e-20 TIES-6.707.38e-16-0.831.81e-01-5.702.99e-17+0.416.36e-01 ISO+3.448.96e-04-0.079.43e-01+4.531.74e-06+13.062.53e-21 TSV+4.041.33e-04+1.791.13e-02+6.102.25e-10+12.971.32e-28 WUDI+1.561.81e-01-0.525.39e-01+4.838.67e-07+12.051.21e-19 OptMerge+3.306.17e-05+1.016.10e-02+6.355.89e-14+10.356.89e-24 DCAT (Ours)+6.533.06e-08+4.251.11e-06+9.172.93e-20+13.756.69e-39 LLaMA 3.1-based TA ∗ (λ = 1.0)+0.703.94e-01+1.821.18e-02+2.881.14e-04+11.555.43e-31 TIES (λ = 1.0)+3.301.15e-04-4.064.81e-08+5.881.44e-13+13.751.74e-39 ISO (λ = 1.0)+2.702.30e-03-22.441.88e-166+5.033.54e-09+11.961.18e-29 TSV ∗ (λ = 1.0)+3.678.75e-06+5.496.86e-15+6.005.26e-15+14.541.27e-45 WUDI (λ = 1.0)+1.961.49e-02-40.750.00e+00+4.353.66e-08+13.896.65e-40 OptMerge ∗ (λ = 1.0)+4.671.20e-07-47.690.00e+00+6.234.68e-14+15.596.01e-47 TA ∗ (λ = 0.2)+4.741.16e-06-8.132.98e-24+2.181.44e-02+9.433.83e-17 TIES (λ = 0.2)+1.709.89e-02-6.364.20e-15+4.433.71e-07+5.806.91e-06 ISO (λ = 0.2)+1.332.07e-01+5.915.03e-14+3.808.24e-06+11.642.07e-20 TSV ∗ (λ = 0.2)+3.261.17e-03+2.087.87e-03+4.651.02e-07+7.546.81e-10 WUDI (λ = 0.2)+4.113.67e-05+7.156.29e-21+3.557.58e-05+11.459.42e-22 OptMerge ∗ (λ = 0.2)+2.411.91e-02+6.028.59e-15+5.485.69e-10+9.258.34e-14 DCAT (Ours)+6.253.85e-09+8.607.24e-23+10.186.03e-30+18.365.51e-57 The results in Table A3 for Audio+Vision and Table A4 for Video+Vision show that our method yields statistically significant improvements across all reported settings. In the tables,p-values below 0.05 are highlighted in yellow, and those below 0.01 are highlighted in red. While baseline merging methods sometimes improve performance, their gains are not consistently significant and vary depending on the benchmark and backbone. In contrast, our method achieves positive improvements with statistically significantp-values in all cases, demonstrating that the performance gains reported in the main text are not only numerically favorable but also consistently supported at the sample level. F.2 Additional quantitative results For some of the results presented in Tab.2, we reported only the overall or average scores rather than detailed subcategory results due to space constraints. In this section, we provide these detailed subcat- egory results, as well as additional benchmark results on EgoSchema[47] and LongVideoBench [78]. The results are reported in Tab.A5 and Tab.A6. These results show that DCAT consistently improves performance across diverse subcategories and additional benchmarks, while maintaining a clear performance advantage over other merging methods. 36 Table A5: Detailed results for MVBench benchmark. We report the specific performance measured from subcategories on MVBench. MVBench Action Antonym Action Count Action Localization Action Prediction Action Sequence Character Order Counterfactual Inference Egocentric Navigation Episodic Reasoning Fine Grained Action Vicuna-based Original51.5038.0027.5029.0035.0047.0034.0027.0032.0040.00 TA62.0032.5036.0046.0048.5045.0034.0034.0040.5038.00 TIES55.5016.5026.0024.5032.0028.5031.5025.0022.5030.50 ISO54.5038.0028.0049.0046.0046.5041.0034.0037.5041.00 TSV60.5036.5031.0048.0050.0046.0038.5034.0041.0040.50 WUDI63.0032.0034.0046.5050.0046.5035.0032.0038.5039.50 OptMerge 61.5036.5028.5049.0052.5044.0043.0031.5043.5038.50 DCAT (Ours)65.0037.0037.0049.0053.0049.0046.0034.0045.5042.00 LLaMA 3.1-based Original61.5018.0030.0047.0045.0050.0039.5021.0033.5036.50 TA ∗ (λ = 1.0)71.0045.5026.0044.0045.5049.0031.0025.0044.5036.50 TIES (λ = 1.0)75.0039.5027.5046.0050.0055.5032.5033.5047.0040.50 ISO (λ = 1.0)64.5044.5027.5047.0050.0052.0036.0037.0046.0040.00 TSV ∗ (λ = 1.0)76.5041.0028.5045.5050.5055.0036.0026.0045.0039.50 WUDI (λ = 1.0)69.0042.0030.5045.0053.0055.5030.0029.5046.0038.00 OptMerge ∗ (λ = 1.0)71.0036.5032.5045.0053.0057.0033.0034.5046.0041.50 TA ∗ (λ = 0.2)63.5028.0024.0045.0054.0044.5035.501.5045.0038.50 TIES (λ = 0.2)64.5027.5021.0044.0057.0049.5032.5036.5049.5036.50 ISO (λ = 0.2)58.0027.5025.5040.5056.5048.0040.0036.5051.0035.50 TSV ∗ (λ = 0.2)62.0028.0024.5040.5059.0050.0039.5038.0050.5036.50 WUDI (λ = 0.2)61.5023.0027.5039.0055.0049.5035.5037.5049.5038.50 OptMerge ∗ (λ = 0.2)64.0028.5026.5042.5058.5049.5040.0037.5050.5034.50 DCAT (Ours)75.5048.5033.5048.5060.0056.0039.5039.5051.5042.00 MVBench Fine Grained Pose Moving Attribute Moving Count Moving Direction Object Existence Object Interaction Object Shuffle Scene Transition State Change Unexpected Action Average Vicuna-based Original28.0043.5022.0027.5047.0052.5038.5065.5039.5044.5038.48 TA33.0047.0030.0030.0054.5054.0034.5082.0045.0047.0043.68 TIES 31.0042.0020.5019.5039.0047.0030.0054.0039.0041.0032.78 ISO29.5047.5034.5027.0053.0051.5035.0078.5039.0049.0043.00 TSV32.0049.5029.0031.5052.5055.0037.0082.5046.5050.0044.58 WUDI30.5047.0027.5028.5055.0052.0034.0082.0045.0047.5043.30 OptMerge33.5049.0028.5031.5047.0056.5037.0083.0046.5055.5044.83 DCAT (Ours)35.0051.0036.5033.0054.5058.5041.0083.0048.5054.5047.65 LLaMA 3.1-based Original34.5045.5031.0025.0051.0058.5031.5079.0038.5043.5041.00 TA ∗ (λ = 1.0)35.5048.0038.0022.5053.0054.0029.5082.0044.5052.5043.88 TIES (λ = 1.0)34.0047.0032.0030.5053.5067.0035.5087.0045.5058.5046.88 ISO (λ = 1.0)30.5046.0036.5031.0048.0055.5032.0088.0043.5065.0046.03 TSV ∗ (λ = 1.0)36.5048.0034.5033.0053.0064.0034.0087.0046.0060.5047.00 WUDI (λ = 1.0)35.0044.5036.0027.0053.5059.5030.0083.5041.5058.0045.35 OptMerge ∗ (λ = 1.0)36.0045.0035.0032.5053.0064.5035.5087.0045.0061.0047.23 TA ∗ (λ = 0.2)36.0046.0038.0034.0048.5065.0030.0076.0045.5065.0043.18 TIES (λ = 0.2)37.0046.5040.0035.0048.0063.5031.5078.0043.5067.0045.43 ISO (λ = 0.2)35.0045.5035.5022.5049.0059.5033.5089.0040.0067.5044.80 TSV ∗ (λ = 0.2)37.0045.5037.0026.0048.0063.5032.0082.5045.0068.0045.65 WUDI (λ = 0.2)35.5044.0038.5028.0044.5063.0028.5087.0040.0065.5044.55 OptMerge ∗ (λ = 0.2)36.5044.0039.5034.0045.5064.5033.0088.0042.5070.0046.48 DCAT (Ours)39.0049.0041.0035.0053.0068.0036.5090.0047.0070.5051.18 F.3 Results on vision benchmarks We use vision MLLMs as donor models to improve the performance of audio and video MLLMs. A natural question is how the vision performance of the resulting transferred models changes after DCAT. We investigate this question using Vicuna-based models, and report the results in Tab.A7. We evaluate on a diverse set of vision benchmarks, including GQA[25], MMBench [43], MMStar [8], POPE [38], and Text-VQA [63]. When the audio MLLM is used as the recipient, the vision performance is largely preserved or slightly improved, indicating only minor perturbations. In contrast, when the video MLLM is used as the recipient, we observe some degree of degradation in vision performance. This behavior is expected because DCAT transfers alignment rather than modality-specific capabilities. We emphasize that our goal is not to obtain a single model that performs well on both modalities, but rather to transfer the capability of a strongly aligned MLLM from one modality to another in order to improve performance on the target modality. F.4 Results on audio-video DCAT Beyond using a vision MLLM as the donor in DCAT, we also conducted experiments with Vicuna- based models in two additional settings: Video-to-Audio, where a video MLLM is used as the donor and an audio MLLM as the recipient, and Audio-to-Video, where an audio MLLM is used as the donor and a video MLLM as the recipient. The results are reported in Tab.A8. In the Video-to-Audio setting, DCAT improves the performance of the audio MLLM, whereas the improvement in the Audio-to-Video setting is limited. We hypothesize that this discrepancy arises from differences in the 37 Table A6: Detailed results for MLVU, TempCompass benchmark and other benchmarks. We report the specific performance measured from subcategories on MLVU, TempCompass(Multi Choice, Caption Matching, Yes/No), and other two benchmarks(EgoSchema, LongVideoBench). MLVUTempCompass - Multi Choice Count Topic Reasoning EgoOrderPlotQANeedle Anomaly Reco. AverageActionAttr.Dir.OrderSpeedOverall Vicuna-based Original27.1857.7928.4123.1730.8033.8027.0032.5970.4135.0733.1337.7532.8142.28 TA34.4765.4034.6623.1750.0945.0761.0044.8482.8445.1437.0138.7435.3348.29 TIES25.7358.5627.2723.1732.4731.8333.0033.1563.6133.3332.5435.7630.6039.56 ISO30.1067.3042.3322.3949.3544.2361.5045.3181.6646.1835.5239.4035.6548.10 TSV33.9870.7242.9023.9448.9842.2553.5045.1886.6947.5736.4242.3835.0250.06 WUDI33.0165.7835.5123.1750.2843.3859.5044.3882.2545.4936.1240.0736.2848.48 OptMerge30.5871.1036.0825.1043.6040.5656.0043.2979.2944.1035.2237.7538.1747.34 DCAT (Ours) 35.7472.3744.2425.7351.7346.0462.3346.3486.2748.1638.0643.5438.4551.62 LLaMA 3.1-based Original3.4050.1934.6616.2243.0435.4939.5031.7987.5746.8836.1237.0935.9649.24 TA ∗ (λ = 1.0)37.8670.7243.7523.9447.5044.7948.0045.2286.9845.1433.7337.0936.9148.48 TIES (λ = 1.0)38.3573.0046.0223.1751.2147.6150.5047.1291.1249.3134.0335.1042.2750.89 ISO (λ = 1.0)24.2767.3042.0525.1049.1749.5859.5045.2867.4638.1926.2718.2117.0333.86 TSV ∗ (λ = 1.0) 38.3574.5247.7323.5551.0247.8953.5048.0890.5348.6133.1334.7743.8550.70 WUDI (λ = 1.0)38.3570.7246.3123.5550.0949.0154.5047.5117.168.686.574.646.628.86 OptMerge ∗ (λ = 1.0)39.3274.1448.0126.2551.9549.0156.0049.246.515.214.180.992.213.86 TA ∗ (λ = 0.2)13.1170.7247.1614.6750.4648.4542.0040.9493.2048.9634.9336.7647.0052.72 TIES (λ = 0.2)21.3654.3751.1411.9741.1942.5447.5038.5889.6543.7528.9631.7937.8546.96 ISO (λ = 0.2)21.8473.3849.1523.5547.6846.7649.0044.4890.5348.6134.0337.0942.2751.01 TSV ∗ (λ = 0.2)17.4868.8249.1513.5143.0441.9749.0040.4292.6048.9633.7334.1141.6450.76 WUDI (λ = 0.2)13.5975.2948.3021.6248.2450.1449.5043.8192.3149.6536.4238.0843.5352.53 OptMerge ∗ (λ = 0.2)17.9669.2048.5818.9245.8345.3547.0041.8391.7247.5731.9435.4342.2750.32 DCAT (Ours)37.3876.8151.7027.0352.6950.9954.5050.1593.5850.5037.7539.2547.9053.15 TempCompass - Caption MatchingTempCompass - Yes/NoOthers ActionAttr.Dir.OrderSpeedOverallActionAttr.Dir.OrderSpeedOverallEgoSchema LongVideo Bench Vicuna-based Original85.8659.3855.9649.6759.1161.8879.3962.7254.0553.6646.3760.0148.6039.04 TA67.0054.1754.4351.6747.0854.8981.0857.5948.5855.2453.8260.2548.8037.62 TIES86.8758.3349.5448.6752.9259.0878.7257.1453.8556.2857.3661.6038.2035.90 ISO80.4757.2952.6047.0055.3358.4277.5356.4749.8051.5750.8458.2147.2038.07 TSV82.4962.1556.5751.0056.0161.5480.5756.9249.6053.4050.6559.2351.2040.02 WUDI64.6555.2152.9149.6745.7053.6380.0758.0448.3854.7153.4559.8949.6038.30 OptMerge81.4857.9953.2146.3357.3959.1581.0861.3851.8255.5049.5360.7047.6040.46 DCAT (Ours)86.0860.6757.2351.2859.7762.6781.2463.5655.6156.2359.0263.1152.0540.54 LLaMA 3.1-based Original78.1132.9948.3241.0032.3046.7176.3550.4549.6047.3847.4955.4037.2040.17 TA ∗ (λ = 1.0)67.0044.7942.5147.6742.2748.7778.0455.3650.0050.7954.0058.7441.8040.31 TIES (λ = 1.0)58.5941.6741.2837.6738.8343.5873.4843.5347.1739.0126.6347.0948.4044.35 ISO (λ = 1.0)54.5529.1729.0532.0037.4636.3325.3429.9125.1010.4712.6621.0441.2045.25 TSV ∗ (λ = 1.0)82.4954.8657.1956.3354.3061.0179.5654.2448.7950.0051.9658.0946.6044.35 WUDI (λ = 1.0) 2.361.740.610.330.691.1329.3920.5420.246.026.8917.3734.8042.78 OptMerge ∗ (λ = 1.0)0.670.351.530.000.690.677.945.586.284.191.685.2237.2045.10 TA ∗ (λ = 0.2)49.4937.8536.0945.0046.0542.7853.5531.2537.4528.0130.5437.2249.2048.09 TIES (λ = 0.2) 53.2031.2532.7232.3342.2738.2657.9438.1745.7542.6749.7247.7044.6046.67 ISO (λ = 0.2)87.2156.9452.6056.0062.5462.8777.0351.7952.6347.3853.2657.6842.8046.30 TSV ∗ (λ = 0.2)75.7650.0044.3455.0059.1156.6272.9745.0949.3942.9348.2353.0446.8048.17 WUDI (λ = 0.2)93.9464.5856.8862.0063.2367.9377.8748.8849.3946.8652.3356.4247.4048.54 OptMerge ∗ (λ = 0.2) 93.6063.5456.8861.0067.3568.2674.6647.1050.0045.2951.7755.0847.0047.64 DCAT (Ours)94.2565.3658.1063.3568.1569.1478.3052.7054.0548.5054.2558.5449.8049.06 intrinsic alignment quality of the donor model, which is consistent with the conjecture discussed in Sec.6 of the main manuscript. F.5 Calibration-matched baselines We compare DCAT with LoRA fine-tuning (LoRA rank=256, batch size=8, 128 steps), RegMean [30], and RegMean++ [27] under a matched calibration budget of 256 AVQA samples. Tab. A9 report results on Vicuna-based models. DCAT outperforms the strongest calibration-aware baseline on both MMAU and Video-MME (by 1.9 and 1.3 points overall, respectively) across all categories in each benchmark, suggesting that DCAT’s gains arise from its alignment-transfer objective, which leverages the donor model to jointly improve subspace overlap, spectral diversity, and text-compatible directions. F.6 Calibration sample robustness To study the robustness of DCAT with respect to the calibration sample, we randomly sampled 256 AVQA training examples using 10 different random seeds and applied DCAT to both Audio+Vision and Video+Vision settings. The results in Tab. A10 show low overall standard deviations of 0.33 on MMAU and 0.14 on VideoMME across calibration subsets, demonstrating that DCAT is robust to calibration sampling and random seed within AVQA. 38 Table A7: Results on vision benchmarks (Vicuna-based). The performance of Vicuna based audio- vision(A+V) and video-vision(Vd+V) models are measured from five different benchmarks: GQA, MMBench-Dev(EN), MMStar, POPE, TextVQA-Val. GQAMMBenchMMStarPOPEText-VQAAverage Original61.9163.8335.4484.3147.7658.65 Merging MethodA+VVd+VA+VVd+VA+VVd+VA+VVd+VA+VVd+VA+VVd+V TA61.7658.0862.9760.4035.6234.6985.3484.8349.1045.3958.9656.68 TIES43.8953.0257.9959.3633.9833.6179.5685.0642.9946.2651.6855.46 ISO58.7557.6261.4360.7434.0732.4885.3182.5148.1546.8957.5456.05 TSV60.9260.7563.8363.6634.2633.5886.0285.5849.8647.8758.9858.29 WUDI61.6458.2863.0660.8235.3933.9785.3284.3648.8845.7158.8656.63 OptMerge58.0859.3462.7162.2034.0633.8286.0886.2049.3548.7158.0658.05 DCAT (Ours)61.4340.6263.7551.4633.5335.0685.1361.0849.3024.3458.6342.51 Table A8: Audio-video cross-modal transfer results. We report DCAT performance when transfer- ring alignment between audio and video models. The left table evaluates video-to-audio transfer on MMAU [69], while the right table evaluates audio-to-video transfer on Video-MME [16]. MMAU (Video-to-Audio) ModelMusic Sound Speech Overall Original 54.79 60.66 31.2348.90 DCAT57.49 66.07 37.2453.60 Video-MME (Audio-to-Video) ModelShort Medium Long Overall Original 39.8934.2230.89 35.00 DCAT37.7833.7829.33 33.63 F.7 Language ability on alignment-transferred models We evaluate general language ability after DCAT on 13 text benchmarks [5,13,14,21,32,49,55,57, 58,66,76,87,90] for Vicuna-based Audio+Vision and Video+Vision models in Tab. A11. McNemar’s test shows no significant change for Audio+Vision (p = 0.506) and a significant improvement for Video+Vision (p≪ 0.01), indicating no degradation. F.8 Additional ablation study In this section, in addition to the component-level ablation studies conducted in Tab. 3, we present ablation studies on key hyperparameters. Specifically, we study the balancing coefficientβin L cross (τ ), the balancing coefficientηin Eq. 10, the subspace dimensionkused to constructP k , and the size of the calibration dataset. All experiments are conducted with Vicuna-based models. For each hyperparameter, we select the sweep range of the parameters to the values where marked performance degradation is observed on target-modality benchmarks such as MMAU and Video-MME. The results of these additional ablation studies are reported in Tab. A12. We observe that appropriate choices ofβandηare important for achieving strong performance. In contrast, DCAT maintains robust performance with respect to the subspace dimensionkand the number of calibration samples. These experiments provide empirical rationale for our hyperparameter choices. Table A13: Summary of benchmarks used in our evaluation. Each benchmark’s question type, example prompt, number of questions, and evaluation method are listed. For benchmarks’s prompt where we modified the default lmms-eval prompt, both the default and custom prompts are provided. ModalityBenchmarkQuestion Type Example Prompt# QEvaluation Method Audio MMAUMCQ"Based on the given audio, identify the source of the speaking voice. A. Man B. Woman C. Child D. Robot Answer with the option’s letter from the given choices directly." 1000rule and based parsing and matching continued on next page 39 Table A13 – continued from previous page ModalityBenchmark Question Type Example Prompt# Q Evaluation Method AIR-Bench- Foundation MCQ"In terms of mood, what does this music suggest? A. party B. emotional C. soundscape D. trailer Answer with the option’s letter from the given choices directly." 24683 rule and based parsing and matching MuChoMusicMCQ"Which instrument initiates the piece? (A) Synthesizer (B) Rueful tune (C) Acoustic guitar (D) Vocals Answer with the option’s letter from the given choices directly: " 1187 rule-based parsing and matching Clotho-AQA (filtered binary subset) binary QA "Are people talking? Answer the question using a single word only. " 1442 exact match VocalSound6-class classifi- cation *We use the custom prompt for both Vicuna and LLaMA evaluation. Default: "Classify the human vocal sound to VocalSound in English: " Custom: "What type of vocal sound is in the audio? Answer with exactly one of: Laughter, Sniff, Throat clearing, Cough, Sigh, Sneeze." 3591 rule-based parsing and matching Video VideoMMEMCQ*We use the custom prompt for Vicuna evaluation. Default: "Select the best answer to the following multiple-choice question based on the video and the subtitles. Respond with only the letter (A, B, C, or D) of the correct option. How many national flags appear in the video? A. 3. B. 4. C. 2. D. 5. Answer with the option’s letter from the given choices directly." Custom: "Select the best answer to the following multiple-choice question based on the video. Respond with only the letter (A, B, C, or D) of the correct option. How many national flags appear in the video? A. 3. B. 4. C. 2. D. 5. The best answer is:" 2700 rule-based parsing and matching continued on next page 40 Table A13 – continued from previous page ModalityBenchmark Question Type Example Prompt# Q Evaluation Method TempCompass (multi-choice) MCQ*We use the custom prompt for Vicuna evaluation. Default: "What is the man doing in the video? A. dunking a basketball B. dribbling a basketball C. passing a basketball Please directly give the best option:" Custom: "What is the man doing in the video? A. dunking a basketball B. dribbling a basketball C. passing a basketball The best answer is:" 1580 rule-based parsing and matching / calls gpt-3.5-turbo- 1106 [6] for judging on rule failure TempCompass (caption- matching) MCQ*We use the custom prompt for Vicuna evaluation. Default: "Which description is a more suitable match for the video? Option 1: The man is dribbling a basketball. Option 2: A man is dunking a basketball. Please directly give the best option:" Custom: "Which description is a more suitable match for the video? Option 1: The man is dribbling a basketball. Option 2: A man is dunking a basketball. The best answer is:" 1503 rule-based parsing and matching / calls gpt-3.5- turbo-1106 for judging on rule failure TempCompass (yes/no) binary QA *We use the custom prompt for Vicuna evaluation. Default: "Is the man dunking? Please answer yes or no:" Custom: "Is the man dunking? Please answer yes or no:" 2453 rule-based parsing and matching / calls gpt-3.5- turbo-1106 for judging on rule failure MVBenchMCQ*We use the custom prompt for Vicuna evaluation. Default: "Question:What is the action performed by the person in the video? Option: (A) Not sure (B) Scattering something down (C) Piling something up Only give the best option. " Custom: "Question:What is the action performed by the person in the video? Option: (A) Not sure (B) Scattering something down (C) Piling something up The best answer is:" 4000rule and based parsing and matching continued on next page 41 Table A13 – continued from previous page ModalityBenchmark Question Type Example Prompt# Q Evaluation Method MLVUMCQ"What color is the water on the tropical beach in the video? (A) Dark blue (B) Clear (C) Green (D) Azure blue Only give the best option. Best option: (" 2174 rule-based parsing and matching * uses per- task-type accuracy, not overall accuracy EgoSchema (subset) MCQ"Taking into account all the actions performed by c, what can you deduce about the primary objective and focus within the video content? A. C is cooking. B. C is doing laundry. C. C is cleaning the kitchen. D. C is cleaning dishes. E. C is cleaning the bathroom. Answer with the option’s letter from the given choices directly." 500 rule-based parsing and matching LongVideo- Bench (Val) MCQ"In front of an ancient building, a woman walks by holding something. When the subtitle says ’temples scattered throughout the’, what color is the woman’s clothing at that moment? A. yellow B. green C. blue D. red Answer with the option’s letter from the given choices directly. " 1337 rule-based parsing and matching Vision GQAFree- form short- answer "Who is wearing the dress? Answer the question using a single word or phrase." 12578 exact match MMBench (Dev_EN) MCQ"Identify the emotion expressed in this image. There are several options: A. happiness B. sadness C. anger D. loneliness Answer with the option’s letter from the given choices directly." 1164 rule-based parsing and matching / calls gpt-4o-2024- 11-20 [26] for parsing if rule-based parsing fails *adapts circular evaluation MMStarMCQ"What seems to be the theme of the image? Options: A: Hanging Posters, B: Music, C: Home decoration, D: Playing guitar Answer with the option’s letter from the given choices directly" 1500 rule-based parsing and matching * uses per- task-type accuracy, not overall accuracy POPEbinary QA "Is there a snowboard in the image? Answer the question using a single word or phrase." 9000 exact match continued on next page 42 Table A13 – continued from previous page ModalityBenchmark Question Type Example Prompt# Q Evaluation Method TextVQA (Val) Free- form short- answer "What is the brand of this camera? Answer the question using a single word or phrase." 5000 VQA soft accuracy F.9 Efficiency study All efficiency numbers are measured on a single NVIDIA L40S 48GB GPU with the same numerical precision as the original checkpoints. For each method and recipient setting, we time only the core merging algorithm, excluding per-checkpoint adapter loading and saving so that the reported cost reflects merging rather than I/O. All methods use the merging hyperparameters from their respective references (300Adam iterations for WUDI and300SGD iterations for OptMerge) from the source of OptMerge [75]. Tab. A14 shows that DCAT remains practical despite using calibration activations. For the LLaMA 3.1-based settings, DCAT has a merging time comparable to stronger baselines such as TSV and OptMerge:28.45/34.57minutes for LLaMA-Audio/Video, compared with25.61/33.02for TSV and 29.96/35.55for OptMerge. Its peak memory is also comparable, requiring16.72GB mainly to store layer-wise outputs for a small number of calibration samples, while TSV and OptMerge use12.12GB and18.66GB. For Vicuna, DCAT is more expensive than lightweight baselines because it merges the full weights after merging LoRA adapters, making its cost closer to the full-weight LLaMA setting. Even so, DCAT finishes within about24–25minutes and uses14.56GB on a single L40S GPU, indicating that its one-time merging cost remains highly feasible. Audio-Vision merged Vision benchmarkVision benchmark Video-Vision merged Figure A8: Failure case on alignment–performance correlation. Although our experiments across hundreds of model weights and diverse benchmarks demonstrate a strong correlation between the alignment metric and performance, we observe a small number of benchmarks for which this correlation is not pronounced. These cases suggest that further study may be needed to explain the underlying factors driving performance improvements. G Limitation First, although our analyses across hundreds of model checkpoints and diverse benchmarks reveal a strong and statistically meaningful correlation between the proposed Alignment score and down- stream performance, this relationship is not universal across all evaluation settings. As shown in Fig. A8, a small number of benchmarks exhibit weak or negligible correlation, suggesting that factors beyond modality–text alignment may dominate performance in certain tasks. For example, some benchmarks may depend more heavily on modality-specific perception, reasoning format, dataset bias, or evaluation protocol than on the alignment properties captured by our score. Therefore, our Alignment score should be interpreted as a useful and broadly predictive indicator for cross-modal transfer, rather than as a complete explanation of all performance variations. Understanding when alignment is insufficient, and identifying complementary factors that better explain such exceptional cases, is an important direction for future work. Second, DCAT assumes that the models to be 43 Table A9: Calibration-matched comparison. We compare DCAT with LoRA fine-tuning, Reg- Mean [30], and RegMean++ [27] under a matched calibration budget of 256 AVQA samples. MMAU is evaluated with Audio+Vision Vicuna-based models, and Video-MME with Video+Vision Vicuna- based models. LoRA is trained with rank 256, batch size 8, and 128 steps. The best results are in bold, and the second-best are underlined. Method MMAUVideo-MME MusicSoundSpeechOverallShortMediumLongOverall Original54.860.731.248.939.934.230.935.0 LoRA fine-tuned53.662.233.049.644.235.731.337.1 RegMean [30]58.566.537.954.345.838.434.839.7 RegMean++ [27]58.866.535.553.646.939.334.640.3 DCAT (ours)60.269.141.756.248.040.536.241.6 Table A10: Robustness to calibration sampling. We randomly sample 256 AVQA training examples with 10 different random seeds and apply DCAT to the Audio+Vision (MMAU) and Video+Vision (Video-MME) settings. We report per-seed results, together with the mean and standard deviation over the 10 calibration subsets. Seed MMAUVideo-MME MusicSoundSpeechOverallShortMediumLongOverall 059.3868.5740.0456.0048.6440.5335.5341.57 159.0868.2739.7455.7048.7639.9835.5341.42 260.8867.6739.7456.1048.0940.3135.7641.39 359.6866.4740.3455.5047.6441.0935.3141.35 457.8967.6740.3455.3048.7640.9835.2041.64 5 57.8968.8741.5456.1047.8741.0935.0941.35 659.6868.2740.3456.1047.5341.0935.0941.24 759.6867.6740.9456.1047.9840.6435.6441.42 858.1967.9740.3455.5048.2040.2035.2041.20 960.2867.6740.6456.2048.3140.3135.0941.24 Mean59.2667.9140.4055.8648.1840.6235.3441.38 Std.1.010.660.540.330.440.420.250.14 merged have compatible parameter shapes, as it operates directly in weight space. This assumption is shared by many training-free merging methods, but it limits applicability to in-the-wild MLLMs whose language backbones differ in tokenizer size, embedding matrix size, LM head size, or other architectural components. Such layers are particularly important in LLMs, since they directly de- termine token representation and generation behavior; naïve resizing, padding, or partial copying can severely degrade model performance and would also introduce confounding factors beyond the merging method itself. For this reason, we restrict our experiments to models with shape-compatible language-model parameters and do not apply additional retraining or ad-hoc architectural surgery to force compatibility. Extending directional cross-modal alignment transfer to models with different vocabularies, embedding spaces, or heterogeneous backbone shapes would substantially broaden the coverage of DCAT and represents a promising future research direction. H Broader Impacts This work proposes an efficient model-merging framework that transfers alignment across MLLMs trained for different modalities, improving target-modality performance without additional large-scale training. This may have positive societal impact by reducing the data and compute costs required to develop capable multimodal systems, especially for data-scarce, high-volume, or domain-specific modalities such as audio, video, medical, or scientific data. As a result, the approach could improve accessibility for researchers and organizations with limited computational resources. However, stronger audio and video understanding capabilities may also be misused for surveillance, sensitive attribute inference, large-scale profiling, or downstream systems that make unfair or unsafe decisions. In addition, cross-modality alignment transfer may propagate or amplify biases, privacy risks, or 44 Table A11: Language ability before and after DCAT. We evaluate general language ability on 13 text benchmarks for the Vicuna-based Audio+Vision and Video+Vision models before and after applying DCAT.∆denotes After−Before. McNemar’s test shows no significant change for Audio+Vision (p = 0.506) and a significant improvement for Video+Vision (p≪ 0.01). Task Audio+VisionVideo+Vision BeforeAfter∆BeforeAfter∆ MMLU0.4920.491 −0.0010.5010.510+0.009 ARC0.5500.514 −0.0360.5060.521+0.015 HellaSwag0.7630.745 −0.0180.7770.775 −0.002 WinoGrande0.7310.710 −0.0210.7220.7220.000 PIQA0.7870.784 −0.0030.7780.786+0.008 BoolQ0.8210.841+0.0200.8370.844+0.007 SciQ0.8900.900+0.0100.8860.902+0.016 OpenBookQA0.4460.454+0.0080.4500.462+0.012 CommonsenseQA0.5670.699+0.1320.6510.708+0.057 Social IQa 0.5060.500 −0.0060.4790.491+0.012 RACE0.4260.435+0.0090.4260.434+0.008 CoPA0.8700.860 −0.0100.8700.860 −0.010 IFEval0.3140.279 −0.0350.3070.311+0.004 Mean0.6280.632+0.0040.6300.641+0.011 Table A12: Hyperparameter study on DCAT. We conduct a hyperparameter study for DCAT. We evaluate a range of options forβ,η,k, and the calibration sample size, and provide the rationale behind our hyperparameter choices. Study subject MMAUVideo-MME MusicSoundSpeechOverallShortMediumLongOverall Original54.7960.6631.2348.9038.3333.6730.6734.22 Base (DCAT)60.2269.0641.7456.2046.3339.6734.3340.11 β = 0.1 in Eq. 650.9060.6634.5348.7037.3329.0026.3330.89 β = 0.2 in Eq. 661.0866.9739.6455.1049.3336.6734.6740.22 β = 0.4 in Eq. 660.1867.2739.3454.8049.3337.0037.6741.33 β = 0.5 in Eq. 658.9866.6739.6454.3049.6737.0038.3341.67 β = 0.6 in Eq. 658.6865.4735.4452.4048.3337.6735.0040.33 β = 1.0 in Eq. 651.5058.5633.6347.9033.0026.3327.3328.89 η = 10 3 in Eq. 1037.7244.4425.2335.801.331.671.671.56 η = 10 4 in Eq. 1056.2966.6737.5453.5046.6734.3334.6738.56 η = 5× 10 4 in Eq. 1058.3866.3739.0454.6049.6738.0036.0041.22 η = 5× 10 5 in Eq. 1057.4966.6738.7454.3050.6737.3334.3340.78 η = 10 6 in Eq. 1058.8966.6738.1453.9050.0037.0035.3340.78 η = 10 7 in Eq. 1056.8966.6733.3352.3045.6737.3334.6739.22 k = 3256.8966.3733.3352.2041.0034.6730.0035.22 k = 64 58.6866.6739.3454.9049.3337.6735.6740.89 k = 25658.0866.3737.2453.9049.6737.0037.3341.33 k = 51258.6866.9733.6353.1044.3337.6734.6738.89 #calib. samples = 6455.9965.1735.4452.2040.3338.0032.0036.78 #calib. samples = 102456.8967.5734.8353.1044.3340.0032.6739.00 failure modes inherited from the donor model, and incorrect multimodal understanding could cause harm when used in high-stakes applications. To mitigate these risks, practical deployments should include careful evaluation across demographic groups and domains, privacy-preserving calibration data, safety testing before and after merging, restrictions on sensitive use cases, and human oversight in high-risk settings. Although this work is primarily foundational, its direct path to improving multimodal capability warrants consideration of these potential impacts. 45 Table A14: Efficiency study of model merging methods. We compare the computational cost of DCAT with existing merging baselines across four recipient settings: Vicuna-Audio, Vicuna-Video, LLaMA-Audio, and LLaMA-Video. We report the wall-clock merging time and peak GPU memory measured during the merging procedure. All results are measured on the same hardware and with the same precision. Method Merging Time (min.)↓Peak GPU Memory (GB)↓ Vicuna-Audio Vicuna-Video LLaMA-Audio LLaMA-VideoVicuna-Audio Vicuna-Video LLaMA-Audio LLaMA-Video TA0.140.450.740.930.000.002.942.94 TIES1.180.511.811.640.000.0022.5122.51 ISO0.580.538.9612.030.040.048.098.09 TSV 0.680.6925.6133.020.050.0512.1212.12 WUDI 5.765.7523.2023.202.292.294.894.89 OptMerge5.865.7929.9635.552.332.3318.6618.66 DCAT23.6425.3928.4534.5714.5614.5616.7216.72 46 NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We identify a phenomenon in which merging MLLMs trained on different modalities using existing model-merging methods can substantially improve performance on specific target modalities. We theoretically characterize this phenomenon and leverage it to propose a method for improving the performance of modality-specific models in settings where both models and data are scarce. These contributions are clearly presented in the abstract and introduction. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We’ve included limitations in Sec. G. Guidelines: •The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. •The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs 47 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We present Theorem 1 and provide detailed proofs of the theorem and its accompanying propositions in Appendix Sec. B. We also explicitly state the Gaussian assumption required for the theorem. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. •All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced. •All assumptions should be clearly stated or referenced in the statement of any theorems. •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We describe the experimental settings and implementation details in the main manuscript Sec. 5.1. To ensure fair comparison, we report the hyperparameter values used for the baselines in Sec. 5.2. Additional details are provided in Appendix Sec. D. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. •Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 48 (d)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All datasets used for our experiments and evaluation are open-source and publicly accessible, and we will release our code upon acceptance. In addition, reviewers are provided with instructions to reproduce the main experimental results in theREADME.md file included with the supplementary code. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). •The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.c/public/guides/CodeSubmissionPolicy) for more details. •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. •At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). •Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: We provide detailed implementation and experimental details, including hyper- parameters, the models used, and the analysis procedures, in the main manuscript Sec. 5.1, Appendix Sec. C, and Sec. D. Guidelines: • The answer [N/A] means that the paper does not include experiments. •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] 49 Justification: We report Pearson and Spearman correlation coefficients with their correspond- ingp-values in Fig. 2. We also include standard deviations in Fig. 3 and Fig. 4. In Appendix Sec. F.1, we conduct statistical significance tests for performance changes across all results presented in the main manuscript and report the complete results. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. •The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). •It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We report the compute resources used in our experiments in Appendix Sec. C.1. The computational costs of our method and the baselines are provided in Appendix Sec. F.9. Guidelines: • The answer [N/A] means that the paper does not include experiments. •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. •The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: The authors have reviewed the NeurIPS Code of Ethics. Guidelines: •The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. 50 •The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We provide broader impacts in Appendix Sec. H. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. •If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. •Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. •If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: Our experiments are conducted using existing pretrained models and do not aim to introduce new capabilities. Therefore, safeguard design is outside the scope of this work, and we rely on the safeguards already incorporated in the original models. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. •Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 51 Answer: [Yes] Justification: We appropriately cite the papers from which the models were obtained, provide citations for all baselines, and credit in-the-wild models by including direct links to their sources. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. •For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. •If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets,paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. •For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide details of the provided code in supplementary README.md. Guidelines: • The answer [N/A] means that the paper does not release new assets. •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. •The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. •Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper. •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects 52 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. •Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. •For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [Yes] Justification: Our experiments include some benchmarks that use an LLM-as-a-judge and parsing. Guidelines: •The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described. 53

Instant research discovery