Paper deep dive
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai, Dongkai Wang, Maolin Wang, Mingjie Wang, Jianzhuang Liu, He Yan, Zhao Kang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/11/2026, 2:18:38 AM
Summary
The paper introduces HRIL (Higher-order Representation and Information Learning), a framework for self-supervised multimodal representation learning that explicitly captures synergistic information. Unlike traditional pairwise contrastive methods that focus on shared information, HRIL models higher-order statistical dependencies among modalities using an empirical cross-moment tensor and Tucker decomposition. A synergy-aware regularizer is applied to the core tensor to prevent energy concentration, thereby preserving the capacity to capture multi-way interactions essential for tasks where synergy is dominant.
Entities (10)
Relation Signals (7)
HRIL → includes → Synergy-Aware Regularizer
confidence 95% · complemented by a synergy-aware regularizer that prevents energy concentration
HRIL → uses → Tucker Decomposition
confidence 95% · HRIL employs Tucker decomposition to obtain a core tensor
HRIL → uses → Cross-Moment Tensor
confidence 95% · constructs an empirical cross-moment tensor over modality embeddings
Partial Information Decomposition → decomposes → Synergy
confidence 90% · PID ... decomposes task-relevant information into redundancy, uniqueness, and synergy
Synergy → modeledby → Higher-Order Statistical Dependence
confidence 90% · synergistic information is reflected in higher-order statistical dependence among modalities
HRIL → outperforms → CLIP
confidence 80% · demonstrate consistent improvements over existing multimodal contrastive methods
HRIL → outperforms → CoMM
confidence 80% · demonstrate consistent improvements over existing multimodal contrastive methods
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.12393v1
- Canonical: https://arxiv.org/abs/2610.12393v1
Trouble viewing inline? Open PDF directly →
Full Text
129,479 characters extracted from source content.
Expand or collapse full text
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling Qun Dai Affiliation: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics Email: daiqun1124@gmail.com Liangjian Wen Affiliation: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics Email: wlj6816@gmail.com Jiang Duan Affiliation: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics Affiliation: Chengdu Everimaging Science and Technology Co., Ltd Yong Dai Affiliation: X-Humanoid Dongkai Wang Affiliation: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics Maolin Wang Affiliation: Hong Kong Institute of AI for Science, City University of Hong Kong Mingjie Wang Affiliation: Zhejiang Sci-Tech University Jianzhuang Liu Affiliation: Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences HE YAN Affiliation: Chengdu Everimaging Science and Technology Co., Ltd Zhao Kang Affiliation: University of Electronic Science and Technology of China Affiliation: Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province Abstract Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL. 11footnotetext: Corresponding author. 1 Introduction Self-supervised multimodal representation learning constructs joint representations from heterogeneous data sources whose underlying statistical structures differ fundamentally. Since distinct modalities arise from disparate data-generating processes and reside in qualitatively different statistical spaces, the central challenge is to leverage complementary and heterogeneous features across modalities for representations that are maximally informative for downstream tasks. Foundational pairwise contrastive methods such as CLIP [36] and ALIGN [18] have achieved remarkable transferability across vision, language, and audio domains, yet they operate under the multiview redundancy assumption [41, 38, 48], which posits that task-relevant information is largely shared across modalities. When we examine multimodal interactions across two or more modalities through the lens of Partial Information Decomposition (PID) [50, 4] — which decomposes task-relevant information into redundancy (shared across modalities), uniqueness (exclusive to individual modalities), and synergy (accessible only via joint observation of multiple modalities) — the limitations of redundancy-centric pairwise contrastive learning become apparent. For instance, in the M-IMDb dataset [2], accurate genre prediction necessitates the synergistic integration of textual plot descriptions and visual poster elements, as the genre semantics emerge from their cross-modal combination rather than being fully encapsulated in either modality alone. Consequently, synergy is of fundamental importance in multimodal representation learning, as it embodies the irreducible cross-modal dependency that justifies multimodal representation learning and cannot be inferred from any single modality [11, 46]. A natural question is whether stronger pairwise contrastive learning can capture synergy. [11] demonstrates that cross-modal contrastive learning under the multiview redundancy assumption primarily captures shared information in bimodal settings. In this work, we further prove that a representation which is sufficient for all pairwise contrastive learning yet minimal in information content provably discards synergistic information accessible only through the joint observation of all modalities in scenarios with more than two modalities, because such information is not required for pairwise sufficiency, building on [43]. Several recent methods have moved beyond pairwise formulations. CoMM [11] allows synergy to emerge implicitly through augmented fusion, while Symile [37] and ConFu [23] target higher-order dependence via total correlation or subset fusion. However, these approaches either lack explicit guarantees for synergy preservation or model higher-order dependence without formally connecting it to task-relevant synergistic information. To bridge this gap, we propose Higher-order Representation and Information Learning (HRIL), a synergy-aware fusion framework for multimodal contrastive representation learning. Guided by our analysis linking synergistic information to higher-order statistical dependence, HRIL models multi-way interactions explicitly by constructing an empirical cross-moment tensor over modality embeddings. HRIL then applies Tucker decomposition to obtain a compact interaction core and introduces a synergy-aware regularizer that prevents degenerate interaction structures and preserves diverse higher-order interaction patterns during training. Figure 1: PID decomposes joint mutual information I(X1,X2,Y)I(X_1,X_2;Y) into shared R, unique U1/U2U_1/U_2, and synergistic S. Our main contributions are threefold. (1) We formally establish the insufficiency of pairwise contrastive learning for synergy recovery among more than two modalities and identify the pure synergistic information as a conditional total-correlation difference, providing a precise information-theoretic criterion for when higher-order modeling is necessary. (2) We propose HRIL, which models higher-order interactions via empirical cross-moment tensorization and Tucker decomposition, coupled with a synergy-aware regularizer that penalizes excessive energy concentration in the Tucker core to preserve capacity for multi-way interactions. (3) We conduct extensive experiments on real-world benchmarks spanning healthcare and affective computing — which contain diverse modalities (e.g., image, text, audio, etc.) — complemented by controlled synthetic evaluations. Our results demonstrate consistent improvements over state-of-the-art methods, with pronounced gains on synergy-heavy tasks. 2 Theoretical Analysis In this section, we first formalize a latent-factor framework to demonstrate the insufficiency of pairwise contrastive learning for capturing synergy in scenarios involving more than two modalities, and then quantify the synergy gap via conditional total correlation, deriving an information-theoretic criterion for the necessity of higher-order modeling. In our theoretical analysis, we focus on the case of more than two modalities. Our analysis of pairwise contrastive learning focuses on its minimal sufficient representations. All proofs are provided in Appendix G. 2.1 A Latent-Factor Framework for Multimodal Representations Task setup. Given X=X1,X2,…,XMX=\X_1,X_2,…,X_M\ with M modalities and a downstream task label Y, our objective is to learn a multimodal representation =f(X) Z=f(X) that is informative for Y. Throughout the paper, we use X1:M:=(X1,…,XM)X_1:M:=(X_1,…,X_M) and XA:=Xi:i∈AX_A:=\X_i:i∈ A\ for any index set A⊆1,…,MA \1,…,M\. Partial Information Decomposition (PID) [50, 4] conceptually decomposes task-relevant information into redundancy, uniqueness, and synergy (see Fig. 1 for a visual depiction). Motivated by this view, we adopt the latent-factor framework below. Assumption 1 (Tripartite Latent Factor Decomposition). We assume the existence of latent factors r,1,…,M,syn\ q_r, q_1,…, q_M, q_syn\, and all task-relevant information is mediated by these latent factors, i.e., Y⟂X1:M∣(r,1,…,M,syn),Y X_1:M ( q_r, q_1,…, q_M, q_syn), where r q_r is shared factors across modalities, m q_m denotes the m-th modality-specific factor, i.e., X−m⟂m|(r,−m,syn)X_-m q_m ( q_r, q_-m, q_syn), where X−m=X1,…,M∖mX_-m=X_\1,…,M\ \m\. And syn q_syn denotes synergistic factors that capture task-relevant information emerging only from joint observation. These latent factors are mutually independent. Throughout, ⟂ denotes conditional independence. Based on the above, we further state that (r,1:M,syn)( q_r, q_1:M, q_syn) represents a decomposition of the task-relevant information contained in the complete multimodal observation X1:MX_1:M. Accordingly, once X1:MX_1:M is given, (r,1:M,syn)( q_r, q_1:M, q_syn) provides no further information about Y, which is expressed as Y⟂(r,1:M,syn)∣X1:M.Y ( q_r, q_1:M, q_syn) X_1:M. Among these latent factors, syn q_syn is substantially harder to capture because it corresponds to task-relevant dependencies that emerge only through joint multimodal observation and therefore requires a more precise characterization [11, 46]. We formalize this accessibility constraint as follows. Definition 1 (Pure Synergistic Factor). We call syn q_syn a pure synergistic factor relative to r q_r if I(syn;X1:M∣r)>0,andI(syn;XA∣r)=0,∀A⊊1,…,M.I( q_syn;X_1:M q_r)>0, I( q_syn;X_A q_r)=0,\ ∀\,A \1,…,M\. Under Assumption 1, the observations are generated as X1:M∼p(X1:M∣r,1:M,syn)X_1:M p(X_1:M q_r, q_1:M, q_syn), where each unique factor m q_m influences only XmX_m. Motivated by this latent-factor framework, we interpret the learned multimodal representation Z as being organized into three functional subspaces: ≈Vrfr(r)+Vufu(1,2,…,M)+Vsynfsyn(syn)+ϵ, Z≈ V^rf_r(q_r)+V^uf_u(q_1,q_2,…,q_M)+V^synf_syn(q_syn)+ε, (1) where VrV^r spans a shared subspace, VuV^u captures modality-specific structures, VsynV^syn denotes a synergistic subspace, and ϵε is a noise term. 2.2 Minimal Sufficient Representation Analysis Let pairmin Z_pair denote a representation that is sufficient for all pairwise inter-modal alignments and minimal in the sense of minimizing I(;X1:M)I( Z;X_1:M) over the class of such representations. Inspired by [43], we adopt the following shared information characterization for our analysis: I(pairmin;X1:M∣r)=0,I( Z_pair ;X_1:M q_r)=0, (2) meaning that pairmin Z_pair retains only information captured by the shared factor r q_r. Prior work [40, 43] has shown that minimal sufficient representations in pairwise contrastive learning can be inadequate for downstream tasks. Here, we sharpen this limitation within this latent-factor framework (Assumption 1), isolating the synergistic component excluded by pairwise minimal sufficiency. Theorem 1 (Insufficiency of Pairwise Minimal Sufficient Representation). Under Assumption 1, suppose that syn q_syn is a pure synergistic factor in the sense of Definition 1 and that I(Y;syn∣r,1:M)>0I(Y; q_syn q_r, q_1:M)>0. Then I(Y,pairmin)≤I(Y,r).I(Y; Z_pair )≤ I(Y; q_r). Moreover, the total information gap is lower bounded as I(Y;X1:M)−I(Y;pairmin)≥I(Y;1:M∣r)⏟unique gap+I(Y;syn∣r,1:M)⏟synergistic gap.I(Y;X_1:M)-I(Y; Z_pair )\;≥\; I(Y; q_1:M q_r)_unique gap\;+\; I(Y; q_syn q_r, q_1:M)_synergistic gap. Theorem 1 reveals that pairwise contrastive learning even after recovers all shared and unique factors (r,1:M)( q_r, q_1:M), an irreducible synergistic gap I(Y;syn∣r,1:M)>0I(Y; q_syn q_r, q_1:M)>0 persists and can only be closed by information about syn q_syn. Inspired by [23], we turn to total correlation [45] as a measure of higher-order dependence to characterize pure synergistic information. Specifically, total correlation characterizes the overall statistical coupling among multiple random variables by the Kullback–Leibler divergence between their joint distribution and the product of their marginals. The conditional difference below makes its connection to pure synergy explicit. Proposition 1 (Conditional Total Correlation Decomposition). For M modalities, the conditional total correlation admits the chain-rule decomposition TC(X1:M∣r)=∑k=2MI(Xk;X1,…,Xk−1∣r).TC(X_1:M q_r)= _k=2^MI(X_k;\,X_1,…,X_k-1 q_r). In particular, for M=3M=3: TC(X1,X2,X3∣r)=I(X1;X2∣r)+I(X3;X1,X2∣r)TC(X_1,X_2,X_3 q_r)=I(X_1;X_2 q_r)+I(X_3;X_1,X_2 q_r), and symmetrically for all permutations. Building on this decomposition, we now establish the precise quantitative link between synergistic information and higher-order statistical dependence. Theorem 2 (Synergy Total Correlation Identity). Under Definition 1, the synergistic information satisfies the identity I(syn;X1:M∣r)=TC(X1:M∣r,syn)−TC(X1:M∣r).I( q_syn;X_1:M q_r)=TC(X_1:M q_r, q_syn)-TC(X_1:M q_r). Interpretation. We unpack Theorem 2 in the following steps. (1) Synergy changes joint dependence without altering single-modality marginals. By Definition 1, conditioning on syn q_syn leaves each marginal entropy H(Xm∣r)H(X_m q_r) unchanged, but reduces the joint entropy H(X1:M∣r)H(X_1:M q_r). Since TC=∑mH(Xm∣⋅)−H(X1:M∣⋅)TC= _mH(X_m ·)-H(X_1:M ·), this reduction increases TC exactly by I(syn;X1:M∣r)I( q_syn;X_1:M q_r). (2) The resulting dependence is higher-order. Conditional on syn q_syn also preserves every pairwise marginal distribution given r q_r; hence the TC difference in Theorem 2 can only be attributed to irreducible multi-way dependence, which pairwise objectives cannot access. 3 Method Figure 2: The overall pipeline of HRIL. Given M modalities, we apply data augmentation to construct two views X′X and X′X , and encode them into unimodal representations m′m=1M\ Z_m \_m=1^M and m′m=1M\ Z_m \_m=1^M, as well as fused representations f′ Z_f and f′ Z_f . The contrastive objective ℒInfoNCEL_InfoNCE performs cross-view alignment between each unimodal representation and the fused one, while ℒsynL_syn and ℒalignL_align are computed from unimodal representations in both views. The overall pipeline of HRIL is illustrated in Fig. 2. Let v(m)=fm(Xv(m)) h_v^(m)=f_m(X_v^(m)) denote the unimodal encoder features, a shared Transformer g produces v,u(m)=g(v(m)) Z_v,u^(m)=g( h_v^(m)) and v,f=g(v(1),…,v(M)) Z_v,f=g( h_v^(1),…, h_v^(M)), where v∈1,2v∈\1,2\ indexes the augmented views. Given the encoded modality embeddings, HRIL optimizes three complementary components. First, a modality-fusion contrastive objective aligns each unimodal representation with the fused representation to extract cross-modal task-relevant signals. Second, the synergy-aware regularization preserves the capacity of the fused representation to encode higher-order synergistic interactions. Third, an auxiliary cross-view alignment objective stabilizes the shared and modality-specific subspaces while preventing representational collapse. 3.1 Higher-Order Dependence Modeling for the Synergistic Subspace Theorems 1 and 2 isolate purely cross-modal, higher-order dependence as the sole information-theoretic carrier of synergy. To translate this criterion into a practical objective, we require a mechanism to distinguish genuine multi-way interactions in the embedding space. Inspired by [1, 21, 53], we employ the empirical cross-moment tensor that aggregates cross-modal dependencies across all interaction orders. Here ⊗ denotes the vector outer product and ×m×_m denotes multiplication along tensor mode m. For a fixed augmented view, given a mini-batch of B samples with modality embeddings m(b)∈ℝdm Z_m^(b) ^d_m, the M-th order empirical cross-moment tensor T is defined as :=1B∑b=1B1(b)⊗2(b)⊗⋯⊗M(b)∈ℝd1×d2×⋯×dM,T:= 1B _b=1^B Z_1^(b) Z_2^(b) ·s Z_M^(b)\;∈\;R^d_1× d_2×·s× d_M, (3) whose (i1,…,iM)(i_1,…,i_M)-th entry estimates the mixed moment [Z1,i1⋯ZM,iM]E[Z_1,i_1·s Z_M,i_M]. Its Tucker decomposition [21] ≈×11⋯×MT ×_1A_1·s×_MA_M, where m∈ℝdm×rmA_m ^d_m× r_m are factor matrices, isolates the core tensor G whose multilinear rank characterizes the multilinear complexity of the modeled cross-moment tensor. If the modalities interact only through a low-dimensional shared factor, T inherits a low multilinear rank and its energy concentrates into a few separable components; preserving capacity for higher-order dependencies motivates discouraging excessive energy concentration in G. This suggests a natural regularization strategy: penalize energy concentration in G to preserve representational capacity for higher-order interactions, and thus for synergy. The following definition and lemmas formalize this diagnostic. Definition 2 (Redundant Low-Order Structure). Multimodal embeddings ii=1M\ Z_i\_i=1^M are said to exhibit a redundant low-order structure if i≈∑j≠ijj Z_i≈ _j≠ iW_ij Z_j with rank(ij)=rij≪djrank(W_ij)=r_ij d_j, where the approximation holds in the population sense: the reconstruction residual i≔i−∑j≠ijj δ_i Z_i- _j≠ iW_ij Z_j satisfies [‖i‖2]→0E[\| δ_i\|^2]→ 0 under the shared-factor dominance regime. Under Definition 2, when the shared component dominates the embedding energy, the low-rank predictor ijW_ij explains nearly all of i Z_i, and the embeddings collapse into the redundant low-order regime. We formally state this result in Proposition 2. The following two lemmas connect redundant low-order structure to multilinear complexity and characterize single-component degeneration of the interaction core. Lemma 1 (Redundancy Implies Low Multilinear Complexity). If the embeddings exhibit a redundant low-order structure in the sense of Definition 2, then the cross-moment tensor =[⨂m]T=E[ _m Z_m] admits a low multilinear-complexity approximation. In particular, the Tucker core G of T has at least one mode whose effective multilinear rank satisfies: rank(k)()≤∑j≠irijfor modality k=i.rank_(k)(G)≤ _j≠ ir_ij modality k=i. Lemma 2 (Energy Concentration Indicates Degenerate Structure). Let ρ():=maxi1,…,iMi1,…,iM2‖F2.ρ(G):= _i_1,…,i_M G_i_1,…,i_M^2\|G\|_F^2. If ρ()≥1−δρ(G)≥ 1-δ for some δ∈[0,1]δ∈[0,1], then G is within relative Frobenius error at most δ δ of a one-sparse core tensor. For small δ, this indicates dominance by a single separable component in the Tucker core. Definition 2, Lemma 1, and Lemma 2 connect information theory and representation geometry: when shared factors dominate the embeddings, the cross-modal predictor is low-rank, which constrains the multilinear rank of the core tensor G; extreme energy concentration indicates dominance by a single separable component. Penalizing this concentration therefore preserves the representational capacity needed to encode the higher-order dependence that, by Theorem 2, carries pure synergistic information. To avoid the cost of computing the full-dimensional cross-moment tensor, we first project each embedding into an rpr_p-dimensional lower-dimensional subspace. For each augmented view v∈1,2v∈\1,2\, we define p,v(m,b)=v,u(m,b)(m) Z_p,v^(m,b)= Z_v,u^(m,b)P^(m), where (m)∈ℝdm×rpP^(m) ^d_m× r_p is constructed by QR decomposition of a random Gaussian matrix with independent entries and rp<minmdmr_p< _md_m. This projection preserves pairwise geometry up to a multiplicative distortion factor by the Johnson–Lindenstrauss lemma (Lemma 3) [10]. We then compute the projected empirical cross-moment tensor for each view over the mini-batch: rp,v:=1B∑b=1Bp,v(1,b)⊗p,v(2,b)⊗⋯⊗p,v(M,b)∈ℝrp×rp×⋯×rp,T_r_p,v:= 1B _b=1^B Z_p,v^(1,b) Z_p,v^(2,b) ·s Z_p,v^(M,b)\;∈\;R^r_p× r_p×·s× r_p, (4) and perform Tucker decomposition rp,v≈v×11,v⋯×M,vT_r_p,v _v×_1A_1,v·s×_MA_M,v, yielding the core tensor vG_v. To prevent excessive core-energy concentration, we introduce the following margin-based synergy-aware regularization grounded in Lemma 2: ℒsyn=12∑v=12[ξ−(1−ρ(v))]+,L_syn= 12 _v=1^2 [ξ- (1-ρ(G_v) ) ]^+, (5) where ξ∈(0,1)ξ∈(0,1) is a margin hyperparameter controlling the minimum energy dispersion. Throughout this paper, we adopt the notation [⋅]+=ReLU(⋅)[·]^+=ReLU(·). By penalizing ρ(v)>1−ξρ(G_v)>1-ξ in each view, ℒsynL_syn discourages one-sparse degeneration in the Tucker cores, thereby preserving multi-way coupling capacity. While ℒsynL_syn preserves the capacity for higher-order dependencies, it does not actively recover synergistic information. To learn such information, we employ a contrastive alignment strategy from [11], which aligns each unimodal representation with the fused multimodal representation, i.e., ℒInfoNCE(m):=−12(I^NCE(1,u(m),2,f)+I^NCE(2,u(m),1,f)).L_InfoNCE^(m):=- 12\! ( I_NCE( Z_1,u^(m), Z_2,f)+ I_NCE( Z_2,u^(m), Z_1,f) ). (6) Here I^NCE I_NCE is the InfoNCE mutual-information estimator [33]: I^NCE(,′)=,′pos∼p(,′)′neg∼p(′)[logexpsim(,′pos)expsim(,′pos)+∑′negexpsim(,′neg)]. I_NCE( Z, Z )=E_ subarrayc z, z _pos p( Z, Z )\\ z _neg p( Z ) subarray [ ( z, z _pos) ( z, z _pos)+ _ z _neg ( z, z _neg) ]. We use sim(⋅,⋅)sim(·,·) to denote cosine similarity throughout this work. 1,u(m) Z_1,u^(m) and 2,u(m) Z_2,u^(m) denote the m-th unimodal embeddings from the two views, and 1,f Z_1,f, 2,f Z_2,f denote the corresponding fused representations. Critically, although the above loss objective only captures shared and unique information in [11], the proposed ℒsynL_syn discourages excessive energy concentration within the core tensor to maintain the higher-order representational capacity of f Z_f. Consequently, the identical contrastive loss further enables the model to extract synergistic components. 3.2 Contrastive alignment for shared and unique subspaces Although Eq. (5) preserves capacity for synergistic interactions, it does not by itself control the geometry of unimodal embeddings. This is important because multimodal InfoNCE can suffer from competing alignment directions and attraction–repulsion conflicts, leading to modality gaps and unstable optimization [51]. Inspired by [3, 56], we introduce an auxiliary alignment regularizer that encourages cross-view consistency while preserving variation in modality-specific directions. For each view v∈1,2v∈\1,2\, let ^i∈ℝB×D Z_i ^B× D denote the ℓ2 _2-normalized representation that concatenates all unimodal embeddings along the feature dimension for the i-th view, where B is the batch size and D is the total feature dimension. The alignment objective is: ℒalign=ℒinv+ℒvar,L_align=L_inv+L_var, (7) where ℒinv=1B∑b=1B(1−sim(^1(b),^2(b)))2andℒvar=1D∑d=1D[γ−Var(¯d)+ϵ]+,L_inv= 1B _b=1^B (1-sim( Z_1^(b), Z_2^(b)) )^2 _var= 1D _d=1^D [γ- Var\! ( Z_d )+ε ]^+, with ¯=(^1+^2)/2 Z=( Z_1+ Z_2)/2, where γ is the variance threshold and ϵε is a small constant. Among these two terms, ℒinvL_inv encourages cross-view agreement of shared factors at the sample level, and ℒvarL_var penalizes standard deviations below γ (0.5 by default) along each embedding dimension to discourage coordinate-wise collapse. Because each modality embedding v(m) Z_v^(m) is individually normalized before concatenation, modality-specific directions are preserved in ^v Z_v; ℒvarL_var then encourages variance along all dimensions, helping preserve modality-specific variation during alignment. Overall objective. The final loss combines all three components: ℒtotal=∑m=1MℒInfoNCE(m)+λaℒalign+λsℒsyn,L_total= _m=1^ML_InfoNCE^(m)+ _aL_align+ _sL_syn, (8) where λa _a and λs _s are hyperparameters. 4 Experiments We evaluate HRIL on both synthetic and real-world benchmarks to validate its effectiveness in capturing synergistic interactions and enhancing downstream task performance. In the synthetic setting, we follow the Trifeature dataset protocol from [11], which provides a controlled environment to explicitly isolate redundancy, uniqueness, and synergy. To assess practical utility, we conduct extensive experiments on Multibench [27], a comprehensive multimodal benchmark spanning diverse domains and modality combinations (e.g., healthcare, affective computing, etc.). First, we evaluate the trimodal setting, which naturally exhibits higher-order statistical dependencies essential for synergy, thereby directly testing our method’s capacity to model complex multi-way interactions. Subsequently, we evaluate generalization in bimodal configurations, which constitute the predominant configuration in practical applications. For evaluation, we use linear probing, i.e., freezing the pre-trained feature extractor and training a linear classifier on top of the learned representations. The resulting downstream performance serves as a direct proxy for the discriminative quality and generalization capacity of the learned multimodal representations. Detailed experimental settings are provided in Appendix B. 4.1 Experiments on Synthetic Datasets with Trifeature Table 1: Linear probing accuracy (in %) for redundancy (shape), uniqueness (texture), and synergy (color and texture) on the Trifeature dataset. ♣ denotes results from [11]. Model Redundancy↑ Uniqueness↑ Synergy↑ Cross♣ [36] 100.0 11.611.6 50.050.0 Cross+Self♣ [52] 99.799.7 86.986.9 50.050.0 FactorCL♣ [26] 99.899.8 62.562.5 46.546.5 CoMM [11] 99.9±0.0699.9_± 0.06 86.8±2.9986.8_± 2.99 71.4±3.4771.4_± 3.47 HRIL (ours) 99.7±0.1499.7_± 0.14 92.6±3.04 92.6_± 3.04 82.3±1.61 82.3_± 1.61 We follow the Trifeature [17] synthetic protocol of CoMM [11]. For redundancy and uniqueness, shape is treated as shared and texture as modality-specific. The dataset consists of image pairs sharing identical shapes but exhibiting independent textures. We formulate two linear probing objectives: (1) predicting the common shape class (redundancy), and (2) identifying the texture from a single modality (uniqueness), both yielding a 10% random-guessing baseline. To evaluate synergy, we impose a deterministic cross-modal mapping ℳM that links the texture of X1X_1 to the color of X2X_2 (e.g., stripes ↦ red), restricting training exclusively to pairs that adhere to this constraint. Evaluation proceeds via a binary compliance task defined as Y=[(tex(X1),col(X2))∈ℳ]Y= 1[\,(tex(X_1),col(X_2)) \,], with a 50% chance baseline. Experimental results are summarized in Tab. 1. Cross-modality constraints with InfoNCE [36] ("Cross") effectively capture redundant shape information but yield near-chance performance on uniqueness. Self-supervised constraints on each encoder ("Cross+Self"), FactorCL [26], and CoMM [11] all improve uniqueness performance, with CoMM achieving the best result as it implicitly models the synergistic factor and is the only baseline that significantly improves synergy capture. In comparison, HRIL achieves the highest uniqueness and synergy performance, outperforming CoMM by 5.8% on uniqueness and 10.9% on synergy. 4.2 Experiments on Multibench To further validate the practical utility of HRIL, we conduct extensive experiments on Multibench [27], which includes multiple datasets with different modalities and domains. These datasets provide a comprehensive evaluation of the model’s ability to learn multimodal representations that are effective for downstream tasks. Further dataset details are provided in Appendix C. 4.2.1 Experiments with 3 Modalities: UR-FUNNY, MOSI, and MOSEI Table 2: Linear probing top-1 accuracy (in %) for classification tasks on trimodal Multibench. Model UR-FUNNY↑ MOSI↑ MOSEI↑ Average Tri_CLIP 60.6±0.7960.6_± 0.79 60.1±3.4360.1_± 3.43 64.1±0.4264.1_± 0.42 61.661.6 CMC [39] 60.1±0.7860.1_± 0.78 62.4±1.6762.4_± 1.67 62.7±0.7262.7_± 0.72 61.7361.73 ConFu [23] 61.3±0.9661.3_± 0.96 62.5±2.6662.5_± 2.66 64.8±0.6164.8_± 0.61 62.8762.87 CoMM [11] 64.8±1.1364.8_± 1.13 66.0±2.2766.0_± 2.27 69.9±0.3469.9_± 0.34 66.966.9 HRIL (ours) 65.5±0.56 65.5_± 0.56 67.2±1.28 67.2_± 1.28 70.5±0.20 70.5_± 0.20 67.73 67.73 We first evaluate the trimodal setting since it naturally exhibits higher-order statistical dependencies essential for synergy. The data preprocessing steps can be found in Appendix C. Those experiments are conducted in trimodal configurations using identical encoders and modality arrangements, with all models trained directly on encoded representations. We compare against several established multimodal contrastive learning frameworks: Tri_CLIP (trained with pairwise contrastive objectives across three modalities), ConFu [23], CMC [39], and CoMM [11]. As shown in Tab. 2, while ConFu extends Tri_CLIP by incorporating a higher-order contrastive loss, it yields only limited performance gains. In contrast, HRIL consistently surpasses all baselines across the three datasets and outperforms the strongest competitor — CoMM — by 0.7% on UR-FUNNY, 1.2% on MOSI, and 0.6% on MOSEI. These empirical results demonstrate that HRIL’s explicit modeling of higher-order interactions effectively enhances representation quality and downstream task performance, thereby validating the theoretical principles established in our information-theoretic analysis. 4.2.2 Experiments with 2 Modalities Beyond the trimodal setting, we further evaluate whether HRIL remains effective in the more common bimodal scenario and test the practical robustness of our synergy-aware objective across heterogeneous domains, including healthcare and affective understanding. Adopting the data preprocessing steps established in [26], Table 3: Linear probing top-1 accuracy (in %) for classification tasks on bimodal Multibench. ♣ denotes results are from [26]. Model Classification MIMIC↑ MOSI↑ UR-FUNNY↑ MUSTARD↑ Average∗↑^* Cross♣Cross [36] 66.7±0.166.7_± 0.1 47.8±1.847.8_± 1.8 50.1±1.950.1_± 1.9 53.5±2.953.5_± 2.9 54.5254.52 Cross+Self♣Cross+Self [52] 65.49±0.065.49_± 0.0 49.0±1.149.0_± 1.1 59.9±0.959.9_± 0.9 53.9±4.053.9_± 4.0 57.0757.07 FactorCL♣FactorCL [26] 67.3±0.067.3_± 0.0 51.2±1.651.2_± 1.6 60.5±0.860.5_± 0.8 55.80±0.955.80_± 0.9 58.758.7 CoMM [11] 66.4±0.466.4_± 0.4 63.7±2.563.7_± 2.5 63.3±0.563.3_± 0.5 64.4±1.164.4_± 1.1 64.4564.45 HRIL (ours) 68.0±0.7 68.0_± 0.7 65.8±1.2 65.8_± 1.2 63.9±0.9 63.9_± 0.9 66.6±2.7 66.6_± 2.7 66.08 66.08 we compare our method with Cross [36], Cross+Self [52], FactorCL [26], and CoMM [11]. Tab. 3 summarizes the results. HRIL achieves the best performance on all four bimodal classification tasks. Compared with the strongest baseline, CoMM, HRIL improves accuracy by 1.6 percentage points on MIMIC, 2.1 on MOSI, 0.6 on UR-FUNNY, and 2.2 on MUSTARD, leading to an average gain of 1.63 percentage points. These results indicate that the proposed synergy-aware regularization is not limited to interactions among three or more modalities; it also improves representation learning in practical bimodal settings by preserving non-degenerate cross-modal dependence. 5 Ablation Studies Loss functions. To evaluate the contribution of each loss term in HRIL, we compare different loss combinations on the Trifeature dataset for capturing multimodal interactions. Table 4: Linear probing accuracy (in %) of multimodal interactions on the Trifeature dataset under different combinations of loss terms in Eq. 8. Loss components Redundancy↑ Uniqueness↑ Synergy probe↑ ℒInfoNCEL_InfoNCE ℒalignL_align ℒsynL_syn × ✓ ✓ 28.5±2.7428.5_± 2.74 12.5±3.1512.5_± 3.15 50.0±0.0050.0_± 0.00 ✓ × × 99.9±0.1099.9_± 0.10 87.2±2.1387.2_± 2.13 50.0±0.0050.0_± 0.00 ✓ × ✓ 99.9±0.0599.9_± 0.05 86.6±2.7986.6_± 2.79 72.4±3.1672.4_± 3.16 ✓ ✓ × 99.9±0.0999.9_± 0.09 85.1±6.8685.1_± 6.86 78.0±0.4478.0_± 0.44 ✓ ✓ ✓ 99.7±0.1499.7_± 0.14 92.6±3.0492.6_± 3.04 82.3±1.6182.3_± 1.61 As shown in Tab. 4, both ℒInfoNCEL_InfoNCE alone and ℒalign+ℒsynL_align+L_syn perform poorly on synergy, achieving only chance-level accuracy (50.0%). The variant without ℒInfoNCEL_InfoNCE performs worse overall, with substantially lower redundancy and uniqueness accuracy. Adding ℒsynL_syn to ℒInfoNCEL_InfoNCE increases synergy accuracy to 72.4%, while adding ℒalignL_align increases it to 78.0%. The full objective further improves synergy accuracy from 78.0% to 82.3% and uniqueness accuracy from 85.1% to 92.6% compared with ℒInfoNCE+ℒalignL_InfoNCE+L_align. Meanwhile, redundancy remains nearly saturated. These results suggest that HRIL benefits from the complementary contributions of all components. Table 5: Quantitative evaluation of InterSHAP on Multibench [27] datasets. × denotes the model without ℒsynL_syn, and ✓ denotes the full objective of HRIL. ℒsynL_syn InterSHAP Value (%) Average UR-FUNNY MOSI MOSEI × 7.78±4.567.78_± 4.56 6.82±3.736.82_± 3.73 1.35±0.531.35_± 0.53 5.32 ✓ 9.25±5.569.25_± 5.56 9.90±4.819.90_± 4.81 1.47±0.401.47_± 0.40 6.87 Improvement +1.47 +1.47 +3.08 +3.08 +0.12 +0.12 +1.55 +1.55 Interaction attribution. To evaluate whether the proposed synergy-aware regularization (Eq. (5)) increases cross-modal interaction attribution, we report InterSHAP [49] as an attribution-based interaction metric. Tab. 5 reports InterSHAP values on three trimodal datasets from Multibench [27]. Without ℒsynL_syn (Eq. (5)), the interaction attribution remains modest. In contrast, adding the proposed synergy-aware regularization consistently raises mean InterSHAP on all datasets, improving the average score from 5.32% to 6.87%. The gains are most pronounced on MOSI and UR-FUNNY, indicating stronger use of cross-modal interaction patterns after applying the synergy-aware regularization. The slight increase on MOSEI indicates a smaller mean change in interaction attribution. Moreover, since interaction attributions in real-world benchmarks are often numerically small [49], the consistent upward shift across datasets provides complementary behavioral evidence that ℒsynL_syn encourages the model to exploit cross-modal interactions. The metric definition and further comparisons with CoMM and a multilinear-rank control are provided in Appendix D.1. Table 6: Linear probing accuracy (in %) of multimodal interactions on the Trifeature dataset. Concat+Linear: concatenate the modality embeddings and pass through a linear layer. Fusion R U S Average Concat+Linear 99.9±0.199.9_± 0.1 65.4±10.7365.4_± 10.73 50.0±0.0050.0_± 0.00 71.7771.77 HRIL 99.7±0.1499.7_± 0.14 92.6±3.04 92.6_± 3.04 82.3±1.61 82.3_± 1.61 91.83 91.83 Fusion module. We compare the attention-based fusion module with a linear fusion baseline on the Trifeature dataset. As shown in Tab. 6, the linear fusion module fails to capture synergistic information, whereas HRIL substantially improves the performance of synergy. This result suggests that synergy-aware regularization is effective only when the fusion module has sufficient capacity to model higher-order cross-modal interactions. It also provides empirical support for Definition 2: if the information in one modality can be linearly approximated from another modality, the resulting representation is unlikely to encode genuinely synergistic information. 6 Related Work Multimodal contrastive learning. Multimodal contrastive learning aligns cross-modal representations by maximizing mutual information through constructed positive and negative pairs [36, 33, 39]. Positive pairs are typically constructed by augmenting the same instance, leveraging intra- and cross-modal variations as mutual supervision signals [19, 5, 7, 56]. The effective utilization of the three forms of multimodal information (i.e., redundancy, uniqueness, and synergy) is driving the advancement of representation learning [47]. Predominant pairwise contrastive frameworks primarily capture shared information across modalities [36, 18], often at the expense of modality-unique features. MV-DHEL [22] jointly aligns all views of the same sample while imposing view-specific uniformity to decouple alignment from uniformity. While recent efforts have pivoted toward disentangling and preserving modality-specific information [26, 44, 15, 35], and methods such as CoMM [11] and InfMasking [46] implicitly capture synergistic interactions, the explicit modeling and recovery of synergy remain an open challenge. Building on CoMM’s fusion and alignment paradigm, HRIL adds an explicit higher-order structural regularizer. Higher-order multimodal dependence. Recent self-supervised methods move beyond pairwise alignment by explicitly modeling multi-way cross-modal dependence. Symile [37] uses a scalar multilinear critic to optimize a contrastive lower bound on total correlation, while ConFu [23] aligns fused modality subsets with the remaining modality alongside pairwise alignment. M3G [34] promotes joint consistency across views by minimizing the gap between ground-truth and optimal matching costs under entropy-regularized multi-marginal optimal transport. Geometric criteria provide an alternative route to joint alignment: GRAM [9] aligns modalities by minimizing the Gramian volume spanned by their embeddings, and TRIANGLE [8] enforces tri-modal alignment by replacing cosine similarity with a triangle-area similarity computed in the embedding span. Nevertheless, these approaches largely target joint dependence or alignment, while HRIL additionally regularizes the empirical cross-moment structure to preserve capacity for synergistic interactions. Tensor-based multimodal fusion. Tensor fusion provides an explicit mechanism for modeling multiplicative cross-modal interactions [1, 21]. TFN [53] enumerates unimodal, bimodal, and trimodal interactions by forming outer products of modality embeddings, but incurs exponential complexity. LMF [28] alleviates this via low-rank factorization while retaining multiplicative coupling. Diverging from tensor-based fusion paradigms, we instead employ Tucker decomposition [21] as a regularization prior, explicitly enforcing a non-degenerate multi-way structure in the fused representation. 7 Conclusion We presented HRIL, a multimodal contrastive representation learning framework that addresses the synergistic gap left by pairwise contrastive learning. By proving under the condition of pure synergy that synergistic information equals a conditional total-correlation difference recoverable only through higher-order dependence, we motivated synergy-aware regularization on the Tucker-decomposed cross-moment core that preserves multi-way interaction capacity alongside multimodal contrastive objectives. Experiments on synthetic and real-world benchmarks show consistent gains, especially on the controlled synergy task. Current limitations include the assumption of cleanly separated latent factors; future work will explore relaxing this assumption and extending the framework to large multimodal models. Further discussions can be found in Appendix A. Acknowledgments This work was supported by the Sichuan Provincial Innovation Group Project (Grant No. 2024NSFTD0054), the Major Science and Technology Special Project of the Sichuan Provincial Department of Science and Technology (Grant No. 2024ZDZX0002), Fundamental Research Funds for the Central Universities (JBK202511081), the National Natural Science Foundation of China (No. U24A20323 , No. 62502397, No.12526646, No 126011059). We thank anonymous reviewers for their insightful feedback. References [1] A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, and M. Telgarsky (2014) Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research 15 (1), p. 2773–2832. Cited by: §3.1, §6. [2] J. Arevalo, T. Solorio, M. Montes-y-Gómez, and F. A. González (2017) Gated multimodal units for information fusion. In International Conference on Learning Representations - Workshops (ICLR-W), Cited by: §C.3, §1. [3] A. Bardes, J. Ponce, and Y. LeCun (2021) Vicreg: variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906. Cited by: §3.2. [4] N. Bertschinger, J. Rauh, E. Olbrich, J. Jost, and N. Ay (2014) Quantifying unique information. Entropy 16 (4), p. 2161–2183. Cited by: §1, §2.1. [5] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin (2020) Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33, p. 9912–9924. Cited by: §6. [6] S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria (2019) Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), p. 4619–4629. Cited by: 5th item. [7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), p. 1597–1607. Cited by: Table 9, §6. [8] G. Cicchetti, E. Grassucci, and D. Comminiello (2025) A triangle enables multimodal alignment beyond cosine similarity. arXiv preprint arXiv:2509.24734. Cited by: §6. [9] G. Cicchetti, E. Grassucci, L. Sigillo, and D. Comminiello (2025) Gramian multimodal representation learning and alignment. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, External Links: Link Cited by: §6. [10] S. Dasgupta and A. Gupta (2003) An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms 22 (1), p. 60–65. Cited by: Appendix G, §3.1, Lemma 3. [11] B. Dufumier, J. Castillo-Navarro, D. Tuia, and J. Thiran (2025) What to align in multimodal contrastive learning?. In International Conference on Learning Representations, Cited by: Appendix B, Appendix B, 1st item, §C.1, §D.6, Table 11, Table 11, Table 9, Table 9, §1, §1, §2.1, §3.1, §3.1, §4.1, §4.1, §4.2.1, §4.2.2, Table 1, Table 1, Table 1, Table 2, Table 3, §4, §6. [12] C. Eckart and G. Young (1936) The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: Appendix G. [13] GroupLens research (2015) MovieLens dataset. External Links: Link Cited by: §C.3. [14] Y. Guo, c. guo, A. Sun, H. He, X. Yang, Y. Lu, Y. Zhang, X. Guo, D. Zhang, J. Liu, J. Duan, Y. Xiao, L. Wen, H. Xu, and Y. Dai (2026) Web-cogreasoner: towards multimodal knowledge-induced cognitive reasoning for web agents. In International Conference on Learning Representations, C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (Eds.), Vol. 2026, p. 36186–36210. External Links: Link Cited by: Appendix A. [15] Z. Guo, Z. Shen, X. Xie, L. Wen, and Z. Kang (2025) Disentangling homophily and heterophily in multimodal graph clustering. arXiv preprint arXiv:2507.15253. Cited by: §6. [16] M. K. Hasan, W. Rahman, A. Zadeh, J. Zhong, M. I. Tanveer, L. Morency, et al. (2019) UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2046–2056. Cited by: 4th item. [17] K. Hermann and A. Lampinen (2020) What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS) 33, p. 9995–10006. Cited by: §C.1, §4.1. [18] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), p. 4904–4916. Cited by: §1, §6. [19] L. Jing and Y. Tian (2020) Self-supervised visual feature learning with deep neural networks: a survey. IEEE transactions on pattern analysis and machine intelligence 43 (11), p. 4037–4058. Cited by: §6. [20] A. E. Johnson, T. J. Pollard, L. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark (2016) MIMIC-I, a freely accessible critical care database. Scientific data 3 (1), p. 1–9. Cited by: 1st item. [21] T. G. Kolda and B. W. Bader (2009) Tensor decompositions and applications. SIAM review 51 (3), p. 455–500. Cited by: §3.1, §3.1, §6. [22] P. Koromilas, E. Georgiou, G. Bouritsas, T. Giannakopoulos, M. A. Nicolaou, and Y. Panagakis (2025) A principled framework for multi-view contrastive learning. arXiv preprint arXiv:2507.06979. Cited by: §6. [23] S. Koutoupis, M. A. Zervou, K. Kontras, M. De Vos, P. Tsakalides, and G. Tsagkatakis (2025) The more, the merrier: contrastive fusion for higher-order multimodal alignment. arXiv preprint arXiv:2511.21331. Cited by: Appendix B, §1, §2.2, §4.2.1, Table 2, §6. [24] P. M. Kroonenberg (2008) Applied multiway data analysis. John Wiley & Sons. Cited by: Appendix B. [25] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg (2020) Making sense of vision and touch: learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics 36 (3), p. 582–596. Cited by: 6th item, §D.5. [26] P. P. Liang, Z. Deng, M. Ma, J. Zou, L. Morency, and R. Salakhutdinov (2023) Factorized contrastive learning: going beyond multi-view redundancy. Advances in Neural Information Processing Systems (NeurIPS) 36, p. 32971–32998. Cited by: 1st item, 2nd item, 3rd item, §4.1, §4.2.2, §4.2.2, Table 1, Table 3, Table 3, Table 3, §6. [27] P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu, et al. (2021) MultiBench: multiscale benchmarks for multimodal representation learning. In Neural Information Processing Systems (NeurIPS) – Track on Datasets and Benchmarks, Vol. 1. Cited by: 1st item, §C.2, §C.3, Table 11, Table 11, NeurIPS Paper Checklist, §4.2, §4, Table 5, Table 5, §5. [28] Z. Liu, Y. Shen, V. B. Lakshminarasimhan, P. P. Liang, A. B. Zadeh, and L. Morency (2018) Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2247–2256. Cited by: §6. [29] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: Appendix B. [30] O. A. Malik and S. Becker (2018) Low-rank tucker decomposition of large tensors using tensorsketch. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, p. . External Links: Link Cited by: §D.6. [31] L. Mirsky (1960) Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics 11 (1), p. 50–59. Cited by: Appendix G. [32] N. Mu, A. Kirillov, D. Wagner, and S. Xie (2022) SLIP: Self-supervision meets language-image pre-training. In European Conference on Computer Vision (ECCV), p. 529–544. Cited by: Table 9. [33] A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: Appendix F, §3.1, §6. [34] Z. Piran, M. Klein, J. Thornton, and M. Cuturi (2024) Contrasting multiple representations with the multi-marginal matching gap. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 40827–40842. External Links: Link Cited by: §6. [35] C. Qian, S. Xing, S. Li, Y. Zhao, and Z. Tu (2025) Decalign: hierarchical cross-modal alignment for decoupled multimodal representation learning. arXiv preprint arXiv:2503.11892. Cited by: §6. [36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), p. 8748–8763. Cited by: Table 9, Table 9, §1, §4.1, §4.2.2, Table 1, Table 3, §6. [37] A. Saporta, A. Puli, M. Goldstein, and R. Ranganath (2024) Contrasting with symile: simple model-agnostic representation learning for unlimited modalities. Advances in Neural Information Processing Systems 37, p. 56919–56957. Cited by: §1, §6. [38] K. Sridharan and S. M. Kakade (2008) An information theoretic framework for multi-view learning. In Annual Conference on Learning Theory, Cited by: §1. [39] Y. Tian, D. Krishnan, and P. Isola (2020) Contrastive multiview coding. In European Conference on Computer Vision (ECCV), p. 776–794. Cited by: §4.2.1, Table 2, §6. [40] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola (2020) What makes for good views for contrastive learning?. Advances in neural information processing systems (NeurIPS) 33, p. 6827–6839. Cited by: §2.2. [41] Y. H. Tsai, Y. Wu, R. Salakhutdinov, and L. Morency (2020) Self-supervised learning from a multi-view perspective. arXiv preprint arXiv:2006.05576. Cited by: §1. [42] C. E. Tsourakakis (2010) MACH: fast randomized tensor decompositions. In Proceedings of the SIAM International Conference on Data Mining, SDM 2010, April 29 - May 1, 2010, Columbus, Ohio, USA, p. 689–700. External Links: Link, Document Cited by: §D.6. [43] H. Wang, X. Guo, Z. Deng, and Y. Lu (2022) Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16041–16050. Cited by: §1, §2.2, §2.2. [44] Y. Wang, C. M. Albrecht, N. A. A. Braham, C. Liu, Z. Xiong, and X. X. Zhu (2024) Decoupling common and unique representations for multimodal self-supervised learning. In European Conference on Computer Vision, p. 286–303. Cited by: §6. [45] S. Watanabe (1960) Information theoretical analysis of multivariate correlation. IBM Journal of research and development 4 (1), p. 66–82. Cited by: §2.2. [46] L. Wen, Q. Dai, J. Liu, J. Zheng, Y. Dai, D. Wang, Z. Kang, J. Wang, Z. Xu, and J. Duan (2025) InfMasking: unleashing synergistic information by contrastive multimodal interactions. arXiv preprint arXiv:2509.25270. Cited by: §1, §2.1, §6. [47] L. Wen, L. Li, J. Duan, Y. Dai, J. Liu, and Z. Kang (2026) Dependency, compression, and synergy: a unified information-theoretic view of multimodal learning. arXiv preprint arXiv:2609.14421. Cited by: §6. [48] L. Wen, X. Wang, J. Liu, and Z. Xu (2024) MVEB: self-supervised learning with multi-view entropy bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (9), p. 6097–6108. Cited by: §1. [49] L. Wenderoth, K. Hemker, N. Simidjievski, and M. Jamnik (2025) Measuring cross-modal interactions in multimodal models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 21501–21509. Cited by: §D.1, §5. [50] P. L. Williams and R. D. Beer (2010) Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515. Cited by: §1, §2.1. [51] W. Yin, P. Zhou, Z. Xiao, J. Liu, S. Yu, J. Sonke, and E. Gavves (2026) Towards uniformity and alignment for multimodal representation learning. arXiv preprint arXiv:2602.09507. Cited by: §3.2. [52] X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y. Wang, M. Maire, A. Kale, and B. Faieta (2021) Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 6995–7004. Cited by: §4.2.2, Table 1, Table 3. [53] A. Zadeh, M. Chen, S. Poria, E. Cambria, and L. Morency (2017) Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 conference on empirical methods in natural language processing, p. 1103–1114. Cited by: §3.1, §6. [54] A. Zadeh, R. Zellers, E. Pincus, and L. Morency (2016) Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intelligent Systems 31 (6), p. 82–88. Cited by: 2nd item. [55] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L. Morency (2018) Multimodal language analysis in the wild: cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2236–2246. Cited by: 3rd item. [56] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny (2021) Barlow twins: self-supervised learning via redundancy reduction. In International conference on machine learning, p. 12310–12320. Cited by: §3.2, §6. Appendix Appendix A Limitations and Future Work Limitations. Our theoretical analysis relies on Assumption 1. In particular, the core results are stated under the pure-synergy condition in Definition 1, where the synergistic factor carries no mutual information with any proper subset of modalities once the shared factor is conditioned on. This assumption is useful for isolating the failure mode of pairwise contrastive learning and for deriving a clean connection between synergistic information and conditional total correlation. However, real multimodal data may exhibit mixed interaction structures: synergistic signals can coexist with residual pairwise dependence, partially redundant cues, or modality-specific shortcuts. In such cases, our theory can be interpreted as characterizing an idealized regime that clarifies when higher-order modeling is necessary. Although the empirical results in Sec. 4 and the InterSHAP analysis in Appendix D.1 suggest that the proposed regularization is beneficial beyond purely synthetic settings, a more systematic robustness study under approximate or partially overlapping synergy remains open. Future Work. Future work can extend Definition 1 from pure synergy to approximate and mixed-synergy regimes. One promising direction is to develop data-dependent diagnostics for estimating the degree of redundancy, uniqueness, and synergy in real benchmarks, which could guide when higher-order regularization is most useful and how strongly it should be applied. Moreover, scaling the proposed objective to larger modality sets and integrating it with multimodal alignment objectives or large multimodal models [14] may further test whether synergy-aware learning improves general-purpose multimodal reasoning. Appendix B Experimental Details Implementation details. Unless otherwise specified, all methods are implemented in PyTorch and trained on a single NVIDIA RTX 4090 GPU with 24 GB of VRAM. We employ the AdamW optimizer [29] across all experiments. Each experimental configuration is trained over five independent runs using random seeds in the range [42,46][42,46]. We report the mean performance along with the standard deviation. Models are trained for up to 100 epochs; the best checkpoint is selected based on validation accuracy, with early stopping applied when applicable to prevent overfitting. Detailed hyperparameter settings for each dataset and method are released alongside our source code. Differentiability and Gradient Path of Tucker Decomposition. HRIL supports end-to-end backpropagation through the Tucker decomposition by leveraging the PyTorch autograd engine coupled with the TensorLy library’s differentiable backend. We compute the approximate Tucker decomposition rp,v≈v×1A1,v⋯×MAM,vT_r_p,v _v×_1A_1,v·s×_MA_M,v using the Higher-Order Orthogonal Iteration (HOOI) algorithm [24], initialized via truncated SVD. The autodiff engine dynamically traces the alternating least-squares updates and embedded SVD operations executed during each forward pass. The solver employs a convergence tolerance (10−310^-3) on changes in reconstruction error alongside a maximum of 15 iterations, ensuring that the computational graph records the executed optimization steps. Consequently, gradients propagate backward through the actual HOOI trajectory of the executed finite iterations without truncation or manual intervention. During backpropagation, the gradient with respect to the modality embeddings v,u(m,b) Z_v,u^(m,b) is computed via the chain rule as: ∂ℓsyn,v∂Zv,u,a(m,b)=1B∑j1,…,jMj1…jMPajm(m)∏k≠mZp,v,jk(k,b), ∂ _syn,v∂ Z_v,u,a^(m,b)= 1B _j_1,…,j_MD_j_1… j_MP^(m)_aj_m _k≠ mZ_p,v,j_k^(k,b), where the total derivative =∂ℓsyn,v/∂rp,vD=∂ _syn,v/ _r_p,v encompasses the full derivative chain through the traced HOOI steps and SVD differentials, ℓsyn,v _syn,v denotes the per-view hinge in Eq. (5), and p,v(m,b)=v,u(m,b)P(m) Z_p,v^(m,b)= Z_v,u^(m,b)P^(m). This term is evaluated automatically by the framework via unrolled automatic differentiation, requiring no manual Jacobian derivation. Autograd also accounts for the factor 1/21/2 from two-view averaging. The projection matrix P(m)P^(m) is constructed once via QR decomposition of a random Gaussian matrix with a fixed seed and cached for both views and remains constant throughout training. As a static linear transformation, it introduces no additional optimization variables and provides a gradient path from the interaction core back to the trainable representation modules. We provide the analysis of computational and optimization efficiency in Appendix D.6. Notably, the synergy-aware regularization ℒsynL_syn employs a margin-based activation that, when numerical clamps are inactive, remains dormant in each view unless the core tensor energy concentration exceeds the prescribed threshold 1−ξ1-ξ. The implementation floors ‖v‖F2\|G_v\|_F^2 at 10−610^-6, clips the remaining-energy ratio to [10−6,1−10−6][10^-6,1-10^-6], and averages the two per-view hinges. This sparse activation mechanism sets the direct hinge gradient to zero when inactive, while the implementation still evaluates forward and backward operations through the Tucker decomposition module. Model configuration. For dataset-specific encoder architectures, modality-specific data augmentations, and latent converters, we follow the same configurations as CoMM [11]. The fusion module in HRIL adopts a Transformer-based encoder to integrate modality-specific representations, following the design of CoMM [11]. Each layer comprises a multi-head self-attention mechanism followed by a position-wise feed-forward network, both equipped with residual connections and layer normalization. In bimodal settings, we employ a single-layer Transformer with 8 attention heads, whereas trimodal configurations utilize a 2-layer Transformer with the same head count. A learnable [CLS] token is prepended to the input sequence to aggregate cross-modal information into a unified global representation. For ConFu [23], we replicate the original implementation by using a 2-layer MLP as the subset fusion module. Appendix C Dataset Details C.1 Trifeature The Trifeature dataset [17] serves as a controlled benchmark for probing feature learning in visual neural networks. It comprises three distinct attributes—shape, color, and texture—each instantiated with 10 categorical variants, yielding 1,000 unique compositions. We adopt the standard split of 800 compositions for training and 200 for evaluation. To enhance sample diversity, each training composition is augmented with three randomized instances, where shapes undergo independent rotations uniformly sampled from [−45∘,45∘][-45 ,45 ]. All shapes are rendered within a 128×128128× 128 region, then randomly positioned on a 224×224224× 224 canvas while preserving full visibility; color and texture are applied analogously. Following [11], we construct bimodal pairs from these instances, resulting in 10,000 training and 4,096 test pairs drawn i.i.d. from the same generative distribution. C.2 Multibench According to [27], all datasets below have been systematically de-identified to eliminate personally identifiable information and ensure strict privacy compliance. Multibench [27] provides access to the extracted features and labels for each dataset. • MIMIC [20] is a large-scale critical care dataset comprising de-identified clinical records from over 38,000 patients admitted to intensive care units between 2001 and 2012. Each sample integrates two heterogeneous modalities: (i) a temporal modality consisting of 24-hour longitudinal clinical measurements (12-dimensional vectors), and (i) a static tabular modality encoding demographic features (5-dimensional vectors). We follow the protocol of prior work [27, 26, 11] and formulate the task as binary disease classification: predicting whether a patient’s diagnosis falls within ICD-9 (International Statistical Classification of Diseases and Related Health Problems) code group 7 (codes 460–519). • MOSI [54] contains 2,199 YouTube-sourced video clips curated for multimodal sentiment analysis. Each sample comprises three modalities: visual, audio, and text. Ground-truth sentiment scores are originally annotated on a continuous scale from -3 to 3; following [26], these labels are binarized into positive and negative classes. In our experiments, we utilize the textual and visual modalities in bimodal configurations, while the trimodal setting incorporates all three modalities to evaluate the model’s ability to integrate complementary information across modalities. • MOSEI [55] is a large-scale multimodal benchmark for sentence-level sentiment analysis and emotion recognition, containing almost 23,000 monologue video clips. It comprises more than 65 hours of annotated video content, spanning 250 distinct categories and featuring contributions from over 1,000 speakers. Each sample includes aligned visual, acoustic, and textual modalities, with sentiment labels originally annotated on a continuous scale from -3 to 3. In our experiments, we use its sentiment annotations and, following [26], binarize them into positive and negative classes. We then evaluate the model’s performance in trimodal settings. • UR-FUNNY [16] is a large-scale benchmark for humor detection, curated from 1,866 TED talks and containing 16,514 multimodal samples. Each instance integrates visual frames, acoustic signals, and aligned textual transcripts, with the goal of discerning humorous versus non-humorous content. In our experiments, we utilize the textual and visual modalities in bimodal configurations, while the trimodal setting incorporates all three modalities. • MUSTARD [6] is a multimodal benchmark for sarcasm detection, compiled from clips of popular sitcoms such as Friends. The corpus contains 690 samples, each providing synchronized video, audio, and subtitle streams with binary sarcasm annotations. In our experiments, we utilize the textual and visual modalities in bimodal configurations. • Vision&Touch [25] is a robotic manipulation benchmark with 150 trajectories of 1,000 time steps each. Its observations combine RGB and depth imagery with force measurements and end-effector position and velocity. In our experiments, we utilize the visual and proprioceptive modalities in a bimodal binary classification task to predict whether contact will occur at the next time step. C.3 M-IMDb Multimodal IMDb (M-IMDb) [2] poses movie-genre prediction as a multilabel classification task with 23 categories. Its 25,959 movies are drawn from MovieLens 20M [13] and include posters, plot descriptions, genre annotations, and metadata. We use the posters and plot descriptions as the image and text modalities, respectively, and evaluate weighted and macro F1. Although the dataset is included in MultiBench [27], our experiments train on its original image and text inputs. Appendix D Additional Experiments D.1 Evaluations with InterSHAP Metric definition. Rather than scoring only downstream task performance, this measure inspects how predictions are formed by separating cross-modal interaction effects from modality-specific main effects. Concretely, predictions f(x)f(x) are decomposed with the Shapley Interaction Index (SII), where pairwise terms represent interaction strength and diagonal terms represent unimodal contributions. Let M be the number of modalities and let ϕij _ij denote the SII attribution between modalities i and j (computed either per sample or aggregated over the dataset). The InterSHAP score [49] is then given by: InterSHAP=∑i,j=1i≠jM|ϕij|∑i,j=1M|ϕij|,InterSHAP= _ subarrayci,j=1\\ i≠ j subarray^M| _ij| _i,j=1^M| _ij|, (9) where the numerator aggregates off-diagonal interaction attributions (i≠ji≠ j) and the denominator aggregates all absolute attributions, including diagonal main effects (i=ji=j). Larger values indicate that model decisions rely more on interaction terms than on unimodal effects. Table 7 extends the comparison in Table 5 with CoMM and a multilinear-rank control on the UR-FUNNY and MOSI datasets, where the multilinear-rank control replaces ℒsynL_syn with a low-rank penalty on the Tucker core while keeping the other losses unchanged. We use this control to test whether a multilinear-rank restriction alone explains the mean InterSHAP gains. Its loss is ℒrank=∑m=1M∑i>kmσi2((m))L_rank= _m=1^M _i>k_m _i^2(G_(m)). The penalty sums the squared singular values beyond the leading kmk_m singular values of each mode unfolding (m)G_(m). This control encourages low multilinear rank, whereas ℒsynL_syn activates only when one core coordinate exceeds the prescribed energy share. Table 7: Further comparison with InterSHAP on the UR-FUNNY and MOSI datasets. Model UR-FUNNY MOSI CoMM 7.77±3.057.77_± 3.05 3.82±1.903.82_± 1.90 HRIL without ℒsynL_syn 7.78±4.567.78_± 4.56 6.82±3.736.82_± 3.73 Multilinear-rank control 8.17±6.338.17_± 6.33 5.87±4.265.87_± 4.26 HRIL 9.25±5.569.25_± 5.56 9.90±4.819.90_± 4.81 As shown in Table 7, HRIL has the highest mean InterSHAP score on both datasets. The multilinear-rank control penalty yields a small mean increase over the variant without synergy regularization on UR-FUNNY but a decrease on MOSI. HRIL exceeds it by 1.08 and 4.03 percentage points, respectively. These results support preserving higher-order interaction capacity within the prescribed Tucker subspace. The regularizer ℒsynL_syn discourages excessive dominance by a single core coordinate while allowing the core to remain sparse. D.2 Hyperparameter Sensitivity Table 8 reports sensitivity analyses on trimodal UR-FUNNY and MOSEI, with the remaining hyperparameters fixed. We vary the Tucker rank rmr_m, the projection dimension rpr_p and the margin ξ. The Tucker rank determines the retained dimension along each mode, while rpr_p determines the number of entries rpMr_p^M in the cross-moment tensor. The margin ξ sets the threshold 1−ξ1-ξ above which core-energy concentration is penalized. Table 8: Sensitivity analysis: accuracy (%) Hyperparameter Value UR-FUNNY MOSEI Tucker rank rmr_m 8 65.0±0.5465.0_± 0.54 69.9±0.3169.9_± 0.31 16 64.9±1.3464.9_± 1.34 70.0±0.4770.0_± 0.47 24 65.5±0.5665.5_± 0.56 70.5±0.2070.5_± 0.20 Projection dimension rpr_p 8 64.8±0.5964.8_± 0.59 70.2±0.2670.2_± 0.26 16 64.9±0.6664.9_± 0.66 70.2±0.2070.2_± 0.20 24 65.5±0.5665.5_± 0.56 70.5±0.2070.5_± 0.20 32 65.5±0.9065.5_± 0.90 70.2±0.4570.2_± 0.45 Margin ξ 0.1 65.2±1.1565.2_± 1.15 70.0±0.4070.0_± 0.40 0.3 65.2±1.0565.2_± 1.05 70.5±0.3870.5_± 0.38 0.5 65.5±0.5665.5_± 0.56 70.5±0.2070.5_± 0.20 0.7 65.2±1.0065.2_± 1.00 70.3±0.2570.3_± 0.25 As shown in Table 8, a Tucker rank of 24 gives the highest mean accuracy on both datasets. A projection dimension of 24 also attains the highest mean accuracy in its separate sweep, while increasing it to 32 enlarges the tensor without further mean improvement. Since the tensor is constructed before Tucker decomposition, reducing the decomposition rank does not lower the tensor construction cost. The margin sweep is comparatively stable: mean accuracy varies by 0.3 percentage points on UR-FUNNY and 0.5 on MOSEI, with ξ=0.5ξ=0.5 among the best settings on both datasets. D.3 Batch Size Robustness Since the synergy-aware regularization is computed from the empirical cross-moment tensor (Eq. (4)), we further evaluate the sensitivity of HRIL to the mini-batch size. As shown in Fig. 3, HRIL demonstrates more pronounced robustness on datasets with larger sample sizes. This trend is expected because larger datasets provide more reliable mini-batch estimates of cross-modal moments, whereas smaller datasets may introduce higher estimation variance. In addition, HRIL achieves its best performance with a batch size of 64, and further increasing the batch size does not lead to consistent gains. These results indicate that HRIL does not require excessively large batches for more reliable estimates of cross-modal moments, while maintaining both scalability and computational efficiency. Figure 3: Performance sensitivity to mini-batch size on MOSEI, UR-FUNNY, and MOSI. D.4 Experiments on M-IMDb We further evaluate HRIL on M-IMDb multilabel movie-genre prediction using the original posters and plot descriptions. Table 9 reports linear-probing results, where V and L denote visual and textual inputs. Table 9: Linear probing F1 scores (%) on M-IMDb. △ indicates further training on unlabeled data. ♣ denotes results from [11]. Model Modalities weighted-F1↑ macro-F1↑ SimCLR♣△ [7] V 40.35±0.2340.35_± 0.23 27.99±0.3327.99_± 0.33 CLIP♣ [36] V 51.551.5 40.840.8 L 51.051.0 43.043.0 V+L 58.958.9 50.950.9 SLIP♣△ [32] V+L 56.54±0.1956.54_± 0.19 47.35±0.2747.35_± 0.27 CLIP♣△ [36] V+L 54.49±0.1954.49_± 0.19 44.94±0.3044.94_± 0.30 CoMM(CLIP backbone)_(CLIP backbone) V+L 61.29±0.7361.29_± 0.73 53.79±0.2253.79_± 0.22 HRIL(ours, CLIP backbone)_(ours, CLIP backbone) V+L 62.30±0.2662.30_± 0.26 55.44±0.2655.44_± 0.26 The CLIP results first show that combining posters and plot descriptions improves over either modality alone, indicating that the joint input provides additional predictive value. With the same backbone, HRIL achieves weighted and macro F1 scores of 62.30% and 55.44%, exceeding CoMM by 1.01 and 1.65 percentage points, respectively. Thus, by considering multilabel classification directly from raw image and text inputs, this experiment broadens the evaluation beyond the original MultiBench experiments and further demonstrates the effectiveness of HRIL. D.5 Experiments on Vision&Touch Table 10: Linear probing accuracy (%) on Vision&Touch contact prediction (V&T CP). Model V&T CP↑ Cross 86.3±0.2586.3_± 0.25 Cross+Self 87.6±0.2687.6_± 0.26 CoMM 87.0±1.7787.0_± 1.77 HRIL (ours) 88.6±0.1488.6_± 0.14 To evaluate the effectiveness of HRIL on a dataset with a larger sample size, we further conduct experiments on Vision&Touch [25], which contains 147,000 samples. Given visual and proprioceptive observations, the task is to predict whether contact will occur at the next time step. As shown in Table 10, adding the self-contrastive objective to Cross raises contact-prediction accuracy from 86.3% to 87.6%, making Cross+Self the strongest listed baseline. Our method reaches 88.6%, improving on Cross+Self by 1.0 percentage point and on CoMM by 1.6 points. These results further demonstrate the effectiveness of HRIL on a substantially larger dataset for robotic contact prediction. D.6 Computational Complexity and Runtime Analysis Table 11: Training efficiency comparison between CoMM [11] and HRIL on trimodal Multibench [27] datasets. Both methods use the same multimodal backbone. We report the average wall-clock step time, optimizer update time, and the percentage of each step spent in backward computation. Dataset Model Training Efficiency Metrics Step Time (ms) Optimizer Time (ms) Backward / Step Ratio (%) UR-FUNNY CoMM 90.73 3.86 53.82 HRIL 155.65 3.47 46.23 MOSI CoMM 90.00 3.90 54.07 HRIL 156.95 3.47 46.96 MOSEI CoMM 90.57 4.06 53.99 HRIL 159.25 3.63 47.56 As shown in Tab. 11, HRIL increases the per-step runtime from roughly 90 ms to 156–159 ms across the three datasets compared with CoMM [11]. The optimizer time remains nearly unchanged, indicating that the overhead does not come from additional trainable parameters or a heavier parameter update. Instead, the extra cost mainly comes from the forward and backward computation of the additional loss terms, especially the cross-moment tensor regularization in Eq. (5). For a mini-batch of B samples and M modalities, each projected to rpr_p dimensions, explicit construction of the cross-moment tensor takes O(BrpM)O(Br_p^M) time, and the resulting tensor occupies O(rpM)O(r_p^M) storage before decomposition. With the batch size and model configuration fixed, increasing the dataset size requires more batches per epoch without changing the loss computation per batch. These additional computations are confined to training, as the synergy-aware regularizer is not used during inference. Given the performance improvements in the evaluated settings, we regard the additional training cost as a reasonable trade-off. Potential ways to reduce this cost include TensorSketch-based Tucker approximation [30], which uses compressed representations in Tucker computations, and randomized tensor sparsification [42], which samples and reweights tensor entries before decomposition. Applying these approximations to our Tucker-core regularizer and evaluating their effects on training remain future work. Appendix E Broader Impact This work advances multimodal representation learning by explicitly preserving synergistic interactions across heterogeneous data sources. Its primary societal benefit lies in enabling more robust fusion frameworks for high-stakes domains such as clinical decision support, where joint modality reasoning is critical. Given the methodological nature of the contribution, direct societal risks are minimal; however, we emphasize that downstream deployment should adhere to established safeguards regarding data privacy, algorithmic fairness, and computational sustainability. Appendix F Pseudo code Algorithm 1 summarizes the training procedure of HRIL, using the same notation as the main paper. Here D=∑m=1MdmD= _m=1^Md_m. We use the InfoNCE mutual-information estimator [33] defined in the main paper: I^NCE(,′)=,′pos∼p(,′)′neg∼p(′)[logexpsim(,′pos)expsim(,′pos)+∑′negexpsim(,′neg)], I_NCE( Z, Z )=E_ subarrayc z, z _pos p( Z, Z )\\ z _neg p( Z ) subarray [ ( z, z _pos) ( z, z _pos)+ _ z _neg ( z, z _neg) ], where sim(⋅,⋅)sim(·,·) denotes cosine similarity. Algorithm 1 HRIL training algorithm 1: Multimodal dataset X1,…,XM\X_1,…,X_M\, label-preserving transformations ⋆T , mini-batch size B, unimodal encoders fmm=1M\f_m\_m=1^M, shared Transformer g, QR projection matrices P(m)m=1M\P^(m)\_m=1^M, projection dimension rpr_p, Tucker ranks (r1,…,rM)(r_1,…,r_M), margin ξ, alignment hyperparameters γ,ϵγ,ε, loss weights λa,λs _a, _s. 2: for each sampled mini-batch ((1,b),…,(M,b))b=1B\( x^(1,b),…, x^(M,b))\_b=1^B do 3: draw t1,t2∼⋆t_1,t_2 4: for b←1b← 1 to B do 5: for m←1m← 1 to M do 6: 1(m,b),2(m,b)←t1((m,b)),t2((m,b)) x_1^(m,b), x_2^(m,b)← t_1( x^(m,b)),t_2( x^(m,b)) 7: 1(m,b),2(m,b)←fm(1(m,b)),fm(2(m,b)) h_1^(m,b), h_2^(m,b)← f_m( x_1^(m,b)),f_m( x_2^(m,b)) 8: 1,u(m,b),2,u(m,b)←g(1(m,b)),g(2(m,b)) Z_1,u^(m,b), Z_2,u^(m,b)← g( h_1^(m,b)),g( h_2^(m,b)) 9: p,1(m,b),p,2(m,b)←1,u(m,b)P(m),2,u(m,b)P(m) Z_p,1^(m,b), Z_p,2^(m,b)← Z_1,u^(m,b)P^(m), Z_2,u^(m,b)P^(m) 10: end for 11: 1,f(b)←g(1(1,b),…,1(M,b)) Z_1,f^(b)← g( h_1^(1,b),…, h_1^(M,b)) 12: 2,f(b)←g(2(1,b),…,2(M,b)) Z_2,f^(b)← g( h_2^(1,b),…, h_2^(M,b)) 13: end for 14: for m←1m← 1 to M do 15: ℒInfoNCE(m)←−12(I^NCE(1,u(m),2,f)+I^NCE(2,u(m),1,f))L_InfoNCE^(m)←- 12 ( I_NCE( Z_1,u^(m), Z_2,f)+ I_NCE( Z_2,u^(m), Z_1,f) ) 16: end for 17: for each view v∈1,2v∈\1,2\ do 18: rp,v←1B∑b=1Bp,v(1,b)⊗⋯⊗p,v(M,b)T_r_p,v← 1B _b=1^B Z_p,v^(1,b) ·s Z_p,v^(M,b) 19: Tucker decompose rp,v≈v×11,v⋯×M,vT_r_p,v _v×_1A_1,v·s×_MA_M,v with ranks (r1,…,rM)(r_1,…,r_M) 20: ρ(v)←maxi1,…,iMi1…iM,v2/‖v‖F2ρ(G_v)← _i_1,…,i_MG_i_1… i_M,v^2/\|G_v\|_F^2 21: ~v(b)←[~v(1,b);…;~v(M,b)] Z_v^(b)←[ Z_v^(1,b);…; Z_v^(M,b)] for all b, where ~v(m,b):=v,u(m,b)/‖v,u(m,b)‖ Z_v^(m,b):= Z_v,u^(m,b)/\| Z_v,u^(m,b)\| 22: ^v(b)←~v(b)/‖~v(b)‖ Z_v^(b)← Z_v^(b)/\| Z_v^(b)\| for all b 23: end for 24: ℒsyn←([ξ−(1−ρ(1))]++[ξ−(1−ρ(2))]+)/2L_syn← ([ξ-(1-ρ(G_1))]^++[ξ-(1-ρ(G_2))]^+ )/2 25: ℒinv←1B∑b=1B(1−sim(^1(b),^2(b)))2L_inv← 1B _b=1^B (1-sim( Z_1^(b), Z_2^(b)) )^2 26: ¯←(^1+^2)/2 Z←( Z_1+ Z_2)/2 27: ℒvar←1D∑d=1D[γ−Var(¯d)+ϵ]+L_var← 1D _d=1^D [γ- Var\! ( Z_d )+ε ]^+ 28: ℒalign←ℒinv+ℒvarL_align _inv+L_var 29: ℒtotal←∑m=1MℒInfoNCE(m)+λaℒalign+λsℒsynL_total← _m=1^ML_InfoNCE^(m)+ _aL_align+ _sL_syn 30: update (fm)m∈[1..M](f_m)_m∈[1..M] and g to minimize ℒtotalL_total 31: end for 32: return (fm)m∈[1..M],g(f_m)_m∈[1..M],g Appendix G Proof See 1 Proof. We prove the two claims in order. Step 1: pairwise alignment captures at most r q_r. (a) Pure synergy is inaccessible to individual modality pairs. For any pair i≠ji≠ j with M≥3M≥ 3, the set i,j\i,j\ is a proper subset of the modalities. Definition 1 therefore gives I(syn;Xi,Xj∣r)=0.I( q_syn;X_i,X_j q_r)=0. Thus, a modality pair contains no information about the pure synergistic factor once the shared factor is given. (b) The pairwise-minimal representation excludes non-shared information. For pairmin Z_pair , we use the shared information characterization in Eq. (2). The encoding relation =f(X) Z=f(X) in the task setup ensures that a representation is a function of the multimodal observation. Conditional data processing consequently gives 0 0 ≤I(pairmin;1:M,syn∣r) ≤ I( Z_pair ; q_1:M, q_syn q_r) ≤I(pairmin;X1:M∣r)=0, ≤ I( Z_pair ;X_1:M q_r)=0, where the last equality is Eq. (2). Hence, pairmin⟂(1:M,syn)∣r. Z_pair ( q_1:M, q_syn) q_r. (c) Upper bound on task information. Assumption 1 states that Y⟂X1:M∣(r,1:M,syn)Y X_1:M ( q_r, q_1:M, q_syn). Since pairmin Z_pair is a function of X1:MX_1:M, this also implies Y⟂pairmin∣(r,1:M,syn).Y Z_pair ( q_r, q_1:M, q_syn). Combining this conditional independence with part (b), the chain rule gives I(pairmin;Y,1:M,syn∣r) I( Z_pair ;Y, q_1:M, q_syn q_r) =I(pairmin;1:M,syn∣r)+I(pairmin;Y∣r,1:M,syn) =I( Z_pair ; q_1:M, q_syn q_r)+I( Z_pair ;Y q_r, q_1:M, q_syn) =0. =0. By nonnegativity of conditional mutual information, I(Y;pairmin∣r)=0I(Y; Z_pair q_r)=0. Another application of the chain rule yields I(Y,pairmin) I(Y; Z_pair ) =I(Y,r)+I(Y;pairmin∣r)−I(Y;r∣pairmin) =I(Y; q_r)+I(Y; Z_pair q_r)-I(Y; q_r Z_pair ) ≤I(Y,r). ≤ I(Y; q_r). Step 2: three-layer gap decomposition. By the chain rule applied to the latent factors, I(Y;r,1:M,syn) I(Y; q_r, q_1:M, q_syn) =I(Y;r)+I(Y;1:M∣r) =I(Y; q_r)+I(Y; q_1:M q_r) +I(Y;syn∣r,1:M). +I(Y; q_syn q_r, q_1:M). Assumption 1 and the condition stated immediately after it give, respectively, I(Y;X1:M∣r,1:M,syn) I(Y;X_1:M q_r, q_1:M, q_syn) =0, =0, I(Y;r,1:M,syn∣X1:M) I(Y; q_r, q_1:M, q_syn X_1:M) =0. =0. Expanding I(Y;X1:M,r,1:M,syn)I(Y;X_1:M, q_r, q_1:M, q_syn) in the two possible orders therefore yields I(Y;X1:M)=I(Y;r,1:M,syn).I(Y;X_1:M)=I(Y; q_r, q_1:M, q_syn). Together with the bound in Step 1, this gives I(Y;X1:M)−I(Y;pairmin) I(Y;X_1:M)-I(Y; Z_pair ) ≥I(Y;1:M∣r) ≥ I(Y; q_1:M q_r) +I(Y;syn∣r,1:M). +I(Y; q_syn q_r, q_1:M). The first term is the unique gap; the second is the synergistic gap. The second term is strictly positive by hypothesis, so the total information gap is strictly positive. ∎ See 1 Proof. We provide the proof for M=3M=3, as the extension to general M is straightforward. By definition, TC(X1,X2,X3∣r)=H(X1∣r)+H(X2∣r)+H(X3∣r)−H(X1,X2,X3∣r).TC(X_1,X_2,X_3 q_r)=H(X_1 q_r)+H(X_2 q_r)+H(X_3 q_r)-H(X_1,X_2,X_3 q_r). Using the chain rule, H(X1,X2,X3∣r)=H(X1∣r)+H(X2∣X1,r)+H(X3∣X1,X2,r).H(X_1,X_2,X_3 q_r)=H(X_1 q_r)+H(X_2 X_1, q_r)+H(X_3 X_1,X_2, q_r). Substituting this identity yields TC(X1,X2,X3∣r)=I(X1;X2∣r)+I(X3;X1,X2∣r).TC(X_1,X_2,X_3 q_r)=I(X_1;X_2 q_r)+I(X_3;X_1,X_2 q_r). The symmetric form follows by applying the chain rule under each permutation. ∎ See 2 Proof. We first establish the above equation and further interpret this theorem. Step 1: the identity. By the definition of mutual information, I(syn;X1:M∣r)=H(X1:M∣r)−H(X1:M∣r,syn).I( q_syn;X_1:M q_r)=H(X_1:M q_r)-H(X_1:M q_r, q_syn). Recall that TC(X1:M∣C):=∑mH(Xm∣C)−H(X1:M∣C)TC(X_1:M C):= _mH(X_m C)-H(X_1:M C) for any conditioning variable C. Rewriting both joint entropies: H(X1:M∣r) H(X_1:M q_r) =∑mH(Xm∣r)−TC(X1:M∣r), = _mH(X_m q_r)-TC(X_1:M q_r), H(X1:M∣r,syn) H(X_1:M q_r, q_syn) =∑mH(Xm∣r,syn)−TC(X1:M∣r,syn). = _mH(X_m q_r, q_syn)-TC(X_1:M q_r, q_syn). Definition 1 gives I(syn;Xm∣r)=0I( q_syn;X_m q_r)=0 for every singleton m⊊1,…,M\m\ \1,…,M\, so H(Xm∣r,syn)=H(Xm∣r)H(X_m q_r, q_syn)=H(X_m q_r). Substituting and cancelling: I(syn;X1:M∣r)=TC(X1:M∣r,syn)−TC(X1:M∣r).I( q_syn;X_1:M q_r)=TC(X_1:M q_r, q_syn)-TC(X_1:M q_r). Step 2: proper-subset marginals remain unchanged. Definition 1 gives syn⟂XA|r q_syn X_A q_r for every proper subset A⊊1,…,MA \1,…,M\. Hence conditioning on syn q_syn leaves all proper-subset marginals unchanged, and the TC difference comes from the change in the full joint distribution. For M≥3M≥ 3, this includes every pairwise marginal. ∎ Proposition 2 (Shared-Factor Dominance Implies Redundant Low-Order Structure). Under Assumption 1 and the representation model (Eq. (1)), let Wij∗=argminW‖i−Wj‖2W_ij^*= _WE\| Z_i-W Z_j\|^2 be the population-optimal cross-modal linear predictor. Then: (a) (Rank bound) rank(Wij∗)≤drrank(W_ij^*)≤ d_r, where dr=dim(r)d_r= ( c_r) is the dimension of the shared subspace. (b) (Collapse under dominance) When the shared-to-total energy ratio approaches one, the normalized reconstruction error vanishes and the embeddings exhibit a redundant low-order structure. Proof. (a) Rank bound. Under the model m=Vmrr+Vmum+ϵm Z_m=V_m^r c_r+V_m^u c_m+ ε_m with mutually uncorrelated components, the cross-covariance is Cov(i,j)=VirΣr(Vjr)⊤,Cov( Z_i, Z_j)=V_i^r _r(V_j^r) , where Σr≔Cov(r)≻0 _r ( c_r) 0. This matrix has rank at most min(rank(Vir),rank(Vjr))≤dr (rank(V_i^r),rank(V_j^r))≤ d_r. The auto-covariance of j Z_j is Σj≔Cov(j)=VjrΣr(Vjr)⊤+VjuΣju(Vju)⊤+Σϵj≻0, _j ( Z_j)=V_j^r _r(V_j^r) +V_j^u _j^u(V_j^u) + _ ε_j 0, where Σju≔Cov(j) _j^u ( c_j) and Σϵj≔Cov(ϵj) _ ε_j ( ε_j). Positivity follows from the noise term. The population-optimal predictor is Wij∗=VirΣr(Vjr)⊤Σj−1,W_ij^*=V_i^r _r(V_j^r) _j^-1, whose rank is at most drd_r since col(Wij∗)⊆col(Vir)col(W_ij^*) (V_i^r). (b) Collapse under dominance. For simplicity, we assume that the observed vectors have been centered and normalized sample-wise: [m]=E[Z_m]=0 and ‖m‖2=tr(Σm)=1E\|Z_m\|^2=tr( _m)=1. Define the unique-plus-noise component of modality m as m≔Vmum+ϵm,Σm≔Cov(m)=VmuΣmu(Vmu)⊤+Σϵm⪰0.n_m V_m^uc_m+ ε_m, _n_m (n_m)=V_m^u _m^u(V_m^u) + _ ε_m 0. Then m=Vmrr+mZ_m=V_m^rc_r+n_m with Cov(r,m)=Cov(c_r,n_m)=0. We impose the following regularity conditions: (A1) Σj≻0 _n_j 0 (or at least Σj≻0 _j 0) to ensure the existence of the optimal linear predictor. (A2) The unique-plus-noise components of different modalities are mutually uncorrelated: Cov(i,j)=Cov(n_i,n_j)=0 for i≠ji≠ j. (A3) VjrV_j^r has full column rank drd_r and Σr≻0 _r 0. Because tr(Σm)=1tr( _m)=1, the shared energy ratio simplifies to αm≔tr(VmrΣr(Vmr)⊤)∈[0,1],andtr(Σm)=1−αm. _m (V_m^r _r(V_m^r) )∈[0,1], ( _n_m)=1- _m. Hence αm→1 _m→ 1 implies tr(Σm)→0tr( _n_m)→ 0. Since Σm⪰0 _n_m 0, ‖Σm‖2≤tr(Σm)→0.\| _n_m\|_2 ( _n_m)→ 0. The optimal linear predictor of rc_r given jZ_j is ^r=Σr(Vjr)⊤Σj−1j,Ωj≔Cov(r−^r)=Σr−Σr(Vjr)⊤Σj−1VjrΣr, c_r= _r(V_j^r) _j^-1Z_j, _j (c_r- c_r)= _r- _r(V_j^r) _j^-1V_j^r _r, where Σj=VjrΣr(Vjr)⊤+Σj≻0 _j=V_j^r _r(V_j^r) + _n_j 0. To prove Ωj→ _j 0 rigorously, we bound it by a feasible sub-optimal estimator. By (A3), let Lj=((Vjr)⊤Vjr)−1(Vjr)⊤L_j= ((V_j^r) V_j^r )^-1(V_j^r) be a left inverse of VjrV_j^r. Define ˇr=Ljj c_r=L_jZ_j. Then r−ˇr=−Ljj,Cov(r−ˇr)=LjΣjLj⊤.c_r- c_r=-L_jn_j, (c_r- c_r)=L_j _n_jL_j . By the optimality of the MMSE predictor, Ωj⪯Cov(r−ˇr) _j (c_r- c_r). Taking spectral norms, ‖Ωj‖2≤‖Lj‖22‖Σj‖2→0,\| _j\|_2≤\|L_j\|_2^2\,\| _n_j\|_2→ 0, so Ωj→ _j 0. The best linear predictor of iZ_i from jZ_j is Wij∗=Cov(i,j)Σj−1=VirΣr(Vjr)⊤Σj−1,W_ij^*=Cov(Z_i,Z_j)\, _j^-1=V_i^r _r(V_j^r) _j^-1, whose column space lies in col(Vir)col(V_i^r), hence rank(Wij∗)≤drrank(W_ij^*)≤ d_r. Its normalized mean-squared error can be written as ‖i−Wij∗j‖2‖i‖2=1−tr(VirΣr(Vjr)⊤Σj−1VjrΣr(Vir)⊤)tr(Σi). E\|Z_i-W_ij^*Z_j\|^2E\|Z_i\|^2=1- tr\! (V_i^r _r(V_j^r) _j^-1V_j^r _r(V_i^r) )tr( _i). Using Σr(Vjr)⊤Σj−1VjrΣr=Σr−Ωj _r(V_j^r) _j^-1V_j^r _r= _r- _j, tr(VirΣr(Vjr)⊤Σj−1VjrΣr(Vir)⊤)=tr(VirΣr(Vir)⊤)−tr(VirΩj(Vir)⊤)=αi−tr(VirΩj(Vir)⊤).tr\! (V_i^r _r(V_j^r) _j^-1V_j^r _r(V_i^r) )=tr\! (V_i^r _r(V_i^r) )-tr\! (V_i^r _j(V_i^r) )= _i-tr\! (V_i^r _j(V_i^r) ). The second term satisfies 0≤tr(VirΩj(Vir)⊤)≤‖Vir‖22‖Ωj‖2rank(Vir)→0.0 \! (V_i^r _j(V_i^r) )≤\|V_i^r\|_2^2\,\| _j\|_2rank(V_i^r)→ 0. Because the normalization guarantees tr(Σi)=1tr( _i)=1, we obtain ‖i−Wij∗j‖2‖i‖2=1−αi+o(1). E\|Z_i-W_ij^*Z_j\|^2E\|Z_i\|^2=1- _i+o(1). If αi→1 _i→ 1 and αj→1 _j→ 1 (the latter ensures Ωj→ _j 0), the right-hand side tends to 00. ∎ See 1 Proof. Without loss of generality, assume modality 1 satisfies the redundancy condition. By Definition 2, there exist linear maps 1jW_1j such that 1=∑j=2M1jj+1,with rank(1j)=r1j≪dj and ‖1‖2→0.Z_1= _j=2^MW_1jZ_j+ δ_1, rank(W_1j)=r_1j d_j and E\| δ_1\|^2→ 0. Substituting into =[1⊗2⊗⋯⊗M]T=E[Z_1 _2 ·s _M] and taking the mode-1 unfolding yields (1)=~(1)+ℰ(1),where ~(1)=∑j=2M1j[j(2⊗⋯⊗M)⊤].T_(1)= T_(1)+E_(1), T_(1)= _j=2^MW_1j\,E\! [Z_j(Z_2 ·s _M) ]. By construction, rank(~(1))≤∑j=2Mr1jrank( T_(1))≤ _j=2^Mr_1j; let R=∑j=2Mr1jR= _j=2^Mr_1j. For the modality-wise normalized embeddings, Cauchy–Schwarz gives ‖ℰ(1)‖F≤(‖1‖2)1/2→0\|E_(1)\|_F≤(E\| δ_1\|^2)^1/2→ 0. For any tolerance ϵ>‖ℰ(1)‖Fε>\|E_(1)\|_F, ~(1) T_(1) is a feasible rank-R approximation to (1)T_(1). By the Eckart–Young–Mirsky theorem [12, 31], the optimal rank-R approximation error satisfies minrank(A)≤R‖(1)−A‖F≤‖ℰ(1)‖F<ϵ. _rank(A)≤ R\|T_(1)-A\|_F≤\|E_(1)\|_F<ε. Consequently, at most R singular values of (1)T_(1) can exceed ϵε, i.e., rank(1)(ϵ)()≤R.rank_(1)^(ε)(T)≤ R. Consider the full orthogonal Tucker decomposition of T obtained via the Higher-Order Singular Value Decomposition (HOSVD): =full×1Q(1)×2⋯×MQ(M),T=G^full×_1Q^(1)×_2·s×_MQ^(M), where each Q(m)∈ℝdm×dmQ^(m) ^d_m× d_m is an orthogonal matrix. The mode-1 unfolding gives (1)=Q(1)(1)full(Q(M)⊗⋯⊗Q(2))⊤,T_(1)=Q^(1)G^full_(1)(Q^(M) ·s Q^(2)) , and by orthogonality, σk((1))=σk((1)full) _k(T_(1))= _k(G^full_(1)) for all k. Hence, rank(1)(ϵ)(full)=rank(1)(ϵ)()≤R.rank_(1)^(ε)(G^full)=rank_(1)^(ε)(T)≤ R. The Gaussian-QR projection in Eq. (4) and the orthogonal Tucker factors used in Appendix B cannot increase the rank or residual norm of this approximation. Hence the truncated core also satisfies rank(1)(ϵ)()≤Rrank_(1)^(ε)(G)≤ R. ∎ See 2 Proof. For ≠0G≠ 0, let (i1⋆,…,iM⋆)(i_1 ,…,i_M ) be an index attaining the maximum entry magnitude of G, and define a one-sparse tensor (1)G^(1) that keeps only i1⋆,…,iM⋆G_i_1 ,…,i_M . Then ‖−(1)‖F2=‖F2−maxi1,…,iMi1,…,iM2.\|G-G^(1)\|_F^2=\|G\|_F^2- _i_1,…,i_MG_i_1,…,i_M^2. If ρ()≥1−δρ(G)≥ 1-δ, then ‖−(1)‖F2‖F2≤δ, \|G-G^(1)\|_F^2\|G\|_F^2≤δ, and hence ‖−(1)‖F‖F≤δ. \|G-G^(1)\|_F\|G\|_F≤ δ. Therefore G is close to a one-sparse tensor in relative Frobenius norm, which indicates a degenerate structure dominated by a single separable component when δ is small. ∎ Lemma 3 (Johnson–Lindenstrauss lemma [10]). If the columns of P(m)P^(m) are i.i.d. standard Gaussian vectors that have been orthonormalized, then for any fixed finite sample set Zb(m)b=1B\Z^(m)_b\_b=1^B independent of P(m)P^(m), by choosing rp=O(ϵ−2logB)r_p=O(ε^-2 B), the following holds with high probability: (1−ϵ)rpdm‖Zb(m)−Zb′(m)‖22 (1-ε) r_pd_m\|Z^(m)_b-Z^(m)_b \|_2^2 ≤‖(Zb(m)−Zb′(m))P(m)‖22 ≤\|(Z^(m)_b-Z^(m)_b )P^(m)\|_2^2 ≤(1+ϵ)rpdm‖Zb(m)−Zb′(m)‖22. ≤(1+ε) r_pd_m\|Z^(m)_b-Z^(m)_b \|_2^2. In other words, pairwise distances are approximately preserved up to a common scale under the projection. Proof. The lemma follows from [10], and we do not provide the proof here for brevity. ∎ NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We focus on the synergistic information in self-supervised multimodal representation learning. We first characterizes the synergistic gap for more than two modalities, and then propose HRIL to preserve the information capacity of synergy. Extensive experiments on diverse real-world datasets verify the effectiveness of our method. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: As shown in Appendix A, we discuss the limitations of this work. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: All theorems, formulas, propositions, and proofs in the main text are properly numbered and cross-referenced. The key proofs have been included in the appendix G. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Detailed analysis and results can be found in Sec. 4. Experimental details can be found in Appendix B. After the anonymity period, we will open-source our training code and key hyperparameters for reproducing. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: After the anonymity period, we will open-source our training code and key hyperparameters for reproducing. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: As shown in Appendix B, we provide comprehensive details of training procedures. Detailed implementation code will be released upon acceptance. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: For all experiments, we report the mean and standard deviation across five independent runs with random seeds in the range [42,46][42,46]. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We use a single NVIDIA RTX 4090 GPU with 24GB of VRAM for all experiments. More details can be found in Appendix B. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification: All experiments utilize publicly available benchmark datasets in accordance with their respective licensing terms. Since this work involves neither human subjects nor personally identifiable information, institutional review board (IRB) approval is not required. Comprehensive experimental configurations are documented in Appendix B. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please refer to Appendix E. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: We utilize the pre-processed datasets provided by MultiBench [27], which have been anonymized to safeguard personal privacy. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have provided detailed descriptions and clearly marked the sources and citations for all models and frameworks involved in the paper within the experimental section. For open-source code, we have included comprehensive comments and explanations for all imported packages and foundational code used. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: This work introduces new assets, including the training code and dataset developed for the HRIL representation learning framework. These resources are comprehensively documented in the experimental section, which details the model architecture, training protocols, and dataset specifications, alongside complete citations for all underlying frameworks and libraries. The accompanying open-source codebase features extensive inline annotations that clarify the functionality of external dependencies and the organization of core modules. A standardized documentation template is provided alongside the release, specifying training configurations, confirming dataset licensing compliance, and outlining model limitations. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: This work does not involve crowdsourcing or research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: This work does not involve crowdsourcing or research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: In this work, the LLM is utilized solely for writing and editing purposes. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.