Paper deep dive
Dynamic Alignment and Calibration for Multimodal Learning
Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.
Tags
Links
- Source: https://arxiv.org/abs/2610.07928v1
- Canonical: https://arxiv.org/abs/2610.07928v1
Trouble viewing inline? Open PDF directly →
Full Text
50,766 characters extracted from source content.
Expand or collapse full text
Dynamic Alignment and Calibration for Multimodal Learning Dynamic Alignment and Calibration for Multimodal Learning Jinghao Xu Zhenhua Guo Xiaofeng Zhu Xiaoshuang Shi Abstract Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (i) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods. †email: nolle98xu@gmail.com†email: cszguo@gmail.com†email: seanzhuxf@gmail.com†email: xsshi2013@gmail.com†affiliation: School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China †affiliation: Tianyijiaotong Technology Ltd.,China †affiliation: School of Computer Science and Technology, Hainan University, Haikou, China 1 Introduction Multimodal learning has achieved significant success in various fields, such as medical diagnosis[36, 27, 10, 47], emotion recognition[19, 9, 14], and scene understanding[46, 13, 30, 33], by integrating complementary information from diverse modalities to enhance representation power and generalization performance. As the pivotal mechanisms for achieving these advantages, alignment and fusion constitute two core sub-problems in multimodal learning. However, in real-world open environments, data quality and reliability across different modalities often fluctuate dynamically due to device heterogeneity, environmental noise, and occlusions[41]. This necessitates multimodal modeling that not only captures cross-modal consistency but also adapts to sample-level variations. Unfortunately, existing methods fall short of adequately modeling such dynamics in both the alignment and fusion stages. During the alignment stage, most methods still rely on uniform static constraints, applying the same alignment intensity to all samples[17]. This paradigm overlooks the significant disparities in discriminability across different samples[29]: while strong alignment helps enhance cross-modal consistency for high-confidence samples that are easy to classify, applying the same intensity to samples with semantic ambiguity or high noise levels may be counterproductive. Forcibly compressing the distance between all positive sample pairs inherently diminishes inter-modal diversity, thereby suppressing the expression of complementary information[23, 10]. Ultimately, this leads to feature space degradation and undermines the representational advantages that multimodal models are intended to possess. During the fusion stage, recent studies[12, 42, 3] introduce confidence- or uncertainty-aware dynamic weighting mechanisms to adaptively calibrate modality contributions. However, these methods often overlook feature magnitude differences and confidence differences across modalities. When the feature magnitude difference between modalities is large, their actual contributions to the fused representation are often already significantly differentiated. In such cases, imposing additional strong constraints may lead to unreasonable contribution reallocation and consequently interfere with effective multimodal fusion. On the other hand, when the confidence difference between modalities is small, excessively enforcing confidence-based modulation of modality contributions may restrict potential synergistic modeling across modalities. Figure 1: Overview of the proposed framework ACML. In ℒDCTAL_DCTA, △ and denote different categories, colors represent modalities, and color intensity reflects confidence. To address the aforementioned issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML), as illustrated in Figure 1. ACML consists of two core components: a dynamic cross-modal triplet alignment (DCTA) module and a difference-aware attention calibration (DAC) module. Specifically, modality-specific features are first extracted using dedicated encoders, followed by an independent classification head with multiple stochastic forward passes for sample-level confidence estimation. The features are then projected through independent layers for dimensional alignment and jointly optimized under the proposed dynamic triplet alignment mechanism. This mechanism enhances inter-class discriminability while enforcing cross-modal semantic consistency for high-confidence positive pairs. For low-confidence samples, the alignment constraint is relaxed and an orthogonality encouragement strategy is introduced to preserve potential complementary information and mitigate noise propagation caused by forced alignment. Furthermore, in the DAC module, we impose consistency regularization on the weighted attention distributions across modalities for the same sample, while adaptively adjusting the regularization strength according to feature magnitude differences and confidence differences among modalities. This design enables more reasonable sample-aware fusion while avoiding unnecessary over-regularization on high-magnitude modalities. Extensive experimental results demonstrate that ACML achieves superior effectiveness and robustness under both standard settings and challenging scenarios involving noisy and missing modalities. 1. We propose ACML, an alignment- and calibration-driven multimodal learning framework that achieves robust and adaptive multimodal representation learning through dynamic cross-modal alignment and difference-aware attention calibration. 2. We design DCTA to enhance semantic consistency for high-confidence positive pairs while promoting diversity between high- and low-confidence pairs, thereby preserving complementary information and mitigating noise propagation. 3. We propose DAC, which dynamically adjusts the strength of attention regularization according to feature magnitude differences and confidence differences, thereby avoiding unreasonable over-regularization on attention weights. 2 Related Work 2.1 Multimodal Alignment Multimodal alignment aims to establish semantic correspondences across different modalities (e.g., text and images) by mapping their representations into a unified shared embedding space. According to the underlying alignment mechanisms, existing approaches can be broadly categorized into explicit and implicit alignment strategies[17]. Explicit methods [39, 35, 18] typically rely on similarity matrices to directly measure cross-modal associations, whereas implicit methods[16, 15, 21] learn joint representations in a latent space through translation or prediction-based objectives. Along the methodological evolution, contrastive learning has emerged as a dominant paradigm for multimodal alignment[18]. Its core principle is to pull positive sample pairs closer while pushing negative pairs apart, thereby enforcing discriminative cross-modal consistency in the embedding space [39, 6]. Our method follows the explicit alignment paradigm but addresses the limited consideration of modality dynamics in prior work by introducing a dynamic-aware mechanism. Moreover, we extend conventional contrastive objectives by designing a triplet alignment loss, which aims to strike a better balance between promoting cross-modal consistency and preserving complementary information. 2.2 Multimodal Fusion Multimodal fusion aims to integrate complementary information from different modalities to improve performance on downstream tasks. Existing approaches are typically categorized into early, intermediate, or late fusion based on the fusion stage [2]. Early fusion methods [40, 28] combine multiple modalities directly at the data level, but they often struggle with high-dimensional or heterogeneous data. Intermediate fusion approaches [5] focus on learning joint feature representations across modalities, while late fusion methods [42, 3] emphasize semantic-level integration of individual modality predictions. Recent studies have increasingly focused on dynamic multimodal fusion [22, 37, 44, 41] to address the observation that the relative importance of each modality may vary across samples or over time. Unlike static fusion strategies that assume fixed modality weights regardless of input conditions, dynamic fusion approaches leverage attention mechanisms or uncertainty modeling to adaptively modulate the contribution of each modality. This adaptability enables them to better handle challenging scenarios, such as degraded data quality or cross-domain variations, thereby improving both robustness and generalization. However, existing studies typically impose homogeneous constraints on fusion weights while overlooking the differences in feature norms and confidence across modalities. This oversight may lead to unreasonable constraints on modality pairs with abnormal norm ratios or highly consistent confidence, thereby suppressing the expression of complementary cross-modal information. In contrast, we introduce a pairwise constraint loss that adaptively adjusts the constraint strength for different modality pairs based on both norm and confidence differences. 3 Method 3.1 Feature Extraction and Confidence Estimation Given a mini-batch of N multimodal samples, each of which has M modality-specific inputs and a corresponding class label. The inputs are represented as mm=1M\X^m\_m=1^M, where m=imi=1NX^m=\x_i^m\_i=1^N denotes the m-th modality of N samples. The associated label set is =y1,y2,…,yNY=\y_1,y_2,…,y_N\, with yi∈1,…,Cy_i∈\1,…,C\. For the m-th modality, the input mX^m is first passed through a modality-specific encoder fencm(⋅)f^m_enc(·) to extract high-level feature representations: m=fencm(m),Z^m=f^m_enc (X^m ), (1) where m=[1m,2m,…,Nm]⊤∈ℝN×dmZ^m=[z_1^m,z_2^m,…,z_N^m] ^N× d_m is the feature matrix, im∈ℝdmz_i^m ^d_m represents the feature vector of the i-th sample. To obtain robust and stable confidence estimates during training, we employ a stochastic inference scheme. Specifically, we leverage the classifier fclsm(⋅)f_cls^m(·) to construct two parallel computational paths. In the primary training path, the feature imz_i^m is directly fed into the classifier without activating dropout. This path is supervised by the standard cross-entropy loss ℒclsmL_cls^m, which is to ensure that the model learns fundamental discriminative representations: ℒclsm=−1N∑i=1Nlog([s(fclsm(im))]yi),L_cls^m=- 1N _i=1^N ( [s(f_cls^m(z_i^m)) ]_y_i ), (2) where s(⋅)s(·) denotes the softmax activation function, and [⋅]yi[·]_y_i extracts the probability corresponding to the ground-truth index. Meanwhile, to mitigate the prevalent over-confidence issue associated with a single deterministic forward pass and to enhance the quality of confidence estimation, we introduce a probabilistic auxiliary path. Under this path, we apply a dropout operator δ(⋅)δ(·) to the same feature vector imz_i^m and execute K stochastic forward passes. In this work, we set the dropout probability to 0.10.1 and the number of stochastic forward passes K to 1010. For each pass k∈1,…,Kk∈\1,…,K\, the generated stochastic prediction vector is formulated as: i,km=s(fclsm(δ(im))).p_i,k^m=s(f_cls^m(δ(z_i^m))). (3) The final confidence score cimc_i^m for the i-th sample is defined as the empirical mean of the ground-truth probabilities across these K stochastic predictions: cim=1K∑k=1K[i,km]yi.c_i^m= 1K _k=1^K [p_i,k^m ]_y_i. (4) After obtaining the confidence vector m=[c1m,c2m,…,cNm]⊤∈ℝNc^m=[c_1^m,c_2^m,…,c_N^m] ^N, we detach it from the computational graph to serve as dynamic guidance for subsequent modules. 3.2 Dynamic Cross-modal Triplet Alignment Cross-modal semantic alignment is a fundamental component of multimodal learning [17]. However, due to the varying discriminative capabilities across modalities and the need to preserve complementary information, enforcing uniform semantic consistency constraints on all sample pairs is often suboptimal. To address this issue, we propose a Dynamic Cross-modal Alignment loss, denoted as ℒDCTAL_DCTA. The proposed loss dynamically adjusts alignment strength according to sample confidence and explicitly preserves feature separability based on confidence discrepancy, thereby alleviating noisy semantic aggregation and promoting feature diversity. For the feature representation mZ^m of the m-th modality, we first employ a modality-specific projection layer fprojm(⋅)f_proj^m(·), implemented as an MLP, to map the features into a shared multimodal embedding space: m=fprojm(m)=1m,2m,…,Nm,H^m=f_proj^m(Z^m)=\h_1^m,h_2^m,…,h_N^m\, (5) where im∈ℝdh_i^m ^d denotes the embedded representation of the i-th sample in the m-th modality. Within the embedding space, we compute the cosine similarity matrix m,n∈ℝN×NS^m,n ^N× N for each modality pair (m,n)(m,n), where each element sijm,ns_ij^m,n is defined as: sijm,n=im⊤jn‖im‖⋅‖jn‖.s_ij^m,n= h_i^m h_j^n\|h_i^m\|·\|h_j^n\|. (6) To dynamically regulate the alignment strength according to paired-sample reliability, we define a joint confidence weight: wijm,n=min(cim,cjn).w_ij^m,n= (c_i^m,c_j^n). (7) In this way, high-confidence positive pairs are encouraged to achieve stronger semantic consistency, while unreliable pairs are softly relaxed to alleviate noisy semantic aggregation. The alignment term is formulated as: Alignim,n=∑j=1N(yi=yj)wijm,nexp(sijm,nτ),Align_i^m,n= _j=1^NI(y_i=y_j)\,w_ij^m,n\, ( s_ij^m,nτ ), (8) where (⋅)I(·) denotes the indicator function and τ is a temperature parameter. Although the confidence-weighted alignment effectively alleviates noisy aggregation caused by unreliable samples, it does not explicitly preserve the intrinsic representation diversity within the same class. To this end, we further introduce a confidence-gap-aware decorrelation constraint to encourage more diverse feature representations among positive pairs. Specifically, we define a confidence-gap weight as: w~ijm,n=|cim−cjn|. w_ij^m,n=|c_i^m-c_j^n|. (9) This weight adaptively modulates the strength of the decorrelation constraint according to the confidence discrepancy between positive pairs. Based on this weighting scheme, we further construct an orthogonality-inspired decorrelation objective using the absolute feature similarity. Positive pairs with larger confidence gaps are subjected to stronger decorrelation, encouraging them to maintain lower feature correlation. As a result, the learned representations remain semantically consistent while preserving sufficient diversity, thereby retaining potential complementary information. The decorrelation term is defined as: Decorim,n=∑j=1N(yi=yj)w~ijm,nexp(|sijm,n|τ).Decor_i^m,n= _j=1^NI(y_i=y_j)\, w_ij^m,n\, ( |s_ij^m,n|τ ). (10) For negative pairs (yi≠yjy_i≠ y_j), we further introduce a repulsion term to separate irrelevant category representations in the embedding space: Repelim,n=∑j=1N(yi≠yj)exp(sijm,nτ).Repel_i^m,n= _j=1^NI(y_i≠ y_j)\, ( s_ij^m,nτ ). (11) Finally, we integrate the alignment, decorrelation, and repulsion terms into a unified optimization objective. The batch-wise loss between modality m and modality n is: ℒDCTAm,n=−1N∑i=1NlogAlignim,nAlignim,n+Decorim,n+Repelim,n.L_DCTA^m,n=- 1N _i=1^N Align_i^m,nAlign_i^m,n+Decor_i^m,n+Repel_i^m,n. (12) The final global objective is obtained by summing all bidirectional modality-pair losses: ℒDCTA=∑1≤m<n≤M(ℒDCTAm,n+ℒDCTAn,m).L_DCTA= _1≤ m<n≤ M (L_DCTA^m,n+L_DCTA^n,m ). (13) 3.3 Difference-aware Attention Calibration In dynamic multimodal fusion, modality contributions are typically modulated by predictive confidence or uncertainty [11, 42, 3]. Existing methods typically adopt a unified global attention calibration strategy without adequately considering sample-wise modality differences. However, when the feature magnitude difference between modalities is large, their contributions to the fused representation are often already significantly differentiated, making additional strong calibration potentially unnecessary and even harmful to effective multimodal fusion. Meanwhile, when the confidence difference between modalities is small, excessive calibration may suppress potential synergistic information across modalities. To address these issues, we propose a difference-aware attention calibration constraint, termed ℒDACL_DAC, which adaptively emphasizes modality pairs with distinguishable confidence levels and comparable feature magnitudes, thereby enabling more adaptive and effective multimodal fusion. Specifically, for the i-th sample, we employ a modality-specific attention layer fattnmf^m_attn, implemented using a modality-specific MLP followed by a Sigmoid function, to compute the attention score for each modality. The original features are then reweighted by the corresponding attention scores to obtain the attention-weighted features ~im h_i^m, as follows: αim,~im=fattnm(im). _i^m, h_i^m=f_attn^m(h_i^m). (14) To calibrate the relative attention allocation at the sample level, we define the pairwise attention ratio and confidence ratio for each modality pair (m,n)(m,n) as: AttnRatioim,n=αimαim+αin+ε,ConfRatioim,n=cimcim+cin+ε,AttnRatio_i^m,n= _i^m _i^m+ _i^n+ , _i^m,n= c_i^mc_i^m+c_i^n+ , (15) where AttnRatioim,nAttnRatio_i^m,n measures the relative attention allocation between modalities, while ConfRatioim,nConfRatio_i^m,n represents their relative predictive confidence. Here, ε is a small constant introduced for numerical stability. Furthermore, to selectively enforce reliable calibration signals, we define a difference-aware pairwise weight as: w^im,n=γim,n⋅(|cim−cin|+1), w_i^m,n= _i^m,n· (|c_i^m-c_i^n|+1 ), (16) where γim,n=min(‖im‖,‖in‖)max(‖im‖,‖in‖)+ε _i^m,n= (\|h_i^m\|,\|h_i^n\|) (\|h_i^m\|,\|h_i^n\|)+ measures the similarity between modality magnitudes. When the magnitude difference between modalities is large, the corresponding calibration constraint is adaptively relaxed to avoid unreasonable contribution reallocation. In contrast, modality pairs with comparable feature magnitudes receive relatively larger calibration weights. Meanwhile, the confidence difference term (|cim−cin|+1)(|c_i^m-c_i^n|+1) preserves a basic alignment objective while further emphasizing modality pairs with more distinguishable reliability levels. Therefore, w^im,n w_i^m,n jointly models feature magnitude differences and confidence differences, enabling more adaptive and reliable attention calibration. Finally, the difference-aware attention calibration constraint between modality m and modality n is formulated as: ℒDACm,n=1N∑i=1Nw^im,n(AttnRatioim,n−ConfRatioim,n)2.L_DAC^m,n= 1N _i=1^N w_i^m,n (AttnRatio_i^m,n-ConfRatio_i^m,n )^2. (17) The global difference-aware attention calibration constraint is obtained by summing over all modality pairs: ℒDAC=∑1≤m<n≤MℒDACm,n.L_DAC= _1≤ m<n≤ ML_DAC^m,n. (18) 3.4 Fusion and Optimization Objective After cross-modal alignment and attention calibration, we adopt a mean fusion strategy to aggregate all reweighted modality representations and perform final classification through the classification head fcls(⋅)f_cls(·): ifuse=1M∑m=1M~im,ℒcls=−1N∑i=1Nlog([s(fcls(ifuse))]yi).h_i^fuse= 1M _m=1^M h_i^m, _cls=- 1N _i=1^N ( [s (f_cls(h_i^fuse) ) ]_y_i ). (19) Finally, the overall optimization objective of ACML is formulated as ℒ=ℒCLS+λ1ℒDCTA+λ2ℒDAC,L=L_CLS+ _1L_DCTA+ _2L_DAC, (20) where ℒCLS=ℒcls+∑m=1MℒclsmL_CLS=L_cls+ _m=1^ML_cls^m denotes the summation of the fused classification loss and all unimodal classification losses, while λ1 _1 and λ2 _2 are balancing hyperparameters for ℒDCTAL_DCTA and ℒDACL_DAC, respectively. For clarity, we summarize the whole training procedure of ACML in Algorithm 3.4. algorithm[htbp] ACML algorithmic[1] data D, the number of training iterations T. trained ACML. ∈ [1, T] select a batch of samples mm=1M,\\X^m\^M_m=1,Y\ from D; # Obtain modality-specific representation. mm=1M←fencm(m)m=1M\Z^m\^M_m=1←\f^m_enc(X^m)\^M_m=1 # Get the confidence score for each modal sample. mm=1M←\c^m\_m=1^M← Eqs. (2)-(4) # Map multimodal features to the same dimension. mm=1M←fprojm(m)m=1M\H^m\^M_m=1←\f^m_proj(Z^m)\^M_m=1 # Perform dynamic cross-modal triplet alignment. ℒDCTA←L_DCTA← Eqs. (6)-(13) # Compute attention weights and reweight features. αm,~m←fattnm(m)α^m, H^m← f_attn^m(H^m) # Perform difference-aware attention calibration. ℒDAC←L_DAC← Eqs. (15)-(18) # Perform multimodal fusion and final classification. ,ℒcls←H^fuse,L_cls← Eq. (19) # Integrate multiple constraints into the final optimization objective. ℒ←L← Eq. (20) -propagate ℒL to update model parameters; 4 Experiments and Analysis In this section, we evaluate the effectiveness of the proposed ACML by addressing the following six Research Questions (RQs): • RQ1 (Effectiveness & Robustness): How does ACML compare with SOTA baselines in terms of overall performance and robustness against noise? • RQ2 (Ablation Study): What are the individual contributions of the key components within ACML to its overall performance? • RQ3 (Visualization Analysis): How does ℒDCTAL_DCTA influence the cross-modal feature distribution in the embedding space? • RQ4 (Hyper-parameter Analysis): How do the critical hyper-parameters influence the stability of ACML? • RQ5 (Applicability to Missing Modalities): Can ACML effectively enhance prompt-tuning frameworks as a plug-and-play module in challenging missing-modality scenarios? • RQ6 (Scalability to Multiple Modalities): Does ACML maintain its effectiveness and scalability when extended to complex scenarios with more than two modalities (M>2M>2)? 4.1 RQ1 (Effectiveness & Robustness) Table 1: Classification comparison when 50% of the modalities are corrupted with Gaussian noise, i.e., zero mean with variance of ϵε. We highlight the optimal results using bold text and indicate the suboptimal results with underlining for clarity. 2*Dataset 2*Method ϵ=0.0ε=0.0 ϵ=5.0ε=5.0 ϵ=10.0ε=10.0 Avg±Std Worst Avg±Std Worst Avg±Std Worst 10* Depth V2 Depth 61.97±0.99 60.40 49.65±3.22 43.27 42.42±2.42 36.85 RGB 62.57±0.82 61.32 52.56±2.28 47.55 46.16±3.10 38.84 LateFusion 69.22±0.98 67.43 56.35±2.84 51.22 48.65±3.40 42.36 Concat 68.70±0.67 67.89 55.76±2.23 49.54 48.17±3.36 40.21 TMC 71.42±0.84 70.03 61.44±2.19 56.58 53.36±3.36 45.87 QMF 68.55±1.29 66.82 56.60±3.12 50.31 49.04±3.69 41.44 PDF 68.46±0.69 66.67 56.67±2.69 48.47 48.61±3.56 38.23 LFM 69.27±0.62 68.20 59.17±2.15 53.98 51.35±2.70 44.95 OPM 71.35±0.93 69.88 63.35±1.41 60.40 54.82±2.37 49.54 !20ACML !2072.72±0.81 !2071.56 !2065.43±1.53 !2060.86 !2058.11±2.13 !2049.69 10* RGB-D Depth 52.58±0.67 51.43 39.05±1.06 37.15 34.66±1.61 31.08 RGB 57.46±0.39 56.84 48.05±0.88 45.25 42.45±1.47 38.74 LateFusion 59.90±0.45 59.24 47.78±2.03 43.01 41.27±2.82 35.57 Concat 59.52±0.50 58.58 48.54±1.63 45.12 42.74±1.98 38.96 TMC 61.02±0.37 60.34 50.64±1.69 47.18 43.89±3.22 37.54 QMF 61.11±0.26 60.59 51.44±1.06 49.13 46.45±1.61 42.71 PDF 60.89±0.39 60.23 50.24±1.43 46.71 44.92±2.21 40.44 LFM 61.98±0.25 61.64 52.84±0.96 50.55 47.36±1.72 42.58 OPM 62.23±0.26 61.79 51.72±1.08 49.60 43.62±1.90 40.59 !20ACML !2063.25±0.25 !2062.76 !2056.50±0.87 !2054.15 !2049.90±1.15 !2047.48 10* 101 Img 64.78±0.17 64.54 34.55±0.58 33.66 32.99±0.28 32.54 Text 86.49±0.05 86.44 67.35±0.18 67.04 43.84±0.31 43.39 LateFusion 90.71±0.09 90.61 70.07±2.86 64.56 59.51±0.87 57.81 Concat 88.35±0.57 87.68 63.28±2.35 59.45 51.18±3.71 46.34 TMC 90.26±0.17 89.94 73.69±0.82 72.15 61.51±0.21 61.01 QMF 92.88±0.11 92.70 76.39±0.62 75.30 61.86±0.41 61.74 PDF 92.98±0.17 92.71 75.34±0.55 74.29 62.77±0.23 62.45 LFM 92.80±0.05 92.74 76.15±0.35 75.51 62.57±0.34 61.88 OPM 92.60±0.22 92.21 75.17±0.51 74.37 60.97±0.78 59.66 !20ACML !2093.36±0.10 !2093.23 !2076.64±0.49 !2075.69 !2063.15±0.24 !2062.76 Setting. We evaluate ACML on two representative multimodal tasks: (1) Scene Recognition on the NYU Depth V2[25] and SUN RGB-D [26] datasets, and (2) Image–Text Classification on the UPMC FOOD101 [32] dataset. Detailed dataset descriptions are provided in the supplementary material. All models are implemented in PyTorch and optimized using Adam on an NVIDIA A100 GPU. For the scene recognition task, we adopt ResNet-18 as the backbone network, with a learning rate of 1×10−41× 10^-4, a batch size of 16, and an embedding dimension d of 512. For the image–text classification task, ResNet-152 and BERT are used to extract image and text features, respectively, with a learning rate of 5×10−55× 10^-5, an effective batch size of 640 achieved via gradient accumulation (i.e., batch size 32 with 20 accumulation steps), and an embedding dimension d of 2048. Following [20], we simulate real-world multimodal uncertainty by injecting Gaussian noise into images and blank noise into text. Scene recognition experiments are repeated over 10 random seeds, while image–text classification experiments are conducted with 5 random seeds. We report both the average accuracy (with standard deviation) and the worst-case accuracy. All models are trained for up to 100 epochs, with early stopping applied to prevent overfitting. Analysis. As shown in Table 1, we compare ACML with three categories of methods: traditional fusion approaches (Concat and LateFusion), state-of-the-art dynamic learning methods (TMC[12], QMF[42], and PDF[3]), and modality-imbalance learning methods (LFM[38] and OPM[34]). The results show that ACML consistently achieves the best performance across all datasets under both clean and noisy conditions. Specifically, under the clean setting (ϵ=0ε=0), ACML yields relative gains of 1.82%, 1.64%, and 0.41% in average accuracy over the strongest baseline on NYU Depth V2, SUN RGB-D, and FOOD101, respectively. As the noise level increases, the advantage of ACML becomes more pronounced on NYU Depth V2 and SUN RGB-D. For example, under severe corruption (ϵ=10ε=10), the relative gains in average accuracy further increase to 6.00% and 5.36%, respectively. This trend suggests that ACML not only improves predictive performance under clean conditions but also exhibits stronger robustness against modality corruption. 4.2 RQ2 (Ablation Study) Table 2: Ablation studies on NYU Depth V2. We report both the average performance and the worst-case performance under different noise levels. 2*Configuration ϵ=0.0ε=0.0 ϵ=5.0ε=5.0 ϵ=10.0ε=10.0 (lr)2-3 (lr)4-5 (lr)6-7 Avg±Std Worst Avg±Std Worst Avg±Std Worst 70.87±0.79 69.27 62.61±1.53 58.87 55.02±3.38 46.79 Baseline + ℒDCTAL_DCTA 72.37±0.80 70.80 63.82±1.97 58.56 55.70±2.91 48.62 Baseline + ℒDACL_DAC 71.67±0.82 70.32 63.66±2.58 58.42 53.93±4.33 47.59 !20 ACML 72.72±0.71 71.56 65.43±1.63 60.86 58.11±2.69 49.69 (ℒDCTA→ℒSCL_DCTA _SC) 72.48±0.69 71.10 65.06±1.74 60.86 57.09±2.75 51.07 ACML (ℒDCTA→ℒCSCL_DCTA _CSC) 72.55±0.76 71.41 64.67±1.55 60.09 57.43±2.39 51.38 (ℒDAC→ℒMSEL_DAC _MSE) 72.49±0.71 71.10 64.78±2.57 58.72 57.41±3.15 49.69 Setting. We conduct ablation studies on the NYU Depth V2 dataset to systematically analyze the contributions of the dynamic cross-modal triplet alignment loss (ℒDCTAL_DCTA) and the difference-aware attention constraint (ℒDACL_DAC). Furthermore, based on the final ACML framework, we further investigate the effectiveness of different optimization objectives and constraint formulations. Specifically, we replace ℒDCTAL_DCTA with the standard supervised contrastive loss (ℒSCL_SC) and the confidence-constrained supervised contrastive loss (ℒCSCL_CSC), respectively. Meanwhile, we replace the proposed ℒDACL_DAC with a standard unweighted MSE constraint (ℒMSEL_MSE) to analyze the impact of difference-aware weighting on the final performance. We report the average classification accuracy with standard deviation and the worst-case classification accuracy under different noise conditions. Analysis. As shown in Table 2, introducing either ℒDCTAL_DCTA or ℒDACL_DAC consistently improves the performance compared with the original baseline model under different noise conditions, demonstrating their effectiveness in enhancing cross-modal representation learning. Moreover, the complete ACML framework achieves the best performance in terms of both average classification accuracy and worst-case classification accuracy, further validating the complementarity between ℒDCTAL_DCTA and ℒDACL_DAC. Furthermore, replacing ℒDCTAL_DCTA with either ℒSCL_SC or ℒCSCL_CSC leads to inferior performance, demonstrating the superiority of the proposed dynamic cross-modal triplet alignment strategy in regularizing cross-modal representation distributions. Similarly, replacing ℒDACL_DAC with the standard unweighted ℒMSEL_MSE constraint also degrades the performance, validating the effectiveness of the proposed difference-aware weighting mechanism. Figure 2: 2D t-SNE visualization of multimodal features. △ indicates the image modality, and denotes the text modality. Each color corresponds to a distinct class, with darker shades indicating higher confidence. 4.3 RQ3 (Visualization Analysis) Setting. To verify whether ℒDCTAL_DCTA effectively reshapes the representation space and improves cross-modal semantic alignment, we perform t-SNE visualizations on the FOOD101 dataset to qualitatively examine the feature distributions before and after applying ℒDCTAL_DCTA. Analysis. As shown in Figure 2, the t-SNE visualizations further reveal that, after introducing ℒDCTAL_DCTA, high-confidence samples form more compact clusters with improved cross-modal consistency. In contrast, low-confidence samples (indicated by lighter colors) remain relatively scattered, indicating that the proposed loss effectively promotes cross-modal alignment. 4.4 RQ4 (Hyper-parameter Analysis) Setting. We evaluate the sensitivity of two key hyper-parameters, λ1 _1 and λ2 _2, on the NYU Depth V2 and FOOD101 datasets, which control the constraint strengths of ℒDCTAL_DCTA and ℒDACL_DAC, respectively. The search spaces for both hyper-parameters are set to 0.1,0.5,1,5 \0.1,0.5,1,5 \. We report the mean classification accuracy under three settings, including the clean setting, ϵ=5ε=5, and ϵ=10ε=10 noisy settings, where the final result is obtained by averaging the accuracies across the three settings. Analysis. As illustrated in Figure 3, excessively small or large values of either λ1 _1 or λ2 _2 lead to noticeable performance degradation. In contrast, when the hyper-parameters are within a reasonable range (e.g., 0.50.5 and 11), the proposed method achieves more stable and robust performance. Similar trends can be observed for both hyper-parameters, and the phenomenon becomes more evident on the larger-scale FOOD101 dataset. These results demonstrate that the proposed method maintains good stability and robustness within a reasonable hyper-parameter range. Figure 3: Performance sensitivity to hyperparameters λ1 _1 and λ2 _2 across two datasets. Table 3: Accuracy comparison on FOOD101 validation and test sets under different missing rates (baselines vs. +ACML) 2* Missing rate η Train/Test Validation Set Testing Set (lr)2-3 (lr)4-7 (lr)8-11 Image Text DCP (+ACML) SyP (+ACML) DCP (+ACML) SyP (+ACML) 3*50% 100% 50% 82.33 84.72 82.64 83.68 82.11 84.42 81.93 83.85 50% 100% 89.23 89.44 89.07 89.16 89.12 89.44 89.18 89.39 75% 75% 85.25 85.86 85.20 86.28 85.24 86.03 86.17 86.50 3*70% 100% 30% 79.18 80.61 79.39 80.21 78.87 80.39 79.28 80.27 30% 100% 87.53 88.11 87.62 87.87 87.32 87.77 87.27 87.75 65% 65% 82.38 83.24 82.53 83.55 81.87 83.11 82.25 83.24 3*90% 100% 10% 75.54 77.21 76.07 77.05 75.26 77.54 76.30 77.47 10% 100% 86.26 86.36 86.50 86.59 85.78 86.00 86.19 86.55 55% 55% 80.39 80.77 80.64 81.48 79.87 79.93 79.31 80.59 4.5 RQ5 (Applicability to Missing Modalities) Setting. To evaluate the adaptability of ACML under missing-modality conditions, we integrate it as a plug-and-play module into two prompt-tuning-based approaches, DCP[24] and SyP[43]. Experiments are conducted on the FOOD101 dataset. Following prior works[24, 43], we consider three missing ratios: 50%, 70%, and 90%. For each ratio, we further evaluate three missing patterns: text missing, image missing, and both modalities missing, where each modality is removed at a ratio of 50%. All methods keep the backbone encoders frozen and only update the prompts together with our module. Performance is reported on both the validation and test sets. Analysis. As reported in Table 3, ACML consistently improves the performance of both DCP and SyP across all missing-modality ratios and missing patterns on both validation and test sets. Notably, the gains remain stable even under severe modality absence (e.g., 90% missing ratio), demonstrating the strong robustness of ACML to incomplete multimodal inputs. These results suggest that ACML can effectively enhance representation learning when modality information is partially unavailable and can be seamlessly integrated into different prompt-tuning frameworks as a general and effective enhancement module. 4.6 RQ6 (Generalization and Scalability) Setting. To evaluate the adaptability of ACML in broader multi-modal scenarios, we conduct classification experiments on two medical multi-omics benchmark datasets: BRCA[31] and ROSMAP[1, 4]. These tasks involve three sequential omics modalities (M=3M=3) and particularly emphasize population-level relationship modeling. Specifically, the BRCA dataset contains 875 samples spanning five subtypes, while the ROSMAP dataset includes 351 samples for binary Alzheimer’s disease diagnosis. For the network architecture, each modality employs a lightweight encoder—consisting of a single linear layer followed by a ReLU activation—to project input features into a 2048-dimensional space (dm=2048d_m=2048), with an MLP used for feature dimension alignment (d=2048d=2048). The model is trained for 500 epochs using the Adam optimizer with a learning rate of 10−410^-4. For evaluation, the BRCA dataset uses accuracy, macro F1-score, and weighted F1-score, while the ROSMAP dataset adopts accuracy, F1-score, and AUC to comprehensively assess diagnostic performance. All results are reported as the mean over 20 random seeds. Analysis. We compare ACML against six SOTA multimodal methods that focus on sequence-level modeling: MOGONet [31], MMDyn [12], MLCLNet [44], DMIB [7], GCFANet [45], and HCMAF [8]. As shown in Table 4, ACML achieves significant and consistent performance gains across all tasks. These results demonstrate that our method maintains strong generalization capability across different data types and more diverse multi-modal scenarios. Table 4: Classification results (%) on the BRCA and ROSMAP datasets. 2*Method BRCA ROSMAP (lr)2-4 (lr)5-7 Accuracy Weighted F1 Macro F1 Accuracy F1 AUC 82.9±1.8 82.5±1.7 77.4±1.7 81.5±2.3 82.1±1.2 87.4±1.2 MMDyn 87.7±0.3 88.0±0.5 84.5±0.5 84.2±1.3 84.6±0.7 91.2±0.7 MLCLNet 86.4±1.6 87.8±1.7 82.6±1.8 84.4±1.5 85.2±1.5 89.3±1.1 DMIB 86.0±1.6 86.0±0.8 81.6±0.9 84.9±1.8 85.3±1.7 91.6±0.7 GCFANet 88.6±1.5 88.9±1.6 85.3±1.6 86.3±1.4 88.3±1.6 91.5±1.2 HCMAF 89.9±0.2 90.1±0.2 86.4±0.6 88.0±0.7 87.9±1.2 91.9±0.7 !20 ACML 90.7±0.2 91.1±0.2 88.1±0.2 88.2±0.8 87.6±0.8 92.4±0.7 5 Conclusion This paper proposes an Alignment- and Calibration-driven Multimodal Learning framework (ACML) to address the limitations of existing methods in static alignment, which overlooks sample-wise differences and fusion strategies lacking difference-aware modeling. Specifically, ACML consists of two key modules: (i) a dynamic cross-modal triplet alignment (DCTA) module, which adaptively adjusts alignment strength according to sample confidence, enforcing cross-modal semantic consistency for high-confidence samples while preserving potential complementary information in low-confidence samples through an orthogonality encouragement strategy; and (i) a difference-aware attention calibration (DAC) module, which adaptively adjusts the attention regularization strength according to feature magnitude differences and confidence differences across modalities, enabling more reasonable sample-aware fusion while avoiding unnecessary over-regularization. Experimental results demonstrate that ACML consistently achieves superior performance and robustness across multiple multimodal benchmark datasets, validating the effectiveness of the proposed modules. In future work, we plan to explore the integration of ACML with large-scale pre-trained foundation models, extending its applicability to complex real-world scenarios such as medical multimodal analysis, as well as multimodal understanding and generation tasks. Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 62276052), and the Medico Engineering Cooperation Funds from University of Electronic Science and Technology of China and West China Hospital (No. ZYGX2022YGRH009). References [1] D. A. Bennett, J. A. Schneider, Z. Arvanitakis, and R. S. Wilson (2012) Overview and findings from the religious orders study. Current Alzheimer Research 9 (6). Cited by: §4.6. [2] P. K. Atrey, M. A. Hossain, A. El Saddik, and M. S. Kankanhalli (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16 (6). Cited by: §2.2. [3] B. Cao, Y. Xia, Y. Ding, C. Zhang, and Q. Hu (2024) Predictive dynamic fusion. In ICML, Cited by: §1, §2.2, §3.3, §4.1. [4] P. L. De Jager, Y. Ma, C. McCabe, et al. (2018) A multi-omic atlas of the human frontal cortex for aging and alzheimer’s disease research. Scientific Data 5 (1). Cited by: §4.6. [5] S. Ding, J. Li, J. Wang, S. Ying, and J. Shi (2024) Multimodal co-attention fusion network with online data augmentation for cancer subtype classification. IEEE Transactions on Medical Imaging 43 (11). Cited by: §2.2. [6] B. Dufumier, J. C. Navarro, D. Tuia, and J. Thiran (2025) What to align in multimodal contrastive learning?. In ICLR, Cited by: §2.1. [7] Y. Fang, S. Wu, S. Zhang, et al. (2024) Dynamic multimodal information bottleneck for multimodality classification. In WACV, Cited by: §4.6. [8] Y. Gan, H. Zhao, K. Wang, C. Yan, and G. Zou (2025) HCMAF: hierarchical feature aggregation and cross-modal attention fusion framework for multi-omics patient classification. IEEE Journal of Biomedical and Health Informatics. Cited by: §4.6. [9] A. Geetha, T. Mala, D. Priyanka, and E. Uma (2024) Multimodal emotion recognition with deep learning: advancements, challenges, and future directions. Information Fusion 105. Cited by: §1. [10] K. Han, D. Hu, F. Zhao, et al. (2025) Incomplete multi-modal disentanglement learning with application to alzheimer’s disease diagnosis. IEEE Transactions on Medical Imaging. Cited by: §1, §1. [11] Z. Han, F. Yang, J. Huang, C. Zhang, and J. Yao (2022) Multimodal dynamics: dynamical fusion for trustworthy multimodal classification. In CVPR, Cited by: §3.3. [12] Z. Han, C. Zhang, H. Fu, and J. T. Zhou (2022) Trusted multi-view classification with dynamic evidential fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1, §4.1, §4.6. [13] L. Kong, X. Xu, J. Ren, et al. (2025) Multi-modal data-efficient 3d scene understanding for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1. [14] D. Li, B. Xing, X. Liu, et al. (2025) Deemo: de-identity multimodal emotion recognition and reasoning. In ACM M, Cited by: §1. [15] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, Cited by: §2.1. [16] J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, Cited by: §2.1. [17] S. Li and H. Tang (2026) Multimodal alignment and fusion: a survey: s. li, h. tang. International Journal of Computer Vision 134 (3). Cited by: §1, §2.1, §3.2. [18] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou (2022) Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 35. Cited by: §2.1. [19] W. Liu, J. Qiu, W. Zheng, and B. Lu (2021) Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems 14 (2). Cited by: §1. [20] H. Ma, Z. Han, C. Zhang, et al. (2021) Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. NeurIPS 34. Cited by: §4.1. [21] Q. Ma, M. Zhang, Y. Tang, and Z. Huang (2023) Att-sinkhorn: multimodal alignment with sinkhorn-based deep attention architecture. In ICAC, Cited by: §2.1. [22] A. Nagrani, S. Yang, A. Arnab, A. Jansen, C. Schmid, and C. Sun (2021) Attention bottlenecks for multimodal fusion. NeurIPS 34. Cited by: §2.2. [23] J. Qu, Y. Yang, W. Dong, and Y. Yang (2024) LDS2AE: local diffusion shared-specific autoencoder for multimodal remote sensing image classification with arbitrary missing modalities. In AAAI, Vol. 38. Cited by: §1. [24] T. Shi, W. Feng, F. Shang, L. Wan, et al. (2024) Deep correlated prompting for visual recognition with missing modalities. NeurIPS 37. Cited by: §4.5. [25] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus (2012) Indoor segmentation and support inference from rgbd images. In ECCV, Cited by: §4.1. [26] S. Song, S. P. Lichtenberg, and J. Xiao (2015) Sun rgb-d: a rgb-d scene understanding benchmark suite. In CVPR, Cited by: §4.1. [27] X. Su, P. Hu, D. Li, et al. (2025) Interpretable identification of cancer genes across biological networks via transformer-powered graph representation learning. Nature Biomedical Engineering. Cited by: §1. [28] X. Suo, M. Chen, L. Chen, et al. (2025) Automatic identification of parkinsonism using clinical multi-contrast brain mri: a large self-supervised vision foundation model strategy. EBioMedicine 116. Cited by: §2.2. [29] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Carneiro (2023) Multi-modal learning with missing modality via shared-specific feature modelling. In CVPR, Cited by: §1. [30] Q. Wang, F. Zhu, G. Wu, et al. (2025) Object-level and scene-level feature aggregation with clip for scene recognition. Information Fusion 120. Cited by: §1. [31] T. Wang, W. Shao, Z. Huang, et al. (2021) MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification. Nature Communications 12 (1). Cited by: §4.6, §4.6. [32] X. Wang, D. Kumar, N. Thome, M. Cord, and F. Precioso (2015) Recipe recognition with large multimodal food dataset. In ICMEW, Cited by: §4.1. [33] Z. Wang, Z. Gao, Y. Yang, et al. (2024) Geometric matching for cross-modal retrieval. IEEE Transactions on Neural Networks and Learning Systems 36 (3). Cited by: §1. [34] Y. Wei, D. Hu, H. Du, and J. Wen (2025) On-the-fly modulation for balanced multimodal learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (1). Cited by: §4.1. [35] J. Xu, H. Tang, Y. Ren, et al. (2022) Multi-level feature learning for contrastive multi-view clustering. In CVPR, Cited by: §2.1. [36] C. Xue, S. S. Kowshik, D. Lteif, et al. (2024) AI-based differential diagnosis of dementia etiologies on multimodal data. Nature Medicine 30 (10). Cited by: §1. [37] Z. Xue and R. Marculescu (2023) Dynamic multimodal fusion. In CVPR, Cited by: §2.2. [38] Y. Yang, F. Wan, Q. Jiang, and Y. Xu (2024) Facilitating multimodal classification via dynamically learning modality gap. NeurIPS 37. Cited by: §4.1. [39] X. Yuan, Z. Lin, J. Kuen, J. Zhang, Y. Wang, M. Maire, A. Kale, and B. Faieta (2021) Multimodal contrastive training for visual representation learning. In CVPR, Cited by: §2.1. [40] Y. Zeng, W. Yan, S. Mai, and H. Hu (2024) Disentanglement translation network for multimodal sentiment analysis. Information Fusion 102. Cited by: §2.2. [41] Q. Zhang, Y. Wei, Z. Han, et al. (2026) Multimodal fusion on low-quality data: a comprehensive survey. Information Fusion 135. Cited by: §1, §2.2. [42] Q. Zhang, H. Wu, C. Zhang, et al. (2023) Provable dynamic fusion for low-quality multimodal data. In ICML, Cited by: §1, §2.2, §3.3, §4.1. [43] Z. Zhang, L. Dai, Q. Lin, Y. Diao, G. Jin, Y. Guo, J. Zhang, and X. Hao (2025) Synergistic prompting for robust visual recognition with missing modalities. In ICCV, Cited by: §4.5. [44] X. Zheng, C. Tang, Z. Wan, C. Hu, and W. Zhang (2023) Multi-level confidence learning for trustworthy multimodal classification. In AAAI, Vol. 37. Cited by: §2.2, §4.6. [45] X. Zheng, M. Wang, K. Huang, and E. Zhu (2024) Global and cross-modal feature aggregation for multi-omics data classification and application on drug response prediction. Information Fusion 102. Cited by: §4.6. [46] W. Zhou, S. Dong, J. Lei, and L. Yu (2022) MTANet: multitask-aware network with hierarchical multimodal fusion for rgb-t urban scene understanding. IEEE Transactions on Intelligent Vehicles 8 (1). Cited by: §1. [47] X. Zhu, L. Sun, Y. Liu, et al. (2025) Causal debiasing medical multimodal representation learning with missing modalities. arXiv preprint arXiv:2509.05615. Cited by: §1.