Paper deep dive
Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination
Kaixin Xu, NaiJin Liu, Yulin Kang, Tangyue Jin, Zixuan Yu, Wenxi Zhao, Yibei Liu, Qianle Zhang, Yangyang Wu, Mengying Zhu, Meng Xi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/21/2026, 4:02:31 AM
Summary
The paper introduces Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), a task addressing the challenge where test-time modality combinations differ from training data. The authors propose Contrastive Mixed Prompt Learning (CMPL), a model utilizing label-guided contrastive feature learning and a soft router for mixed modality-combination prompts. Three prompt contrastive learning strategies are employed to enhance generalization to unseen combinations. Experiments on CMU-MOSI, CMU-MOSEI, and SIMS-V2 datasets show CMPL outperforms state-of-the-art methods by over 5% in accuracy.
Entities (8)
Relation Signals (7)
CMPL → evaluatedon → CMU-MOSI
confidence 95% · Extensive experiments on three widely used datasets demonstrate that CMPL achieves more than a 5% improvement... CMU-MOSI
CMPL → evaluatedon → CMU-MOSEI
confidence 95% · Extensive experiments on three widely used datasets demonstrate that CMPL achieves more than a 5% improvement... CMU-MOSEI
CMPL → evaluatedon → SIMS-V2
confidence 95% · Extensive experiments on three widely used datasets demonstrate that CMPL achieves more than a 5% improvement... SIMS-V2
CMPL → solves → IMSAUMC
confidence 95% · we propose the model named Contrastive Mixed Prompt Learning (CMPL) for IMSAUMC.
CMPL → uses → Label-guided Contrastive Feature Learning
confidence 92% · It introduces a label-guided contrastive feature learning mechanism to learn robust and discriminative cross-modal representations.
IMSAUMC → isa → Multimodal Sentiment Analysis
confidence 90% · Incomplete multimodal sentiment analysis has garnered significant attention... we introduce the problem of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC)
CMPL → uses → Soft Router
confidence 90% · we design modality-combination prompts with a soft router to facilitate better learning of various modality combinations.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Incomplete multimodal sentiment analysis has garnered significant attention in recent years. Existing approaches typically assume that data is missing at random or are designed specifically for certain missing patterns, ignoring the modality combination inconsistency between training and testing phases. However, in real-world scenarios, the testing phase often encounters modal combinations that were not present during the training phase, which leads to insufficient generalization capabilities and unstable performance. In this paper, we introduce the problem of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), aiming to enhance model generalization for unseen modality combinations. To address this challenge, we propose the model named $\textbf{C}$ontrastive $\textbf{M}$ixed $\textbf{P}$rompt $\textbf{L}$earning ($\textsf{CMPL}$) for IMSAUMC. It introduces a label-guided contrastive feature learning mechanism to learn robust and discriminative cross-modal representations. Additionally, we design modality-combination prompts with a soft router to facilitate better learning of various modality combinations. Furthermore, we introduce three prompt contrastive learning strategies, which enable effective learning of prompts corresponding to unseen modality combinations, thereby significantly strengthening the model's generalization capabilities in diverse testing scenarios. Extensive experiments on three widely used datasets demonstrate that $\textsf{CMPL}$ achieves more than a 5% improvement in accuracy compared to state-of-the-art approaches.
Tags
Links
- Source: https://arxiv.org/abs/2608.20019v1
- Canonical: https://arxiv.org/abs/2608.20019v1
Trouble viewing inline? Open PDF directly →
Full Text
65,479 characters extracted from source content.
Expand or collapse full text
Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination Kaixin Xu NaiJin Liu Yulin Kang Tangyue Jin Zixuan Yu Wenxi Zhao Yibei Liu Qianle Zhang Yangyang Wu Mengying Zhu Meng Xi Abstract Incomplete multimodal sentiment analysis has garnered significant attention in recent years. Existing approaches typically assume that data is missing at random or are designed specifically for certain missing patterns, ignoring the modality combination inconsistency between training and testing phases. However, in real-world scenarios, the testing phase often encounters modal combinations that were not present during the training phase, which leads to insufficient generalization capabilities and unstable performance. In this paper, we introduce the problem of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), aiming to enhance model generalization for unseen modality combinations. To address this challenge, we propose the model named Contrastive Mixed Prompt Learning (CMPL) for IMSAUMC. It introduces a label-guided contrastive feature learning mechanism to learn robust and discriminative cross-modal representations. Additionally, we design modality-combination prompts with a soft router to facilitate better learning of various modality combinations. Furthermore, we introduce three prompt contrastive learning strategies, which enable effective learning of prompts corresponding to unseen modality combinations, thereby significantly strengthening the model’s generalization capabilities in diverse testing scenarios. Extensive experiments on three widely used datasets demonstrate that CMPL achieves more than a 5% improvement in accuracy compared to state-of-the-art approaches. Introduction Multimodal Sentiment Analysis (MSA) has emerged as a pivotal research area in recent years. By jointly modeling textual utterances, acoustic characteristics, and visual expressions, MSA systems aim to infer human affective states with significantly enhanced robustness compared to unimodal approaches. Many works (7; 40; 10; 19; 28) have achieved promising results by exploiting cross-modal complementarity. However, real-world deployment scenarios frequently violate the full-modality assumption. Practical challenges such as sensor failures, background noise, occlusions, and privacy constraints often lead to missing modalities during inference. Consequently, substantial research efforts have been directed toward MSA with missing modality (22; 20; 8; 14; 15). Figure 1: The difference between the general MSA with missing modality task and the IMSAUMC task. In the general MSA with missing modality task, all possible modality combinations are available during both the training and testing phases. However, in our IMSAUMC task, only partial modality combinations are present during training, while unseen modality combinations may appear during testing. Although these approaches demonstrate resilience against random modality absence, they fundamentally assume that all modality combinations have been observed during training. In real-world data, however, missing patterns are often structured rather than random. For example, a camera failure causes the absence of visual features; a subsequent audio failure then results in missing the visual-audio modality combination entirely. Consequently, the dataset may contain only a subset of possible combinations, and the test set is likely to encounter unseen ones. For another example, users on social media may decline to upload certain modalities (e.g., audio) due to privacy concerns, yielding datasets with only partial combinations. In such scenarios, we aim to train models using only the available partial combinations, while still ensuring strong generalization when more complete modality combinations appear at inference time. For a dataset with n modalities, 2n−12^n-1 possible combinations exist. As illustrated above, the dataset may not cover all missing patterns, and the test set may encounter unseen modality combinations, especially as n grows. Consequently, existing methods often struggle to handle such scenarios effectively, and approaches capable of generalizing to unseen modality combinations are critically needed. We introduce the task of Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinations (IMSAUMC), which handles missing modalities and unseen modality combinations during testing, as shown in Figure 1. While previous work in MSA with missing modalities has achieved notable progress, two key challenges remain: First, many works employ contrastive learning to obtain modality-invariant representations. However, most fail to account for the intrinsic data structure, potentially separating semantically similar samples, which leads to suboptimal representations (CH1). Second, in scenarios involving unseen modality combinations, existing methods often overlook the relations between different modality combinations. As a result struggle to effectively handle test-time data whose modality combinations were not encountered during the training phase (CH2). To address these challenges, we propose Contrastive Mixed Prompt Learning (CMPL) for multimodal sentiment analysis with unseen modality combinations. Specifically, for CH1, we introduce label-guided contrastive feature learning, which incorporates label similarity constraints to pull representations of same-labeled samples closer while maintaining distances proportional to label dissimilarity for others. For CH2, we design mixed modality-combination prompts with a soft routing mechanism that dynamically selects prompts to comprehensively model inter-combinatorial relations. Moreover, we develop three prompt contrastive learning strategies to further enhance generalization to unseen combinations: a modality information preservation strategy, a cross-combination complementarity strategy, and a conditional information alignment strategy. The main contributions are summarized as follows: • We propose CMPL, a novel model for the IMSAUMC task that improves generalization to unseen modality combinations. To the best of our knowledge, this is the first work addressing this problem. • We propose a label-guided contrastive feature learning mechanism, which enforces representation consistency for samples with identical labels while constraining the distance between dissimilar samples proportionally to their label differences. This preserves the structural relationships within sample representations. • We introduce a mixed prompts learning mechanism coupled with three prompt contrastive learning strategies. These comprehensively model inter-combinatorial relationships and enhance the model’s generalization to unseen modality combinations. • Extensive experiments on CMU-MOSI, CMU-MOSEI, and SIMS-V2 datasets demonstrate the effectiveness of our method over state-of-the-art approaches. Related Work Multimodal Sentiment Analysis Multimodal Sentiment Analysis (MSA) aims to infer sentiment by integrating heterogeneous data from multiple modalities, such as text, visual, and acoustic signals. MSA methods (26; 33; 21; 23; 16) leverage cross-modal complementarity to improve robustness and accuracy over unimodal approaches. For instance, 33 proposed self-M (33), jointly training a multimodal main task with unimodal subtasks as pseudo-label supervision to learn inter-modal consistency and cross-modal differences. 23 presented Cube-MLP (23), which mixes features along three axes via MLP units. 16 introduced DMD (16), decoupling homogeneous and heterogeneous features with adaptive cross-modal distillation to enhance modality discriminability. 28 proposed DLF (28), a disentangled language-focused framework that reduces cross-modal redundancy for improved MSA performance. Figure 2: The framework of our CMPL, which consists of three components: the labeled-guided contrastive feature learning module, the mixed prompts learning mechanism, and prompt contrastive learning strategies. Taking the absence of audio-visual and text-audio-visual modality combinations in training data, as well as the missing vision modality in input, as an example. However, real-world data often suffers from missing modalities. Many methods (34; 38; 35; 29; 39) have been developed to address MSA with missing modality. For example, TFR-Net (34) employs a feature reconstruction module to generate missing modality content. LNLN (39) improves robustness by guaranteeing high-quality dominant modality representation. HME (42) leverages cross-sample semantic enrichment and uncertainty-aware fusion, eliminating explicit modality reconstruction while enhancing robustness and generalization. MFMB-Net (24) jointly performs global–local dual-stream fusion and collaborative feature reconstruction to robustly handle missing modalities. However, these methods overlook inconsistent distributions of modality combinations in missing-modality scenarios, where the test set may contain unseen modality combinations that were not present during training. In contrast, our approach focuses on leveraging known modality combination information to enhance generalization to unseen combinations. Prompt Learning Prompt learning has emerged as a powerful paradigm for adapting pre-trained models such as large language models (LLM) to downstream tasks (4; 6; 17; 41). 27 concatenated visual embeddings as prefix prompts to enable frozen language models to generate appropriate captions (27). 13 designed missing-aware prompts for different missing-modality cases to enhance robustness (13). 12 designed branch-aware multi-modal prompts to enhance alignment between language and visual modalities (12). MPLMM (5) generates missing modality features and strengthens intra- and inter-modality learning by designing generative, missing-signal, and missing-type prompts. These methods ignore the relations between modality combinations, limiting generalization to unseen combinations. In contrast, our approach thoroughly explores inter-modal relationships and designs three prompt contrastive learning strategies to address unseen modality combinations. Methodology Problem Formulation Given a multimodal dataset =t,a,vX=\X^t,X^a,X^v\ with three modalities (text, audio, visual), each k=1k,⋯,Nk∈ℝN×lk×dkX^k=\x^k_1,·s,x^k_N\ ^N× l_k× d_k denotes the feature matrix of modality k, where N is the number of samples, and lkl_k, dkd_k are the sequence length and embedding dimension, with k∈t,a,vk∈\t,a,v\. The sentiment labels are ∈ℝNY ^N. For missing modalities, we define mask matrices tM^t, aM^a, vM^v where ik=0M^k_i=0 indicates the i-th sample is missing in modality k and ik=1M^k_i=1 indicates its presence. The textual features are obtained from text embeddings of a LLM, while audio and visual features are extracted using pre-trained toolkits. For these three modalities, excluding the all-absent case, there are T=7T=7 possible modality combinations, denoted as =S1,S2,⋯,ST=t,a,v,t,a,t,v,a,v,t,a,vS=\S_1,S_2,·s,S_T\=\\t\,\a\,\v\,\t,a\,\t,v\,\a,v\,\t,a,v\\. For example, S4S_4 represents the text-audio combination with visual modality missing, with available data =t,ax=\x^t,x^a\. More detailed information can be found in Appendix A. Definition. The objective of IMSAUMC is to train a model for sentiment analysis under the condition that the training data only contains a subset of the modality combinations in S, while the test data includes all possible modality combinations (i.e., S). The model must generalize to unseen modality combinations during testing. Overall Framework Figure 2 shows the framework of our CMPL. First, during the representation learning stage, we extract sequential embeddings for the audio and visual modalities using pre-trained tools and reduce the sequence length via adaptive average pooling. For the textual modality, we leverage text embeddings from the LLM to obtain contextually correlated embeddings. Then, guided by label similarity, we apply contrastive learning across modality embeddings to learn robust representations while preserving semantic structure. Additionally, we equip each modality combination with a prompt. The multimodal embeddings are then fed into a router to generate a mixed prompt. Furthermore, three prompt contrastive learning strategies are employed to exploit relations between modality combinations, improving generalization to unseen modality combinations. Finally, we train CMPL with the following objective function: ℒCMPL=ℒtask+α⋅ℒlcfl+β⋅ℒpcl,L_ CMPL=L_task+α·L_lcfl+β·L_pcl, (1) where ℒtaskL_task, ℒlcflL_lcfl, and ℒpclL_pcl are task-specific loss, label-guided contrastive loss, and prompt contrastive loss, respectively. Here, ℒtaskL_task is used to guide the model’s predictions, and we employ the traditional autoregressive cross-entropy loss from LLMs for this purpose. The parameters α and β are the balanced factors on ℒlcflL_lcfl and ℒpclL_pcl, respectively. Label-guided Contrastive Feature Learning Contrastive learning, as an effective representation learning method, has been widely applied in multimodal domains. Existing methods typically maximize the similarity of representations across multiple modalities while minimizing the similarity between different samples directly. However, these approaches often overlook the structural relationships between samples, potentially separating representations of similarly labeled samples and leading to suboptimal representations. To address these challenges, we propose the label-guided contrastive feature learning (LCFL) module. This mechanism aims to pull together latent representations of different modalities from the same class while preserving similarity between samples of related classes. By incorporating class-aware semantic relationships, this mechanism enables the learned representations to incorporate richer and more robust semantic information, enhancing the model’s understanding of multimodal representations. Specifically, for audio and vision modalities, we first project the available embeddings into the LLM’s text embedding space and use an adaptive pooling to reduce the sequence (31) as follows: ¯a X^a =Pool(σ(a⋅1a)⋅2a), =Pool(σ(X^a·W^a_1)·W^a_2), (2) where σ is the activation function. 1aW^a_1 and 2aW^a_2 are trainable parameters. The formulation for ¯v X^v follows analogously. ¯a∈ℝN×l×d X^a ^N× l× d and ¯v∈ℝN×l×d X^v ^N× l× d are the audio and visual embeddings after projection and adaptive pooling, respectively. Then, we project the embeddings of all modalities into the contrastive learning space (2) as follows: k=σ(k⋅1k)⋅2k, ^k=σ(X^k·W^k_1)·W^k_2, (3) where k∈t,a,vk∈\t,a,v\. k∈ℝN×l×dcH^k ^N× l× d_c is the representation of modality k after projection. For representation iwH^w_i, we treat the instances have the same label as positive pairs, denoted as ju|j=i,u≠w|j≠iH^u_j|_Y_j=Y_i,\ u≠ w|j≠ i, while considering others as negative pairs, denoted as ju|j≠iH^u_j|_Y_j _i, where u,w∈t,a,vu,w∈\t,a,v\. We use the cosine distance to evaluate the similarity between iwH^w_i and juH^u_j: d(iw,ju)=⟨iw,ju⟩/‖iw‖⋅‖ju‖,d(H^w_i,H^u_j)= ^w_i,H^u_j /\|H^w_i\|·\|H^u_j\|, where ⟨⋅,⋅⟩ ·,· is the dot product operator. To effectively enhance the model’s comprehension of multimodal embeddings and explore cross-modal relationships, we design a LCFL loss function ℒlcflL_lcfl. Our method maximizes similarity between positive pairs while maintaining the similarity of negative pairs according to their label relations. This approach effectively mitigates the adverse effects of incorrectly pushing apart embeddings sharing similar labels. Given text and audio modality as an example, the contrastive loss ℒ(t,a)L^(t,a) between tH^t and aH^a can be defined as: ℒ(t,a)=−12N∑w=t,a∑i=1Nwilogiwiw+iw,L^(t,a)=- 12N _w=t,a _i=1^NM^w_i T^w_iT^w_i+N^w_i, (4) where wi=∑j=1N∑u=t,auj⋅[Si,j≠1]⋅e|d(iw,ju)−Si,j|/τN^w_i= _j=1^N _u=t,aM^u_j·I_[S_i,j≠ 1]· e^|d(H^w_i,H^u_j)-S_i,j|/τ and wi=∑j=1N∑u=t,auj⋅[Si,j=1]⋅ed(iw,ju)/τ−e1/τT^w_i= _j=1^N _u=t,aM^u_j·I_[S_i,j=1]· e^d(H^w_i,H^u_j)/τ-e^1/τ. τ is the temperature parameter that adjusts the softness. N represents the number of instances. [Si,j=1]I_[S_i,j=1] is the indicator function that equals 1 iff Si,j=1S_i,j=1. Si,jS_i,j is the similarity between labels of i-th and j-th instance. Here, we employ a simple approach to measure inter-label similarity: Si,j=1−|j−i|max−min,S_i,j=1- |Y_j-Y_i | \Y\- \Y\, (5) where max \Y\ and min \Y\ represent the maximum and minimum values of the labels, respectively. Similarly, we can compute ℒ(t,v)L^(t,v) and ℒ(v,a)L^(v,a). Then, the objective function ℒlcflL_lcfl can be calculated as follows: ℒlcfl=ℒ(t,a)+ℒ(t,v)+ℒ(v,a).L_lcfl=L^(t,a)+L^(t,v)+L^(v,a). (6) By minimizing ℒlcflL_lcfl, the representations of instances with consistent labels are pulled closer, while the similarity between others aligns with their label similarity. This captures more structured representations and effectively enhances multimodal learning in incomplete combinations, effectively facilitating learning for unseen modality combinations. Mixed Prompts Learning Most LLM fine-tuning methods employ LoRA for adaptation. However, in the IMSAUMC task, using LoRA alone to simultaneously train multiple modality combinations fails to distinguish between them. To address this, we design a mixed prompts learning mechanism to enhance the model’s understanding of diverse modality combinations. Specifically, we design modality-combination prompts, i.e., =1,2,⋯,TP=\P^1,P^2,·s,P^T\, where i∈ℝlp×dpP^i ^l_p× d_p is the prompt embedding for the i-th modality combination, with lpl_p and dpd_p being its sequence length and dimension. Traditional methods concatenate each modality-combination prompt with its corresponding input and feed them into the LLM, learning the prompts from available data. However, in IMSAUMC tasks, the training set does not cover all modality combinations appearing in the test set. Consequently, prompts for unseen combinations lack training data and cannot be learned. Recognizing that different modality combinations are not independent but exhibit correlations, we propose a Soft Routing-inspired mixed prompts guidance mechanism. Given multimodal data j=[jt,¯ja]∈ℝN×2l×dX_j=[X_j^t, X_j^a] ^N× 2l× d, where [⋯][·s] denotes sequence concatenation, this mechanism feeds jX_j into a router that automatically selects and weights prompts according to the input representation: G(j)=Softmax(j⋅g),G(X_j)=Softmax(X_j·W_g), (7) where g∈ℝd×TW_g ^d× T is the router’s parameter and G(j)G(X_j) represents the soft assignment weights. Subsequently, we obtain the final mixed prompt via dynamic blending: ¯j=∑i=1TG(j)i⋅i, P_j=Σ _i=1^TG(X_j)_i·P^i, (8) where ¯j P_j is the mixed prompt for the j-th instance. Finally, the mixed prompt ¯j P_j and multimodal input jX_j are jointly fed into the LLM to produce the output: ¯j=LLM(¯j,j,θ), Y_j=LLM( P_j,X_j;θ), (9) where θ represents the LLM’s parameters and ¯j Y_j is the generated text with sentiment class and sentiment score. Following standard LLM training, we adopt next-token prediction loss. Thus, the task loss ℒtaskL_task is: ℒtask=∑i=1N∑j=1K−logP(Li,j|i¯,i,θ),L_task=Σ _i=1^NΣ _j=1^K- P(L_i,j| P_i,X_i,θ), (10) where K is the number of label tokens and Li,jL_i,j is the j-th label token of the ¯i Y_i generated by the i-th instance. Prompt Contrastive Learning In the IMSAUMC task, a key challenge lies in effectively leveraging knowledge from existing modality combinations to enhance the learning of unseen modality combinations. Recognizing that different modality combinations are not isolated but inherently interrelated, we design three prompt contrastive learning strategies to enable mutual learning among prompts: (1) modality information preservation strategy; (2) cross-combination complementarity strategy; (3) conditional information alignment strategy. The core idea is to consider the relationships between various modality combinations to minimize the conditional entropy or maximize the conditional mutual information. To compute conditional entropy and mutual information, we first project the modality-combination prompt embeddings, then average them along the sequence dimension, and finally apply the softmax function, which allows the prompt representation to be interpreted as a probability distribution, enabling entropy and mutual information estimation: ^i=Softmax(Avg(σ(i⋅1i)⋅2i)), P^i=Softmax(Avg(σ(P^i·W^i_1)·W^i_2)), (11) where ^i∈ℝlp′×D P^i ^l _p× D are the normalized prompt embeddings of i-th modality combination. For convenience, we denote the set of unimodal prompts as u=^1,^2,^3P_u=\ P^1, P^2, P^3\, the set of dual modality prompts as d=^4,^5,^6P_d=\ P^4, P^5, P^6\, and the set of full modality prompts as f=^7P_f=\ P^7\. Modality Information Preservation. For a multimodal prompt, it inherently contains the information present in each of its unimodal components. Therefore, it can be argued that when given a multimodal prompt, it should retain the sub-modality-combination prompts it encompasses as much as possible. To achieve this, we minimize the conditional entropy H(^i∣^j)H( P^i P^j) between such sub-modality-combination ^i P^i and multi-modality-combination prompt ^j P^j. Since each element of ^i P^i and ^j P^j can be treated as probability distribution of two variables ziz_i and zjz_j over D classes (11; 9; 18), where D is the dimensionality of ^i P^i and ^j P^j. The joint probability distribution P(m,n)∈ℝD×DP^(m,n) ^D× D can be defined as: Pi,j(m,n)=1l∑k=1l^k,im^k,jn.P^(m,n)_i,j= 1l _k=1^l P^m_k,i P^n_k,j. Let Pd(m,n)P^(m,n)_d and Pd′(m,n)P^(m,n)_d denote the margin probability distributions P(m,n)(zm=d)P^(m,n)(z_m=d) and P(m,n)(zn=d′)P^(m,n)(z_n=d ), which can be obtained by summing the d-th rows and d′d -th columns of P. We can define the loss function between the m-th prompt and the n-th prompt ℒ(m,n)L^(m,n) as follows: ℒ(m,n)=H(^m∣^n)=−∑d=1D∑d′=1DPd,d′(m,n)lnPd,d′(m,n)Pd′(m,n).L^(m,n)=H( P^m P^n)=- _d=1^D _d =1^DP^(m,n)_d,d P^(m,n)_d,d P^(m,n)_d . The loss function ℒmipL_mip can be defined as follows: ℒmip=∑m=1,2ℒ(m,4)+∑m=1,3ℒ(m,5)+∑m=2,3ℒ(m,6)+∑m=16ℒ(m,7).L_mip= _m=1,2L^(m,4)+ _m=1,3L^(m,5)+ _m=2,3L^(m,6)+ _m=1^6L^(m,7). Cross-Combination Complementarity. For two prompts that share a common modality—such as ^4 P^4 (text-audio combination) and ^5 P^5 (text-visual combination), their shared textual information acts as a bridge connecting the other two modalities, i.e., audio and visual. Therefore, when given such prompts ^4 P^4 and ^5 P^5, the uncertainty of the visual and audio prompt ^6 P^6 should also decrease. Hence, we minimize the conditional entropy H(^6|^4,^5)H( P^6| P^4, P^5). More generally, we aim to minimize the conditional entropy H(^l|^m,^n)H( P^l| P^m, P^n), where ^l,^m,^n∈d P^l, P^m, P^n _d and l, m, and n are mutually distinct. Similarly, we firstly define the joint probability distribution P(l,m,n)∈ℝD×D×DP^(l,m,n) ^D× D× D of zlz_l, zmz_m, and znz_n as follows: Pi,j,k(l,m,n)=1l∑t=1l^t,il^t,jm^t,kn.P^(l,m,n)_i,j,k= 1l _t=1^l P^l_t,i P^m_t,j P^n_t,k. The loss function between the l-th, m-th, and n-th prompts can be defined as: ℒ(l,m,n)−∑d1=1D∑d2=1D∑d3=1DPd1,d2,d3(l,m,n)lnPd1,d2,d3(l,m,n)Pd2,d3(l,m,n), ^(l,m,n)- _d_1=1^D _d_2=1^D _d_3=1^DP^(l,m,n)_d_1,d_2,d_3 P^(l,m,n)_d_1,d_2,d_3P^(l,m,n)_d_2,d_3, where Pd2,d3(l,m,n)P^(l,m,n)_d_2,d_3 is the marginal probability distribution by summing the first dimension of P(l,m,n)P^(l,m,n). The total loss function ℒccL_c can be defined as follows: ℒcc=ℒ(3,4,5)+ℒ(4,5,3)+ℒ(5,3,4).L_c=L^(3,4,5)+L^(4,5,3)+L^(5,3,4). (12) Conditional Information Alignment. Given a multimodal combined prompt such as ^6 P^6 containing audio and visual information, for the prompt ^4 P^4 containing audio and text and the prompt ^5 P^5 containing visual and text, their shared text modality should remain consistent and aligned. Therefore, we maximize the conditional mutual information I(^4;^5|^6)I( P^4; P^5| P^6). More generally, we aim to maximize I(^l;^m|^n)I( P_l; P^m| P^n), where ^l,^m,^n∈d P^l, P^m, P^n _d and l, m, and n are mutually distinct. The loss function between l-th, m-th and n-th prompts can be defined as: ℒ^(l,m,n)=−∑d1=1D∑d2=1D∑d3=1DP(l,m,n)d1,d2,d3lnPd1,d2,d3(l,m,n)Pd3(l,m,n)Pd1,d3(l,m,n)Pd2,d3(l,m,n). L^(l,m,n)=- _d_1=1^D _d_2=1^D _d_3=1^DP^(l,m,n)_d_1,d_2,d_3 P^(l,m,n)_d_1,d_2,d_3P^(l,m,n)_d_3P^(l,m,n)_d_1,d_3P^(l,m,n)_d_2,d_3. The total loss function ℒciaL_cia can be defined as follows: ℒcia=ℒ^(3,4,5)+ℒ^(4,5,3)+ℒ^(5,3,4).L_cia= L^(3,4,5)+ L^(4,5,3)+ L^(5,3,4). (13) Finally, the total prompt contrastive learning loss function ℒpclL_pcl can be defined as: ℒpcl=ℒmip+λ1⋅ℒcc+λ2⋅ℒcia,L_pcl=L_mip+ _1·L_c+ _2·L_cia, (14) where λ1 _1 and λ2 _2 are trade-off parameters. Dataset Method Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Acc-2 F1 Acc-2 F1 Acc-2 F1 Acc-2 F1 Acc-2 F1 Acc-2 F1 CMU-MOSI Self-M 65.70 64.81 67.22 66.63 65.55 65.00 68.70 67.33 66.82 66.23 64.43 63.17 CubeMLP 69.41 69.44 69.97 69.88 69.61 69.71 68.50 68.61 70.17 70.27 66.77 66.42 DMD 67.68 67.06 69.92 69.91 69.46 69.50 67.53 67.05 69.87 69.96 67.83 67.94 DLF 67.27 66.81 69.51 69.38 68.80 67.97 69.77 69.89 70.02 69.66 65.85 65.58 TFR-Net 60.06 57.77 47.51 42.94 50.31 42.17 55.44 49.83 53.86 50.73 49.95 44.55 MPLMM 55.03 48.38 53.71 46.35 59.20 55.02 58.64 55.80 56.40 50.50 65.55 64.53 MFMB-Net 67.04 66.01 68.57 68.31 69.14 68.72 69.28 68.12 68.03 67.43 67.96 67.79 LNLN 66.06 65.17 66.57 65.64 67.79 67.32 68.45 68.14 67.63 67.04 67.02 66.46 CMPL 75.36 75.43 75.56 75.55 75.15 75.21 76.88 76.80 76.67 76.79 73.63 73.63 SIMS-V2 Self-M 66.28 64.83 66.34 64.11 63.73 62.65 66.92 64.41 64.22 62.77 64.18 62.89 CubeMLP 67.57 67.13 66.02 65.60 65.25 60.05 65.12 59.95 64.57 59.40 63.06 57.91 DMD 67.47 66.52 67.60 66.94 68.60 68.39 67.89 66.77 68.67 68.25 71.24 70.73 DLF 68.31 67.16 70.31 69.83 71.02 70.79 70.86 70.48 68.47 67.43 70.18 69.84 TFR-Net 65.96 65.42 65.05 65.18 61.77 61.61 64.80 64.24 64.22 60.44 60.80 55.45 MPLMM 62.67 61.29 68.12 67.89 62.99 59.64 64.02 63.34 65.89 63.80 61.77 60.75 MFMB-Net 70.74 70.75 67.96 67.67 71.12 70.98 69.70 69.08 66.96 66.85 70.80 70.46 LNLN 71.02 70.30 70.63 70.44 71.50 70.59 71.47 70.90 71.47 70.94 70.86 70.40 CMPL 77.95 78.00 76.92 77.02 78.34 78.36 77.69 77.75 78.21 78.20 76.72 76.70 CMU-MOSEI Self-M 74.50 72.69 74.53 73.02 74.02 72.21 74.42 72.60 75.06 72.79 73.78 70.83 CubeMLP 74.63 73.02 71.01 64.43 70.88 65.10 74.38 72.86 66.39 61.60 67.19 62.74 DMD 75.73 74.36 75.35 74.47 74.51 73.70 75.60 73.77 75.08 73.38 74.24 73.59 DLF 75.72 74.14 75.50 74.22 75.42 73.90 74.98 73.64 75.30 73.95 74.99 73.96 TFR-Net 71.44 68.50 73.43 71.55 71.71 66.74 73.09 71.15 70.06 67.15 68.35 60.86 MPLMM 72.17 71.23 70.78 70.11 70.06 69.36 73.17 71.76 69.99 69.99 71.42 70.39 MFMB-Net 74.09 70.57 72.94 68.79 74.20 71.94 73.78 70.34 74.11 71.25 72.36 68.76 LNLN 75.43 74.19 75.20 73.98 75.19 73.51 75.22 73.27 75.35 73.46 74.72 73.14 CMPL 77.39 76.59 77.41 76.88 77.27 76.79 76.28 75.86 76.94 76.29 75.94 74.92 Table 1: The performance of different methods on various datasets under six tasks. The best and second-best results are marked in bold and underlined, respectively. Experiments Experiment Setting Datasets. We conducted experiments on three widely used datasets, including CMU-MOSI (36), CMU-MOSEI (37), and SIMS-V2 (32). The CMU-MOSI dataset contains a total of 2,199 video clips, each manually annotated with sentiment scores ranging from strongly negative to strongly positive (-3 to 3). The CMU-MOSEI dataset consists of 22,856 video clips, covering a broader range of topics compared to CMU-MOSI, with sentiment labels also annotated on the same scale (-3 to 3). SIMS-V2 is a Chinese multimodal sentiment analysis dataset containing 4,403 video clips, where sentiment values are labeled from -1 to 1. Task No. Training Test Task 1 S1,S2,S3,S4,S5\S_1,S_2,S_3,S_4,S_5\ S Task 2 S1,S2,S3,S4,S6\S_1,S_2,S_3,S_4,S_6\ S Task 3 S1,S2,S3,S5,S6\S_1,S_2,S_3,S_5,S_6\ S Task 4 S1,S2,S3,S4\S_1,S_2,S_3,S_4\ S Task 5 S1,S2,S3,S5\S_1,S_2,S_3,S_5\ S Task 6 S1,S2,S3,S6\S_1,S_2,S_3,S_6\ S Task 7 S S Table 2: The cases of modality combinations in the training and test sets across the seven tasks. Figure 3: The performance of seven methods across six evaluation metrics on three datasets under Task 7. The center of the radar chart represents the worst results, and the outermost vertices correspond to the best results. Metrics. Due to differences in labels across datasets, we employ different evaluation metrics for different datasets. For CMU-MOSI and CMU-MOSEI, we adopt binary accuracy (Acc-2), five-category accuracy (Acc-5), seven-category accuracy (Acc-7), F1 score, mean absolute error (MAE), and Pearson correlation (Corr) as evaluation metrics. For SIMS-V2, we use Acc-2, three-category accuracy (Acc-3), Acc-5, F1, MAE, and Corr. Here, Acc-2 and F1 follow the non-positive/positive standard. Baselines. In our experiments, we compare with six state-of-the-art methods, including modality-complete methods: Self-M, Cube-MLP, DMD, and DLF; and modality-missing methods: TFRNet, MPLMM, MFMB-Net and LNLN. For methods that require complete modalities during training, we fill in missing modalities with zero. For all methods, we keep the parameters recommended in their original papers or released codes. Implementation Details. We conduct experiments on the Ubuntu 20.04 system with an Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz and a single NVIDIA A40. We adopt Qwen1.5-1.8b (1) as the backbone. For the training process, we use the Adam optimizer with a learning rate of 1×10−41× 10^-4. For reliability, we perform three independent runs for each experiment and report the average results. More details can be found in Appendix B. Main Results In the experiment, we design seven experimental scenarios, i.e., seven different tasks, as shown in Table 2, where SiS_i represents the i-th modality combination and =S1,S2,⋯,S7S=\S_1,S_2,·s,S_7\. For all scenarios, the testing phase has all modality combinations (i.e., S). Table 1 presents the results of seven methods across three datasets under seven different scenarios. From Table 1, it can be observed that our method achieves the best performance in almost all scenarios. Compared to the second-best method, LNLN, our approach improves accuracy by an average of 5.57%. Notably, on the CMU-MOSI dataset, our method outperforms LNLN by an average of 13.26% in F1-score. In particular, under the Task 6 scenario, the F1-score improvement reaches 16.24%. This is because our method, CMPL, leverages label-guided contrastive feature learning to effectively capture multimodal consistency and the semantic structure of instances, employing three prompt contrastive learning strategies to enhance the model’s ability to learn from each modality combination and generalize to unseen modality combinations. Furthermore, to further validate the effectiveness of our method, we conduct experiments under the scenario where both the training and test sets contain all seven modality combinations (i.e., S7S_7), as shown in Figure 3. The results demonstrate that our approach still outperforms others in most cases, highlighting its superiority and robustness. More results can be found in Appendix C. Models Acc-2 F1 MAE Corr TtrainT_train TtestT_test Qwen1.5-1.8B 75.36 75.43 1.068 0.579 6.1m 6.0s Llama3.2-3B 75.71 75.51 1.096 0.581 9.0m 8.4s Llama-2-7B 76.60 76.53 1.087 0.574 25.8m 15.1s Qwen3-8B 77.90 78.03 0.989 0.636 18.6m 16.8s Table 3: The performance with different LLMs on the CMU-MOSI dataset under Task 4, where the units for training time and testing time are minutes (m) and seconds (s), respectively. Comparable Results with Different LLMs To further validate the effectively of CMPL, we conduct experiments using different LLMs of varying scales (i.e., Qwen1.5-1.8B, Llama3.2-3B (3), Llama-2-7B (25), and Qwen3-8B (30)) as the backbone under the Task 1 on the CMU-MOSI dataset. The performance and time of training and testing are presented in Table 3. As shown, models with larger parameter sizes generally achieve higher performance than smaller ones. For instance, Qwen3-8B achieves a 9.84% improvement in Corr compared to Qwen1.5-1.8B, which can be attributed to its greater learning capacity and ability to capture more nuanced knowledge. However, for computational resources, Qwen3-8B requires over three times the training time of Qwen1.5-1.8B. Given that Qwen1.5-1.8B offers a favorable balance between resource efficiency and performance, it serves as a practical and cost-effective choice for common deployment. Ablation Study To validate the effectiveness of each module in our method, we conduct experiments on the CMU-MOSI and SIMS-V2 datasets under the Task 4 scenario. We systematically remove each module and observe the model’s performance changes. The ablation results are presented in Table 4. The ablation results demonstrate that removing any module leads to performance degradation, while the model achieves its optimal performance when all modules are intact. Specifically, on CMU-MOSI dataset, removing the LCFL module results in a 2.79% decrease in Corr, while removing the PCL module causes a 2.94% increase in MAE on SIMS-V2 dataset. What’s more, the removal of the MPL module causes both a 2.64% drop in Acc-2 and a substantial 2.83% in F1 on CMU-MOSI dataset. These results confirm that each module plays a critical role, validating the contributions of each module to the model’s effectiveness. More additional ablation results can be found in Appendix C. Datasets Methods Acc-2 F1 MAE Corr CMU-MOSI w/o LCFL 75.25 75.02 1.107 0.574 w/o MPL 74.90 74.69 1.089 0.574 w/o PCL 75.96 75.85 1.100 0.586 CMPL 76.88 76.80 1.079 0.590 SIMS-V2 w/o LCFL 76.27 76.31 0.375 0.569 w/o MPL 77.24 77.11 0.365 0.574 w/o PCL 76.47 76.49 0.374 0.574 CMPL 77.69 77.75 0.363 0.593 Table 4: The ablation study on both CMU-MOSI and SIMS-V2 datasets under the Task 4. Conclusion In this paper, we propose a novel model named CMPL to address the task of incomplete multimodal sentiment analysis with the unseen modality combination. We introduce a label-guided contrastive feature learning mechanism to maintain multimodal consistency while preserving the structural relationships among data. Furthermore, we develop a mixed prompts learning mechanism incorporating the prompt contrastive learning strategies, which effectively enhances the model’s comprehension of diverse modal combinations and improves its generalization capability to unseen modal combinations. Extensive experiments validate the effectiveness of our approach. References Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: Implementation Details.. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, p. 1597–1607. Cited by: Label-guided Contrastive Feature Learning. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv e-prints, p. arXiv–2407. Cited by: Comparable Results with Different LLMs. Gao et al. (2021) T. Gao, A. Fisch, and D. Chen Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 3816–3830. Cited by: Prompt Learning. Guo et al. (2024) Z. Guo, T. Jin, and Z. Zhao Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1726–1736. Cited by: Prompt Learning. Heinzerling and Inui (2021) B. Heinzerling and K. Inui Language models as knowledge bases: on entity representations, storage capacity, and paraphrased queries. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p. 1772–1791. Cited by: Prompt Learning. Hu and Flaxman (2018) A. Hu and S. Flaxman Multimodal sentiment analysis to explore the structure of emotions. In proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining, p. 350–358. Cited by: Introduction. Huan et al. (2023) R. Huan, G. Zhong, P. Chen, and R. Liang Unimf: a unified multimodal framework for multimodal sentiment analysis in missing modalities and unaligned multimodal sequences. IEEE Transactions on Multimedia 26, p. 5753–5768. Cited by: Introduction. Huang et al. (2020) J. Huang, S. Gong, and X. Zhu Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8849–8858. Cited by: Prompt Contrastive Learning. Huang et al. (2024) J. Huang, J. Zhou, Z. Tang, J. Lin, and C. Y. Chen TMBL: transformer-based multimodal binding learning model for multimodal sentiment analysis. Knowledge-Based Systems 285, p. 111346. Cited by: Introduction. Ji et al. (2019) X. Ji, J. F. Henriques, and A. Vedaldi Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9865–9874. Cited by: Prompt Contrastive Learning. Khattak et al. (2023) M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan Maple: multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19113–19122. Cited by: Prompt Learning. Lee et al. (2023) Y. Lee, Y. Tsai, W. Chiu, and C. Lee Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14943–14952. Cited by: Prompt Learning. Li et al. (2024a) M. Li, D. Yang, Y. Lei, S. Wang, S. Wang, L. Su, K. Yang, Y. Wang, M. Sun, and L. Zhang A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 10074–10082. Cited by: Introduction. Li et al. (2024b) M. Li, D. Yang, Y. Liu, S. Wang, J. Chen, S. Wang, J. Wei, Y. Jiang, Q. Xu, X. Hou, et al. Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning. Advances in Neural Information Processing Systems 37, p. 28515–28536. Cited by: Introduction. Li et al. (2023) Y. Li, Y. Wang, and Z. Cui Decoupled multimodal distilling for emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6631–6640. Cited by: Multimodal Sentiment Analysis. Liang et al. (2022) S. Liang, M. Zhao, and H. Schütze Modular and parameter-efficient multimodal fusion with prompting. In Findings of the Association for Computational Linguistics: ACL 2022, p. 2976–2985. Cited by: Prompt Learning. Lin et al. (2021) Y. Lin, Y. Gou, Z. Liu, B. Li, J. Lv, and X. Peng Completer: incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11174–11183. Cited by: Prompt Contrastive Learning. Liu et al. (2024) Z. Liu, A. Braytee, A. Anaissi, G. Zhang, L. Qin, and J. Akram Ensemble pretrained models for multimodal sentiment analysis using textual and video data fusion. In Companion Proceedings of the ACM Web Conference 2024, p. 1841–1848. Cited by: Introduction. Ma et al. (2021) M. Ma, J. Ren, L. Zhao, S. Tulyakov, C. Wu, and X. Peng Smil: multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 2302–2310. Cited by: Introduction. Mai et al. (2022) S. Mai, Y. Zeng, S. Zheng, and H. Hu Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affective Computing 14 (3), p. 2276–2289. Cited by: Multimodal Sentiment Analysis. Pham et al. (2019) H. Pham, P. P. Liang, T. Manzini, L. Morency, and B. Póczos Found in translation: learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, p. 6892–6899. Cited by: Introduction. Sun et al. (2022) H. Sun, H. Wang, J. Liu, Y. Chen, and L. Lin CubeMLP: an mlp-based model for multimodal sentiment analysis and depression estimation. In Proceedings of the 30th ACM international Conference on Multimedia, p. 3722–3729. Cited by: Multimodal Sentiment Analysis. Tao et al. (2025) C. Tao, J. Li, T. Zang, and P. Gao A multi-focus-driven multi-branch network for robust multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 1547–1555. Cited by: Multimodal Sentiment Analysis. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: Comparable Results with Different LLMs. Truong and Lauw (2019) Q. Truong and H. W. Lauw Vistanet: visual aspect attention network for multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, p. 305–312. Cited by: Multimodal Sentiment Analysis. Tsimpoukelli et al. (2021) M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, p. 200–212. Cited by: Prompt Learning. Wang et al. (2025) P. Wang, Q. Zhou, Y. Wu, T. Chen, and J. Hu DLF: disentangled-language-focused multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 21180–21188. Cited by: Introduction, Multimodal Sentiment Analysis. Xu et al. (2024) W. Xu, H. Jiang, and X. Liang Leveraging knowledge of modality experts for incomplete multimodal learning. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 438–446. Cited by: Multimodal Sentiment Analysis. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Comparable Results with Different LLMs. Yao et al. (2024) L. Yao, L. Li, S. Ren, L. Wang, Y. Liu, X. Sun, and L. Hou Deco: decoupling token compression from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405.20985. Cited by: Label-guided Contrastive Feature Learning. Yu et al. (2020) W. Yu, H. Xu, F. Meng, Y. Zhu, Y. Ma, J. Wu, J. Zou, and K. Yang Ch-sims: a chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 3718–3727. Cited by: Datasets.. Yu et al. (2021) W. Yu, H. Xu, Z. Yuan, and J. Wu Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 10790–10797. Cited by: Multimodal Sentiment Analysis. Yuan et al. (2021) Z. Yuan, W. Li, H. Xu, and W. Yu Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In Proceedings of the 29th ACM international conference on multimedia, p. 4400–4407. Cited by: Multimodal Sentiment Analysis. Yuan et al. (2023) Z. Yuan, Y. Liu, H. Xu, and K. Gao Noise imitation based adversarial training for robust multimodal sentiment analysis. IEEE Transactions on Multimedia 26, p. 529–539. Cited by: Multimodal Sentiment Analysis. Zadeh et al. (2016) A. Zadeh, R. Zellers, E. Pincus, and L. Morency Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intelligent Systems 31 (6), p. 82–88. Cited by: Datasets.. Zadeh et al. (2018) A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L. Morency Multimodal language analysis in the wild: cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2236–2246. Cited by: Datasets.. Zeng et al. (2022) J. Zeng, T. Liu, and J. Zhou Tag-assisted multimodal sentiment analysis under uncertain missing modalities. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 1545–1554. Cited by: Multimodal Sentiment Analysis. Zhang et al. (2024) H. Zhang, W. Wang, and T. Yu Towards robust multimodal sentiment analysis with incomplete data. Advances in Neural Information Processing Systems 37, p. 55943–55974. Cited by: Multimodal Sentiment Analysis. Zhu et al. (2022) T. Zhu, L. Li, J. Yang, S. Zhao, H. Liu, and J. Qian Multimodal sentiment analysis with image-text interaction network. IEEE Transactions on Multimedia 25, p. 3375–3385. Cited by: Introduction. Zhu et al. (2023) Y. Zhu, Y. Wang, J. Qiang, and X. Wu Prompt-learning for short text classification. IEEE Transactions on Knowledge and Data Engineering 36 (10), p. 5328–5339. Cited by: Prompt Learning. Zhuang et al. (2025) Y. Zhuang, L. Minhao, W. Bai, Y. Zhang, W. Li, J. Deng, and F. Ren Hyper-modality enhancement for multimodal sentiment analysis with missing modalities. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Multimodal Sentiment Analysis. No. Available, Missing Available Data S1S_1 (t), (a, v) =tx=\x^t\ S2S_2 (a), (t, v) =ax=\x^a\ S3S_3 (v), (t, a) =vx=\x^v\ S4S_4 (t, a), (v) =t,ax=\x^t,x^a\ S5S_5 (t, v), (a) =t,vx=\x^t,x^v\ S6S_6 (a, v), (t) =a,vx=\x^a,x^v\ S7S_7 (t, a, v), () =t,a,vx=\x^t,x^a,x^v\ Table 5: The seven modality combination cases. Parameters CMU-MOSI SIMS-V2 CMU-MOSEI Learning Rate 1e-4 1e-4 1e-4 Batch Size 64 64 64 α, β 1, 1e-2 1, 1e-1 1, 1e-2 λ1 _1, λ2 _2 10, 1 1e-1, 1 10, 1 Pool Size 32 32 32 Optimizer Adam Adam Adam Table 6: Experimental parameters of three datasets. Dataset Initial Text of Prompt CMU-MOSI Please predict the sentiment intensity of the <content>with in the range [-3.0, +3.0]. SIMS-V2 Please predict the sentiment intensity of the <content>with in the range [-1.0, +1.0]. CMU-MOSEI Please predict the sentiment intensity of the <content>with in the range [-3.0, +3.0]. Table 7: The initial text of the modality combination prompts for three datasets. Appendix A Task Description In the IMSAUMC task, given three modalities, excluding the case where all modalities are absent, there are a total of 23−1=72^3-1=7 modality combinations. The corresponding available and missing states of the modalities, as well as the available data, are shown in Table 5. Appendix B Implementation Details Some parameters involved in the experiment are shown in Table 6. Table 7 displays the initial text of prompts in three datasets, where “< > ” will be replaced according to different modality combinations. For instance, when the input is an audio-vision modality combination, “< > ” will be replaced with “audio and vision content”, and similarly for other cases. Datasets Methods Acc-2 F1 MAE Corr CMU-MOSI w/o MIP 76.02 72.47 1.181 0.549 w/o C 76.68 76.53 1.088 0.577 w/o CIA 76.32 76.42 1.108 0.571 CMPL 76.88 76.80 1.079 0.590 SIMS-V2 w/o MIP 76.89 76.77 0.370 0.580 w/o C 77.53 77.52 0.365 0.587 w/o CIA 76.72 76.83 0.364 0.590 CMPL 77.69 77.75 0.363 0.593 Table 8: Further ablation study on both CMU-MOSI and SIMS-V2 datasets under the Task 4. Appendix C Additional Experimental Results Parameter Analysis We conduct a parameter analysis on the CMU-MOSI dataset for the key hyperparameters used in CMPL. Specifically, we examine the parameter α, which controls the LCFL loss when the PCL strategy is disabled. We then analyze the parameters β, λ1 _1, and λ2 _2, which regulate the MIP, C, and CIA components in the PCL strategy. The parameter analysis results are presented in Figure 4. The results show that all four parameters lead to certain performance variations when ranging from 1e−21e-2 to 1e21e2. However, the overall performance of the model is not highly sensitive to these changes. The model achieves its best performance when α=1α=1, β=1e−2β=1e-2, λ1=10 _1=10, and λ2=1 _2=1. Further Ablation Study To further validate the contribution of each strategy within PCL, we conducted additional ablation studies by individually removing the MIP, C, and CIA strategies. While the ablation results have already been presented in the main text in graphical form, we provide the detailed numerical results in Table 8. As shown, removing any single strategy leads to a performance drop, which further demonstrates the effectiveness of each component in our PCL framework. To further validate the effectiveness of PCL on the IMSAUMC task, we additionally present its performance for each modality combination in the test set under Task 4 on the SIMS-V2 dataset. The results are shown in Figure 5, where “t”, “a”, and “v” denote text, audio, and visual modality, while “t-a” indicates the text-audio modality combination and similarly for others. Here, “w/o MIP”, “w/o C”, “w/o CIA” indicate the absence of each strategy in PCL. As observed, in the absence of any prompt learning strategies, the model performs poorly on unseen modality combinations, with an average accuracy drop of approximately 3.12% compared to CMPL. The incorporation of the PCL module demonstrates additional performance gains in audio-vision (a-v), and text-audio-vision (t-a-v) modality combinations. Furthermore, when the MIP, C, and CIA strategies are successively removed from the CMPL model, the performance decreases by about 1.32%, 0.94%, and 2.26% on unseen modality combinations, respectively, which indicates that each strategy plays a critical role in improving the model’s generalization. The model achieves optimal performance when all three strategies are incorporated. These results demonstrate the effectiveness and necessity of each strategy in PCL. Detailed Experimental Results Table 9 to Table 16 present detailed experimental results of the seven methods across seven tasks on three datasets, evaluated by six metrics. As shown in the results, our model consistently achieves superior performance over baseline methods in most scenarios, clearly validating the effectiveness and superiority of our approach. (a) Parameter Analysis of α (b) Parameter Analysis of β (c) Parameter Analysis of λ1 _1 (d) Parameter Analysis of λ2 _2 Figure 4: Parameter analysis on the CMU-MOSI dataset. Figure 5: The model performance for various modality combinations on SIMS-V2 dataset under Task 4. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 65.70 69.41 67.68 67.27 60.06 55.03 67.04 66.06 75.36 Acc-5 33.82 35.03 33.63 31.58 21.14 18.71 32.19 29.30 40.82 Acc-7 29.74 30.95 29.88 28.96 20.07 18.61 27.68 27.11 34.50 F1 64.81 69.44 67.06 66.81 57.77 48.38 66.01 65.17 75.43 MAE 1.088 1.078 1.086 1.078 1.297 1.378 1.127 1.118 1.068 Corr 0.572 0.569 0.587 0.579 0.378 0.294 0.568 0.586 0.579 SIMS-V2 Acc-2 66.28 67.57 67.47 68.31 65.96 62.67 70.74 71.02 77.95 Acc-3 48.32 59.83 54.06 55.67 52.74 55.80 59.27 59.06 71.44 Acc-5 33.11 40.78 36.94 38.49 35.82 36.69 40.44 40.72 50.45 F1 64.83 67.13 66.52 67.16 65.42 61.29 70.75 70.30 78.00 MAE 0.437 0.433 0.420 0.420 0.427 0.486 0.410 0.415 0.358 Corr 0.429 0.449 0.461 0.478 0.444 0.288 0.499 0.506 0.611 CMU-MOSEI Acc-2 74.50 74.63 75.73 75.72 71.44 72.17 74.09 75.43 77.39 Acc-5 48.94 46.96 49.11 48.33 44.86 42.28 49.59 48.55 49.68 Acc-7 47.84 45.90 48.22 47.33 43.54 42.13 48.48 47.63 48.30 F1 72.69 73.02 74.36 74.14 68.50 71.23 70.57 74.19 76.59 MAE 0.664 0.696 0.659 0.663 0.741 0.773 0.664 0.662 0.660 Corr 0.578 0.551 0.593 0.591 0.544 0.407 0.581 0.595 0.593 Table 9: The performance of different methods on three datasets under Task 1. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 67.22 69.97 69.92 69.51 47.51 53.71 68.57 66.57 75.56 Acc-5 34.11 32.80 33.72 34.01 17.83 21.52 33.73 27.01 42.32 Acc-7 30.32 29.49 28.72 29.59 17.78 20.89 29.32 25.22 35.23 F1 66.63 69.88 69.91 69.38 42.94 46.35 68.31 65.64 75.55 MAE 1.062 1.078 1.075 1.085 1.480 1.340 1.070 1.194 1.121 Corr 0.574 0.583 0.589 0.593 0.139 0.326 0.596 0.551 0.586 SIMS-V2 Acc-2 66.34 66.02 67.60 70.31 65.05 68.12 67.96 70.63 76.92 Acc-3 42.65 54.64 50.23 58.80 45.26 61.38 57.62 58.41 72.11 Acc-5 30.11 37.11 34.78 39.88 30.53 40.14 39.57 39.33 51.84 F1 64.11 65.60 66.94 69.83 65.18 67.89 67.67 70.44 77.02 MAE 0.435 0.447 0.419 0.419 0.450 0.446 0.415 0.414 0.376 Corr 0.421 0.404 0.459 0.511 0.399 0.419 0.495 0.519 0.592 CMU-MOSEI Acc-2 74.53 71.01 75.35 75.50 73.43 70.78 72.94 75.20 77.41 Acc-5 48.84 44.13 49.05 48.20 46.92 41.59 47.62 49.06 48.82 Acc-7 47.82 43.62 48.16 47.33 46.07 41.59 46.93 48.02 47.66 F1 73.02 64.43 74.47 74.22 71.55 70.11 68.79 73.98 76.88 MAE 0.667 0.751 0.663 0.664 0.694 0.790 0.681 0.664 0.667 Corr 0.579 0.536 0.586 0.591 0.551 0.394 0.554 0.595 0.587 Table 10: The performance of different methods on three datasets under Task 2. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 65.55 69.61 69.46 68.80 50.31 59.20 69.14 67.79 75.15 Acc-5 35.37 34.70 29.64 34.21 17.35 22.35 32.59 33.14 40.82 Acc-7 31.78 30.91 26.78 29.93 17.20 21.77 29.07 30.03 33.09 F1 65.00 69.71 69.50 67.97 42.17 55.02 68.72 67.32 75.21 MAE 1.071 1.106 1.096 1.084 1.425 1.325 1.082 1.066 1.165 Corr 0.556 0.542 0.579 0.591 0.340 0.319 0.571 0.578 0.575 SIMS-V2 Acc-2 63.73 65.25 68.60 71.02 61.77 62.99 71.12 71.50 78.34 Acc-3 48.48 44.13 52.90 58.61 45.36 37.56 60.47 61.12 72.40 Acc-5 33.33 30.72 36.52 39.46 31.08 25.73 40.57 41.01 51.71 F1 62.65 60.05 68.39 70.79 61.61 59.64 70.98 70.59 78.36 MAE 0.432 0.452 0.416 0.413 0.490 0.477 0.403 0.418 0.366 Corr 0.427 0.448 0.479 0.514 0.251 0.286 0.514 0.513 0.610 CMU-MOSEI Acc-2 74.02 70.88 74.51 75.42 71.71 70.06 74.20 75.19 77.27 Acc-5 48.44 45.17 48.39 48.74 45.01 41.55 48.92 48.79 49.15 Acc-7 47.45 44.46 47.71 47.76 44.47 41.53 48.01 47.79 47.75 F1 72.21 65.10 73.70 73.90 66.74 69.36 71.94 73.51 76.79 MAE 0.672 0.733 0.665 0.660 0.729 0.784 0.671 0.666 0.660 Corr 0.568 0.572 0.587 0.595 0.511 0.382 0.568 0.592 0.600 Table 11: The performance of different methods on three datasets under Task 3. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 68.70 68.50 67.53 69.77 55.44 58.64 69.28 68.45 76.88 Acc-5 33.58 33.04 31.68 35.47 18.95 23.08 34.03 35.03 42.03 Acc-7 29.98 29.50 28.47 30.56 17.64 22.30 30.21 31.05 34.55 F1 67.33 68.61 67.05 69.89 49.83 55.80 68.12 68.14 76.80 MAE 1.062 1.099 1.093 1.059 1.446 1.329 1.085 1.042 1.079 Corr 0.584 0.547 0.573 0.592 0.326 0.332 0.566 0.592 0.590 SIMS-V2 Acc-2 66.92 65.12 67.89 70.86 64.80 64.02 69.70 71.47 77.69 Acc-3 48.55 44.20 57.35 59.57 52.64 55.06 57.62 60.67 70.54 Acc-5 32.98 30.01 39.52 39.68 35.56 35.78 38.53 40.30 50.42 F1 64.41 59.95 66.77 70.48 64.24 63.34 69.08 70.90 77.75 MAE 0.441 0.467 0.424 0.421 0.456 0.477 0.416 0.419 0.363 Corr 0.423 0.432 0.457 0.498 0.382 0.319 0.487 0.504 0.593 CMU-MOSEI Acc-2 74.42 74.38 75.60 74.98 73.09 73.17 73.78 75.22 76.28 Acc-5 47.98 46.51 47.23 48.62 45.66 41.21 47.85 47.63 49.68 Acc-7 47.14 45.57 46.61 47.68 45.12 41.16 46.96 46.59 47.94 F1 72.60 72.86 73.77 73.64 71.15 71.76 70.34 73.27 75.86 MAE 0.675 0.693 0.669 0.668 0.716 0.785 0.677 0.674 0.671 Corr 0.570 0.549 0.585 0.580 0.511 0.395 0.571 0.594 0.592 Table 12: The performance of different methods on three datasets under Task 4. The best and second-best results are marked in bold and underlined, respectively. Table 13: The performance of different methods on three datasets under Task 5. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 66.82 70.17 69.87 70.02 53.86 56.40 68.03 67.63 76.67 Acc-5 34.50 33.77 32.51 32.31 20.12 20.02 32.19 30.22 41.98 Acc-7 30.90 29.69 28.48 28.67 19.29 19.78 28.52 27.70 35.28 F1 66.23 70.27 69.96 69.66 50.73 50.50 67.43 67.04 76.79 MAE 1.072 1.078 1.076 1.065 1.420 1.341 1.104 1.111 1.120 Corr 0.568 0.573 0.587 0.589 0.337 0.311 0.556 0.579 0.597 SIMS-V2 Acc-2 64.22 64.57 68.67 68.47 64.22 65.89 66.96 71.47 78.21 Acc-3 43.00 57.03 50.29 56.58 47.13 50.87 55.78 53.64 71.15 Acc-5 30.50 38.39 35.11 39.88 32.59 32.95 38.08 36.91 49.90 F1 62.77 59.40 68.25 67.43 60.44 63.80 66.85 70.94 78.20 MAE 0.428 0.471 0.416 0.412 0.464 0.477 0.430 0.414 0.361 Corr 0.438 0.412 0.475 0.481 0.370 0.324 0.455 0.501 0.612 CMU-MOSEI Acc-2 75.06 66.39 75.08 75.30 70.06 69.99 74.11 75.35 76.94 Acc-5 48.82 45.48 48.03 48.42 46.10 41.56 48.52 48.37 49.40 Acc-7 47.73 44.80 47.25 47.47 45.40 41.42 47.46 47.34 48.07 F1 72.79 61.60 73.38 73.95 67.15 69.99 71.25 73.46 76.29 MAE 0.670 0.731 0.671 0.667 0.712 0.779 0.674 0.670 0.658 Corr 0.568 0.571 0.579 0.579 0.515 0.399 0.562 0.593 0.592 Table 14: The performance of different methods on three datasets under Task 5. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 64.43 66.77 67.83 65.85 49.95 65.55 67.96 67.02 73.63 Acc-5 31.58 27.36 27.45 26.04 17.54 20.80 30.66 31.83 38.87 Acc-7 28.67 25.12 25.80 24.30 17.49 20.26 27.33 29.49 31.78 F1 63.17 66.42 67.94 65.58 44.55 64.53 67.79 66.46 73.63 MAE 1.137 1.191 1.170 1.171 1.460 1.342 1.126 1.104 1.182 Corr 0.531 0.498 0.508 0.548 0.107 0.350 0.553 0.566 0.552 SIMS-V2 Acc-2 64.18 63.06 71.24 70.18 60.80 61.77 70.80 70.86 76.72 Acc-3 48.39 41.20 57.99 59.16 46.32 55.74 59.33 60.38 69.70 Acc-5 33.33 29.17 40.39 41.39 30.14 36.04 40.47 39.56 49.68 F1 62.89 57.91 70.73 69.84 55.45 60.75 70.46 70.40 76.70 MAE 0.434 0.475 0.407 0.405 0.515 0.480 0.408 0.421 0.384 Corr 0.424 0.361 0.501 0.501 0.282 0.353 0.500 0.510 0.571 CMU-MOSEI Acc-2 73.78 67.19 74.24 74.99 68.35 71.42 72.36 74.72 75.94 Acc-5 46.89 44.05 47.88 47.30 43.49 41.59 47.97 47.89 46.58 Acc-7 46.22 43.46 47.14 46.44 42.81 41.48 47.17 46.87 45.42 F1 70.83 62.74 73.59 73.96 60.86 70.39 68.76 73.14 74.92 MAE 0.684 0.751 0.674 0.682 0.781 0.773 0.680 0.678 0.710 Corr 0.547 0.528 0.575 0.567 0.477 0.390 0.552 0.583 0.534 Table 15: The performance of different methods on three datasets under Task 6. The best and second-best results are marked in bold and underlined, respectively. Dataset Metrics self-M cube-MLP DMD DLF TFRNet MPLMM MFMB-Net LNLN Ours CMU-MOSI Acc-2 67.73 70.38 67.53 69.26 66.21 61.74 68.89 67.07 77.03 Acc-5 35.86 36.44 32.95 36.00 24.73 21.53 34.92 34.45 43.29 Acc-7 32.27 33.04 29.50 31.97 21.28 20.89 31.10 30.95 36.64 F1 67.06 70.43 67.12 69.13 65.74 60.78 68.39 66.37 77.15 MAE 1.036 1.051 1.065 1.038 1.326 1.365 1.053 1.058 1.101 Corr 0.595 0.576 0.590 0.602 0.454 0.331 0.597 0.600 0.602 SIMS-V2 Acc-2 66.44 62.89 68.38 69.70 63.60 66.60 69.93 71.28 78.47 Acc-3 42.91 42.30 57.99 59.57 51.71 56.77 61.40 57.06 72.28 Acc-5 30.72 28.63 39.43 41.91 35.94 37.52 42.51 37.91 51.90 F1 64.43 57.81 67.36 68.79 62.34 65.93 69.69 70.23 78.57 MAE 0.431 0.480 0.421 0.412 0.446 0.455 0.411 0.423 0.364 Corr 0.424 0.398 0.475 0.493 0.391 0.385 0.493 0.505 0.608 CMU-MOSEI Acc-2 74.78 72.97 74.41 75.71 73.28 73.03 73.47 75.31 75.97 Acc-5 48.01 48.53 49.13 48.95 47.60 42.51 48.65 49.00 49.40 Acc-7 47.15 47.46 48.12 48.00 46.14 42.47 47.69 48.05 47.96 F1 71.86 71.61 73.03 74.01 69.52 71.79 69.95 73.24 75.58 MAE 0.668 0.664 0.662 0.659 0.704 0.775 0.670 0.663 0.662 Corr 0.574 0.583 0.590 0.592 0.541 0.415 0.575 0.593 0.595 Table 16: The performance of different methods on three datasets under Task 7. The best and second-best results are marked in bold and underlined, respectively.