Paper deep dive
URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection
Zhenyu Wang, Weichen Cheng, Weijia Li, Junjie Mou, Zongyou Zhao, Guoying Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/10/2026, 3:42:30 AM
Summary
URMF (Uncertainty-aware Robust Multimodal Fusion) is a framework for multimodal sarcasm detection that addresses the limitation of deterministic fusion by explicitly modeling modality-specific aleatoric uncertainty. It uses a cross-modal interaction module to inject visual evidence into text, followed by Gaussian posterior modeling to estimate uncertainty, which then dynamically regulates modality contributions during fusion. The model is trained with a joint objective including information bottleneck, modality regularization, and uncertainty-driven contrastive learning to improve robustness against noisy or irrelevant inputs.
Entities (4)
Relation Signals (3)
URMF → addresses → Multimodal Sarcasm Detection
confidence 100% · we propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework for robust multimodal sarcasm detection.
URMF → utilizes → Aleatoric uncertainty
confidence 95% · URMF performs unified unimodal aleatoric uncertainty modeling over text, image, and interaction-aware latent representations
URMF → uses → Transformer
confidence 90% · URMF first adopts a single-layer Transformer-based cross-modal interaction module
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image. Although recent methods have improved MSD through cross-modal interaction and incongruity reasoning, they often assume that all modalities are equally reliable. In real-world social media, however, textual content may be ambiguous and visual content may be weakly relevant or even irrelevant, causing deterministic fusion to introduce noisy evidence and weaken robust reasoning. To address this issue, we propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion. URMF first employs multi-head cross-attention to inject visual evidence into textual representations, followed by multi-head self-attention in the fused semantic space to enhance incongruity-aware reasoning. It then performs unified unimodal aleatoric uncertainty modeling over text, image, and interaction-aware latent representations by parameterizing each modality as a learnable Gaussian posterior. The estimated uncertainty is further used to dynamically regulate modality contributions during fusion, suppressing unreliable modalities and yielding a more robust joint representation. In addition, we design a joint training objective integrating task supervision, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven self-sampling contrastive learning. Experiments on public MSD benchmarks show that URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines, demonstrating the effectiveness of uncertainty-aware fusion for improving both accuracy and robustness.
Tags
Links
- Source: https://arxiv.org/abs/2604.06728v1
- Canonical: https://arxiv.org/abs/2604.06728v1
Trouble viewing inline? Open PDF directly →
Full Text
46,794 characters extracted from source content.
Expand or collapse full text
URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection Zhenyu Wang, Weichen Cheng, Weijia Li, Junjie Mou, Zongyou Zhao, Guoying Zhang School of Artificial Intelligence, China University of Mining and Technology, Beijing 2310640139, 2310410210, 2310410120, 2310410319, 2310410234@student.cumtb.edu.cn zgy@cumtb.edu.cn Corresponding author Abstract Multimodal sarcasm detection (MSD) aims to identify sarcastic intent arising from semantic incongruity between text and image. Although recent advances in cross-modal interaction and incongruity reasoning have substantially improved MSD performance, most existing methods implicitly assume that different modalities are equally reliable. This assumption, however, is often violated in real-world social media scenarios, where textual content may be ambiguous or lack crucial context, while visual content may be weakly relevant or even irrelevant. Under such conditions, deterministic fusion tends to inject noisy evidence into the joint representation, dilute key conflict cues, and ultimately undermine the robustness of cross-modal reasoning. To address this issue, we propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during multimodal interaction and fusion. URMF first employs a cross-modal interaction module, where multi-head cross-attention injects visual evidence into textual representations, followed by multi-head self-attention in the fused semantic space to strengthen incongruity-aware reasoning. On this basis, we perform unified unimodal aleatoric uncertainty modeling over the text modality, image modality, and interaction-aware latent modality by parameterizing each representation as a learnable Gaussian posterior, thereby explicitly capturing modality-specific noise and reliability differences. The estimated uncertainty is then used to dynamically regulate modality contributions during fusion, allowing the model to suppress unreliable modalities and produce a more robust joint representation. Furthermore, we introduce a joint training objective consisting of an information-bottleneck-style task loss, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven self-sampling contrastive learning, which together enhance compactness, consistency, and robustness in latent representation learning. Extensive experiments on public MSD benchmark demonstrate that URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines. The results verify that explicitly modeling unimodal uncertainty and incorporating it into cross-modal fusion is an effective way to improve both accuracy and robustness in multimodal sarcasm detection. Keywords Multimodal sarcasm detection ⋅· Cross-modal interaction ⋅· Aleatoric uncertainty ⋅· Dynamic fusion ⋅· Contrastive learning 1 Introduction Sarcasm often conveys stance and emotion through a discrepancy between literal expression and underlying intent, making its recognition heavily dependent on the joint understanding of semantic incongruity and pragmatic context. With the prevalence of image-text posts on social media, multimodal sarcasm detection (MSD) has emerged as an important yet challenging cross-modal understanding task. A reliable MSD model must not only capture textual and visual semantics individually, but also identify the implicit conflicts and fine-grained correspondences between them in order to infer sarcastic intent accurately. To address the central challenge of cross-modal incongruity modeling, prior studies have proposed a variety of effective approaches. Early work mainly relied on feature concatenation or shallow fusion strategies, which are simple but limited in capturing the implicit contradictions and contextual dependencies that sarcasm often requires. Subsequent research has shifted toward stronger cross-modal interaction and explicit reasoning paradigms, including graph-based relational modeling, interaction-driven attention mechanisms, and selective interaction strategies built upon Transformer architectures. These methods improve multimodal representation learning by strengthening cross-modal alignment and incongruity reasoning. Despite their progress, most existing MSD methods implicitly assume that multimodal inputs are deterministic and equally reliable. This assumption is often unrealistic in real-world social media data. Text may be ambiguous, rhetorical, metaphorical, or missing essential context, while images may merely provide decoration, emotional atmosphere, or even irrelevant content. When modality quality and relevance are inherently unstable, deterministic fusion can mistakenly treat noisy modalities as reliable evidence, amplify misleading signals, and dilute key conflict cues, thereby harming robustness and generalization under noisy, missing, or weakly related modalities. Therefore, explicitly modeling modality reliability is crucial for robust multimodal sarcasm understanding. Uncertainty estimation offers a natural way to address this issue. In deep learning, uncertainty is commonly categorized into epistemic uncertainty and aleatoric uncertainty, where the latter captures inherent noise and irreducible ambiguity in observations. For multimodal learning, aleatoric uncertainty is particularly important because different modalities often exhibit different noise levels and reliability. If such uncertainty can be explicitly estimated at the representation level and incorporated into multimodal fusion, the model may adaptively suppress unreliable modalities and thereby produce more stable cross-modal reasoning under challenging conditions. Motivated by these observations, we propose Uncertainty-aware Robust Mul-timodal Fusion (URMF), a unified framework for robust multimodal sarcasm detection. URMF first adopts a single-layer Transformer-based cross-modal interaction module, where multi-head cross-attention injects semantically relevant visual evidence into textual representations, followed by multi-head self-attention to refine token dependencies in the fused semantic space and strengthen conflict-aware reasoning. We then perform unified Gaussian modeling over the text modality, image modality, and interaction-aware latent modality, where both the mean and variance are learned to explicitly characterize modality-specific semantics and aleatoric uncertainty. Based on the estimated uncertainty, URMF dynamically adjusts modality contributions during fusion, allowing low-uncertainty modalities to contribute more to the final decision. In addition, we introduce an uncertainty-driven self-sampling contrastive learning strategy, in which stochastic perturbations induced by posterior distributions are used to construct contrastive views, further improving the robustness of latent representations. The entire framework is trained end-to-end with a unified objective that combines task supervision, information bottleneck regularization, modality prior regularization, cross-modal distribution alignment, and uncertainty-aware contrastive learning. Our main contributions are summarized as follows: • We propose URMF, a unified framework that combines cross-modal interaction with unimodal aleatoric uncertainty modeling to explicitly characterize modality reliability in multimodal sarcasm detection. • We design an uncertainty-guided dynamic fusion mechanism together with a joint optimization objective, including information bottleneck regularization, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven contrastive learning, to improve representation consistency and robustness. • Extensive experiments on public MSD benchmark demonstrate that URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines, validating the effectiveness of uncertainty modeling for robust multimodal sarcasm detection. 2 Related Work 2.1 Multimodal Sarcasm Detection Multimodal sarcasm detection (MSD) aims to identify sarcastic intent by modeling the semantic incongruity between textual expression and visual content. Early MSD studies mainly relied on direct multimodal fusion strategies, where visual and textual features extracted by pre-trained encoders were concatenated and then fed into a classifier [2]. Although simple and effective in controlled scenarios, such methods usually struggle to capture the implicit contradictions and contextual dependencies underlying sarcastic expression. To overcome these limitations, subsequent work has focused on strengthening cross-modal interaction modeling. Graph-based methods and interaction-driven frameworks have been widely adopted to capture fine-grained associations across modalities. RCLMuFN [23], for instance, constructs a heterogeneous relation graph and employs multi-route fusion to explicitly model contextual dependencies between textual tokens and visual objects. Similarly, Dynamic Routing Transformer Network [21] introduces a routing mechanism into the Transformer architecture, enabling the model to adaptively select more informative cross-modal representations while suppressing irrelevant information. These approaches substantially improve multimodal representation learning by enhancing modality interaction. Beyond interaction modeling, recent studies have increasingly emphasized explicit incongruity reasoning, which is widely regarded as a core characteristic of sarcasm understanding. Incongruity-aware Tension Field Network [32] introduces a physically inspired tension-field representation to quantify the intensity of semantic conflict across modalities. Other work has explored incongruity-aware learning from different perspectives. MuMu [22] explicitly aligns conflicting semantic cues to improve sarcasm detection; MICL [8] models complementary incongruity from multiple semantic views; SemIRNet [35] enhances sarcasm recognition through semantic-level interaction modeling. More recently, higher-level reasoning paradigms have also been introduced into MSD. LDGNet [34] incorporates large language models (LLMs) into a debate-guided framework and exploits structured reasoning processes to uncover implicit sarcastic intent. These advances indicate that progressively stronger interaction and reasoning mechanisms can significantly improve sarcasm detection performance. Despite these encouraging results, most existing MSD methods implicitly assume that multimodal inputs are deterministic and equally reliable. In real-world social media scenarios, textual descriptions may be ambiguous, while images may be weakly related or even irrelevant to the sarcastic intent. Under such conditions, deterministic fusion may amplify noisy modalities and impair reasoning performance. Therefore, explicitly modeling modality reliability is particularly important for robust multimodal sarcasm understanding. 2.2 Uncertainty in Multimodal Learning Uncertainty estimation has become an important technique for improving the robustness of deep learning systems. Existing studies commonly categorize uncertainty into epistemic uncertainty and aleatoric uncertainty, where aleatoric uncertainty characterizes inherent noise in observed data [10]. In multimodal learning, aleatoric uncertainty is particularly valuable because different modalities often exhibit different reliability levels and noise characteristics. Recent work on robust multimodal fusion has proposed uncertainty estimation as an adaptive weighting mechanism. Instead of treating all modalities equally, these methods first estimate unimodal uncertainty and then dynamically adjust modality contributions during fusion. For example, uncertainty-aware fusion has been shown to significantly improve performance when one modality is corrupted or partially missing [6]. Inspired by these studies, we argue that uncertainty modeling is particularly suitable for multimodal sarcasm detection, where modality relevance is inherently unstable. Accordingly, we incorporate uncertainty estimation into both the cross-modal interaction and fusion stages, enabling the model to perform more reliable sarcasm reasoning under noisy or ambiguous multimodal inputs. 3 Method Figure 1: Given an image-text pair (I,T)(I,T), URMF performs cross-modal interaction, modality uncertainty modeling, and uncertainty-guided dynamic fusion to learn a joint latent representation for sarcasm prediction. The model is trained end-to-end with information bottleneck, modality regularization, multimodal alignment, and uncertainty contrastive learning losses. 3.1 Overall Framework Given an image-text pair (I,T)(I,T), the goal is to predict its sarcasm label y. The proposed URMF framework consists of four stages: cross-modal interaction, unimodal uncertainty modeling, uncertainty-guided fusion, and joint objective optimization. (1) We first feed deterministic text and image representations into a cross-modal interaction module to obtain an interaction-aware latent modality representation XfX_f. (2) We then perform unimodal aleatoric uncertainty modeling by learning distribution parameters for each modality-specific latent representation and sampling latent variables via reparameterization. (3) Based on the estimated uncertainty, we apply dynamic modality fusion to adaptively combine modality representations and produce a joint representation for prediction. (4) Finally, the entire framework is optimized end-to-end with an information-bottleneck-style task loss, modality prior regularization, cross-modal distribution alignment, and self-sampling contrastive learning. 3.2 Cross-modal Interaction Module Different from standard multimodal Transformer layers, which usually adopt the information flow of “self-attention first, then cross-attention” [21], we instead use a “cross-modal alignment first, intra-modal reasoning later” design. Specifically, we first inject image-context evidence into textual representations through cross-attention, and then apply self-attention in the fused semantic space to model token dependencies, followed by a feed-forward network (FFN) for nonlinear transformation. Each sub-module is equipped with residual connections and layer normalization (LN). Let Xt∈ℝn×dtX_t ^n× d_t and Xi∈ℝm×diX_i ^m× d_i denote the initial text and image representations extracted by the text and image encoders, respectively. A single-layer cross-modal interaction module is formulated as Xtc X_t^c =LN(MHCA(Xt,Xi)+Xt), =LN\! (MHCA(X_t,X_i)+X_t ), (1) Xts X_t^s =LN(MHSA(Xtc)+Xtc), =LN\! (MHSA(X_t^c)+X_t^c ), (2) Xt′ X_t =LN(FFN(Xts)+Xts), =LN\! (FFN(X_t^s)+X_t^s ), (3) where Xt′X_t denotes the output of the single-layer cross-modal interaction module, and XtcX_t^c and XtsX_t^s are the outputs of the MHCA and MHSA modules, respectively. For the multi-head cross-attention (MHCA) module, we define MHCA(Xt,Xi)=concat(head1,…,headH)WO,MHCA(X_t,X_i)=concat\! (head_1,…,head_H )\,W^O, (4) where concat(⋅)concat(·) denotes concatenation, H is the number of heads, and WO∈ℝd×dW^O ^d× d is the output projection matrix. Let d=dt=did=d_t=d_i and dh=d/Hd_h=d/H. For the h-th head, we have headh=softmax(QhKh⊤dh)Vhhead_h=softmax\! ( Q_hK_h d_h )V_h (5) Qh=XtWhQ,Kh=XiWhK,Vh=XiWhVQ_h=X_tW_h^Q, K_h=X_iW_h^K, V_h=X_iW_h^V (6) where WhQ,WhK,WhV∈ℝd×dhW_h^Q,W_h^K,W_h^V ^d× d_h are learnable projection matrices. For the multi-head self-attention (MHSA) module, after obtaining the cross-modally enhanced text sequence XtcX_t^c, we perform self-attention in the same semantic space: MHSA(Xtc)=concat(head1′,…,headH′)W~O,MHSA(X_t^c)=concat\! (head _1,…,head _H )\, W^O, (7) where headh′=softmax(Q~hK~h⊤dh)V~h,head _h=softmax\! ( Q_h K_h d_h ) V_h, (8) Q~h=XtcW~hQ,K~h=XtcW~hK,V~h=XtcW~hV, Q_h=X_t^c W_h^Q, K_h=X_t^c W_h^K, V_h=X_t^c W_h^V, (9) with W~hQ,W~hK,W~hV∈ℝd×dh W_h^Q, W_h^K, W_h^V ^d× d_h and W~O∈ℝd×d W^O ^d× d is the learnable parameters of multi-head self-attention. This module models token dependencies on top of representations already injected with visual context, thus strengthening structured reasoning over cross-modal conflict semantics. Finally, we pool the output cross-modal representation Xt′X_t to obtain a global interaction representation: Xf=Pool(Xt′),X_f=Pool(X_t ), (10) where XfX_f fuses both textual and visual information while emphasizing cross-modal semantic conflict, and can thus be regarded as an interaction-aware latent modality representation. 3.3 Unimodal Aleatoric Uncertainty Modeling After cross-modal interaction, we obtain an interaction-aware latent representation XfX_f. We then represent the text modality, image modality, and interaction-aware latent modality of the k-th sample as tkx_t^k, ikx_i^k, and fkx_f^k, respectively. For convenience, we use mkx_m^k as a unified notation, where m∈t,i,fm∈\t,i,f\. Following unimodal aleatoric uncertainty modeling [6], we model the latent representation under modality m as a multivariate Gaussian random variable mkz_m^k. The mean mk μ_m^k represents the semantic center of the k-th sample under modality m, while the standard deviation mk σ_m^k characterizes its aleatoric uncertainty. We quantify modality-internal aleatoric uncertainty by learning both the mean and variance. Specifically, we use two lightweight fully connected heads to predict the mean vector and the log-variance vector, respectively: mk=fθ,μm(mk),logmk 2=fθ,σm(mk), μ_m^k=f^m_θ,μ(x_m^k), σ_m^k\,2=f^m_θ,σ(x_m^k), (11) where fθ,μm(⋅)f^m_θ,μ(·) and fθ,σm(⋅)f^m_θ,σ(·) have independent parameters. The latent representation mkz_m^k is then modeled by the Gaussian posterior p(mk∣mk)=(mk,mk 2),p(z_m^k _m^k)=N\! ( μ_m^k,\ σ_m^k\,2I ), (12) where I denotes the identity matrix. And sampled via the reparameterization: mk=mk+mkϵ,ϵ∼(0,).z_m^k= μ_m^k+ σ_m^kε, ε (0,I). (13) In this way, the latent space jointly encodes modality semantics and aleatoric uncertainty, providing a learnable confidence signal for subsequent dynamic fusion. 3.4 Uncertainty-guided Dynamic Fusion After obtaining the latent representations mkz_m^k of the interaction-aware latent modality XfX_f and the image modality XiX_i, we further assume that the contribution of each modality to the final prediction is related to its uncertainty: lower uncertainty indicates higher confidence, and thus the modality should contribute more to the joint representation. Since the textual semantics have already been injected into the interaction-aware latent modality XfX_f through cross-modal interaction, we perform final uncertainty-guided fusion over XfX_f and XiX_i only, so as to avoid redundant reuse of textual evidence. To this end, we first aggregate the dimension-wise variance into a scalar uncertainty measure using the average variance: σ¯mk 2=1D∑d=1Dm,dk 2, σ_m^k\,2= 1D _d=1^D σ_m,d^k\,2, (14) where D is the dimensionality of mkz_m^k, and m∈f,im∈\f,i\. We then map lower uncertainty to higher weights via an exponential transformation and normalize the weights to obtain dynamic fusion coefficients: amk=exp(1σ¯mk 2+ε),a_m^k= \! ( 1 σ_m^k\,2+ ), (15) αmk=amk∑s∈f,iask, _m^k= a_m^kΣ _s∈\f,i\a_s^k, (16) where ε is a numerical stability term. The modality-specific contribution representation is then computed by dynamically weighting the mean vector: ^mk=αmkmk. x_m^k= _m^k\, μ_m^k. (17) This fusion strategy adaptively suppresses noise from high-uncertainty modalities, thereby improving the robustness and discriminability of the joint representation. On top of this, we further use a joint network ϕ(⋅)φ(·) to fuse the modality-specific contribution representations ^mk x_m^k into a joint representation: k=ϕ([^fk,^ik]).h^k=φ([ x_f^k, x_i^k]). (18) Following the same unimodal aleatoric uncertainty modeling strategy, we model the joint modality as hk=fθ,μh(k),loghk 2=fθ,σh(k), μ_h^k=f_θ,μ^h(h^k), σ_h^k\,2=f_θ,σ^h(h^k), (19) and apply the reparameterization to obtain the latent representation hkz_h^k of the joint modality. Finally, the joint latent representation of the k-th sample, denoted by hkz_h^k, is fed into a classifier to obtain the prediction probability: y^k=softmax(Whk+b). y^k=softmax(Wz_h^k+b). (20) 3.5 Training Objective To jointly account for task discrimination, latent compression, modality distribution regularization, cross-modal consistency, and uncertainty-driven augmentation, we decompose the overall training objective into four complementary loss terms: ℒ=ℒIB+λ1ℒreg+λ2ℒalign+λ3ℒUCL.L=L_IB+ _1L_reg+ _2L_align+ _3L_UCL. (21) We use classification cross-entropy as the main task loss and impose KL regularization on the latent representation of the final joint modality (see Sec. 3.4) to encourage the model to learn task-sufficient but non-redundant representations, i.e., an information bottleneck constraint [1]: ℒIB=ℒtask(y,y^)+λIBKL(p(hk∣k)∥(0,)),L_IB=L_task(y, y)+ _IB\,KL\! (p(z_h^k ^k)\,\|\,N(0,I) ), (22) where ℒtaskL_task denotes the cross-entropy loss. Furthermore, to explicitly characterize and regularize unimodal aleatoric uncertainty while stabilizing training and improving generalization, we regularize the latent representations of the text modality, image modality, and interaction-aware latent modality (see Sec. 3.3) toward the standard Gaussian prior: ℒreg=∑m∈t,i,fKL(p(mk∣mk)∥(0,)).L_reg= _m∈\t,i,f\KL\! (p(z_m^k _m^k)\,\|\,N(0,I) ). (23) Here, m=tm=t, m=im=i, and m=fm=f correspond to the text modality, image modality, and interaction-aware latent modality, respectively. This term enforces consistent scale and distributional form across different modality-specific latent spaces, thereby providing a stable basis for uncertainty-guided dynamic fusion (see Sec. 3.4) and subsequent cross-modal distribution alignment. Under the constraint of ℒregL_reg, the latent spaces of different modalities become better regularized. On this basis, to align modality-specific representations in the latent space and better exploit the cross-modal interaction module (see Sec. 3.2) for semantic conflict modeling while reducing statistical shift across modalities, we introduce a cross-modal KL alignment term that encourages the text modality and interaction-aware latent modality to approach the image modality in the distributional sense: ℒalign=KL(p(tk∣tk)∥p(ik∣ik))+KL(p(fk∣fk)∥p(ik∣ik)).L_align=KL\! (p(z_t^k _t^k)\,\|\,p(z_i^k _i^k) )+KL\! (p(z_f^k _f^k)\,\|\,p(z_i^k _i^k) ). (24) This term aligns cross-modal representations at the distribution level rather than the point-representation level, which helps alleviate representation drift caused by noise and uncertainty. Finally, aleatoric uncertainty also increases the diversity of modality-specific data. Therefore, we exploit the uncertainty-aware latent representations learned in Sec. 3.3 to perform self-sampling data augmentation. For each modality m∈t,i,fm∈\t,i,f\, we independently sample twice from p(mk∣mk)p(z_m^k _m^k) to obtain an anchor ~mk z_m^k and an augmented positive sample mkz_m^k. In addition, we randomly sample a set of negative instances from other latent distributions of the same modality and adopt the following contrastive learning objective: ℒUCLm=−logexp(sim(~mk,mk)/τ)∑k≠k′(exp(sim(~mk,mk)/τ)+exp(sim(~mk,mk′)/τ)),L_UCL^m=- \! (sim\! ( z_m^k,\,z_m^k )/τ ) _k≠ k ( \! (sim\! ( z_m^k,\,z_m^k )/τ )+ \! (sim\! ( z_m^k,\,z_m^k )/τ ) ), (25) ℒUCL=∑m∈t,i,fℒUCLm,L_UCL= _m∈\t,i,f\L_UCL^m, (26) where τ is a temperature coefficient. Since the sampling noise is controlled by σm _m, this loss explicitly leverages uncertainty-induced stochastic perturbations to construct contrastive views, thereby improving noise robustness in learned representations. 4 Experiments 4.1 Dataset and Evaluation Metrics We conduct experiments on the public Multimodal Sarcasm Detection (MSD) dataset [2]. We follow the official train/validation/test split and adopt the same preprocessing procedure as the original dataset. We report Accuracy, Precision, Recall, and F1-score as evaluation metrics. 4.2 Implementation Details Table 1 summarizes the hyperparameter settings used in our experiments. For modality-specific representation learning, we employ publicly available pre-trained encoders obtained. Specifically, the text encoder is RoBERTa, while the image encoder is ViT. Unless otherwise specified, all experiments are performed on a single NVIDIA GeForce RTX 4090 GPU. Symbol Value Description n 100100 maximum length of text tokens m 4949 number of image patches dtd_t 768768 dimension of text embeddings did_i 768768 dimension of image embeddings λIB _IB 10−310^-3 weight of the KL regularization term in ℒIBL_IB λ1 _1 10−310^-3 weight of ℒregL_reg λ2 _2 10−510^-5 weight of ℒalignL_align λ3 _3 10−310^-3 weight of ℒUCLL_UCL Table 1: Hyperparameter settings used in our experiments. 4.3 Baseline Models To evaluate the performance of URMF, we compare it with a wide range of competitive baselines, which can be grouped into image-only, text-only, and multimodal methods. Image-only methods: ResNet [2], ViT [4]. Text-only methods: Bi-LSTM [7], SIARN [20], SMSD [28], BERT [3], Ro-BERTa [18]. Multimodal methods: HFM [2], D&R Net [29], Bridge [24], InCrossMGs [12], CMGCN [13], HKE-model [14], Att-BERT [16], DIP [26], KnowleNet [31], DMSD-CL [9], AMIF [11], G2SAM [25], FSICN [15], Multi-view CLIP [18], MuMu [22], MIL-Net [17], MICL [8], DynRT-Net [21], DCPNet [5], InterARM [30], KFGC-Net [36], CIRM [33], SCI-GDFN [27]. In addition, we include MLLM-based models [19], namely LLaVA1.5 and LLaVA1.5-VIDR, which are fine-tuned on the MSD training set using LoRA-based parameter-efficient adaptation. 4.4 Main Results Modality Model Acc(%) P(%) R(%) F1(%) Image ResNet [2] 64.76 54.51 70.80 61.53 ViT [4] 67.83 57.93 70.07 63.43 Text Bi-LSTM [7] 81.90 76.66 78.42 77.53 SIARN [20] 80.57 75.55 75.70 75.63 SMSD [28] 80.90 76.46 75.18 75.82 BERT [3] 83.85 78.72 82.27 80.22 RoBERTa [18] 93.97 90.39 94.59 92.45 Multimodal HFM [2] 86.63 83.84 84.18 84.01 D&R Net [29] 84.02 77.97 83.42 80.60 Bridge [24] 88.51 82.95 89.39 86.05 InCrossMGs [12] 86.10 81.38 84.36 82.84 CMGCN [13] 87.55 83.63 84.69 84.16 HKE-model [14] 87.36 81.84 86.48 84.09 Att-BERT [16] 86.05 78.63 83.31 80.90 DIP [26] 89.59 87.76 86.58 87.17 KnowleNet [31] 88.87 88.59 84.18 86.33 DMSD-CL [9] 88.95 84.89 87.90 86.37 AMIF [11] 90.10 86.55 89.68 88.09 G2SAM [25] 90.48 87.95 89.02 88.48 FSICN [15] 90.55 89.93 89.51 89.72 Multi-view CLIP [18] 88.33 82.66 88.65 85.55 MuMu [22] 90.73 88.81 88.44 88.62 MIL-Net [17] 89.50 85.16 89.16 87.11 MICL [8] 92.08 90.05 90.61 90.33 DynRT-Net [21] 93.59 93.06 93.60 93.31 LLaVA1.5 [19] 93.67 93.70 93.14 93.40 LLaVA1.5-VIDR [19] 89.97 89.26 89.58 89.42 DCPNet [5] 89.48 87.46 88.73 88.10 InterARM [30] 92.28 91.79 92.23 92.01 KFGC-Net [36] 90.97 88.28 89.56 88.91 CIRM [33] 94.02 93.46 94.14 93.76 SCI-GDFN [27] 94.06 93.50 94.17 93.80 URMF (Ours) 95.02 94.69 95.19 94.91 Table 2: Main results on the MSD dataset. Best results are shown in bold and second-best results are underlined. Table 2 presents the overall comparison on the MSD dataset. URMF achieves the best performance on all four evaluation metrics, reaching 95.02% Accuracy, 94.69% Precision, 95.19% Recall, and 94.91% F1-score. These results consistently outperform all unimodal, multimodal, and MLLM-based baselines, demonstrating the effectiveness of uncertainty-aware robust fusion for multimodal sarcasm detection. Compared with unimodal methods, URMF shows clear advantages over both text-only and image-only models. Although the RoBERTa-based text model already achieves strong performance, it still falls behind URMF by a noticeable margin, suggesting that textual cues alone are insufficient to fully capture sarcasm triggered by image-text incongruity. Meanwhile, image-only methods perform substantially worse, indicating that visual information alone is usually not sufficiently discriminative and is most useful when jointly modeled with text. These observations confirm that effective MSD requires not only strong textual understanding but also explicit modeling of cross-modal semantic conflict. URMF also establishes a new state of the art among strong multimodal baselines. For example, compared with DynRT-Net, URMF improves Accuracy and F1 by 1.43 and 1.60 percentage points, respectively. It also consistently outperforms recent competitive approaches such as CIRM and SCI-GDFN across all evaluation metrics. This superiority is particularly meaningful because most existing methods focus mainly on cross-modal interaction or incongruity reasoning, while still implicitly assuming comparable reliability across modalities. In contrast, URMF explicitly models unimodal aleatoric uncertainty and dynamically regulates modality contributions during fusion, thereby reducing the interference of noisy or weakly related modalities and yielding more robust discriminative representations. Moreover, URMF achieves the best Precision and Recall simultaneously. The higher Recall indicates that the model is better able to preserve sarcasm-related conflict cues, while the higher Precision suggests that uncertainty-aware fusion effectively suppresses misleading information introduced by unreliable modalities. Their joint improvement ultimately leads to the best F1-score, showing that URMF achieves a better balance between recognition performance and robustness. It is also worth noting that URMF surpasses MLLM-based models such as LLaVA1.5 and LLaVA1.5-VIDR. This result suggests that, for MSD tasks requiring fine-grained modeling of image-text conflict, a task-specific framework that explicitly combines cross-modal interaction with uncertainty estimation can be more effective than directly adapting general-purpose multimodal foundation models. 4.5 Ablation Study To systematically assess the contribution of each key component in URMF, we conduct ablation studies on the MSD dataset. Specifically, starting from the full model, we construct six variants by removing one component at a time, including the four training objectives (ℒalignL_align, ℒIBL_IB, ℒregL_reg, and ℒUCLL_UCL), the uncertainty-guided dynamic fusion mechanism, and the proposed cross-modal interaction order. Unless otherwise specified, each variant differs from the full model by the removal of only one component. The results are reported in Table 3. Variant Acc(%) P(%) R(%) F1(%) w/o ℒalignL_align 94.60 94.24 94.88 94.49 w/o ℒIBL_IB 94.85 94.51 95.06 94.74 w/o ℒregL_reg 94.44 94.07 94.69 94.32 w/o ℒUCLL_UCL 94.27 93.90 94.25 94.15 w/o Dynamic Fusion 94.65 94.32 94.78 94.52 standard Transformer 92.61 92.27 92.66 92.44 URMF (full) 95.02 94.69 95.19 94.91 Table 3: Ablation results of URMF on the MSD dataset. Removing any key component consistently degrades performance, verifying the effectiveness of each module. Figure 2: t-SNE visualization of joint latent representations on the MSD test set for major ablation variants and the full URMF. Overall, all ablated variants perform worse than the full URMF, indicating that each component contributes positively to the final performance. Among them, replacing the proposed interaction order with a standard Transformer leads to the largest performance drop, suggesting that performing cross-modal alignment before intra-modal semantic reasoning is more suitable for modeling image-text incongruity in MSD. Removing Dynamic Fusion also results in clear degradation, indicating that static fusion is insufficient for handling sample-level variation in modality reliability. In addition, removing ℒregL_reg weakens the overall performance, showing that latent-space regularization plays an important role in stabilizing modality-specific representation distributions, improving uncertainty estimation, and supporting robust downstream fusion. To further provide an intuitive view of how different components affect the learned representation space, we visualize the joint latent representations of several major ablation variants in Fig. 2. From left to right, the four panels correspond to standard Transformer, w/o Dynamic Fusion, w/o ℒregL_reg, and URMF (full), respectively. The standard Transformer variant exhibits the weakest class separability, while removing Dynamic Fusion or ℒregL_reg leads to relatively more mixed samples around the boundary between sarcastic and non-sarcastic instances. By contrast, the full URMF forms clearer inter-class separation and more compact class-wise distributions. This qualitative evidence is consistent with the quantitative results in Table 3, and further supports the effectiveness of the proposed interaction design, uncertainty-guided fusion mechanism, and latent-space regularization strategy. 5 Conclusion In this paper, we propose URMF, an uncertainty-aware robust multimodal fusion framework for multimodal sarcasm detection. URMF explicitly models image-text semantic conflict through a cross-modal interaction module, performs aleatoric uncertainty modeling over the text modality, image modality, and interaction-aware latent modality, and dynamically regulates modality contributions during fusion based on the estimated uncertainty. To further improve representation quality, we introduce a unified training objective that combines task supervision, information bottleneck regularization, modality prior regularization, cross-modal distribution alignment, and uncertainty-driven self-sampling contrastive learning. Extensive experiments on the MSD dataset show that URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines across Accuracy, Precision, Recall, and F1-score. Further ablation studies confirm the effectiveness of each key component and demonstrate that explicitly modeling unimodal aleatoric uncertainty can effectively alleviate the interference caused by noisy or weakly related modalities, thereby improving both accuracy and robustness in multimodal sarcasm detection. Overall, our findings suggest that uncertainty is valuable not only as a measure of modality reliability, but also as an active signal for cross-modal representation learning and robust optimization. In future work, we plan to explore the transferability of this framework to larger-scale multimodal datasets and multimodal foundation models, as well as finer-grained uncertainty decomposition and reasoning mechanisms for more complex open-world multimodal understanding scenarios. References [1] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy (2016) Deep variational information bottleneck. arXiv preprint arXiv:1612.00410. Cited by: §3.5. [2] Y. Cai, H. Cai, and X. Wan (2019) Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 2506–2515. Cited by: §2.1, §4.1, §4.3, §4.3, Table 2, Table 2. [3] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §4.3, Table 2. [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: §4.3, Table 2. [5] Y. Fang, L. Zhang, S. Wang, W. Zhang, Y. Wang, Y. Liu, X. Wei, and X. Yang (2026) DCPNet: a comprehensive framework for multimodal sarcasm detection via graph topology extraction and multi-scale feature fusion. Frontiers of Computer Science 20 (7), p. 2007336. Cited by: §4.3, Table 2. [6] Z. Gao, X. Jiang, X. Xu, F. Shen, Y. Li, and H. T. Shen (2024) Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 26876–26885. Cited by: §2.2, §3.3. [7] A. Graves and J. Schmidhuber (2005) Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks 18 (5-6), p. 602–610. Cited by: §4.3, Table 2. [8] D. Guo, C. Cao, F. Yuan, Y. Liu, G. Zeng, X. Yu, H. Peng, and P. S. Yu (2025) Multi-view incongruity learning for multimodal sarcasm detection. In Proceedings of the 31st International Conference on Computational Linguistics, p. 1754–1766. Cited by: §2.1, §4.3, Table 2. [9] M. Jia, C. Xie, and L. Jing (2024) Debiasing multimodal sarcasm detection with contrastive learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 18354–18362. Cited by: §4.3, Table 2. [10] A. Kendall and Y. Gal (2017) What uncertainties do we need in bayesian deep learning for computer vision?. Advances in neural information processing systems 30. Cited by: §2.2. [11] K. Li, Y. Chen, Q. Wu, W. Mai, F. Li, and Y. Xue (2025) Ambiguity-aware multi-level incongruity fusion network for multi-modal sarcasm detection. In Proceedings of the 31st International Conference on Computational Linguistics, p. 380–391. Cited by: §4.3, Table 2. [12] B. Liang, C. Lou, X. Li, L. Gui, M. Yang, and R. Xu (2021) Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In Proceedings of the 29th ACM international conference on multimedia, p. 4707–4715. Cited by: §4.3, Table 2. [13] B. Liang, C. Lou, X. Li, M. Yang, L. Gui, Y. He, W. Pei, and R. Xu (2022) Multi-modal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), p. 1767–1777. Cited by: §4.3, Table 2. [14] H. Liu, W. Wang, and H. Li (2022) Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 4995–5006. Cited by: §4.3, Table 2. [15] Q. Lu, Y. Long, X. Sun, J. Feng, and H. Zhang (2024) Fact-sentiment incongruity combination network for multimodal sarcasm detection. Information Fusion 104, p. 102203. Cited by: §4.3, Table 2. [16] H. Pan, Z. Lin, P. Fu, Y. Qi, and W. Wang (2020) Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 1383–1392. Cited by: §4.3, Table 2. [17] Y. Qiao, L. Jing, X. Song, X. Chen, L. Zhu, and L. Nie (2023) Mutual-enhanced incongruity learning network for multi-modal sarcasm detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 9507–9515. Cited by: §4.3, Table 2. [18] L. Qin, S. Huang, Q. Chen, C. Cai, Y. Zhang, B. Liang, W. Che, and R. Xu (2023) MMSD2. 0: towards a reliable multi-modal sarcasm detection system. In Findings of the association for computational linguistics: ACL 2023, p. 10834–10845. Cited by: §4.3, §4.3, Table 2, Table 2. [19] B. Tang, B. Lin, H. Yan, and S. Li (2024) Leveraging generative large language models with visual instruction and demonstration retrieval for multimodal sarcasm detection. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 1732–1742. Cited by: §4.3, Table 2, Table 2. [20] Y. Tay, L. A. Tuan, S. C. Hui, and J. Su (2018) Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1010–1020. Cited by: §4.3, Table 2. [21] Y. Tian, N. Xu, R. Zhang, and W. Mao (2023) Dynamic routing transformer network for multimodal sarcasm detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2468–2480. Cited by: §2.1, §3.2, §4.3, Table 2. [22] J. Wang, Y. Yang, Y. Jiang, M. Ma, Z. Xie, and T. Li (2024) Cross-modal incongruity aligning and collaborating for multi-modal sarcasm detection. Information Fusion 103, p. 102132. Cited by: §2.1, §4.3, Table 2. [23] T. Wang, J. Li, G. Su, Y. Zhang, D. Su, Y. Hu, and Y. Sha (2025) RCLMuFN: relational context learning and multiplex fusion network for multimodal sarcasm detection. Knowledge-Based Systems 319, p. 113614. Cited by: §2.1. [24] X. Wang, X. Sun, T. Yang, and H. Wang (2020) Building a bridge: a method for image-text sarcasm detection without pretraining on image-text data. In Proceedings of the first international workshop on natural language processing beyond text, p. 19–29. Cited by: §4.3, Table 2. [25] Y. Wei, S. Yuan, H. Zhou, L. Wang, Z. Yan, R. Yang, and M. Chen (2024) Gˆ 2sam: graph-based global semantic awareness method for multimodal sarcasm detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 9151–9159. Cited by: §4.3, Table 2. [26] C. Wen, G. Jia, and J. Yang (2023) Dip: dual incongruity perceiving network for sarcasm detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2540–2550. Cited by: §4.3, Table 2. [27] Z. Xi, B. Yu, and H. Wang (2025) Multimodal sarcasm detection based on sentiment-clue inconsistency global detection fusion network. Expert Systems with Applications 275, p. 127020. Cited by: §4.3, Table 2. [28] T. Xiong, P. Zhang, H. Zhu, and Y. Yang (2019) Sarcasm detection with self-matching networks and low-rank bilinear pooling. In The world wide web conference, p. 2115–2124. Cited by: §4.3, Table 2. [29] N. Xu, Z. Zeng, and W. Mao (2020) Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th annual meeting of the association for computational linguistics, p. 3777–3786. Cited by: §4.3, Table 2. [30] T. Yue, R. Mao, X. Shi, and E. Cambria (2026) InterARM: interpretable affective reasoning model for multimodal sarcasm detection. IEEE Transactions on Affective Computing. Cited by: §4.3, Table 2. [31] T. Yue, R. Mao, H. Wang, Z. Hu, and E. Cambria (2023) KnowleNet: knowledge fusion network for multimodal sarcasm detection. Information Fusion 100, p. 101921. Cited by: §4.3, Table 2. [32] J. Zhang, C. P. Chen, S. Li, and T. Zhang (2025) Incongruity-aware tension field network for multi-modal sarcasm detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14499–14508. Cited by: §2.1. [33] H. Zhao, Y. Kong, Y. Xu, G. Gou, H. Xu, Y. Wang, and H. Zhang (2025) MMSD3. 0: a multi-image benchmark for real-world multimodal sarcasm detection. arXiv preprint arXiv:2510.23299. Cited by: §4.3, Table 2. [34] H. Zhou, J. Yan, Y. Chen, R. Hong, W. Zuo, and K. Jin (2025) Ldgnet: llms debate-guided network for multimodal sarcasm detection. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §2.1. [35] J. Zhou, Y. Wu, Y. Zhang, Y. Zhang, Y. Liu, B. Huang, and C. Yuan (2025) SemIRNet: a semantic irony recognition network for multimodal sarcasm detection. In 2025 10th International Conference on Information and Network Technologies (ICINT), p. 158–162. Cited by: §2.1. [36] X. Zhuang, F. Zhou, and Z. Li (2025) Multi-modal sarcasm detection via knowledge-aware focused graph convolutional networks. ACM Transactions on Multimedia Computing, Communications and Applications 21 (5), p. 1–22. Cited by: §4.3, Table 2.