Paper deep dive

HCIG: A Hierarchical Cross-Modal Incongruity Graph Network for Multimodal Sarcasm and Cyberbullying Detection

Bhavana Verma, Priyanka Meel, Dinesh Kumar Vishwakarma

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 56

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 7/20/2026, 4:23:53 AM

Summary

This paper introduces HCIG (Hierarchical Cross-modal Incongruity Graph Network) and GCCN (Graph-based Cross-modal Contradiction Network) for multimodal sarcasm and cyberbullying detection. HCIG models cross-modal incongruity at token, phrase, and global levels using graph attention networks, while GCCN uses contradiction-aware pooling. Both architectures outperform existing baselines on the MMSD and MultiBully datasets, demonstrating that hierarchical multi-granularity incongruity modeling is more effective than conventional fusion strategies.

Entities (9)

GCCN · model · 98%HCIG · model · 98%MMSD · dataset · 95%MultiBully · dataset · 95%GATv2 · model · 92%RoBERTa · model · 92%Vision Transformer · model · 92%Hateful Memes Challenge · dataset · 90%MMSD2.0 · dataset · 90%

Relation Signals (9)

HCIG → evaluatedon → MMSD

confidence 98% · The proposed models are evaluated on the MMSD sarcasm benchmark

GCCN → evaluatedon → MultiBully

confidence 98% · evaluated on... the MultiBully cyberbullying dataset

Vision Transformer → usedby → HCIG

confidence 95% · images with a ViT-Base/16 Vision Transformer

RoBERTa → usedby → HCIG

confidence 95% · Both architectures share a common backbone: text is encoded with RoBERTa-base

MMSD → usedfor → sarcasm detection

confidence 95% · MMSD sarcasm benchmark

MultiBully → usedfor → cyberbullying detection

confidence 95% · MultiBully cyberbullying dataset

HCIG → uses → Graph Attention Networks

confidence 95% · models cross-modal incongruity at token, phrase, and global levels using graph attention networks

GCCN → uses → contradiction-aware pooling

confidence 95% · GCCN... performs graph-based reasoning using contradiction-aware pooling

HCIG → outperforms → existing multimodal approaches

confidence 90% · Experimental results demonstrate that HCIG achieves the best performance on MMSD... providing more effective multimodal reasoning than conventional fusion strategies

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal sarcasm and cyberbullying detection remain challenging because the intended meaning often emerges from incongruity between textual and visual information rather than from either modality alone. Existing multimodal approaches primarily rely on feature fusion or cross-modal attention, which may not effectively capture hierarchical semantic inconsistencies across different levels of representation. To address this limitation, this paper proposes HCIG (Hierarchical Cross-modal Incongruity Graph Network), a novel framework that models cross-modal incongruity at token, phrase, and global levels using graph attention networks and adaptively integrates these representations through a learned hierarchical attention mechanism. As a complementary architecture, we also introduce GCCN (Graph-based Cross-modal Contradiction Network), which performs graph-based reasoning using contradiction-aware pooling for efficient multimodal interaction learning. The proposed models are evaluated on the MMSD sarcasm benchmark and the MultiBully cyberbullying dataset, together with comprehensive ablation studies and cross-task transfer experiments. Experimental results demonstrate that HCIG achieves the best performance on MMSD with 85.74% accuracy and 85.29% macro-F1, while GCCN attains the highest macro-F1 (68.66%) on MultiBully and HCIG achieves the highest accuracy (69.62%) and bullying-class F1 (74.90%). The findings demonstrate that hierarchical multi-granularity incongruity modeling provides more effective multimodal reasoning than conventional fusion strategies, offering a robust framework for sarcasm and cyberbullying detection in social media.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

54,555 characters extracted from source content.

Expand or collapse full text

HCIG: A Hierarchical Cross-Modal Incongruity Graph Network for Multimodal Sarcasm and Cyberbullying Detection Bhavana Verma Priyanka Meel Dinesh Kumar Vishwakarma Multimodal Data Analytics Research Laboratory Department of Information Technology Delhi Technological University bhavana.verma2905 priyankameel86 dvishwakarma@gmail.com Abstract Multimodal sarcasm and cyberbullying detection remain challenging because the intended meaning often emerges from incongruity between textual and visual information rather than from either modality alone. Existing multimodal approaches primarily rely on feature fusion or cross-modal attention, which may not effectively capture hierarchical semantic inconsistencies across different levels of representation. To address this limitation, this paper proposes HCIG (Hierarchical Cross-modal Incongruity Graph Network), a novel framework that models cross-modal incongruity at token, phrase, and global levels using graph attention networks and adaptively integrates these representations through a learned hierarchical attention mechanism. As a complementary architecture, we also introduce GCCN (Graph-based Cross-modal Contradiction Network), which performs graph-based reasoning using contradiction-aware pooling for efficient multimodal interaction learning. The proposed models are evaluated on the MMSD sarcasm benchmark and the MultiBully cyberbullying dataset, together with comprehensive ablation studies and cross-task transfer experiments. Experimental results demonstrate that HCIG achieves the best performance on MMSD with 85.74% accuracy and 85.29% macro-F1, while GCCN attains the highest macro-F1 (68.66%) on MultiBully and HCIG achieves the highest accuracy (69.62%) and bullying-class F1 (74.90%). The findings demonstrate that hierarchical multi-granularity incongruity modeling provides more effective multimodal reasoning than conventional fusion strategies, offering a robust framework for sarcasm and cyberbullying detection in social media. Keywords: multimodal sarcasm detection; cyberbullying detection; cross-modal incongruity; hierarchical graph attention networks; MMSD; MultiBully. 1 Introduction Sarcasm and cyberbullying are both pragmatic phenomena that can be signaled as much by what a message shows as by what it says. A caption reading “what a beautiful day” paired with an image of a flooded street is sarcastic precisely because of the mismatch between text and image; a comment that reads as neutral in isolation can become bullying when paired with a mocking or humiliating image. Text-only classifiers, including large pretrained language models [8], systematically miss this class of example because the incongruity is only observable across modalities. This motivates multimodal approaches that explicitly model the relationship, rather than only the concatenation, between textual and visual signals. Multimodal sarcasm detection has been studied through the MMSD benchmark of image–text pairs collected from social media [4, 30], with methods spanning hierarchical fusion [4], cross-modality contrast modeling [39], intra-/inter-modality incongruity modeling [27], and graph-based cross-modal reasoning [19, 20] intended to explicitly capture incongruity rather than relying on fusion alone. A revised, de-biased version of the benchmark, MMSD2.0, was later released to remove spurious textual cues [28]. Cyberbullying detection has a longer history as a predominantly unimodal text classification problem [15, 2]; multimodal cyberbullying datasets that pair short text with an associated image, such as MultiBully [25], are comparatively recent, drawing on the broader multimodal hate-speech and meme-analysis literature established by benchmarks such as the Hateful Memes Challenge [14] and related multimodal hate-speech studies [10, 18, 40, 12, 7, 21, 26] as shown in Figure 1. It remains an open question whether incongruity-oriented architectures designed for sarcasm generalize to bullying detection, where the relevant signal (mockery, threat, or humiliation) need not involve semantic contradiction between modalities in the same way sarcasm does. Figure 1: Representative examples from a multimodal cyberbullying dataset highlighting the role of image–text interaction in distinguishing cyberbullying from non-bullying content. This paper addresses that gap with three contributions. First, and primarily, we propose HCIG, a hierarchical cross-modal incongruity graph network that computes incongruity independently at token, phrase, and global granularity and combines the three via a learned softmax attention gate, allowing the model to adaptively emphasize whichever granularity is most informative for a given example. Second, we implement GCCN, a structurally simpler graph-based architecture that represents text tokens and image patches as nodes in a similarity-thresholded graph reasoned over with a graph attention network (GATv2 [3]) and augmented with a single learned contradiction-pooling score, which serves both as an independent point of comparison and as a design precursor that motivates HCIG’s hierarchical extension. Third, we evaluate both architectures, together with unimodal and late-fusion baselines, on both MMSD (sarcasm) and MultiBully (cyberbullying), including in-domain training, MMSD-to-MultiBully fine-tuning, a bidirectional cross-task direct-transfer stress test, and component-level ablation studies for each architecture. 2 Related Work 2.1 Multimodal sarcasm detection Schifanella et al.[30] first framed multimodal sarcasm detection as a joint text–image classification problem. Cai et al.[4] introduced the MMSD benchmark used in this paper and a hierarchical fusion baseline; Qin et al. [28] later identified and corrected spurious textual cues and mis-annotations in the original release, producing MMSD2.0. Subsequent work has increasingly modeled the text–image relationship explicitly rather than fusing modalities directly: Xu et al.[39] proposed a decomposition-and-relation network to capture cross-modality contrast and semantic association; Pan et al. [27] modeled intra- and inter-modality incongruity with transformer-based attention; and Liang et al. introduced in-modal and cross-modal graph representations reasoned over with graph convolutional and graph attention mechanisms in two related studies [19, 20], the line of work most directly related to the GCCN and HCIG architectures compared here. A separate line of work, MUStARD, extended multimodal sarcasm detection to video and text modalities drawn from television dialogue rather than static image–text social media posts [5]. 2.2 Multimodal cyberbullying and hate-speech detection Cyberbullying detection has traditionally been treated as a unimodal text classification problem, with a substantial body of work on lexical, syntactic, and pretrained-language-model features [15, 2, 1]. Multimodal treatments are more recent. The Hateful Memes Challenge established that unimodal models systematically fail on memes constructed so that hatefulness emerges only from the text–image combination [14], motivating a growing body of multimodal hate-speech architectures [10, 18, 40, 12, 7, 21, 26]. Maity et al. extended this direction to cyberbullying specifically with MultiBully, a code-mixed, multi-label (bully, sentiment, emotion, sarcasm) meme dataset collected from Twitter and Reddit, which we use as the cyberbullying benchmark in this paper [25]. To our knowledge, no prior work has directly compared graph- and hierarchy-based cross-modal incongruity architectures originally developed for sarcasm detection on this dataset, nor examined whether sarcasm-trained incongruity representations transfer to bullying detection. 2.3 Multimodal sentiment, emotion, and meme analysis Beyond sarcasm and bullying specifically, a broader literature on multimodal sentiment and emotion analysis informs the fusion and attention design choices used here. Recent surveys taxonomize fusion strategies, learning paradigms, and attention mechanisms across the field [36], and graph convolutional approaches have been proposed for speaker-aware multimodal emotion recognition [37] and for conversational emotion recognition more broadly [13], motivating the use of graph attention as a cross-modal reasoning mechanism in this work. Attention-based frameworks have also been proposed specifically for multimodal sentiment analysis in memes [35], the same content format used in MultiBully. Graph neural networks more generally have been surveyed extensively [38], and their foundational variants [17, 34, 3] underpin the graph-reasoning modules in both GCCN and HCIG. 2.4 Architectural components GCCN and HCIG both build on Transformer-based encoders [33]: RoBERTa [22] for text and Vision Transformer [9] for images, trained with the AdamW optimizer [23], a decoupled-weight-decay variant of Adam [16]. Both use graph attention networks for cross-modal reasoning, specifically GATv2 [3], which was proposed to correct a static-attention limitation in the original graph attention formulation [34]; the related graph convolutional network formulation [17] is noted here as an alternative, non-attentive graph reasoning mechanism not used in this work. Where relevant we also note CLIP-style joint vision–language pretraining [29] and ViLBERT-style two-stream co-attentional pretraining [24] as alternative encoder strategies not used in this study, and Grad-CAM [31] as the visualization technique underlying part of the exploratory analysis in Section 5. Beyond the graph-attention formulation adopted here, entropic optimal transport [6] and Dirichlet-based evidential uncertainty estimation [32] have both been proposed as alternative mechanisms for cross-modal alignment and confidence-aware prediction, respectively; neither is used by GCCN or HCIG in the present study, but both remain relevant directions for future extensions of this work. 2.5 Summary comparison Table 1 summarizes representative prior work by task, modality, core technique, and dataset, situating HCIG and GCCN relative to the existing literature. Table 1: Comparison of representative multimodal sarcasm and cyberbullying detection methods. Study Year Task Core Technique Dataset(s) Schifanella et al. [30] 2016 Sarcasm Hand-crafted + CNN visual features Twitter/Instagram Cai et al. [4] 2019 Sarcasm Hierarchical fusion (text/image/attribute) MMSD Xu et al. [39] 2020 Sarcasm Decomposition and relation network MMSD Pan et al. [27] 2020 Sarcasm Intra-/inter-modality transformer attention MMSD Liang et al. [19] 2021 Sarcasm In-modal + cross-modal graphs MMSD Liang et al. [20] 2022 Sarcasm Cross-modal graph convolutional network MMSD Qin et al. [28] 2023 Sarcasm Debiased benchmark + baseline re-evaluation MMSD2.0 Kiela et al. [14] 2020 Hate speech Multimodal meme benchmark + baselines Hateful Memes Maity et al. [25] 2022 Cyberbullying Multitask sentiment/emotion/sarcasm-aware network MultiBully This work (GCCN) – Sarcasm + Bullying Similarity-graph + GATv2 + contradiction pooling MMSD, MultiBully This work (HCIG) – Sarcasm + Bullying Hierarchical token/phrase/global GATv2 + attention gate MMSD, MultiBully 3 Proposed Methodology Figure 2 provides a conceptual overview of the shared multimodal processing pipeline and positions HCIG and GCCN as alternative task-specific architectures. The detailed computational flows of GCCN and HCIG are shown in Figures 3 and 4, respectively; the exact operations implemented by the models are defined mathematically in Sections 3.4 and 3.5. Figure 2: Conceptual overview of the shared multimodal input, preprocessing, encoder, cross-modal interaction, and graph-construction stages, followed by the alternative HCIG and GCCN prediction architectures. 3.1 Problem Formulation and Notation We formulate multimodal sarcasm and cyberbullying detection as binary classification over paired text–image inputs. Let =(ti,vi,yi)i=1ND=\(t_i,v_i,y_i)\_i=1^N denote a dataset of N text–image pairs, where tit_i is a social-media post’s text, viv_i is its associated image, and yi∈0,1y_i∈\0,1\ is a binary label (sarcastic/non-sarcastic for MMSD, bully/non-bully for MultiBully). The text tit_i is tokenized into a sequence of n subword tokens w1,…,wn\w_1,…,w_n\ and encoded by a text Transformer to produce contextual token representations Hit∈ℝn×dH^t_i ^n× d and a pooled sentence representation hCLS,it∈ℝdh^t_CLS,i ^d. The image viv_i is divided into m fixed-size patches and encoded by a vision Transformer to produce patch representations Hiv∈ℝm×dH^v_i ^m× d and a pooled image representation hCLS,iv∈ℝdh^v_CLS,i ^d, with d the shared hidden dimensionality of both encoders. The goal is to learn a function fθ:(ti,vi)↦y^if_θ:(t_i,v_i) y_i that predicts y^i≈yi y_i≈ y_i while explicitly modeling a cross-modal incongruity signal δ(Hit,Hiv)δ(H^t_i,H^v_i) that captures the degree of semantic mismatch between the text and image, rather than relying only on a fused joint representation [hCLS,it;hCLS,iv][h^t_CLS,i;h^v_CLS,i]. We denote by ziz_i the learned incongruity-aware representation produced by an architecture’s cross-modal reasoning module, which is combined with the pooled unimodal representations and passed to a classification head gθ(⋅)g_θ(·): y^i=gθ([zi;hCLS,it;hCLS,iv]) y_i=g_θ ([\,z_i\,;\,h^t_CLS,i\,;\,h^v_CLS,i\,] ) (1) GCCN and HCIG, described below, differ in how ziz_i is computed and, in HCIG’s case, in whether hCLS,ith^t_CLS,i and hCLS,ivh^v_CLS,i are concatenated again at the classification head or are already subsumed into ziz_i (Section 3.5). 3.2 Backbone Encoders Both architectures share a common backbone: text is encoded with RoBERTa-base [22] and images with a ViT-Base/16 Vision Transformer [9], both fine-tuned end-to-end, following the general Transformer encoder design of Vaswani et al. [33]. Text sequences are tokenized to a maximum length of 80 subwords; images are resized to 224×224224× 224. Image features require an explicit alignment step because ViT backbones expose a class token concatenated with patch tokens, which must be separated before cross-modal reasoning. Writing ϕViT(vi)∈ℝ(m+1)×dv _ViT(v_i) ^(m+1)× d_v for the raw ViT feature sequence and Wv∈ℝd×dvW_v ^d× d_v for a learned projection to the shared hidden size, H~iv=WvϕViT(vi)∈ℝ(m+1)×d,hCLS,iv=H~iv[0],Hiv=H~iv[1:], H^v_i=W_v\, _ViT(v_i) ^(m+1)× d, h^v_CLS,i= H^v_i[0], H^v_i= H^v_i[1:], (2) so that hCLS,ivh^v_CLS,i and the m patch representations HivH^v_i used downstream are disjoint; backbones without a class token instead pool a single pseudo-patch and reuse it as hCLS,ivh^v_CLS,i. On the text side, the RoBERTa output sequence contains a beginning-of-sequence (BOS) token, an end-of-sequence (EOS) token, and right-padding, none of which should participate in token-level cross-modal reasoning. Let ai,j∈0,1a_i,j∈\0,1\ be the attention mask returned by the tokenizer and ℓi=∑jai,j _i= _ja_i,j the true sequence length (including BOS/EOS). The BOS position is used as hCLS,ith^t_CLS,i; the remaining n−1n-1 positions form HitH^t_i, with a token validity mask ℳit[j]=[j<ℓi−1],j=1,…,n−1,M^t_i[j]=1 [\,j< _i-1\, ], j=1,…,n-1, (3) that excludes BOS, EOS, and padding, so that only genuine subword tokens can enter the contradiction- and incongruity-scoring modules described below. We write i=j:ℳit[j]=1T_i=\j:M^t_i[j]=1\ for the resulting set of valid token indices. 3.3 Shared Cross-Modal Graph Construction and Reasoning GCCN and HCIG both reason over text–image node sets using the same graph-construction-and-propagation primitive, applied at different granularities; we describe it once here and reuse it by reference in Sections 3.4–3.5. Given a set of text-side nodes x1,…,xp\x_1,…,x_p\ (individual tokens, pooled phrases, or a single CLS vector, depending on level) and image-side nodes xp+1,…,xp+q\x_p+1,…,x_p+q\ (patches or a single CLS vector), each node is first tagged with a learned modality embedding emod:0,1→ℝde_mod:\0,1\ ^d, x^u=xu+emod([u is an image node]),u=1,…,p+q, x_u=x_u+e_mod(1[u is an image node]), u=1,…,p+q, (4) so the subsequent graph attention can distinguish node modality independently of content. A weighted graph is then built by thresholding pairwise cosine similarity, with self-loops inserted explicitly rather than learned: suw=x^u⋅x^w∥x^u∥∥x^w∥+ϵ,uw=[suw>τ]∨ 1[u=w],s_uw= x_u· x_w x_u \, x_w +ε, _uw=1[\,s_uw>τ\,]\ \ 1[u=w], (5) giving a sparse edge set ℰ=(u,w):uw=1E=\(u,w):A_uw=1\ with each retained edge carrying its similarity as an edge attribute, euw=suwe_uw=s_uw. A two-layer GATv2 network [3] propagates over (x^u,ℰ,euw)(\ x_u\,E,\e_uw\); for a single attention head, the update at node u over neighborhood (u)N(u) is αuw=exp⁡(a⊤LeakyReLU(W1x^u+W2x^w+Weeuw))∑w′∈(u)exp⁡(a⊤LeakyReLU(W1x^u+W2x^w′+Weeuw′)), _uw= (a LeakyReLU(W_1 x_u+W_2 x_w+W_e\,e_uw) ) _w (u) (a LeakyReLU(W_1 x_u+W_2 x_w +W_e\,e_uw ) ), (6) X(1)=GELU([∑w∈(u)αuw(h)W2(h)x^w]h=1H),X(2)=GATv2(X(1),ℰ,euw),X^(1)=GELU ( [ _w (u)α^(h)_uwW_2^(h) x_w ]_h=1^H ), X^(2)=GATv2 (X^(1),E,\e_uw\ ), (7) where the first layer concatenates H attention heads (Eq. 6 instantiated per head) and the second layer uses a single head to return node states in ℝdR^d. We denote this two-stage operator compactly as GraphReason(xu,ℰ)GraphReason(\x_u\,E) in the remainder of this section. 3.4 GCCN: Graph-based Cross-modal Contradiction Network GCCN instantiates a single application of GraphReasonGraphReason over the full set of valid text tokens hjt:j∈i\h^t_j:j _i\ and all m+1m+1 image nodes (patches plus CLS), following Eqs. 4–7 with threshold τ=0.3τ=0.3, and mean-pools the resulting node states into a single graph-level embedding: zigraph=1|i|+m+1∑uXu(2)z^graph_i= 1|T_i|+m+1 _uX^(2)_u (8) In parallel, a dedicated contradiction-pooling module scores every valid token–patch pair directly, independently of the graph pathway. For token j∈ij _i and patch l∈1,…,ml∈\1,…,m\, sjl=hjt⋅hlv∥hjt∥∥hlv∥+ϵ,djl=clip(1−sjl, 0, 2),s_jl= h^t_j· h^v_l h^t_j \, h^v_l +ε, d_jl=clip(1-s_jl,\,0,\,2), (9) and a small learned gate, conditioned on similarity, distance, and squared similarity, controls how much each pair contributes to the contradiction score: Cjl=σ(MLPc([sjl;djl;sjl2]))⋅djlC_jl=σ (MLP_c([\,s_jl\,;\,d_jl\,;\,s_jl^2\,]) )· d_jl (10) A single scalar contradiction score per example is obtained by masked top-k pooling (k=10k=10) over the valid entries of the contradiction matrix Ci=Cjlj∈i,l≤mC_i=\C_jl\_j _i,\,l≤ m, ci=1min⁡(k,|valid(Ci)|)∑(j,l)∈TopKk(valid(Ci))Cjl,c_i= 1 (k,|valid(C_i)|) _(j,l)\,∈\,TopK_k(valid(C_i))C_jl, (11) so that cic_i reflects the most strongly contradicted token–patch pairs for the example rather than an average dominated by well-aligned pairs. GCCN instantiates Eq. 1 with zi=[zigraph;ci]z_i=[\,z^graph_i\,;\,c_i\,] (12) The end-to-end GCCN processing sequence, including modality encoding, graph construction, adaptive GATv2 reasoning, contradiction-aware pooling, graph readout, and task-specific prediction, is illustrated conceptually in Figure 3. Figure 3: Conceptual architecture of GCCN, showing multimodal encoding, cross-modal graph construction, adaptive GATv2 reasoning, contradiction-aware pooling, graph readout, and task-specific classification. 3.5 HCIG: Hierarchical Cross-modal Incongruity Graph Network HCIG, the primary architecture proposed in this work, computes incongruity at three levels of granularity rather than a single graph pass, motivated by the observation that cross-modal incongruity in sarcasm and bullying content can manifest as an individual word–object mismatch (token level), a short phrase-level mismatch (phrase level), or a single global tone mismatch between the overall message and image (global level). Token and phrase levels share the same three-stage procedure — graph reasoning, incongruity-guided alignment, residual fusion — applied to different text granularities against the same set of image patches; the global level instead operates directly on the pooled [CLS] representations. Phrase construction. Before phrase-level reasoning, valid text tokens are mean-pooled into non-overlapping, mask-aware groups of three: hi,gt,phr=∑r=13μi,g,rhi, 3(g−1)+rt∑r=13μi,g,r+ϵ,μi,g=⋁r=13μi,g,r,h^t,phr_i,g= _r=1^3 _i,g,r\,h^t_i,\,3(g-1)+r _r=1^3 _i,g,r+ε, _i,g= _r=1^3 _i,g,r, (13) for g=1,…,⌈n/3⌉g=1,…, n/3 , where μi,g,r∈0,1 _i,g,r∈\0,1\ marks whether the r-th token of group g is a valid (non-padding) token per Eq. 3; groups with no valid tokens are masked out entirely by μi,g=0 _i,g=0. Level graph reasoning. At each level lvl∈tok,phrlvl∈\tok,phr\, the level’s text units (tokens or phrases) and the m image patches are passed through GraphReasonGraphReason (Eqs. 4–7) using level-specific GATv2 weights, producing graph-updated text units h~ulvl\ h^lvl_u\ and graph-updated patches h~lv\ h^v_l\. Incongruity-guided cross-modal alignment. For each graph-updated text unit u at level lvl and each graph-updated patch l, sul=cos⁡(h~ulvl,h~lv),dul=clip(1−sul,0,2),ιul=σ(MLPι([sul;dul;sul2]))⋅dul,s_ul= ( h^lvl_u, h^v_l), d_ul=clip(1-s_ul,0,2), _ul=σ (MLP_ ([\,s_ul\,;\,d_ul\,;\,s_ul^2\,]) )· d_ul, (14) using a single MLPιMLP_ shared across the token and phrase levels. Rather than aligning each text unit to its most similar patch, HCIG deliberately attends toward the most divergent patches, since it is precisely the mismatched regions of the image that carry incongruity signal: wul=exp⁡(dul)∑l′exp⁡(dul′),a~u=∑lwulh~lvw_ul= (d_ul) _l (d_ul ), a_u= _lw_ul\, h^v_l (15) Each text unit is then updated with a residual fusion of itself and its incongruity-aligned image counterpart, ru=LN(MLPr([h~ulvl;a~u;|h~ulvl−a~u|;h~ulvl⊙a~u])),r_u=LN (MLP_r ([\, h^lvl_u\,;\, a_u\,;\,| h^lvl_u- a_u|\,;\, h^lvl_u a_u\,] ) ), (16) and the level representation is obtained by pooling residual states, weighted by each unit’s peak incongruity across patches, ωu=maxl⁡ιul _u= _l _ul: zilvl=∑uωuru∑uωu+ϵ,ιilvl=1|i|⋅m∑u∈i∑lιul,z^lvl_i= _u _u\,r_u _u _u+ε, ^lvl_i= 1|U_i|· m _u _i _l _ul, (17) where iU_i is the set of valid units at that level (iT_i for tokens, valid phrase groups for phrases) and ιilvl ^lvl_i is a scalar level-incongruity score used both for reporting and for the fusion gate below. Global level. The pooled [CLS] representations are compared directly, without graph reasoning, via the same pairwise-interaction pattern used at the token and phrase levels: ziglob z^glob_i =LN(MLPg([hCLS,it;hCLS,iv; =LN (MLP_g ([\,h^t_CLS,i\,;\,h^v_CLS,i\,; (18) |hCLS,it−hCLS,iv|;hCLS,it⊙hCLS,iv])), 47.00003pt\,|h^t_CLS,i-h^v_CLS,i|\,;\,h^t_CLS,i h^v_CLS,i\,] ) ), ιiglob ^glob_i =1−cos⁡(hCLS,it,hCLS,iv) =1- (h^t_CLS,i,h^v_CLS,i) Level-attention fusion. The three level representations and their scalar incongruity scores are combined with a learned softmax attention gate rather than fixed or uniform weighting: i=softmax(MLPα([zitok;ziphr;ziglob;ιitok;ιiphr;ιiglob]))∈Δ2, α_i=softmax (MLP_α ([\,z^tok_i\,;\,z^phr_i\,;\,z^glob_i\,;\, ^tok_i\,;\, ^phr_i\,;\, ^glob_i\,] ) )∈ ^2, (19) so the gate can condition its weighting on how incongruent each level already appears to be, not only on the content of each level’s representation. The gated level representations are then concatenated — preserving each level’s identity within the fused vector rather than collapsing them into a single d-dimensional average — and passed to the classifier: zi=[αitokzitok;αiphrziphr;αiglobziglob]∈ℝ3d,y^i=gθ(zi),z_i= [\,α^tok_i\,z^tok_i\,;\,α^phr_i\,z^phr_i\,;\,α^glob_i\,z^glob_i\, ] ^3d, y_i=g_θ(z_i), (20) which departs from the general fusion form of Eq. 1: because ziglobz^glob_i (Eq. 18) already integrates hCLS,ith^t_CLS,i and hCLS,ivh^v_CLS,i through their pairwise interaction, HCIG’s classifier operates directly on ziz_i rather than re-concatenating the raw CLS vectors, avoiding redundant duplication of the same information. A single scalar incongruity score is additionally defined as the attention-weighted combination of the three level scores, ιi=αitokιitok+αiphrιiphr+αiglobιiglob _i=α^tok_i\, ^tok_i+α^phr_i\, ^phr_i+α^glob_i\, ^glob_i (21) All GATv2 layers across the three levels are genuinely active in the forward pass, correcting an implementation issue present in an earlier development version of this architecture in which GAT layers were declared but not used; this is noted explicitly because it is directly relevant to interpreting the corresponding ablation results in Section 5.2. Figure 4 summarizes the hierarchical HCIG workflow, from multimodal feature extraction and cross-modal alignment to token-, phrase-, and global-level graph reasoning, hierarchical attention fusion, and task-specific prediction. Figure 4: Conceptual architecture of HCIG, showing token-, phrase-, and global-level incongruity reasoning, GATv2-based graph processing, hierarchical attention fusion, and task-specific classification. 3.6 Training Objective Both architectures are trained end-to-end with a standard binary cross-entropy classification loss on y^i y_i against yiy_i, using the AdamW optimizer [23] with a linear warmup schedule. No auxiliary losses (e.g., contrastive, counterfactual, or uncertainty-regularization terms) are used for either GCCN or HCIG in this work; both rely solely on the classification objective operating on the incongruity-aware representation ziz_i defined above. 3.7 Algorithms Algorithm 1 summarizes the GCCN forward pass (Section 3.4), and Algorithm 2 summarizes the HCIG forward pass (Section 3.5), which is the main architectural contribution of this paper. Algorithm 1 GCCN Forward Pass 1:Text tokens t, image v 2:Prediction y y, contradiction score c 3:Ht,hCLSt←RoBERTa(t)H^t,h^t_CLS (t); Hv,hCLSv←ViT(v)H^v,h^v_CLS (v) ⊳ Eq. 2 4:←j:ℳt[j]=1T←\j:M^t[j]=1\ ⊳ Valid token mask, Eq. 3 5:G←BuildSimilarityGraph(Hjtj∈,Hv∪hCLSv,τ)G (\H^t_j\_j ,H^v∪ h^v_CLS,τ) ⊳ Eqs. 4–5 6:X←GraphReason(G)X (G) ⊳ Eqs. 6–7 7:zgraph←MeanPool(X)z^graph (X) ⊳ Eq. 8 8:C←GatedContradiction(Hjtj∈,Hv)C (\H^t_j\_j ,H^v) ⊳ Eqs. 9–10 9:c←MaskedTopK(C,k=10)c (C,k=10) ⊳ Eq. 11 10:z←[zgraph;c]z←[z^graph;c] ⊳ Eq. 12 11:y^←Classifier([z;hCLSt;hCLSv]) y ([z;h^t_CLS;h^v_CLS]) ⊳ Eq. 1 12:return y^,c y,c Algorithm 2 HCIG Forward Pass 1:Text tokens t, image v 2:Prediction y y, level-attention weights α 3:Ht,hCLSt←RoBERTa(t)H^t,h^t_CLS (t); Hv,hCLSv←ViT(v)H^v,h^v_CLS (v) ⊳ Eq. 2 4:Gtok←BuildSimilarityGraph(Ht,Hv,τ)G_tok (H^t,H^v,τ); H~t,H~v←GraphReason(Gtok) H^t, H^v (G_tok) 5:ztok,ιtok←IncongruityAlign(H~t,H~v)z^tok, ^tok ( H^t, H^v) ⊳ Eqs. 14–17 6:Ht,phr←PhrasePool(Ht,size=3)H^t,phr (H^t,size=3) ⊳ Eq. 13 7:Gphr←BuildSimilarityGraph(Ht,phr,Hv,τ)G_phr (H^t,phr,H^v,τ); H~t,phr,H~phrv←GraphReason(Gphr) H^t,phr, H^v_phr (G_phr) 8:zphr,ιphr←IncongruityAlign(H~t,phr,H~phrv)z^phr, ^phr ( H^t,phr, H^v_phr) ⊳ Eqs. 14–17 9:zglob,ιglob←GlobalContradiction(hCLSt,hCLSv)z^glob, ^glob (h^t_CLS,h^v_CLS) ⊳ Eq. 18 10:←Softmax(MLP([ztok;zphr;zglob;ιtok;ιphr;ιglob])) α (MLP([z^tok;z^phr;z^glob; ^tok; ^phr; ^glob])) ⊳ Level-attention gate, Eq. 19 11:z←[αtokztok;αphrzphr;αglobzglob]z←[α^tokz^tok;α^phrz^phr;α^globz^glob] ⊳ Eq. 20 12:y^←Classifier(z) y (z) ⊳ Eq. 20 13:return y^, y, α 4 Implementation Details 4.1 Dataset and Exploratory Analysis 4.1.1 Multimodal Sarcasm Detection (MMSD) The raw MMSD release [4] comprises 33,696 text–image pairs labeled sarcastic or non-sarcastic. After removing rows with missing or corrupted image files, 24,472 pairs (72.6% of the raw set) remained for experiments. The drop was not uniform across splits: 31.9% of training rows were removed for missing images, versus 0% of validation and test rows, because the validation and test partitions had already been curated in the source release. This asymmetry is reported explicitly because it means the effective training set is smaller, and its class balance and image-quality profile is not guaranteed to match the untouched validation/test partitions. Table 2 and Figure 5 summarize the cleaned splits and their class distributions. 4.1.2 MultiBully The MultiBully dataset [25] pairs short social-media text with an associated image and a binary bully/non-bully label, collected in a code-mixed language setting from Twitter and Reddit. After removing 61 pairs (1.0%) with missing or unreadable images, 5,793 pairs remained, comprising 3,188 bully and 2,605 non-bully examples (Table 2, Figure 6), a moderate class imbalance retained without resampling in all reported experiments. Table 2: Statistics of the cleaned MMSD and MultiBully dataset. Dataset Split Total pairs Negative class Positive class MMSD train 19,694 11,067 (Non-sarcastic) 8,627 (Sarcastic) MMSD val 2,394 1,441 (Non-sarcastic) 953 (Sarcastic) MMSD test 2,384 1,428 (Non-sarcastic) 956 (Sarcastic) MultiBully all (cleaned) 5,793 2,605 (Non-bully) 3,188 (Bully) Figure 5: MMSD class distribution by split (train/val/test), sarcastic vs. non-sarcastic. Figure 6: Label and sentiment distribution in the cleaned MultiBully dataset. 4.2 Baselines Three baselines are reported: (i) Text-BERT, a BERT-base-uncased classifier [8] using only the text modality; (i) Image-ResNet, a fine-tuned ResNet-50 classifier [11] using only the image modality; and (i) Late Fusion, which concatenates pooled text and image representations from the shared RoBERTa/ViT backbone before a classification head, without any explicit incongruity modeling. 4.3 Training Configuration and Experimental Setup Table 3 summarizes the shared training configuration used across all models, baselines, ablations, and cross-task experiments unless the ablated component was itself the subject of the change. Three evaluation regimes are reported: (1) in-domain MMSD, where all five models (three baselines, GCCN, HCIG) are trained and evaluated on the MMSD train/validation/test splits described in Section 4.1; (2) in-domain and fine-tuned MultiBully, where the same five architectures are trained from randomly initialized task heads on MultiBully (“in-domain”), and GCCN/HCIG are additionally initialized from their complete MMSD-trained checkpoints before continued training on MultiBully (“fine-tuned”). In this fine-tuning regime, the existing two-class MMSD classification head is retained rather than reinitialized or replaced, and all loaded parameters are further optimized on MultiBully; and (3) a bidirectional cross-task direct-transfer stress test, where models trained only on MMSD are applied directly to the MultiBully test set with no target-task training, and vice versa. Table 3: Experimental settings and training hyperparameters. Setting Value / Description Optimizer AdamW [23], linear warmup schedule Batch size 6 (constrained by combined RoBERTa, ViT, and graph-module memory footprint) Max epochs 10, with early stopping (patience =3=3) on validation macro-F1 Text max length 80 subword tokens Image size 224×224224× 224 Graph similarity threshold 0.3 Contradiction top-k (k) 10 Phrase group size 3 tokens Evaluation regimes In-domain MMSD; in-domain and fine-tuned MultiBully; bidirectional cross-task direct-transfer stress test Reported metrics Accuracy, Precision, Recall, F1, Macro-F1, ROC-AUC 5 Results 5.1 MMSD in-domain results Table 4 reports test-set performance for all five models on MMSD . HCIG achieves the best accuracy, F1, and macro-F1 of any model; GCCN achieves the best ROC-AUC. Both proposed architectures exceed all three baselines on accuracy, F1, and macro-F1. Figure 7 visualizes the same comparison across metrics, and Figure 8 shows the corresponding confusion matrices; Figure 9 shows ROC and precision–recall curves. Table 4: Performance comparison on the MMSD test set. Model Acc. Prec. Rec. F1 Macro-F1 AUC PR-AUC Image-ResNet (baseline) 69.92 59.20 80.44 68.20 69.84 77.92 65.28 Text-BERT (baseline) 83.72 77.79 83.16 80.38 83.24 90.54 85.38 Late Fusion (baseline) 84.35 78.33 84.31 81.21 83.90 90.27 84.18 GCCN (ours) 85.15 78.61 86.51 82.37 84.77 90.99 83.19 HCIG (ours) 85.74 80.37 85.25 82.74 85.29 90.30 84.34 Figure 7: Performance comparison of baseline and proposed models on MMSD. Figure 8: Confusion matrices of the evaluated MMSD models. Figure 9: ROC curves (left) and precision–recall curves (right) for the MMSD models. 5.2 Ablation studies Component ablations were run for GCCN and HCIG on MMSD, holding the training configuration fixed and removing or isolating one architectural component at a time (Tables 5–6, Figure 10). ROC-AUC for the full models is reported only in Table 4, which is derived from the canonical main-evaluation prediction archive; the ablation tables are therefore restricted to accuracy and F1-family metrics from the common ablation evaluation. Table 5: Ablation study of GCCN on MMSD. Variant Acc. Prec. Rec. F1 Macro-F1 GCCN (full) 85.15 84.46 85.37 82.37 84.77 w/o graph reasoning 84.48 83.75 84.37 81.24 84.00 w/o contradiction pooling 84.35 83.68 83.85 80.64 83.76 text-only (no image branch) 84.36 83.15 84.20 81.87 84.09 Table 6: Ablation study of HCIG on MMSD. Variant Acc. Prec. Rec. F1 Macro-F1 HCIG (full) 85.74 85.04 85.66 82.74 85.29 token-level only 85.44 84.74 85.50 82.54 85.03 w/o global level 84.65 83.93 84.70 81.61 84.22 w/o graph reasoning 84.65 83.96 84.89 81.81 84.26 text-only (no image branch) 84.23 84.15 84.83 81.88 84.74 Figure 10: Decrease in positive-class F1 and macro-F1 after removing or isolating individual GCCN and HCIG components, relative to each corresponding full model. 5.3 MultiBully in-domain and fine-tuning results Table 7 reports MultiBully results under two training regimes: models trained in-domain from randomly initialized heads, and GCCN/HCIG initialized from their complete MMSD checkpoints and then further optimized on MultiBully. The fine-tuning runs retain the existing two-class MMSD classification head and jointly update it with the remaining model parameters; the head is not reinitialized or replaced. Using macro-F1 as the primary class-balanced metric, GCCN obtains the highest in-domain score (68.66%), whereas HCIG obtains the highest accuracy (69.62%) and bullying-class F1 (74.90%). Figure 11 shows the in-domain comparison across accuracy, positive-class F1, macro-F1, and ROC-AUC. Table 7: Performance comparison on the MultiBully dataset. Model Regime Acc. Prec. Rec. F1 Macro-F1 AUC Text-BERT In-domain 64.33 63.88 63.66 68.43 63.71 68.78 Image-ResNet In-domain 64.56 64.11 63.83 68.83 63.88 71.60 Late Fusion In-domain 64.67 64.36 63.46 70.17 63.43 69.64 GCCN In-domain 69.39 69.17 68.54 73.45 68.66 75.02 HCIG In-domain 69.62 70.08 68.20 74.90 68.21 74.91 GCCN Fine-tuned from MMSD 68.24 68.54 66.80 73.76 66.77 73.18 HCIG Fine-tuned from MMSD 65.59 67.35 63.25 73.47 62.27 70.73 Figure 11: MultiBully in-domain comparison across accuracy, positive-class F1, macro-F1, and ROC-AUC. 5.4 Bidirectional cross-task direct-transfer stress test Models trained exclusively on MMSD were applied directly to the MultiBully test set (MMSD → MB), and models trained exclusively on MultiBully were applied directly to the MMSD test set (MB→ ), in both cases without target-task training (Figure 12). Because MMSD predicts sarcasm whereas MultiBully predicts bullying, the source and target labels represent different constructs; this experiment is therefore treated as a bidirectional cross-task direct-transfer stress test rather than conventional zero-shot domain transfer. For both GCCN and HCIG, direct-transfer accuracy in both directions falls in the 51–56% range, close to the corresponding majority-class baseline, and neither architecture shows a clear advantage under this cross-task classifier reuse. The in-domain differences between GCCN and HCIG (Tables 4 and 7) are not preserved in the stress-test setting. Figure 12: Accuracy and macro-F1 for GCCN and HCIG across MMSD in-domain evaluation, MultiBully in-domain evaluation, bidirectional cross-task direct-transfer stress tests, and MMSD-initialized MultiBully fine-tuning. 5.5 Explainability and qualitative analysis Figure 13 shows HCIG’s learned level-attention weights (token/phrase/global) by label, and Figure 14 shows GCCN’s contradiction score distribution by label. HCIG’s level-attention weights show a strong and consistent preference for the token-level branch across both classes, with the global branch contributing a secondary signal and the phrase-level branch contributing comparatively little, consistent with the ablation finding (Table 6) that a token-only variant already recovers most of HCIG’s full performance. Figure 13: HCIG level-attention weights (token/phrase/global) by label. Figure 14: GCCN contradiction score by label. 6 Conclusion and Future Work This paper presented HCIG, a novel Hierarchical Cross-modal Incongruity Graph Network for multimodal sarcasm and cyberbullying detection by explicitly modeling semantic incongruity between textual and visual modalities at token, phrase, and global levels. Unlike conventional multimodal fusion approaches that primarily focus on feature aggregation, HCIG performs hierarchical graph-based reasoning to capture fine-grained cross-modal inconsistencies and adaptively integrates multi-level representations through an attention-guided fusion mechanism. In addition, a lightweight Graph-based Cross-modal Contradiction Network (GCCN) was introduced as a complementary architecture to investigate contradiction-aware multimodal reasoning. Extensive experiments on the MMSD and MultiBully benchmark datasets demonstrate the effectiveness of the proposed frameworks. HCIG achieved the best performance on MMSD with an accuracy of 85.74% and a macro-F1 score of 85.29%, whereas GCCN obtained the highest macro-F1 score (68.66%) on MultiBully while HCIG achieved the highest accuracy (69.62%) and bullying-class F1 (74.90%). Furthermore, the ablation studies demonstrate the effectiveness of hierarchical incongruity modeling, highlighting that token-level semantic interactions provide the strongest discriminative cues while phrase-level and global representations contribute complementary contextual information. The cross-task transfer analysis further indicates that sarcasm and cyberbullying exhibit distinct semantic characteristics, emphasizing the importance of task-specific multimodal reasoning for effective social media understanding. The rapid advancement of multimodal artificial intelligence and foundation models is expected to drive the next generation of multimodal content understanding systems. Future research should focus on developing more context-aware frameworks capable of reasoning over implicit sarcasm, offensive intent, cultural references, conversational history, and temporal dependencies across multiple modalities. Integrating graph reasoning with large language models and large multimodal models offers promising opportunities for improving semantic alignment, cross-modal reasoning, and generalization across diverse social media domains. Future benchmark datasets should also include multilingual, multicultural, and real-world multimodal content while adopting standardized annotation protocols and evaluation settings to improve reproducibility and facilitate meaningful comparison across studies. In addition, enhancing explainability, adversarial robustness, fairness, computational efficiency, and human-centered AI will be essential for building reliable and trustworthy multimodal moderation systems. We believe that the proposed HCIG framework provides a strong foundation for future research toward robust, interpretable, and scalable multimodal sarcasm and cyberbullying detection systems. Data and Code Availability All result tables reported in this paper are provided as supplementary CSV files. MMSD is available from [4]; MultiBully is available from [25]. References [1] H. Allwaibed, M. Anbar, S. Manickam, and A. Bintang (2025) Cyberbullying detection approaches for Arabic texts: a systematic literature review. Frontiers in Artificial Intelligence. External Links: Document Cited by: §2.2. [2] V. Balakrisnan and M. Kaity (2023) Cyberbullying detection and machine learning: a systematic literature review. Artificial Intelligence Review 56, p. 1375–1416. External Links: Document Cited by: §1, §2.2. [3] S. Brody, U. Alon, and E. Yahav (2022) How attentive are graph attention networks?. In International Conference on Learning Representations, External Links: Document Cited by: §1, §2.3, §2.4, §3.3. [4] Y. Cai, H. Cai, and X. Wan (2019) Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 2506–2515. External Links: Document Cited by: §1, §2.1, Table 1, §4.1.1, Data and Code Availability. [5] S. Castro, D. Hazarika, V. Pérez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria (2019) Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4619–4629. External Links: Document Cited by: §2.1. [6] M. Cuturi (2013) Sinkhorn distances: lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26, p. 2292–2300. External Links: Document Cited by: §2.4. [7] A. Das, J. S. Wahi, and S. Li (2020) Detecting hate speech in multi-modal memes. arXiv preprint arXiv:2012.14891. External Links: Document Cited by: §1, §2.2. [8] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, p. 4171–4186. External Links: Document Cited by: §1, §4.2. [9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations, External Links: Document Cited by: §2.4, §3.2. [10] R. Gómez, J. Gibert, L. Gomez, and D. Karatzas (2020) Exploring hate speech detection in multimodal publications. In 2020 IEEE Winter Conference on Applications of Computer Vision, p. 1459–1467. External Links: Document Cited by: §1, §2.2. [11] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778. External Links: Document Cited by: §4.2. [12] E. Hossain, O. Sharif, M. M. Hoque, and S. M. Preum (2024) Deciphering hate: identifying hateful memes and their targets. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 8347–8359. External Links: Document Cited by: §1, §2.2. [13] J. Hu, Y. Liu, J. Zhao, and Q. Jin (2021) MMGCN: multimodal fusion via deep graph convolution network for emotion recognition in conversation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, p. 5666–5675. External Links: Document Cited by: §2.3. [14] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine (2020) The hateful memes challenge: detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems 33, External Links: Document Cited by: §1, §2.2, Table 1. [15] S. Kim, A. Razi, G. Stringhini, P. J. Wisniewski, and M. De Choudhury (2021) A human-centered systematic literature review of cyberbullying detection algorithms. Proceedings of the ACM on Human-Computer Interaction 5 (CSCW2), p. 1–34. External Links: Document Cited by: §1, §2.2. [16] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations, External Links: Document Cited by: §2.4. [17] T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, External Links: Document Cited by: §2.3, §2.4. [18] R. K. Lee, R. Cao, Z. Fan, J. Jiang, and W. Chong (2021) Disentangling hate in online memes. In Proceedings of the 29th ACM International Conference on Multimedia, External Links: Document Cited by: §1, §2.2. [19] B. Liang, C. Lou, X. Li, L. Gui, M. Yang, and R. Xu (2021) Multi-modal sarcasm detection with interactive in-modal and cross-modal graphs. In Proceedings of the 29th ACM International Conference on Multimedia, p. 4707–4715. External Links: Document Cited by: §1, §2.1, Table 1. [20] B. Liang, C. Lou, X. Li, M. Yang, L. Gui, Y. He, W. Pei, and R. Xu (2022) Multi-modal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, p. 1767–1777. External Links: Document Cited by: §1, §2.1, Table 1. [21] P. Lippe, N. Holla, S. Chandra, S. Rajamanickam, G. Antoniou, E. Shutova, and H. Yannakoudakis (2020) A multimodal framework for the detection of hateful memes. arXiv preprint arXiv:2012.12871. External Links: Document Cited by: §1, §2.2. [22] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. External Links: Document Cited by: §2.4, §3.2. [23] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Document Cited by: §2.4, §3.6, Table 3. [24] J. Lu, D. Batra, D. Parikh, and S. Lee (2019) ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems 32, External Links: Document Cited by: §2.4. [25] K. Maity, P. Jha, S. Saha, and P. Bhattacharyya (2022) A multitask framework for sentiment, emotion and sarcasm aware cyberbullying detection from multi-modal code-mixed memes. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 1739–1749. External Links: Document Cited by: §1, §2.2, Table 1, §4.1.2, Data and Code Availability. [26] M. Mozafari, R. Farahbakhsh, and N. Crespi (2020) Hate speech detection and racial bias mitigation in social media based on BERT model. PLOS ONE 15 (8), p. e0237861. External Links: Document Cited by: §1, §2.2. [27] H. Pan, Z. Lin, P. Fu, Y. Qi, and W. Wang (2020) Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 1383–1392. External Links: Document Cited by: §1, §2.1, Table 1. [28] L. Qin, S. Huang, Q. Chen, C. Cai, Y. Zhang, B. Liang, W. Che, and R. Xu (2023) MMSD2.0: towards a reliable multi-modal sarcasm detection system. In Findings of the Association for Computational Linguistics: ACL 2023, p. 10834–10845. External Links: Document Cited by: §1, §2.1, Table 1. [29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, p. 8748–8763. External Links: Document Cited by: §2.4. [30] R. Schifanella, P. de Juan, J. Tetreault, and L. Cao (2016) Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM International Conference on Multimedia, p. 1136–1145. External Links: Document Cited by: §1, §2.1, Table 1. [31] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision, p. 618–626. External Links: Document Cited by: §2.4. [32] M. Sensoy, L. Kaplan, and M. Kandemir (2018) Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems 31, p. 3183–3193. External Links: Document Cited by: §2.4. [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems 30, p. 5998–6008. External Links: Document Cited by: §2.4, §3.2. [34] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. In International Conference on Learning Representations, External Links: Document Cited by: §2.3, §2.4. [35] B. Verma, P. Meel, and D. K. Vishwakarma (2025) MHAM: a novel framework for multimodal sentiment analysis in memes. Knowledge and Information Systems 67 (11), p. 10355–10394. External Links: Document Cited by: §2.3. [36] B. Verma, P. Meel, and D. K. Vishwakarma (2025) Navigating sentiment analysis through fusion, learning, utterance, and attention methods: an extensive four-fold perspective survey. Engineering Applications of Artificial Intelligence 156, p. 111195. External Links: Document Cited by: §2.3. [37] B. Verma, P. Meel, and D. K. Vishwakarma (2026) SAMER-Net: speaker-aware multimodal emotion recognition with context-aware attention and graph convolutional networks. Applied Intelligence 56 (5), p. 162. External Links: Document Cited by: §2.3. [38] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu (2021) A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems 32 (1), p. 4–24. External Links: Document Cited by: §2.3. [39] N. Xu, Z. Zeng, and W. Mao (2020) Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 3777–3786. External Links: Document Cited by: §1, §2.1, Table 1. [40] C. Yang, F. Zhu, G. Liu, J. Han, and S. Hu (2022) Multimodal hate speech detection via cross-domain knowledge transfer. In Proceedings of the 30th ACM International Conference on Multimedia, p. 4505–4514. External Links: Document Cited by: §1, §2.2.

Instant research discovery