Paper deep dive

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

Yujia Li, Yiqun Zhang, Zihan Cheng, Yijie Huang, Tenglong Ye, Zihan Wang, Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 81

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 8/23/2026, 2:27:03 AM

Summary

The paper introduces HarmTrace, a framework for fine-grained target identification in harmful memes, addressing the gap between general harmfulness detection and specific target identification. It proposes Meme3W, a unified dataset with human-verified annotations, and Joint Record Accuracy (JRA) as a strict evaluation metric. HarmTrace utilizes Entity-aware Supervised Fine-Tuning (E-SFT) and Conditional Target-identification Policy Optimization (CTPO) with a Virtual Positive Anchor (VPA) to improve performance, significantly increasing JRA on models like Qwen3-VL-8B.

Entities (8)

HarmTrace · methodframework · 98%Meme3W · dataset · 97%Joint Record Accuracy · metric · 96%Qwen3-VL-8B · model · 95%Conditional Target-identification Policy Optimization · technique · 94%Entity-aware Supervised Fine-Tuning · technique · 94%Virtual Positive Anchor · concept · 93%InternVL3-8B · model · 92%

Relation Signals (6)

HarmTrace → improves → Joint Record Accuracy

confidence 96% · HarmTrace improves both JRA and harmfulness accuracy across the evaluated backbones...

HarmTrace → appliedto → Qwen3-VL-8B

confidence 95% · JRA on the Qwen3-VL-8B backbone increasing from 17.58% to 52.51%.

HarmTrace → uses → Entity-aware Supervised Fine-Tuning

confidence 95% · HarmTrace first performs entity-aware supervised initialization...

HarmTrace → uses → Conditional Target-identification Policy Optimization

confidence 95% · It then applies Conditional Target-identification Policy Optimization (CTPO)...

Conditional Target-identification Policy Optimization → uses → Virtual Positive Anchor

confidence 94% · CTPO uses a Virtual Positive Anchor (VPA) as a fully correct reference...

Meme3W → supports → Joint Record Accuracy

confidence 93% · To support this task, we introduce Meme3W... We further introduce Joint Record Accuracy (JRA)...

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal harmful meme detection is typically formulated as image--text harmfulness classification. A model may correctly predict harmfulness while misidentifying the attacked target or its supporting evidence. We therefore extend harmful meme detection with fine-grained target identification, asking what type of target is attacked, who is targeted, and where the target appears in the meme. The model predicts harmfulness for every meme and, for harmful memes, outputs the target category, target entity, textual mention, and visual region. To support this task, we introduce Meme3W, which unifies multiple public harmful meme datasets and provides human-verified annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requiring the harmfulness label and all target-identification fields to be jointly correct. Experiments with representative multimodal large language models reveal a substantial gap between harmfulness accuracy and JRA. To narrow this gap, we propose HarmTrace, an anchor-calibrated decoupled optimization framework. HarmTrace strengthens target-entity supervision through entity-aware supervised fine-tuning. It then applies Conditional Target-identification Policy Optimization (CTPO) to decouple harmfulness and target-identification advantages, restricting target-identification optimization to label-correct responses for harmful examples. CTPO uses a Virtual Positive Anchor (VPA) as a fully correct reference for target-identification advantage normalization. HarmTrace improves both JRA and harmfulness accuracy across the evaluated backbones, with JRA on the Qwen3-VL-8B backbone increasing from 17.58\% to 52.51\%. Our code is publicly available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

78,016 characters extracted from source content.

Expand or collapse full text

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes Yujia Li 1 , Yiqun Zhang 2 , Zihan Cheng 1 , Yijie Huang 1 , Tenglong Ye 1 , Zihan Wang 1 , Xiaocui Yang 1 , Shi Feng 1∗ , Yifei Zhang 1 , Daling Wang 1 1 School of Computer Science and Engineering, Northeastern University Shenyang 110819, China 2 Apple Inc., Beijing 100006, China liyujia@mails.neu.edu.cn, fengshi@cse.neu.edu.cn Abstract Multimodal harmful meme detection is typically formulated as image–text harmfulness classification. A model may cor- rectly predict harmfulness while misidentifying the attacked target or its supporting evidence. We therefore extend harmful meme detection with fine-grained target identification, asking what type of target is attacked, who is targeted, and where the target appears in the meme. The model predicts harmfulness for every meme and, for harmful memes, outputs the target cat- egory, target entity, textual mention, and visual region. To sup- port this task, we introduce Meme3W, which unifies multiple public harmful meme datasets and provides human-verified annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requir- ing the harmfulness label and all target-identification fields to be jointly correct. Experiments with representative mul- timodal large language models reveal a substantial gap be- tween harmfulness accuracy and JRA. To narrow this gap, we propose HarmTrace, an anchor-calibrated decoupled opti- mization framework. HarmTrace strengthens target-entity su- pervision through entity-aware supervised fine-tuning. It then applies Conditional Target-identification Policy Optimization (CTPO) to decouple harmfulness and target-identification advantages, restricting target-identification optimization to label-correct responses for harmful examples. CTPO uses a Virtual Positive Anchor (VPA) as a fully correct reference for target-identification advantage normalization. HarmTrace improves both JRA and harmfulness accuracy across the eval- uated backbones, with JRA on the Qwen3-VL-8B backbone increasing from 17.58% to 52.51%. Our code is publicly avail- able at https://github.com/llly1234/HarmTrace-for-Harmful- Memes. Introduction Most existing methods formulate multimodal harmful meme detection primarily as image–text harmfulness classification (Kmainasi et al. 2026; Cheng et al. 2026; Wang et al. 2026; Hou et al. 2026). However, a correct harmfulness label alone does not establish whether the attacked target has been cor- rectly identified (Mia and Fahim 2025). It also does not in- dicate which textual mention and visual region support that target identification. We therefore extend harmful meme de- tection with fine-grained target identification, predicting ∗ Corresponding author. 74.88 73.38 78.27 75.82 72.53 71.97 47.95 44.75 38.81 38.13 17.58 6.62 0 20 40 60 80 Gemini3-FlashGLM-5vKimi k2.5Qwen3.5-397BQwen3VL-8BInternVL3-8B S c o r e ( % ) Label Acc. JRA Label: Target Category: Target Entity: Text Mention: Visual Region: Harmful health condition people with physical disabilities “man” [388,53,908,995] Harmful gender woman null [49,247,480,992] JRA Score: 0 JRA Score: 1 Label: Target Category: Target Entity: Text Mention: Visual Region: Figure 1: JRA requires the harmfulness label and all target- identification fields to be jointly correct. Under this criterion, representative MLLMs show a clear gap between harmful- ness accuracy and fine-grained target identification. harmfulness for every meme and, for harmful memes, addi- tionally identifying the target category, target entity, textual mention, and visual region. For harmful memes, these fields form a traceable record linking the harmfulness judgment to the attacked target and the textual and visual evidence supporting its identification. This record allows reviewers to assess whether the target and evidence are consistent with the final judgment. The EU Digital Services Act emphasizes that moderation decisions should be reasoned and open to review and contestation by affected users (Chiarella 2022). Fine-grained target identi- fication provides the structured information needed for this examination by making the attacked target and its multimodal grounding explicit. The reliability of this record depends on the joint cor- rectness of the harmfulness label and all target-identification fields. Existing harmful meme datasets provide annotations for only subsets of these fields (Hee, Chong, and Lee 2023; Shah et al. 2024; Bui, von der Wense, and Lauscher 2025). They therefore cannot support record-level evaluation of their joint correctness. To address this gap, we introduce Meme3W, a dataset for fine-grained target identification in harmful memes. Meme3W integrates multiple public datasets under a unified schema and provides human-verified arXiv:2608.16622v1 [cs.CV] 17 Aug 2026 annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requir- ing the harmfulness label and all target-identification fields to be jointly correct. Evaluation of representative MLLMs on Meme3W reveals a clear gap between harmfulness pre- diction and jointly correct fine-grained target identification. As shown in Figure 1, harmfulness accuracy consistently ex- ceeds JRA across model scales. The best JRA among general- purpose MLLMs is 47.95%, while smaller MLLMs generally remain below 25%. To narrow this gap, we propose HarmTrace, an anchor- calibrated decoupled optimization framework for fine- grained target identification in harmful memes. HarmTrace first performs entity-aware supervised initialization and then applies conditional policy optimization. Unlike standard SFT with uniform token weighting, Entity-aware Super- vised Fine-Tuning (E-SFT) upweights target-entity tokens, strengthening supervision for the short field that connects the target category to textual and visual grounding. Rather than using a single aggregate reward, Conditional Target- identification Policy Optimization (CTPO) separately nor- malizes harmfulness and target-identification advantages. It also restricts target-identification updates to label-correct harmful responses. Within CTPO, a Virtual Positive Anchor (VPA) adds a virtual reward representing complete correct- ness during target-identification advantage normalization. Experiments on two MLLM backbones show substantial im- provements. HarmTrace raises JRA from 17.58% to 52.51% on Qwen3-VL-8B and from 6.62% to 49.09% on InternVL3- 8B. It also improves all evaluated target-identification fields on both backbones. Our contributions are as follows: • We extend harmful meme detection with fine-grained tar- get identification and construct Meme3W. It augments target-category annotations with human-verified target entities and available textual and visual grounding. We further introduce Joint Record Accuracy (JRA) for strict record-level evaluation. • We propose HarmTrace, an anchor-calibrated decou- pled optimization framework. It strengthens target- entity supervision, decouples harmfulness and target- identification credit assignment, and calibrates target- identification advantages with VPA. • We evaluate representative MLLMs on Meme3W, reveal a clear gap between harmfulness prediction and fine- grained target identification, and show that HarmTrace improves JRA across the evaluated backbones. Related Work Multimodal Hateful Meme Detection Harmful meme understanding is moving beyond binary clas- sification toward target prediction. Hateful Memes focuses on hateful/non-hateful classification (Kiela et al. 2020), while MAMI adds misogyny-type prediction (Fersini et al. 2022). Harm-C and PrideMM provide harmfulness levels or coarse target annotations (Pramanick et al. 2021; Shah et al. 2024); MemeMind and MemeIntel add chain-of-thought or explanation-oriented supervision (Gu et al. 2025; Kmainasi et al. 2025); and MemeLens unifies multilingual and mul- titask meme resources (Shahraur, Bayan et al. 2026). How- ever, these datasets do not provide unified annotations for the concrete target entity and its supporting textual and visual evidence, nor formulate these fields jointly with harmful- ness prediction. Recent work improves harmful meme de- tection through multimodal representation learning (Burbi et al. 2023) and retrieval- or knowledge-augmented adapta- tion (Mei et al. 2024; Tzelepi and Mezaris 2025; Mei et al. 2025). Other methods introduce multimodal debate or gen- erated explanations (Lin et al. 2024; Hee and Lee 2025; Mei et al. 2026), while DR-HM adopts reasoning-enhanced train- ing (Cheng et al. 2026). These methods primarily optimize binary harmfulness classification. Fine-grained target iden- tification is not included in their prediction objectives. Reinforcement Learning for MLLMs Reinforcement learning has become an important post- training paradigm for improving the reasoning and alignment of LLMs and MLLMs. PPO stabilizes policy updates through clipping and a learned value function (Schulman et al. 2017), DPO directly optimizes preference pairs (Rafailov et al. 2023), and GRPO estimates advantages within sampled groups without a critic (Shao et al. 2024). Recent variants fur- ther refine group-based optimization through dynamic sam- pling, process-level rewards, and negative-enhanced signals for all-negative groups (Yu et al. 2025; Tan et al. 2026; Nan et al. 2025). These methods are mainly studied in mathemat- ical, coding, and general reasoning tasks. In harmful meme detection, HarmTrace applies group-based optimization to fine-grained target identification. Meme3W: A Fine-Grained Target Identification Dataset for Harmful Memes Task Definition and Output Schema We extend multimodal harmful meme detection with fine- grained target identification. The task predicts harmfulness for every meme and, for harmful memes, additionally iden- tifies the target category, target entity, textual mention, and visual region. This extension enables evaluation beyond la- bel correctness by making the attacked target and its avail- able textual and visual evidence explicit. Given a dataset D = (x i ,a ⋆ i ) N i=1 , each input x i = (I i ,T i ) consists of a meme image I i and its associated text T i . The structured annotation is defined as a ⋆ i = (l ⋆ i ,c ⋆ i ,e ⋆ i ,s ⋆ i ,B ⋆ i ),(1) where l ⋆ i ∈ harmful, non-harmful denotes the harmful- ness label, c ⋆ i denotes the category of the attacked target, e ⋆ i denotes the specific target entity, s ⋆ i denotes the textual target mention, and B ⋆ i denotes a list of visual target regions represented by bounding boxes. For harmful samples, the category of the attacked target and the target entity are annotated, while the textual and visual grounding fields are populated only when the cor- responding evidence is present. For non-harmful samples, there is no attacked target to identify. Therefore, all non-box The meme is harmful as it mocks historical persecution and dehumanizes a protected group based on race/ethnicity. This falls under 'group background'. The image depicts a historical slave-sale scene, showing Black people gathered in the foreground and white traders nearby, with ships in the background. The text reads "original black friday sale." The meme equates the forced capture and enslavement of Black people with a commercial 'sale' du- ring a holiday. This is a direct mockery of the historical trauma, dehumanization, and racial persecu- tion of Black people. It trivializes slavery by framing it as a 'deal'. The white traders and ships are contextual or perpetrating elements, not the attacked target. The attacked target is Black people, who are represented as the people being sold. The target is present in both modalities. The text explicitly mentions "black", and the image explic- itly depicts Black people in the slave-sale context. Original black friday sale black people Religion Caste "Label": Harmful, "Target Category":"Group Background", "Target Entity": "Black people", "Text Mention": "black", "Visual Region":[110, 334, 439, 868] Output Analyze Input What is the target category? Where does the target appear? Who is targeted? Decode Distinguish Determine Classify Group Background Gender Health Condition 1 2 3 4 5 Original black friday sale OCR Text Figure 2: A representative Meme3W annotation, showing the analysis process and a final structured output for fine-grained target identification, answering where the target appears, who is targeted, and which target category applies. target fields are set to null, while the visual-region field is set to []. Dataset Construction We construct Meme3W by curating samples from four public datasets: PrideMM (Shah et al. 2024), MAMI (Fersini et al. 2022), Hateful Memes (Kiela et al. 2020), and Harm-C (Pra- manick et al. 2021). Their original annotations do not jointly identify the attacked target and its textual and visual ground- ing. We therefore re-annotate the selected samples under the unified schema in Eq. 1. Before annotation, we manually screened the collected samples and removed those whose visual content was unclear or could not be reliably recognized. The resulting benchmark contains 10,662 samples, including 4,418 harmful samples. Harmful samples are organized into four broad target cat- egories: group background, gender, religion and caste, and health condition. Before splitting, we removed exact dupli- cates and assigned verified perceptual near-duplicate groups to the same split. Meme3W is partitioned into training (85%), validation (5%), and test (10%) sets. Annotation Pipeline We construct the structured annotations through a three- stage MLLM-assisted human annotation pipeline. Figure 2 presents a representative Meme3W annotation, including the analysis process and the resulting fine-grained target- identification output. Stage 1: MLLM Candidate Initialization. We employ Gemini-3-Flash (Google 2025), GPT-5.2 (OpenAI 2025), and Qwen3.5-397B (Qwen Team 2026a) to independently generate one candidate annotation for each sample. All three models follow the same annotation prompt, which specifies five analysis steps and requires a final structured annotation. The outputs are normalized to the predefined format, and malformed fields are removed. The model identities are hid- den, and the three candidates are randomly ordered for each sample to reduce model-specific anchoring. Stage 2: Human Expert Annotation. Five trained grad- uate annotators with strong English proficiency and NLP backgrounds participate in this stage. They are trained on the task definition, output schema, and annotation guidelines. Each sample is independently reviewed by two annotators. The MLLM outputs serve only as editable references, and an- notators may revise any field based on the sample evidence. Before adjudication, the field-specific agreement scores are Cohen’s κ = 0.908 for target category, normalized agree- ment of 0.813 for target entity, token-F1 of 0.824 for textual mention, and IoU@0.75 of 0.818 for visual regions. For tar- get entities, semantically equivalent mentions are mapped to a shared canonical form before agreement computation. Stage 3: Human Expert Adjudication. Disagreements are independently reviewed by a third annotator and resolved through discussion among all three annotators. We further conduct a candidate-blind audit on 200 sampled cases, in which annotators receive only the raw samples and anno- tation guideline. Their annotations achieve an unweighted mean of 0.827 across the four field-specific agreement scores against the final gold annotations. Method Framework Overview To improve the joint correctness of harmfulness and target- identification predictions, we propose HarmTrace, an anchor-calibrated decoupled optimization framework. Fig- ure 3 provides an overview. HarmTrace consists of Entity- aware Supervised Fine-Tuning (E-SFT) and Conditional Target-identification Policy Optimization (CTPO). E-SFT 퐀 퐀 = 1 퐀 퐀 =0.35 Think Answer r lab = 1 y 1 r tag = 0.75 Label All Label Rewards Norm G A lab Label-correct Harmful Subset C 0.350.550.75 10.350.550.75 w/o VPA +VPA 0.550.550.55 1 0.550.550.55 w/o VPA +VPA C r lab =1 y 1 y 2 y 3 1) Unequal Incomplete 2) Tied Incomplete VPA Norm C A tag Entity-aware Token Weighting Input [woman] Think->Answer [:] Filtered Training Trajectory [...] ["Text Mention"][:] [:] Rollout Responses Label ["Think"] [Harmful] ["Answer"] ["Label"] ["Target Entity"][:] [...] [...][...] Filter = 0 A tag = 0 ≠ 0 A tag ≠ 0 A i = A lab +A tag r tag + Label Branch Target Branch Rollout Response σ ~ μ ~ Best partial A tag Signal restored Virtual Positive Anchor Entity token X λ ent X λ ent Rollout Response Think Answer r lab = 1 r tag = 0.35 y 1 Think Answer r lab = 1 r tag = 0.55 y 2 OCR Text the pimp hand strong enough for man ... ... Visual Region Label Target Category Label Target Entity Text Mention Output Teacher MLLM y 2 y 3 E-SFT Policy Required Output Fields σ ~ σ ~ ... Think Answer r lab = 0 r tag = N/A y 4 Other tokens × 1 Entity-aware SFT (E-SFT) Conditional Target-identification Policy Optimization (CTPO) [...] λ ent W t 1 Figure 3: Overview of HarmTrace. The left panel shows an input and its required output schema. E-SFT upweights target-entity tokens, while CTPO normalizes label rewards over all rollouts and target-identification rewards over the label-correct harmful subsetC. The bottom panel shows VPA for unequal and tied rewards among incomplete responses. upweights target-entity tokens to strengthen entity supervi- sion. CTPO decouples harmfulness and target-identification advantages and applies target-identification optimization only to label-correct harmful responses. Within CTPO, a Vir- tual Positive Anchor (VPA) augments target-identification advantage normalization with a virtual fully correct response. For training, HarmTrace adopts an explain-then-answer se- quence: y i =think : z i , answer : a i ,(2) where the answer field follows Eq. (1), and the think field serves as an intermediate reasoning scaffold for generating the structured answer. Entity-aware Supervised Fine-Tuning (E-SFT) Before policy optimization, we initialize the policy through cold-start supervised fine-tuning. We use Gemini-3-Flash to generate one explain-then-answer trajectory for each training meme x i . Both trajectory generation and filtering use only the training split. We retain only trajectories whose struc- tured answers are valid and consistent with the verified gold annotations: D cold =(x i ,y i )| Match(y i ,a ⋆ i ) = 1,(3) where Match(·) extracts the structured answer from y i and checks its format and consistency with a ⋆ i . The filtered tra- jectories are used for cold-start training, whereas CTPO uses the full training set. Standard SFT and E-SFT share the same trajectories and differ only in token weighting. The target entity identifies who is attacked and links the target category to textual and visual evidence, yet its short value contributes little to the sequence loss under uniform weighting. E-SFT therefore upweights thetarget_entity value tokens. For (x i ,y i )∈D cold , we optimize L E-SFT =− P T t=1 w t logp θ (y i,t | x i ,y i,<t ) P T t=1 w t , (4) where w t = λ ent , t∈I ent , 1,otherwise, (5) I ent denotes the target-entity value positions and λ ent > 1 controls their supervision weight. The resulting policy ini- tializes CTPO. Conditional Target-identification Policy Optimization (CTPO) For each input meme x, the old policy π θ old samples a group of responsesy i G i=1 . Responses violating the output format or label-conditioned schema receive zero rewards. For valid responses, r lab i ∈ 0, 1 measures harmfulness-label cor- rectness, whiler tag i ∈ [0, 1] combines applicable field scores with schema-applicability consistency. The latter penalizes missing required grounding and hallucinated non-applicable grounding, and is computed only for label-correct harmful responses, with r tag i = 1 indicating complete correctness. Detailed reward weights and matching functions are pro- vided in Appendix B.3. Combining the two rewards would mix label and target- identification credit. CTPO therefore computes their advan- tages separately. The label advantage is normalized over the full rollout group: A lab i = r lab i − μ lab σ lab + ε norm ,(6) where μ lab and σ lab are computed from r lab j G j=1 , and ε norm > 0 is a small constant for numerical stability. Target-identification learning is restricted to C =j | r lab j = 1, l ⋆ = harmful,(7) and responses outsideC receive zero target-identification ad- vantage. Normalizing target-identification rewards only within C has two limitations when all responses inC are incomplete. With unequal rewards, the best partial response is normalized only against other incomplete responses and can therefore receive a strong positive advantage. With tied rewards, the reward variance becomes zero and all target-identification advantages vanish. We therefore introduce a Virtual Positive Anchor (VPA) by adding a virtual fully correct score r max = 1 to the normalization multiset: e R tag C =r tag j | j ∈C⊎r max .(8) When C is not empty, VPA is included only in the normal- ization statistics and contributes no policy-loss term. Let m =|C|. The resulting statistics are eμ tag C = P j∈C r tag j + r max m + 1 , eσ tag C = v u u t 1 m + 1 X r∈ e R tag C r− eμ tag C 2 . (9) The target-identification advantage is A tag i =      r tag i − eμ tag C eσ tag C + ε norm , i∈C, 0,i /∈C. (10) If C is empty, we set A tag i = 0 for all responses. The final advantage combines the separately normalized terms: A i = A lab i + A tag i .(11) CTPO uses A i in the standard clipped GRPO objective, with the policy obtained after E-SFT fixed asπ ref . Letρ i,t (θ) denote the token-level probability ratio betweenπ θ andπ θ old . The optimization objective is J CTPO (θ) = E x,y i ∼π θ old 1 G G X i=1 1 |y i | |y i | X t=1 min ρ i,t (θ)A i , clip ρ i,t (θ), 1− ε clip , 1 + ε clip A i − βD KL π θ ∥π ref . (12) Experiments Experimental Setup Baselines. We evaluate HarmTrace on Meme3W against three baseline groups. 1) General-purpose MLLMs, in- cluding Gemini 3 Flash (Google 2025), GPT-5.2 (OpenAI 2025), GLM-5V-Turbo (GLM-V Team 2026), Qwen3-VL models at multiple parameter scales (Qwen Team 2025), Qwen3.5/3.6 (Qwen Team 2026a,b), InternVL3/3.5 (Chen et al. 2023; InternVL Team 2025), Gemma (Gemma Team 2024), and Kimi-K2.5 (Kimi Team 2026). Table 1 lists all models. 2) Harmful meme detection methods. We adapt EXPO-HM (Mei et al. 2026) from our standard SFT check- point and apply its RL optimization method to the fine- grained target-identification task. 3) RL-based optimization methods. Under the same E-SFT initialization, we compare HarmTrace with PPO (Schulman et al. 2017), GRPO (Shao et al. 2024), DAPO (Yu et al. 2025), and PAPO (Tan et al. 2026). Metrics. We report three groups of metrics. 1) Harmful- ness detection. We evaluate binary harmfulness classifica- tion using accuracy and F1 over all test samples. 2) Joint record correctness. Joint Record Accuracy (JRA) is com- puted over all gold-harmful samples. A record is correct only when the meme is predicted as harmful and every target- identification field satisfies its respective matching criterion. For a field that is absent in the gold record, the prediction must also indicate absence. Let H = i | l ⋆ i = harmful denote the set of gold-harmful samples andF =c,e,s,B the target-identification fields. Here, ˆ l i and l ⋆ i denote the pre- dicted and gold harmfulness labels. For each field f, ˆ f i , f ⋆ i , S f , and τ f denote its predicted value, gold value, matching score, and correctness threshold, respectively. We define JRA = 1 |H| X i∈H 1 h ˆ l i = l ⋆ i i Y f∈F 1 h S f ( ˆ f i ,f ⋆ i )≥ τ f i , (13) where (S c ,S e ,S s ,S B ) = (EM, F1 rel , F1 tok , IoU) and (τ c ,τ e ,τ s ,τ B ) = (1, 0.7, 0.7, 0.5). Semantically equivalent target-entity mentions are canonicalized before computing F1 rel . Following prior token-overlap-based relaxed evalua- tion (Heo et al. 2025), we set τ e = τ s = 0.7 to allow minor lexical differences in target entities and boundary variations in textual mentions. For visual regions, IoU is computed between the minimum enclosing rectangles of all predicted and gold boxes (Yu et al. 2016). 3) Field-level diagnosis. On gold-harmful samples, we report EM for target category, F1 rel for target entity, EM and token-F1 for textual mention, and set accuracy for visual regions at IoU thresholds of 0.5 and 0.75. Settings. All trainable methods were run for 3 epochs on 2×NVIDIA H200 GPUs using LoRA (Hu et al. 2021) with rank = 64 andα = 128. RL training uses a rollout group size of 8. Additional SFT and RL hyperparameters are provided in Appendix B.1. Main Results Table 1 shows that current MLLMs achieve higher harm- fulness accuracy than JRA, revealing a clear gap between harmfulness detection and jointly correct fine-grained target identification. Among the general-purpose MLLMs, the best JRA is 47.95%, while most smaller general-purpose MLLMs remain below 25% despite substantially higher harmfulness accuracy. Strong performance on individual fields also does not necessarily translate into a jointly correct record. The Model Label JRA Category EntityText MentionVisual Region Acc.F1EMF1 rel EMF1IoU50 IoU75 Closed-source MLLMs Gemini3-Flash74.88 73.66 47.9579.9173.29 61.87 63.01 65.9859.59 GPT-5.275.92 69.01 28.3162.5660.73 43.38 46.80 38.5833.33 GLM-5V-Turbo73.38 72.28 44.7574.8968.26 60.96 62.56 62.7957.31 Open-source Large MLLMs Kimi-K2.578.27 75.86 38.8176.2673.0655.48 56.85 59.5955.02 Gemma-4-26B-A4B78.46 74.24 28.5467.8157.08 52.97 54.11 62.7941.78 Qwen3.5-397B-A17B75.82 74.58 38.1375.1170.78 59.36 63.70 73.0659.13 Qwen3-VL-235B-A22B76.01 70.99 21.4663.4752.97 37.44 39.50 56.3947.49 Qwen3.6-27B76.95 73.80 36.9969.6361.87 53.42 55.25 63.2459.82 Qwen3.5-27B75.16 73.65 35.8474.8965.98 52.05 53.65 63.9359.36 Qwen3-VL-32B75.26 73.51 29.0076.0367.35 50.91 52.05 65.9850.68 Open-source Small MLLMs EXPO-HM (Qwen3-VL-8B) 78.93 74.58 48.8670.5565.53 57.76 59.36 62.1058.68 Qwen3.5-9B75.54 68.22 23.0653.6545.43 41.10 42.47 52.5142.92 Qwen3-VL-8B73.75 69.03 17.5860.0546.35 34.02 37.67 53.2039.95 InternVL3.5-8B68.77 67.95 14.6164.8444.06 36.76 38.36 56.8534.93 InternVL3-8B71.97 68.966.6260.0546.35 17.81 19.63 53.2036.53 Qwen3-VL-4B72.53 66.20 10.9657.9937.90 24.89 27.40 50.0031.05 Ours HarmTrace (InternVL3-8B)80.0676.0249.0971.2366.21 58.22 62.84 60.7356.85 HarmTrace (Qwen3-VL-8B) 80.15 76.77 52.5172.6068.72 60.73 64.93 66.6762.56 Table 1: Main results of different models on Meme3W. Best results are bolded, and second-best results are underlined. Setting Label JRA CategoryEntityText MentionVisual Region Acc.F1EMF1 rel EMF1IoU50IoU75 Qwen3-VL-8B73.7569.0317.5860.0546.3534.0237.6753.2039.95 +SFT77.8073.0043.1565.9861.4252.0556.4257.5353.88 +E-SFT (λ ent = 5)77.3372.9544.7567.8162.7955.9459.9258.9054.34 +E-SFT (λ ent = 10)78.6574.2645.8967.5863.7054.7958.6761.1957.31 +E-SFT (λ ent = 15)77.1472.3544.9866.6764.1655.0258.6757.5354.11 Table 2: Ablation study of SFT and E-SFT with different entity-token weights λ ent . absence of a consistent advantage for larger models further suggests that scaling alone does not resolve the difficulty of producing jointly correct target-identification outputs. Harm- Trace narrows this gap on both evaluated backbones, raising JRA from 17.58% to 52.51% on Qwen3-VL-8B and from 6.62% to 49.09% on InternVL3-8B, corresponding to abso- lute improvements of 34.93 and 42.47 points. Harmfulness accuracy and F1 also improve on both backbones, indicat- ing that the gains in target identification do not come at the expense of harmfulness detection. On Qwen3-VL-8B, Harm- Trace also improves all evaluated target-identification fields over the base model. These gains are consistent with the design of HarmTrace, which strengthens fine-grained tar- get identification while maintaining harmfulness detection through decoupled credit assignment. Overall, HarmTrace improves both field-level target identification and the joint correctness of harmfulness and target-identification outputs. Ablation and Mechanism Analysis To understand how each component contributes to Harm- Trace, we conduct targeted ablations on E-SFT, decoupled optimization, and the VPA. Entity-Aware Supervision Weight. Since the target en- tity links harmfulness detection to supporting textual and visual evidence, we examine whether stronger entity su- pervision improves the supervised initialization. Using the same explain-then-answer trajectories, Table 2 compares standard SFT with E-SFT variants using different entity- token weights. Standard SFT raises JRA from 17.58% to Init.RL Label JRA Acc.F1 w/o SFTGRPO74.5171.5218.72 +Dec.74.4171.4320.32 +VPA75.1671.8022.37 +SFTGRPO77.9974.8945.21 +Dec.77.4273.7447.26 +VPA78.6574.3549.54 +E-SFTGRPO78.7474.4949.09 +Dec.79.12 75.3350.91 +VPA80.1576.7752.51 Table 3: RL-stage ablation of HarmTrace under different starting points. w/o SFT denotes initializing RL from the base model; +Dec. adds decoupled reward optimization; and +VPA further adds VPA to +Dec. Category EM Entity F1 rel Text F1 Visual IoU50 0 25 50 75 100 Score (%) 68.72 65.98 61.70 59.82 GRPO (+E-SFT) Category EM Entity F1 rel Text F1 Visual IoU50 0 2 4 6 8 10 Gain (%) +2.74 +2.06 +3.14 +2.51 +3.88 +2.74 +3.23 +6.85 Additional gain over GRPO GRPO+Dec.+Dec.+VPA Figure 4: Target-identification results under E-SFT initializa- tion. The left panel shows GRPO scores, and the right shows gains from decoupled optimization and VPA over GRPO. 10210410610810 Step 0.0 0.1 0.2 0.3 Ratio Collapsed incomplete groups 10210410610810 Step 0.0 0.2 0.4 0.6 0.8 1.0 Target-identification advantage Positive advantage of best incomplete responses Dec. (w/o VPA)Dec. + VPA Figure 5: Effect of VPA under E-SFT initialization. The left panel shows the zero-advantage group ratio, while the right tracks positive advantage of the best incomplete responses. 43.15%, while all E-SFT variants provide further gains. Among the tested weights, λ ent = 10 provides the most balanced performance, with a JRA of 45.89% and strong re- sults on harmfulness and visual-region metrics. Although the other weights perform better on a few individual fields, their lower JRA suggests less consistent performance across the full output. Overall, E-SFT consistently improves over stan- dard SFT across the tested weights, with λ ent = 10 showing a favorable balance between JRA, harmfulness detection, and visual-region performance. RL-Stage Components. To examine whether decoupled optimization and VPA remain effective under different su- Method Label JRA Acc.F1 PPO77.6172.3346.80 GRPO78.7474.4949.09 DAPO78.0872.9447.03 PAPO78.4674.6148.40 HarmTrace80.1576.7752.51 Table 4: Performance comparison of different RL methods under the same E-SFT-initialized Qwen3-VL-8B backbone. Method Category EntityTextVisual EMF1 rel EM F1 IoU50 IoU75 PPO63.9361.42 54.57 58.09 58.68 55.02 GRPO68.7265.98 57.9961.7059.82 56.39 DAPO65.3061.19 55.02 59.10 60.27 57.08 PAPO70.55 66.2157.76 61.12 63.9360.73 HarmTrace 72.6068.72 60.73 64.93 66.67 62.56 Table 5: Performance comparison of RL methods on target- identification fields under the same E-SFT initialization. pervised initializations, Table 3 evaluates their contributions under three starting points. Under E-SFT initialization, de- coupled optimization raises JRA from 49.09% to 50.91%, and adding VPA further raises it to 52.51%. The same step- wise improvement is observed under the other two initializa- tions. Figure 4 further shows that, under E-SFT initialization, both components improve all evaluated target-identification fields, indicating that the JRA improvement is not driven by a single field. To further examine the role of VPA, Fig- ure 5 analyzes its effect on credit assignment. VPA reduces zero-advantage incomplete groups and lowers the positive advantage of the best incomplete responses. Overall, decou- pled optimization and VPA consistently improve JRA across initialization settings, and their gains under E-SFT extend across all evaluated target-identification fields. Comparison with RL Baselines To determine whether the gains arise from the optimization design of HarmTrace rather than from applying RL alone, we compare HarmTrace with PPO, GRPO, DAPO, and PAPO under the same E-SFT initialization. As shown in Tables 4 and 5, the generic RL methods show different strengths across metrics, while HarmTrace achieves the highest scores on all reported harmfulness and target-identification metrics. Among the generic RL methods, GRPO obtains the highest JRA of 49.09%, whereas HarmTrace reaches 52.51%. Over- all, the optimization design of HarmTrace yields additional gains beyond those obtained by generic RL under the same E-SFT initialization. Conclusion We study fine-grained target identification in harmful memes, where models jointly predict harmfulness, target category, target entity, textual mention, and visual region. Together, these outputs form a structured record that can support mod- eration review. We introduce Meme3W with unified, human- verified annotations and Joint Record Accuracy (JRA) for strict record-level evaluation. We further propose Harm- Trace, an anchor-calibrated decoupled optimization frame- work that combines entity-aware supervision, decoupled credit assignment, and a Virtual Positive Anchor. Experi- mental results show that HarmTrace improves JRA and all reported fine-grained target-identification fields. Future work will evaluate HarmTrace on larger MLLMs. References Bui, M. D.; von der Wense, K.; and Lauscher, A. 2025. Multi 3 Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), 9714–9731. Albuquerque, New Mexico: Associa- tion for Computational Linguistics. Burbi, G.; Baldrati, A.; Agnolucci, L.; Bertini, M.; and Del Bimbo, A. 2023. Mapping memes to words for mul- timodal hateful meme classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2832–2836. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; Li, B.; Luo, P.; Lu, T.; Qiao, Y.; and Dai, J. 2023. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual- Linguistic Tasks. Cheng, Z.; Ma, J.; Yang, X.; Wang, P.; Zhang, W.; Feng, S.; Wang, D.; Zhang, Y.; and Zhang, M. 2026. DR-HM: Distill-then-Reinforce Training with Cognition-Aware Data Synthesis for Harmful Meme Detection. In Findings of the Association for Computational Linguistics: ACL 2026, 42975–42993. San Diego, California, United States: Associ- ation for Computational Linguistics. Chiarella, M. L. 2022. Digital Markets Act (DMA) and Digital Services Act (DSA): New Rules for the EU Digital Environment. Athens Journal of Law. Fersini, E.; Gasparini, F.; Rizzi, G.; Saibene, A.; Chulvi, B.; Rosso, P.; Lees, A.; and Sorensen, J. 2022. SemEval-2022 task 5: Multimedia automatic misogyny identification. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), 533–549. Gemma Team. 2024. Gemma: Open models based on gemini research and technology. arXiv:2403.08295. GLM-V Team. 2026.GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents. arXiv:2604.26752. Google. 2025. Gemini 3 Flash: frontier intelligence built for speed. https://blog.google/products/gemini/gemini-3-flash/. Accessed: 2026-07-27. Gu, H.; Yu, Q.; Liu, Y.; Li, Z.; Hou, S.; Zhao, J.; and He, Z. 2025. MemeMind: A Large-Scale Multimodal Dataset with Chain-of-Thought Reasoning for Harmful Meme Detection. arXiv:2506.18919. Hee, M. S.; Chong, W.-H.; and Lee, R. K.-W. 2023. Decod- ing the Underlying Meaning of Multimodal Hateful Memes. arXiv:2305.17678. Hee, M. S.; and Lee, R. K.-W. 2025. Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions. arXiv:2502.11073. Heo, R.; Seo, Y.; Lee, J.; and Lee, D. 2025. Can Large Language Models be Effective Online Opinion Miners? In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, 23097–23136. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6. Hou, W.; Tu, H.; Wang, Y.; Zhang, Y.; Liu, Y.; Zhu, D.; Gao, L.; and Zhou, B. 2026. Beyond Single-View Detection: A Dual-Space Reasoning Framework for Interpretable Harm- ful Meme Understanding. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10526–10544. San Diego, Cal- ifornia, United States: Association for Computational Lin- guistics. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. InternVL Team. 2025. InternVL3.5: Advancing Open- Source Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv:2508.18265. Kiela, D.; Firooz, H.; Mohan, A.; Goswami, V.; Singh, A.; Ringshia, P.; and Testuggine, D. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. Ad- vances in neural information processing systems, 33: 2611– 2624. Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276. Kmainasi, M. B.; Hasnat, A.; Hasan, M. A.; Shahroor, A. E.; and Alam, F. 2025. MemeIntel: Explainable Detection of Propagandistic and Hateful Memes. arXiv:2502.16612. Kmainasi, M. B.; Kutlu, M.; Shahroor, A. E.; Hasnat, A.; and Alam, F. 2026. Can Thinking Models Think to Detect Hateful Memes? arXiv:2603.01225. Lin, H.; Luo, Z.; Gao, W.; Ma, J.; Wang, B.; and Yang, R. 2024. Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Mod- els. In Proceedings of the ACM Web Conference 2024. Mei, J.; Chen, J.; Lin, W.; Byrne, B.; and Tomalin, M. 2024. Improving Hateful Meme Detection through Retrieval- Guided Contrastive Learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), 5333–5347. Mei, J.; Chen, J.; Yang, G.; Lin, W.; and Byrne, B. 2025. Ro- bust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection. In Conference on Em- pirical Methods in Natural Language Processing. Mei, J.; Sun, M.; Chen, J.; Qin, P.; Li, Y.; Chen, D.; and Byrne, B. 2026. ExPO-HM: Learning to Explain-then- Detect for Hateful Meme Detection. arXiv:2510.08630. Mia, M. A.; and Fahim, M. 2025. BanHateME: Under- standing Hate in Bangla Memes thorough Detection, Cat- egorization, and Target Profiling. In Proceedings of the Second Workshop on Bangla Language Processing (BLP- 2025), 180–195. Mumbai, India: Association for Computa- tional Linguistics. Nan, G.; Chen, S.; Huang, J.; Lu, M.; Wang, D.; Xie, C.; Xiong, W.; Zeng, X.; Zhou, Q.; Li, Y.; et al. 2025. Ngrpo: Negative-enhanced group relative policy optimiza- tion. arXiv:2509.18851. OpenAI. 2025.Update to GPT-5 System Card: GPT-5.2. https://cdn.openai.com/pdf/3a4153c8-c748-4b71- 8e31-aecbde944f8d/oai_5_2_system-card.pdf. Accessed: 2026-07-27. Pramanick, S.; Sharma, S.; Dimitrov, D.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021. MOMENTA: A mul- timodal framework for detecting harmful memes and their targets. In Findings of the association for computational linguistics: EMNLP 2021, 4439–4455. Qwen Team. 2025.Qwen3-VL Technical Report. arXiv:2511.21631. Qwen Team. 2026a. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5. Accessed: 2026- 07-27. Qwen Team. 2026b. Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. https://qwen.ai/blog?id=qwen3.6- 27b. Accessed: 2026-07-27. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct Preference Optimiza- tion: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, vol- ume 36. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. Shah, S. B.; Shiwakoti, S.; Chaudhary, M.; and Wang, H. 2024. Memeclip: Leveraging clip representations for mul- timodal meme classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, 17320–17332. Shahraur, A.; Bayan, M.; et al. 2026. MemeLens: A Mul- timodal, Multilingual Benchmark for Meme Understanding. arXiv:2601.12539. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Rea- soning in Open Language Models. arXiv:2402.03300. Tan, Z.; Yu, Z.; Lin, B.; Geng, Z.; Geng, H.; Zhang, Y.; Zhang, M.; Chen, Y.; Hu, S.; Yin, Z.; Zhang, C.; and Bai, L. 2026. PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization. arXiv:2603.26535. Tzelepi, M.; and Mezaris, V. 2025. Improving Multi- modal Hateful Meme Detection Exploiting LMM-Generated Knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 202–211. Wang, X.; Su, Y.; Li, W.; Wang, X.; Li, Z.; and Liu, A. 2026. SGoT-R1: Social Graph of Thought Reasoning-Enhanced Multimodal Large Language Model for Harmful Meme De- tection. In AAAI Conference on Artificial Intelligence. Yu, J.; Jiang, Y.; Wang, Z.; Cao, Z.; and Huang, T. 2016. Unit- box: An advanced object detection network. In Proceedings of the 24th ACM international conference on Multimedia, 516–520. Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476. A Meme3W Construction and Validation This section details the construction and validation of Meme3W, including data collection and filtering, the an- notation pipeline, annotation quality assessment, and dataset release. Disclaimer. This paper contains harmful content, which has the potential to be offensive and may disturb readers. A.1 Data Curation, Unified Schema, and Annotation Prompt Data sources and screening. Meme3W is curated from four existing multimodal meme datasets, namely PrideMM (Shah et al. 2024), MAMI (Fersini et al. 2022), Hateful Memes (FHM) (Kiela et al. 2020), and Harm-C (Pra- manick et al. 2021). These datasets provide harmful and non-harmful examples, but their original annotations cover only subsets of the target-identification fields and do not jointly identify the attacked target and its textual and vi- sual grounding. FHM covers all target categories considered in Meme3W, although racial, ethnic, and religious targets dominate its harmful subset. To broaden and balance tar- get coverage, we supplement it with harmful memes from MAMI, PrideMM, and Harm-C, which focus on misogyny, LGBTQ-related harm, and COVID-19-related harm, respec- tively. After removing samples with unclear or unreliable visual content, we re-annotate the retained harmful memes under the unified Meme3W schema. Table 6 compares the annotation fields provided by the source datasets with those defined in Meme3W. FHM category distribution before augmentation. Ta- ble 7 reports the target-category distribution of the retained FHM harmful subset, which contains 3,548 memes after ap- plying the same filtering and deduplication criteria used for Meme3W. Harmfulness definition. A meme is considered harm- ful when it directs harmful behavior toward a clear target. Harmful behavior includes targeted satire, insults, mockery, degradation, stereotyping, dehumanization, threats, exclu- sion, segregation, discrimination, assertions of inferiority, comparisons to animals or objects, mockery of hate crimes or historical suffering, and gender- or sexuality-based harm. Original annotations DatasetTopic#Total#Harm.Harm.Tgt. cat.Tgt. ent.TextBox PrideMM (Shah et al. 2024)LGBTQ+5,0632,482✓– MAMI (Fersini et al. 2022)Misogyny11,0005,500✓– Hateful Memes (Kiela et al. 2020)General hate10,0005,000✓– Harm-C (Pramanick et al. 2021)COVID-193,5441,249✓– Meme3WMixed10,6624,418✓ Table 6: Annotation fields in the source datasets and Meme3W. Harm. denotes harmfulness labels. Tgt. cat. denotes source- specific target-category annotations defined under different taxonomies. Tgt. ent., Text, and Box denote target entities, textual mentions, and visual bounding boxes, respectively. A check mark indicates field availability. Attack targetCountShare (%) Group background1,46341.23 Religion and caste1,09030.72 Gender67719.08 Health condition3188.96 Total3,548100.00 Table 7: Attack-target distribution of the retained FHM harm- ful subset before augmentation with MAMI, PrideMM, and Harm-C. Counts use the same filtering criteria as the final Meme3W dataset. Ordinary humor, non-targeted profanity, general criticism, and attacks on criminals, terrorists, or criminal activities are considered non-harmful. Unified schema and taxonomy. Each meme is annotated with a harmfulness label, target category, target entity, textual mention, and visual region. Because non-harmful memes contain no attacked target, all target-identification fields other than the visual-region field are set to null, while the visual- region field is set to an empty list[]. For harmful memes, the target category and target entity are always annotated. The textual mention and visual region are included only when corresponding evidence is available. Otherwise, the textual mention is set to null, and the visual region is set to []. A prediction is considered correct for an absent field only when it likewise indicates the field’s absence. The unified target-category taxonomy contains four categories: • group background includes ethnicity, race, nationality, regional or ethnic origin, minority background, and im- migration or migrant status. • religion and caste includes religion, religious belief, re- ligious identity, followers or believers of a religion, reli- gious people, religious groups, and caste. • gender includes biological sex, gender identity, and sex- ual orientation. • health condition includes disability, disease, vaccination-related targets, infection, bodily condi- tion, medical treatment, public-health measures, and health-related policy framing. Unified output schema. Each complete model response operationalizes the explain-then-answer sequence in Eq. (2) of the main paper as a single flat JSON object. The think field contains auxiliary reasoning. The remaining five fields are label, target_category, target_entity, text_mention, and visual_region. Together, these five fields form the structured prediction a i and follow the structured annotation defined in Eq. (1) of the main paper. In Eq. (2), answer : a i denotes these five fields collectively rather than a literal answer key. Only these five fields are used for semantic matching and target-reward computation, while think is not semantically scored. Before scoring, the response parser extracts them as the input to the verifier. Field applicability and absence values follow the unified schema defined above. Shared prompting protocol. The three MLLMs used for candidate generation and the MLLMs evaluated in our ex- periments follow the same task definition, five-step analy- sis procedure, and structured output schema, with model- specific conversation templates applied when necessary. The operational prompt instructions are provided below. 1. Analyze the visual and textual content. First enumerate all important visible elements in the image, rather than considering only the most salient one. These may include every visible person or group, object, symbol, animal, scene, gesture, item of clothing, flag, religious marker, medical marker, and visible text region. Do not omit an element solely because it may not be the attacked target. Identify the element first and determine its role afterward. Also identify the relevant cues in the accompanying text. 2. Decode the meme semantics. Analyze relevant linguis- tic and contextual cues, including puns, double mean- ings, metaphors, euphemisms, coded expressions, offen- sive terms, symbolic associations, stereotypes, and his- torical, cultural, or social context. Consider whether the meme conveys insults, discrimination, dehumanization, exclusion, inferiority, or mockery of hate crimes, histori- cal persecution, violence, disability, disease, or suffering. If no such semantic cue is relevant, state this briefly. 3. Distinguish the attacked target from contextual ele- ments. Identify visually salient figures and symbols, and determine whether each is itself being attacked or instead functions as a perpetrator, criminal, terrorist, background object, metaphor, symbol, or visual vehicle used to attack another target. Identify the actual attacked target based on the complete meaning of the meme. 4. Determine the target evidence. For a harmful meme, record the exact textual mention only when the attacked target is explicitly referred to in the accompanying text, and identify the corresponding visual region or regions only when the attacked target is explicitly depicted in the image. Otherwise, assign the predefined absence value to the corresponding field. 5. Classify the meme and target category. Explicitly de- termine whether harmful behavior is directed toward a clear target and identify the behavior type that is present or absent. Relevant behavior types include targeted satire, insult, mockery, degradation, stereotyping, dehumaniza- tion, threats, exclusion, segregation, discrimination, and assertions of inferiority. If no harmful behavior is directed toward a clear target, classify the meme as non-harmful. For a harmful meme, determine the target category and target entity. Finally, return the result using the unified output schema. Output format and normalization. Candidate models re- ceive the meme image, its associated text, the task instruc- tions, and the unified output schema. Each complete re- sponse contains intermediate analysis in think together with the five prediction fields in a single flat JSON object. Thetext_mention field contains the exact text span refer- ring to the attacked target, while the visual_region field contains a list of target bounding boxes, each represented as [x 1 ,y 1 ,x 2 ,y 2 ]. All bounding-box coordinates are normal- ized to the range [0, 1000], where (x 1 ,y 1 ) and (x 2 ,y 2 ) de- note the top-left and bottom-right corners, respectively. The parser extracts the five prediction fields, and the verifier vali- dates their values. The resulting predictions are presented to annotators as editable references. Field absence follows the unified schema defined above. A.2 Human Annotation and Quality Control Annotation guidelines. Annotators follow the require- ments below: 1. Determine harmfulness and target category. Annota- tors first determine whether the meme is harmful. For each harmful meme, the attacked target is assigned to exactly one category from the taxonomy defined in Ap- pendix A.1. Samples whose target category cannot be determined reliably, or that independently attack multiple categories, are excluded rather than forced into a single category. 2. Interpret the meme from the author’s perspective. Harmfulness and target identity are determined from the intended meaning conveyed by the meme author, based only on the image–text content. Annotators consider satire, metaphor, insinuation, double meanings, stereo- types, and relevant historical, cultural, or social context rather than relying only on the literal wording. 3. Identify the actual attacked entity. The target entity must precisely describe the person or group being at- tacked at an appropriate level of granularity. Visually salient perpetrators, symbols, background figures, and contextual objects must not be annotated as targets un- less they are themselves being attacked. When the target Figure 6: Candidate-assisted human-annotation interface. Annotators inspect the source text and three anonymized MLLM candidates, revise all structured fields, and draw or adjust image-target boxes before saving the annotation. is conveyed indirectly, the entity is inferred from the com- plete meaning of the meme. 4. Copy textual mentions exactly. When the attacked target is explicitly referred to in the associated text, annotators copy the exact text span without paraphrasing, expanding, or correcting it. If no explicit textual mention is present, the field is set to null. 5. Mark complete visual target regions. When the attacked target is visually depicted, each bounding box should tightly cover the complete visible extent of the target rather than only its face or another salient part. Multiple boxes are used when multiple visual instances of the same target are present. If the target is not explicitly depicted, the visual-region field is set to []. 6. Ensure accurate multimodal grounding. Textual men- tions and visual regions must accurately correspond to the attacked target in the respective modality. Independent annotation and adjudication. Each meme is independently annotated by two annotators using editable MLLM candidates as references. The identity of the model producing each candidate is hidden, and candidate positions are randomly ordered in the annotation interface. Disagree- ments are independently reviewed by a third annotator and then resolved through discussion to produce the final gold annotation. Figure 6 shows the interfaces used for the initial independent annotation and the third-annotator review. Entity canonicalization. Before computing target-entity agreement and evaluation metrics, we apply a deterministic canonicalization procedure to normalize surface-form varia- tion. We normalize casing, punctuation, and whitespace, and map referentially equivalent spelling, plurality, abbreviation, and referring-expression variants to a shared canonical form. Broader groups and their subgroups, as well as modifiers that alter the target identity, remain distinct. The canonicaliza- tion lexicon was manually reviewed, finalized prior to model evaluation, and kept fixed across all experiments. The canon- icalized mentions are subsequently compared using F 1 rel . For JRA, the target-entity field is considered correct when F 1 rel ≥ 0.7. Entity AEntity BSame Form womenwomanYes Muslim peopleMuslimsYes Jewish peopleJewsYes LGBTQ peopleLGBTQ communityYes immigrantsrefugeesNo Asian peopleChinese peopleNo Black womenwomenNo Table 8: Examples of target-entity canonicalization. Yes in- dicates that two mentions map to the same canonical form; No indicates that they remain distinct. Final target-entity cor- rectness is evaluated using F 1 rel . Figure 7: Candidate-blind annotation interface without ac- cess to MLLM candidates. Candidate-blind audit. We conduct a candidate-blind au- dit on 200 cases sampled to cover all four target categories and different textual and visual grounding conditions. Anno- tators receive only the raw meme and annotation guidelines, without access to the MLLM candidate annotations. The blind annotations are compared with the final gold annota- tions using the same field-specific metrics as in the main annotation process. Figure 7 shows the candidate-blind an- notation interface. On this subset, re-evaluation against the blind annotations preserved the relative model ordering obtained with the final gold annotations, suggesting that candidate assistance did not alter the main comparative conclusion. A.3 Dataset Statistics and Split Integrity Dataset statistics. Meme3W contains 10,662 memes, in- cluding 4,418 harmful and 6,244 non-harmful examples. Table 10 reports the exact numbers of examples in each split. Figure 8 shows the overall attack-target distribution and target-modality composition of the harmful subset. Split integrity. We verify that no exact image duplicates occur across the training, validation, and test splits us- ing pixel-level hashing. Near-duplicate candidates identified through perceptual hashing are manually reviewed, and ver- FieldMetricBlind vs. Gold Target categoryCategory agreement0.900 Target entityNormalized F10.797 Textual mentionToken-F10.808 Visual regionIoU0.803 Unweighted mean –0.827 Table 9: Candidate-blind audit against the final gold annota- tions. 320 523 478 662 249 303 241 398 212 270 316 446 02004006008001000120014001600 Health condition Religion and caste Gender Group background text-onlyimage-onlytext+image 34% 25% 23% 18% Group backgroundReligion and caste GenderHealth condition (a) Attacked Target Distribution(b) Target Modality Composition Figure 8: Attack-target distribution and target-modality com- position among the 4,418 harmful memes in Meme3W. ified near-duplicate groups are assigned to the same split. Recurring meme templates with different textual or visual content are retained as distinct samples. Data release and responsible use. We will release the structured annotations, fixed data splits, annotation guide- lines, shared prompt, and evaluation code. Raw images or source identifiers will be distributed in accordance with the licenses of the corresponding source datasets. Meme3W con- tains offensive and discriminatory content and is intended for research on harmful-content understanding and moderation. B Experimental Details and Implementation B.1 Implementation and Evaluation Protocol Shared evaluation prompt. All evaluated MLLMs use the shared task instructions, five-step analysis procedure, and structured output schema described in Appendix A. Each re- sponse follows the explain-then-answer format. The think field contains intermediate analysis, while only the five pre- diction fields are used for evaluation. Model and decoding settings. For API-based baselines, we use the official model endpoints available at the time of evaluation. For open-source baselines, we use the publicly released instruction-tuned versions of the models reported in the main-results table. All models follow the same task defini- tion, output schema, and parsing rules. Malformed structured outputs are treated as invalid predictions. ExPO-HM follows an SFT-to-RL pipeline. Because its task-specific SFT data are not publicly available, we initialize it from our standard SFT checkpoint trained on Meme3W and subsequently apply the ExPO-HM optimization procedure. Training hyperparameters. We use ms-swift for super- vised fine-tuning and verl for reinforcement learning. All SplitHarmfulNon-harmfulTotal Training3,7595,3079,066 Validation221312533 Test4386251,063 Total4,4186,24410,662 Table 10: Numbers of harmful and non-harmful memes in each Meme3W split. ParameterValue LoRA rank / alpha64 / 128 Epochs3 Learning rate1× 10 −4 Max sequence length4096 Per-device train batch size8 Gradient accumulation steps2 Effective batch size32 Table 11: SFT hyperparameter settings. trainable methods use LoRA (Hu et al. 2021) with rank 64 and α = 128. The key hyperparameters are summarized in Tables 11 and 12. All hyperparameters and checkpoint- selection criteria for the reported systems were determined exclusively on the validation split. Test-set results were com- puted only after the corresponding configurations had been fixed, and no test result was used for hyperparameter or check- point selection. B.2 Teacher-Trajectory Generation and Filtering We use Gemini 3 Flash to generate one explain-then-answer trajectory for each of the 9,066 training samples. Each trajec- tory first provides intermediate analysis in the think field and then produces the five prediction fields in the same flat JSON object. After automatic filtering, 6,005 trajectories are retained. A trajectory is retained only when its response is parseable and all five fields satisfy the corresponding gold- matching requirements. The filtered trajectories are used only for cold-start SFT and E-SFT. Standard SFT and E-SFT use the same filtered trajectory set and differ only in token weighting. The trajec- tory filtering does not remove samples from subsequent pol- icy optimization. The subsequent policy-optimization stage uses the full training set. B.3 Reward Components and Active Target-Identification Fields Format and label reward. Following Eq. (1) of the main paper, only the five prediction fields in a i enter semantic reward computation. The response parser extracts these fields from the flat response object. The verifier then checks the resulting five-field JSON representation against the unified output schema defined in Appendix A.1 and validates the prediction-field value types and bounding-box coordinates. Invalid JSON, missing or additional prediction fields, invalid field values, and invalid bounding boxes fail the format check. ParameterValue LoRA rank / alpha64 / 128 Epochs3 Train batch size32 PPO mini-batch size32 Rollout group size8 Clip ratio0.2 KL coefficient0.02 Rollout temperature0.8 Table 12: RL hyperparameter settings. The binary label reward is r lab i = 1[format valid]1[ ˆ l i = l ⋆ i ].(14) Responses that fail the format check receive zero label re- ward. For a non-harmful prediction, target_category, target_entity, and text_mention must be null, while visual_region must be []. Target-identification rewards are computed only for format-valid, label-correct responses to gold-harmful samples. Applicable-field reward. The target-identification reward provides graded credit for partially correct predictions while preserving complete-record consistency. For a gold-harmful sample, let c, e, s, B, and q denote target category, target en- tity, textual mention, visual region, and schema consistency, respectively. The active component set is A i =c,e,q∪s : s ⋆ i ̸= null∪B : B ⋆ i ̸=∅. (15) Category, entity, and schema consistency are always active. Textual and visual components are included only when the corresponding gold evidence is present. The active scores are combined as r tag i = P f∈A i w f r i,f P f∈A i w f ,(16) where the fixed weights are 0.25 for category, 0.25 for entity, 0.20 for textual mention, 0.20 for visual region, and 0.10 for schema consistency. These weights sum to 1. When textual or visual evidence is absent, the denominator renormalizes the remaining active weights. Therefore, r tag i ∈ [0, 1] and reaches 1 when all active components receive a score of 1. The category component uses exact match. The entity component is r i,e = max(S rel ,λ F1 ent + (1− λ) EM ent ),(17) where λ = 0.7, and S rel follows the entity canonicaliza- tion and relation matching described in Appendix A.2. The textual component is r i,s = max(EM tok ,λ F1 tok + (1− λ) EM tok ). (18) The schema-consistency component equals 1 only when the predicted category is valid and the predicted presence or absence of both textual and visual evidence matches the gold annotation. Otherwise, it equals 0. ModelIoU 0.5 IoU 0.6 IoU 0.7 IoU 0.75 IoU 0.8 GRPO49.09 47.49 46.3546.35 44.52 HarmTrace 52.51 52.05 50.4650.23 50.00 ∆+3.42 +4.57 +4.11 +3.88 +5.48 Table 13: Sensitivity of JRA to the minimum-enclosing- rectangle IoU threshold with τ e = τ s = 0.7. Values are percentages on the 438 gold-harmful test memes, and ∆ de- notes HarmTrace minus GRPO. Visual-region reward and penalties. The visual compo- nent uses soft matching to provide graded credit for spatially close predictions, including imperfect box matches. For a predicted box b and gold box g, the pairwise score is S B (b,g) = max          IoU, γ IoU min 1, IoU τ soft + γ Cov Cov + γ Prec Prec + γ ctr S ctr          , (19) where (γ IoU ,γ Cov ,γ Prec ,γ ctr ) = (0.55, 0.20, 0.15, 0.10) and τ soft = 0.6. Here, Cov, Prec, and S ctr denote gold- region coverage, prediction precision, and center proximity. For multiple regions, predicted and gold boxes are greedily matched and their scores are averaged. The minimum en- closing rectangle is also considered when multiple predicted boxes jointly cover one gold region. Each additional pre- dicted box with best-match score below 0.20 incurs a 0.05 penalty. If the gold annotation contains no visual region, the visual component is inactive and any predicted region sets the schema-consistency score to 0. B.4 Threshold Sensitivity and Visual Matching Robustness Entity and text threshold sensitivity. We set τ e = τ s = 0.7 for target-entity and textual-mention matching. To as- sess sensitivity to these choices, we vary τ e and τ s over 0.5, 0.6, 0.7, 0.8, 1.0 and evaluate JRA on the complete 5×5 grid. Figure 9 shows the JRA difference between Harm- Trace and GRPO under each threshold combination. Harm- Trace remains above GRPO for all 25 settings, with ∆JRA ranging from 3.20 to 4.80 points, indicating that the model ordering is not sensitive to the selected entity or textual- mention threshold. Visual IoU threshold sensitivity. With τ e = τ s = 0.7 fixed, we vary the minimum-enclosing-rectangle IoU thresh- old. As shown in Table 13, HarmTrace remains above GRPO at every evaluated threshold. Visual-region matching robustness. We evaluate whether the JRA comparison depends on the visual-region matching rule. In addition to minimum-enclosing-rectangle IoU, we consider one-to-one bipartite matching with no unmatched boxes and geometric set IoU over the unions of predicted and gold regions. All rules use an IoU threshold of 0.5, with the remaining JRA criteria unchanged. HarmTrace 0.50.60.70.81.0 Text threshold τ s 0.5 0.6 0.7 0.8 1.0 Entity threshold τ e 4.344.333.653.883.88 4.114.113.423.653.66 4.114.113.423.653.66 3.893.883.203.423.42 4.794.804.114.334.34 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 ΔJRA (HarmTrace − GRPO) Figure 9: Sensitivity of the JRA improvement to the target- entity threshold τ e and textual-mention threshold τ s . Each cell reports the JRA difference between HarmTrace and GRPO in percentage points. Visual ruleGRPO HarmTrace∆JRA Enclosing-rectangle IoU 49.0952.51 +3.42 Bipartite set matching46.1250.00 +3.88 Geometric set IoU49.7752.28 +2.51 Table 14: JRA under alternative visual-region matching rules. HarmTrace remains above GRPO under all three rules. remains above GRPO under all three matching rules, indicating that the model ordering is not determined by the visual-region matching implementation. B.5 Additional Ablations and Robustness Analyses We report supplementary statistical and robustness analyses that are not included in the main paper. Statistical significance analysis. We quantify uncertainty in the primary JRA metric using 10,000 paired percentile- bootstrap resamples of the 438 gold-harmful test memes, applying the same resampled indices to both systems in each comparison. As shown in Table 15, HarmTrace achieves 52.51over E-SFT (45.89%) by 6.62 percentage points, with a paired 95interval of [2.51, 10.96]. Under the same E-SFT initialization, it also outperforms GRPO (49.09%) by 3.42 points, with a paired 95% confidence interval of [0.23, 6.85]. Category-wise target identification. To examine whether the overall JRA gains extend across target categories, we compare E-SFT, GRPO, and HarmTrace. Table 16 reports category-wise JRA on gold-harmful test memes using the same joint correctness criteria as the overall evaluation. HarmTrace achieves the highest overall JRA, outperform- ing E-SFT and GRPO by 6.62 and 3.42 points, respectively. Its gains are concentrated in group background and gender, ComparatorJRA HarmTrace∆JRA Paired 95% CI E-SFT45.8952.51+6.62[2.51, 10.96] GRPO49.0952.51+3.42[0.23, 6.85] Table 15: Paired comparisons of JRA on the 438 gold- harmful test memes. All values are percentages. Confidence intervals are percentile intervals from 10,000 paired boot- strap resamples of test instances. Cat.#E-SFTGRPOOurs Group/bg.14646.5850.0056.16 Religion/caste11160.3663.9662.16 Gender10344.6647.5756.31 Health cond.7825.6428.2126.92 Overall43845.8949.0952.51 Table 16: Category-wise JRA for E-SFT, GRPO, and Harm- Trace on gold-harmful test memes. where it improves over GRPO by 6.16 and 8.74 points and over E-SFT by 9.58 and 11.65 points. On religion and caste and health condition, HarmTrace remains above E-SFT but is slightly below GRPO by 1.80 and 1.29 points. These differ- ences correspond to only two and one test examples, respec- tively. Overall, the gains on group background and gender outweigh these minor decreases. C. Qualitative Case Studies Figures 10 and 11 present two representative successful cases. In both cases, HarmTrace correctly predicts the harm- fulness label and all target-identification fields. These fields include the target category, target entity, textual mention, and visual region. The resulting records therefore satisfy all JRA criteria. Figures 12 and 13 illustrate two different forms of cross- modal target misidentification. In the first case, the text ex- plicitly mentions a man. However, “dishwasher” is used as a derogatory reference to the woman shown in the image. HarmTrace follows the explicit mention of the man. It there- fore fails to associate the implicit textual attack with the woman. In the second case, the attacked target is a girl men- tioned only in the text. The image, however, depicts a Black man. HarmTrace relies on the visible person and incorrectly predicts and grounds the man as the target. These cases sug- gest that the model may over-rely on a single modality when textual and visual cues are not directly aligned. This can lead to incorrect identification of the attacked target. D. Ethical Considerations Data use and release. Meme3W is derived from four pub- licly released research datasets, namely PrideMM, MAMI, Hateful Memes, and Harm-C. We will comply with their respective licenses and redistribution requirements. When raw-image redistribution is not permitted, we will release only structured annotations, source identifiers, and process- ing code. Because the data may contain identifiable individ- uals, slurs, or stigmatizing content, the release will include a content warning and a procedure for reviewing removal requests. Annotator welfare and oversight. Five graduate-student annotators were informed in advance that the task involved potentially offensive and discriminatory content. Participa- tion was voluntary, and annotators could skip individual ex- amples or withdraw from the task. They were compensated at a rate of $10 per hour, consistent with institutional re- quirements and above the applicable local minimum wage. MLLM outputs were used only as editable annotation can- didates. Disagreements were resolved through independent human review. Intended use. Meme3W is intended to support research on harmful-content understanding, safety evaluation, model analysis, and content-moderation review. Its structured an- notations make the attacked target and the associated textual and visual evidence explicit. "Label": "Harmful", "Target Category":"Health condition", "Target Entity": "people with Down syndrome", "Text Mention": null, "Visual Region":[100, 546, 894, 1000] "Label": "Harmful", "Target Category":"Health condition", "Target Entity": "people with Down syndrome", "Text Mention": null, "Visual Region":[120, 540, 890, 999] Gold Record 1. **Analyze Visuals & Text**: The meme consists of two panels. The top panel shows lines of white powder (crack cocaine) and a plastic bag, labeled 'upper'. The bottom panel shows a man with Down syndrome smiling, labeled 'downer'. 2. **Decode Semantics**: The meme uses a pun on the word 'downer', which refers to a depressant drug, but also refers to Down syndrome. By pairing a person with Down syndrome with the term 'downer', the meme mocks and degrades the individual based on their genetic condition. This is a common harmful trope that uses people with disabilities as the punchline of a joke. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The man in the bottom panel is the victim/target. He is being mocked for his physical appearance and condition. 4. **Determine Where the Attacked Target Appears**: The attacked target (a person with Down syndrome) is explicitly shown in the image. The text 'downer' is a coded reference to the condition but does not explicitly name the group 'people with Down syndrome'. 5. **Classify the meme and target category**: The meme is harmful as it mocks a person based on a health/physical condition. It maps to 'Health condition'. The target entity is 'people with Down syndrome'. Gold vs. Model Grounding Think: Gold annotation Model prediction Input Case 1 1 2 3 Structured Prediction Figure 10: Case 1 demonstrates correct image-grounded target identification. "Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black people", "Text Mention": black, "Visual Region":[135, 70, 890, 1000] "Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black people", "Text Mention": black, "Visual Region":[131, 66, 887, 995] Gold Record Gold vs. Model Grounding Think: Gold annotation Model prediction Input Case 2 1 2 1. **Analyze Visuals & Text**: The image shows a Black woman wearing clown makeup (white face paint, red nose, red lips, black teardrop) and a top hat. The text says 'how you look when youre black and you celebrate the 4th of july'. 2. **Decode Semantics**: The clown makeup references the racist history of minstrelsy, portraying Black people as clowns. The meme mocks Black people by linking them to a degrading stereotype during a national celebration. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The woman in the image is Black and is being used to represent the target group. The clown makeup is a derogatory symbol used to mock her and the group she represents. The attacked target is Black people. 4. **Determine Where the Attacked Target Appears**: The target is explicitly named in the text ('black') and a representative of the target group is shown in the image. Thus, the location is 'both'. 5. **Classify the meme and target category**: The meme is harmful as it uses a racist stereotype to mock a group based on their race. This falls under 'group background'. The target entity is 'Black people'. The text span is 'black'. The image target is the woman. 3 Structured Prediction Figure 11: Case 2 demonstrates correct joint text–image target identification. "Label": "Harmful", "Target Category":"Gender", "Target Entity": "men", "Text Mention": null, "Visual Region":[270, 220, 992, 992] "Label": "Harmful", "Target Category":"Gender", "Target Entity": "women", "Text Mention": null, "Visual Region":[319, 43, 684, 875] Gold Record Gold vs. Model Grounding Think: Gold annotation Model prediction Input Case 3 1 2 "1. **Analyze Visuals & Text**: The image shows a man and a woman in an intimate setting. The text says “a real man... loads the dishwasher every night!!”. 2. **Decode Semantics**: The phrase “a real man” with “loads the dishwasher” implies a sexist stereotype that defines masculinity through domestic roles, degrading men by reducing their identity to household labor. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The man in the image is the subject of the text's claim. He is being labeled as a 'real man' based on a sexist standard. The meme attacks men by reducing their identity to a domestic role. The woman is a visual element in the scene but is not the target of the attack; the attack is directed at the concept of masculinity. 4. **Determine Where the Attacked Target Appears**: The target (men) is explicitly referred to in the text ('a real man') and visually represented by the man in the image. 5. **Classify the meme and target category**: The meme is harmful because it promotes a sexist stereotype and degrades men based on their gender. It fits the 'gender' category. The target entity is 'men'. The text span is 'a real man'. The image target is the man. 3 Structured Prediction Figure 12: Case 3 shows target-entity and visual-grounding errors despite correct harmfulness and target-category predictions. "Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black men", "Text Mention": null, "Visual Region":[500, 0, 990, 495], [500, 500, 990, 990] "Label": "Harmful", "Target Category":"Gender", "Target Entity": "women", "Text Mention": "girl", "Visual Region":[] Gold Record Gold vs. Model Grounding Think: Gold annotation Model prediction Input Case 4 1 2 "1. **Analyze Visuals & Text**: The meme has two panels. The top shows a Black man smiling with the text “Finally getting to date a girl with thick thighs.” The bottom shows him looking shocked with the text “But she won't crush your head in between them.” 2. **Decode Semantics**: The meme uses a harmful stereotype that hypersexualizes Black men and associates them with sexual violence. The phrase “crush your head in between them” is a coded reference to assault, reinforcing a racial stereotype and making light of such violence. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The Black man in the image is the visual vehicle for the stereotype. The meme attacks Black men as a group by associating them with this violent behavior. The 'girl' mentioned is the hypothetical victim of the implied violence. 4. **Determine Where the Attacked Target Appears**: The attacked target (Black men) is explicitly represented by the man in the image. The text does not explicitly name the racial group. 5. **Classify the meme and target category**: The meme is harmful as it promotes a violent stereotype against a group based on race. This falls under 'group background'. The target entity is 'Black men'. 3 Structured Prediction Figure 13: Case 4 shows cascading target-identification errors despite a correct harmfulness prediction.

Instant research discovery