Paper deep dive
Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models
Aditi Sarker, Nazreen Shah, Rafi Ibn Sultan, Rhongho Jang, Dongxiao Zhu, Prashant Khanduri
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/15/2026, 7:45:11 AM
Summary
The paper introduces PURGE, a framework for mitigating spurious object-background correlations in Large Vision-Language Models (LVLMs) that cause hallucinations. PURGE consists of structured dataset construction strategies (PURGE-D, PURGE-H, PURGE-M) to partition data into retain and forget sets based on object evidence versus background cues, and partition-aware unlearning methods (Gradient Ascent, KL-regularized, NPO) to suppress spurious associations while preserving object-grounded reasoning. The framework is evaluated on multiple LVLMs and benchmarks, showing consistent reduction in hallucinations.
Entities (18)
Relation Signals (16)
PURGE → consistsof → PURGE-M
confidence 95% · The framework consists of: -- (1) Structured dataset construction ... PURGE-D ... PURGE-H ... PURGE-M
PURGE → consistsof → PURGE-D
confidence 95% · The framework consists of: -- (1) Structured dataset construction ... PURGE-D ... PURGE-H ... PURGE-M
PURGE → consistsof → PURGE-H
confidence 95% · The framework consists of: -- (1) Structured dataset construction ... PURGE-D ... PURGE-H ... PURGE-M
PURGE → evaluatedon → Qwen3.5-9B
confidence 95% · We evaluate the PURGE framework across multiple LVLMs, including ... Qwen3.5-9B
PURGE → evaluatedon → Qwen3-VL-8B-Instruct
confidence 95% · We evaluate the PURGE framework across multiple LVLMs, including ... Qwen3-VL-8B-Instruct
PURGE → evaluatedon → LLaVA-1.6-7B
confidence 95% · We evaluate the PURGE framework across multiple LVLMs, including LLaVA-1.6-7B
PURGE → mitigates → Spurious Correlation
confidence 95% · PURGE ... a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs.
PURGE → evaluatedon → Waterbirds
confidence 90% · on a diverse suite of benchmarks, including ... Waterbirds
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (\underline{P}artition-aware \underline{U}nlearning for \underline{R}emoving spurious-correlation \underline{G}enerated \underline{E}rrors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of: -- (1) Structured dataset construction wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and -- (2) Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the \algo~framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs.
Tags
Links
- Source: https://arxiv.org/abs/2608.29996v1
- Canonical: https://arxiv.org/abs/2608.29996v1
Trouble viewing inline? Open PDF directly →
Full Text
97,620 characters extracted from source content.
Expand or collapse full text
Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models Aditi Sarker† Nazreen Shah‡ Rafi Ibn Sultan† Rhongho Jang Dongxiao Zhu and Prashant Khanduri†Department of Computer Science, Wayne State University, MI, USA‡Department of Electronics & Communications Engineering, IIIT Delhi, India∗Institute for AI and Data Science (AIDaS), Wayne State University, MI, USAEmail: aditi1, rafis, r.jang, dzhu, khanduri.prashant@wayne.edu, nazreens@iiitd.ac.in Abstract Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of: – ❶ Structured dataset construction wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and – ❷ Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the PURGE framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, M-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance in most evaluated settings, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs. 1 Introduction and motivation Large Vision-Language Models (LVLMs) have achieved strong performance on multimodal tasks such as visual question answering, image captioning, and visual reasoning [1, 2, 3, 4, 5, 6]. Despite these advances, LVLMs remain prone to hallucination, where models generate or predict objects that are not present in the image [7, 8, 9]. Such failures limit their reliability in safety-critical and high-stakes applications, including assistive technologies, autonomous driving, and medical image understanding, where predictions must be grounded in visual evidence rather than contextual shortcuts. One important driver of these failures is the presence of spurious correlations in training data [10, 11, 12], where objects frequently co-occur with particular backgrounds, scenes, or contextual cues. Figure 1: Spurious correlations in LVLMs. (a) Natural co-occurrence statistics showing strong object-background dependencies. (b) Debiased co-occurrence suppresses these dependencies, reducing reliance on spurious correlations. Figure 2: PURGE framework: Structured data construction followed by partition-aware unlearning Existing approaches attempt to mitigate this problem through representation learning and multimodal alignment [13, 14, 15], region-level supervision [16], causal or invariant learning [17, 18, 19], and inference-time decoding or prompting strategies [20, 21, 22]. While these methods improve model behavior, they provide limited mechanisms for explicitly identifying and removing spurious object-background associations learned by LVLMs. Training-based methods may still inherit biases from correlated data distributions, whereas inference-time methods adjust model outputs without directly modifying the underlying learned associations. As a result, LVLMs may continue to rely on background shortcuts even when the target object evidence is absent, ambiguous, or contradicted. A central challenge is that existing evaluations often measure hallucination or robustness at the aggregate level, without systematically disentangling whether a prediction is supported by object-relevant visual evidence or induced by correlated background cues. For example, in real-world datasets such as MSCOCO [23], objects frequently appear in characteristic contexts, creating strong object-background co-occurrence patterns. As shown in Fig. 1a, such correlations are highly visible in the original data distribution and can encourage models to associate objects with backgrounds rather than with object-specific evidence. In contrast, Fig. 1b illustrates a debiased distribution with reduced object-background co-occurrences, which translates into improved performance as shown later in Fig. 3. This motivates the key idea of our work: Key idea Spurious object-background correlations can be diagnosed and mitigated through structured data partitioning. By separating object-grounded examples from shortcut-prone ones, we obtain both a controlled measure of background reliance and targeted retain/forget partitions for LVLM unlearning, enabling the model to suppress spurious associations while preserving object-based reasoning. In this work, we propose PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework for constructing, benchmarking, and mitigating spurious object-background correlations in LVLMs. As illustrated in Fig. 2, PURGE has two main components: Figure 3: Object hallucination comparison measured using CHAIRi (↓ ); lower values indicate fewer hallucinations. – ❶ We construct structured retain/forget partitions using three complementary strategies that expose spurious correlations at different levels: PURGE-D, a data-level disentanglement strategy that separates object and background information via masking and inpainting; PURGE-H, a hybrid behavior-driven strategy that combines ground-truth annotations, model predictions, and counterfactual inputs to identify shortcut-driven errors; and PURGE-M, a model-informed semantic strategy that partitions examples by object presence and contextual cues to isolate background-driven predictions. – ❷ We apply partition-aware unlearning, which uses these structured partitions to suppress background-driven predictions while preserving correct object reasoning, using standard unlearning methods like Gradient Ascent (GA), KL-regularized unlearning (KL), and Negative Preference Optimization (NPO) [24]. We evaluate PURGE across multiple LVLMs, including LLaVA-1.6-7B [25], Qwen3-VL-8B-Instruct [6], and Qwen3.5-9B [26], together with CLIP as a vision-language encoder, on a diverse suite of benchmarks including CHAIR [7], POPE [27], Causal-HalBench, M-SpuBench [28], AMBER [8], MMHal [29], and Waterbirds. Across these settings, PURGE consistently reduces hallucination and spurious-correlation-driven errors while maintaining or improving overall performance, as shown in Fig. 3. Our contributions are summarized as follows: ➤ We identify spurious object-background correlations as an important contributor to hallucination in LVLMs, and introduce structured object-grounded and context-only retain/forget partitions that enable joint diagnosis and targeted unlearning of these dependencies. ➤ We introduce PURGE, a unified two-stage framework for constructing, benchmarking, and mitigating spurious-correlation-induced errors through structured retain/forget partitioning and partition-aware unlearning. – We develop three complementary data construction strategies: data-level disentanglement, hybrid behavior-driven partitioning, and model-informed semantic partitioning, which capture spurious correlations across pixel-level evidence, model behavior, and contextual semantics. – We propose partition-aware unlearning as a targeted mechanism for suppressing spurious object-background associations while preserving object-grounded reasoning, using standard unlearning objectives including GA, KL, and NPO. ➤ We demonstrate consistent improvements across diverse LVLMs and vision-language models, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, Qwen3.5-9B, and CLIP, on multiple hallucination and spurious-correlation benchmarks, reducing hallucination and improving robustness while preserving competitive overall utility. 2 Related work Spurious correlations have been widely studied in LLMs and LVLMs [17, 30], where models may rely on background cues [31], language priors [32], or dataset co-occurrence statistics rather than object-relevant evidence [12]. In multimodal settings, these shortcuts are especially challenging because visual and textual representations are often entangled, amplifying hallucinations and degrading out-of-distribution generalization. Existing mitigation methods can be broadly grouped into training-time, inference-time, and unlearning-based approaches. Training-time debiasing. Training-time methods aim to reduce spurious correlations by modifying model parameters, representations, or supervision signals. Multimodal contrastive debiasing approaches separate shortcut cues from semantic alignments by discouraging spurious image-text associations [13, 14]. Prompt-based methods steer models away from biased directions using designed or reweighted prompts, offering parameter-efficient alternatives to full finetuning [33, 34]. Region-aware methods identify and suppress spurious image regions, showing that many failures arise from background and contextual cues [16, 35]. Representation-level methods further manipulate the joint embedding space through feature masking, residual correction, or neuron-level intervention to suppress biased components [15, 36, 37]. Recent causal approaches also attribute hallucinations to co-occurrence-driven entanglement and seek to disentangle object and context representations within the model [19]. While effective, these methods typically require retraining or additional supervision and may still inherit biases from correlated training distributions. Inference-time debiasing. Inference-time methods mitigate hallucination or bias without updating model parameters. Embedding-level approaches adjust intermediate representations through projection or input-adaptive transformations to suppress spurious components while preserving utility [38, 39]. Decoding-level methods operate directly on output distributions: visual contrastive decoding promotes predictions consistent with visual evidence [21], instruction-based contrastive decoding suppresses language-driven hallucinations using perturbed prompts [20], and OPERA penalizes over-reliance on internal priors during generation [22]. Although lightweight and deployable, these methods primarily adjust outputs at inference time and do not directly remove spurious associations encoded in model representations. Unlearning-based approaches. Recent work has explored multimodal unlearning as a mechanism for removing harmful or undesirable associations while preserving overall model utility [40, 41, 42, 43]. These methods apply targeted forgetting to suppress biased, unsafe, or unwanted dependencies encoded in model representations. Subsequent approaches extend unlearning to finer-grained settings, including concept-level and domain-level removal, through attention guidance, targeted updates, or cross-modal interventions [44, 45, 46, 47]. However, most existing unlearning methods assume that the information to forget can be specified directly and separated from task-relevant knowledge. In LVLMs, object evidence and contextual cues are often tightly entangled, making it difficult to define what should be forgotten without also damaging useful object-grounded reasoning. Despite substantial progress, existing methods leave two key gaps. First, they provide limited mechanisms for constructing controlled partitions that isolate when LVLM predictions are object-grounded versus shortcut-driven. Second, they either mitigate shortcuts during training, correct outputs at inference time, or perform unlearning without explicitly modeling object-background entanglement. PURGE addresses these gaps by combining structured data partitioning with partition-aware unlearning: the partitions provide a controlled evaluation protocol for measuring shortcut reliance and define targeted retain/forget sets to remove spurious object-background associations while preserving object-based reasoning 3 PURGE: Structured data partitioning for spurious correlation analysis In this section, we present the data construction component of PURGE, which builds structured retain and forget partitions for controlled evaluation and mitigation of spurious object-background correlations. We model an input image as v=(o,z)v=(o,z), where o denotes object-specific visual evidence and z denotes contextual or background features. Given a target concept c, an LVLM with parameters θ receives a multimodal input x=(v,q)x=(v,q), where q is a text query, and predicts Pθ(c∣x)P_θ(c x). Our goal is to construct structured retain and forget sets, denoted by DrD_r and DfD_f, that separate object-grounded evidence from shortcut-inducing contextual cues. We instantiate this idea using MSCOCO [23] and Waterbirds [12], and we design three complementary partitioning strategies, as discussed next. Figure 4: Three complementary partitioning strategies in PURGE: (a) PURGE-D performs data-level disentanglement via masking and inpainting, (b) PURGE-H applies hybrid behavior-driven partitioning based on prediction correctness, and (c) PURGE-M performs model-informed semantic disentanglement via cue scoring. 3.1 PURGE-D (Data-level disentanglement) As illustrated in Fig. 4a, PURGE-D separates object and background information directly at the image level using masking (or inpainting). Given an image v and an object mask M(v)M(v), we construct two complementary views: vr=v⊙M(v)v_r=v M(v), vf=v⊙(1−M(v)),v_f=v (1-M(v)), where vrv_r preserves object-centric information and vfv_f removes the object while retaining contextual cues. We then partition the dataset into retain and forget sets as: PURGE-D Retain set Dr(1)D_r^(1): object-focused samples constructed using vrv_r, encouraging predictions based on true object evidence. Forget set Df(1)D_f^(1): context-only samples constructed using vfv_f, augmented with additional object-absent hard negatives that expose spurious contextual dependencies. We further consider an inpainting-based variant [48], where removed regions are filled to produce more natural counterfactual images. This variant is reported in App. C. While PURGE-D provides a direct pixel-level intervention, it may not fully capture higher-level semantic or model-specific shortcut behavior. 3.2 PURGE-H (Hybrid behavior-driven partitioning) As illustrated in Fig. 4b, PURGE-H combines object-level supervision with model behavior under original and counterfactual inputs. For each image v, we use ground-truth annotations to determine whether the target object is present, denoted by o∈0,1o∈\0,1\. We then query the LVLM with a binary question, such as “Is there a c in the image?”, and obtain the prediction y y. To probe reliance on contextual cues, we also construct a counterfactual image v′v by removing the target object and query the model again. This yields four cases: (o=1,y^=1)⏟Correct detections,(o=1,y^=0)⏟Missed detections,(o=0,y^=1)⏟Hallucinations,(o=0,y^=0)⏟Correct rejections, (o=1, y=1)_Correct detections,~ (o=1, y=0)_Missed detections,~ (o=0, y=1)_Hallucinations,~ (o=0, y=0)_Correct rejections, using this, we partition the dataset as: PURGE-H Retain set Dr(2)D_r^(2): Reliable prediction cases where the model behaves correctly, including correct detections (1,1)(1,1) and correct rejections (0,0)(0,0). Forget set Df(2)D_f^(2): Failure cases including missed detections (1,0)(1,0) and hallucinations (0,1)(0,1), where predictions deviate from the true object presence. In particular, hallucination cases (0,1)(0,1) correspond to scenarios in which the model predicts objects purely from contextual cues despite the absence of objects, directly capturing spurious correlations. Compared to purely data-driven approaches, PURGE-H explicitly captures model-specific biases and failure modes, making it effective for reducing hallucinations. However, imperfect counterfactual generation may introduce noise and partial model dependence. 3.3 PURGE-M (Model-informed semantic partitioning) As illustrated in Fig. 4c, PURGE-M separates object evidence from contextual cues at the semantic level utilizing the learned model. Let C denote the set of images where the target object is present, and let CcC^c denote its complement. Similarly, let S denote the set of images containing a candidate spurious background cue, and let ScS^c denote the corresponding low-cue set. Following [31], we first use a language model to propose candidate co-occurring cues and then use an open-vocabulary detector to compute per-image cue scores s(v)s(v), which are thresholded to define S and ScS^c. This produces four semantic partitions: C∩S,C∩Sc,Cc∩S,Cc∩ScC∩ S,~C∩ S^c,~C^c∩ S,~C^c∩ S^c which are then utilized to partition the dataset as: PURGE-M Retain set Dr(3)=C∩ScD_r^(3)=C∩ S^c: Object-centric samples where the target object is present without strong contextual bias, providing reliable supervision for object-grounded learning. Forget set Df(3)=Cc∩SD_f^(3)=C^c∩ S: Hallucination-prone samples where contextual cues are present despite object absence, directly capturing spurious object-background dependencies. The remaining partitions are excluded from training: C∩SC∩ S contains entangled object-context evidence, while Cc∩ScC^c∩ S^c contains trivial negatives with limited disentanglement signal. Both are retained for evaluation. Using Dr(3)D_r^(3) and Df(3)D_f^(3), PURGE-M encourages the model to suppress context-driven predictions P(c∣z)P(c z) while preserving object-based predictions P(c∣o)P(c o). Further details of the PURGE-M cue-discovery, scoring, selection, and partition-construction pipeline are provided in Appendix E. 4 PURGE: Partition-aware unlearning for spurious correlation mitigation We now present the partition-aware unlearning component of PURGE, which uses the structured partitions from Sec. 3 to suppress context-driven predictions while preserving object-grounded reasoning. Ideally, predictions should be grounded in object evidence, so that Pθ(c∣x)P_θ(c x) is primarily determined by Pθ(c∣o)P_θ(c o) rather than by contextual cues. However, under spurious object-background correlations, an LVLM may instead rely on the context z, leading to high Pθ(c∣z)P_θ(c z) even when the target object is absent. This produces hallucinations in cases where o=0o=0 but background cues associated with the concept are present. Let θ′θ denote the parameters after unlearning. Our goal is to suppress context-driven predictions while preserving object-grounded reasoning: Pθ′(c∣z)→0,whilePθ′(c∣o)≈Pθ(c∣o).P_θ (c z)→ 0, P_θ (c o)≈ P_θ(c o). (1) We formulate this goal as a targeted unlearning problem over the structured partitions constructed in Sec. 3. For each partitioning strategy k∈1,2,3k∈\1,2,3\, we utilize the retain set Dr(k)D_r^(k) containing samples that support object-grounded reasoning, and the forget set Df(k)D_f^(k) containing samples that expose spurious or failure-inducing context dependence. The general objective is ℒ(k)=ℒretain(k)+λℒforget(k),L^(k)=L_retain^(k)+ _forget^(k), (2) where ℒretain(k)L_retain^(k) preserves useful object-based predictions on Dr(k)D_r^(k), ℒforget(k)L_forget^(k) suppresses spurious predictions on Df(k)D_f^(k), and λ>0λ>0 controls the strength of forgetting. Training setup. We train a LVLM on multimodal inputs x=(v,q)x=(v,q), where v denotes the input image and q is a text query (e.g., “Is there a c in the image?”). The model predicts Pθ(y∣v,q),y∈yes,noP_θ(y v,q), y∈\yes,no\. We adopt parameter-efficient fine-tuning θ=ϕ,θLoRAθ=\φ, _LoRA\ where ϕφ denotes the multimodal projector parameters and θLoRA _LoRA represents the LoRA parameters of the language model. The vision encoder and backbone of LVLM remain frozen. Let (vr,qr,yr)∼Dr(k)(v_r,q_r,y_r) D_r^(k) and (vf,qf,yf)∼Df(k)(v_f,q_f,y_f) D_f^(k). We instantiate Eq. (2) with three unlearning objectives, Gradient Ascent (GA), KL-regularized unlearning (KL), and Negative Preference Optimization (NPO) [24], each offering a different trade-off between forgetting strength and stability: – ❶ GA preserves correct predictions on the retain set while reducing the likelihood of spurious labels on the forget set: ℒGA(k) _GA^(k) ≔−Dr(k)logPθ(yr∣vr,qr)+λDf(k)logPθ(yf∣vf,qf). -\,E_D_r^(k) P_θ(y_r v_r,q_r)+\,λ\,E_D_f^(k) P_θ(y_f v_f,q_f). (3) Since the objective is minimized, the second term decreases the probability assigned to the forget labels. GA is simple and aggressive, but it does not explicitly constrain the updated model to remain close to the original model, which can cause instability or unintended degradation. – ❷ KL-regularized unlearning suppresses forget-set predictions while keeping the updated model close to a reference model Pθ0P_ _0 on retain samples to improve stability of GA: ℒKL(k) _KL^(k) =λDf(k)logPθ(yf∣vf,qf)+βDr(k)KL(Pθ(⋅|vr,qr)∥Pθ0(⋅|vr,qr)). =λ\,E_D_f^(k) P_θ(y_f v_f,q_f)+β\,E_D_r^(k)KL\! (P_θ(·|v_r,q_r)\,\|\,P_ _0(·|v_r,q_r) ). (4) Here, θ0 _0 denotes the original model before unlearning. The KL term reduces drift on retain samples, although strong regularization can weaken forgetting when retain and forget objectives conflict. – ❸ NPO penalizes spurious predictions relative to the reference model rather than only in absolute likelihood. This discourages the updated model from assigning high confidence to forget-set labels while maintaining retain-set performance: ℒNPO(k) _NPO^(k) =−Dr(k)logPθ(yr∣vr,qr)+2λβDf(k)log(1+(Pθ(yf|vf,qf)Pθ0(yf|vf,qf))β). =-\,E_D_r^(k) P_θ(y_r v_r,q_r)+ 2λβ\,E_D_f^(k) \! (1+ ( P_θ(y_f|v_f,q_f)P_ _0(y_f|v_f,q_f) )^\!β ). (5) The hyperparameter β controls the sharpness of the relative penalty. Compared with GA, NPO provides a more targeted forgetting signal by focusing on forget labels whose confidence remains high relative to the reference model. Together, these objectives span different forgetting behaviors, from aggressive forgetting with GA to more conservative regularization with KL and relative preference-based suppression with NPO. The three partitioning strategies in Sec. 3 instantiate the retain and forget sets as follows: Partition-aware unlearning: PURGE PURGE-D: Retain samples are object-only views and forget samples are background-only views, with yr=yf=yesy_r=y_f=yes, so the forget term suppresses spurious background confidence. Since Df(1)D_f^(1) contains only background-only images without a natural negative label, an auxiliary set Df−D_f^- provides explicit no supervision via an additional term ℒneg(1)=−Df−logPθ(no∣v,q)L_neg^(1)=-E_D_f^- P_θ(no v,q), giving ℒ(1)=ℒretain(1)+λℒforget(1)+γℒneg(1)L^(1)=L_retain^(1)+λ\,L_forget^(1)+ _neg^(1) with γ>0γ>0. PURGE-H: yry_r corresponds to the model’s correct predictions and yfy_f corresponds to the model’s spurious predictions covering both object-present and object-absent cases. Note that the (0,0)(0,0) cases in Dr(2)D_r^(2) already provide negative supervision, as object-absent samples with no labels are naturally included in the retain set. PURGE-M: yr=yf=yesy_r=y_f=yes, with retain drawn from C∩ScC∩ S^c and forget from Cc∩SC^c∩ S. Since all forget samples are object-absent, the forget objective itself directly penalizes false-positive predictions on object-absent images, making explicit negative supervision redundant. 5 Experimental evaluation We evaluate PURGE along three dimensions: – ❶ Whether partition-aware unlearning reduces hallucination and spurious-correlation-driven errors, – ❷ Whether the improvements generalize across LVLM architectures and benchmarks, and – ❸ How different partitioning strategies and trainable components affect performance. Models. We evaluate the PURGE framework on LLaVA-1.6-7B [25], Qwen3-VL-8B-Instruct [6], and Qwen3.5-9B [26]. LLaVA follows a modular design with a CLIP-based [49] vision encoder and language model, while Qwen3-VL adopts a unified multimodal architecture with strong vision-language alignment. For Qwen3.5-9B, vision-language fusion is tightly integrated within the multimodal architecture; therefore, we do not train a separate multimodal projector. Adaptation is performed through LoRA-based updates while the integrated multimodal fusion architecture is preserved. Baselines. We compare against the corresponding vanilla models without unlearning. To isolate the role of partition-aware data construction, we evaluate standard unlearning objectives, including GA, KL, and NPO, applied to the constructed partitions. In addition, we compare with prior multimodal debiasing methods such as RAVL [16], and with CLIP-based debiasing on Waterbirds [13]. Benchmarks and metrics. We evaluate on a diverse set of benchmarks covering object hallucination, object existence, causal consistency, and robustness to spurious correlations, including CHAIR [7], AMBER [8], POPE [27], MMHal [29], Causal-HalBench [50], M-SpuBench [28], and Waterbirds [12]. Details of each benchmark are provided in App. B. In all tables, green shading indicates the best result and red shading indicates the second-best result. Table 1: CLIP on Waterbirds. Method Avg. Acc. ↑ WGA ↑ CLIP baselines Pre-trained CLIP 90.8 44.9 Fine-tuned CLIP 81.3 77.1 Existing methods ERM 93.5 54.4 Group DRO [12] 83.3 73.7 MSC [13] 84.7 77.5 MSC[variant] [13] 83.2 77.5 Partition-aware unlearning PURGE-D-NPO 83.5 75.1 PURGE-H-NPO 86.3 79.3 PURGE-M-NPO 90.9 79.6 Spurious-correlation robustness on Waterbirds with CLIP. Table 1 evaluates CLIP-RN50 on Waterbirds [12]. Starting from Waterbirds, we construct PURGE-specific retain and forget partitions based on object and background cues and use these partitions for partition-aware unlearning. PURGE-M-NPO achieves the highest WGA among the evaluated methods (79.679.6) while maintaining a competitive average accuracy of 90.990.9, demonstrating reduced reliance on spurious background correlations while preserving overall performance. Comparison methods. Pre-trained CLIP denotes the original CLIP-RN50 evaluated without task-specific updates, whereas Fine-tuned CLIP denotes CLIP-RN50 initialized from the pre-trained checkpoint and adapted using our Waterbirds-derived training data. For our CLIP experiments, we initialize from CLIP-RN50, freeze the text encoder, and update the visual encoder and classification head. ERM denotes empirical-risk-minimization training from the same CLIP-RN50 initialization, while Group DRO follows Sagawa et al. [12]. MSC and MSC[variant] denote the two multimodal spurious-correlation mitigation configurations from Yang et al. [13]. Robustness across models. Table 2 evaluates PURGE-M on the held-out test split of our Table 2: Robustness across models. Model Method Overall ↑ WGA ↑ Prior work RAVL [16] 70.2 40.8 LLaVA-1.6-7B Baseline 54.4 32.0 PURGE-M-KL 72.1 46.4 PURGE-M-NPO 77.5 60.1 Qwen3-VL 8B-Instruct Baseline 62.7 46.3 PURGE-M-KL 76.8 54.2 PURGE-M-NPO 82.9 62.8 Qwen3.5-9B Baseline 59.8 47.5 PURGE-M-KL 74.1 51.4 PURGE-M-NPO 77.2 59.2 constructed object-context dataset, using average classification accuracy (Overall) and worst-group accuracy (WGA) over the four object-context groups. Across all evaluated LVLMs, PURGE-M consistently improves both overall accuracy and WGA over the corresponding baselines, with especially large gains in WGA. These results suggest that the proposed partitions expose shortcut-driven behavior in a way that enables effective and architecture-agnostic mitigation. Spurious-correlation robustness and causal consistency. Table 3 further evaluates causal consistency on Causal-HalBench. Here, QcQ_c and QaQ_a denote accuracies for contextual-object and absent-object questions on the original image, while Qc′Q_c and Qa′Q_a denote the corresponding accuracies on the counterfactual image. CAC is computed as Qc−Qc′Q_c-Q_c , and AAC as Qa′−QaQ_a -Q_a. CHR denotes the counterfactual-object hallucination rate. Lower CAC, AAC, and CHR indicate better causal robustness. Across models, PURGE improves QcQ_c and QaQ_a, reduces CAC and AAC, and lowers CHR. These results show that PURGE not only improves accuracy but also makes predictions more stable under counterfactual changes. Detailed group-wise M-SpuBench results are reported in App. O. Table 3: Evaluation on Causal-HalBench. Method QcQ_c QaQ_a Qc′Q_c Qa′Q_a Acc. ↑ CAC ↓ AAC ↓ CHR ↓ LLaVA-1.6-7B Baseline 80.2 81.1 75.7 82.9 85.6 4.5 1.8 12.9 PURGE-D-NPO 85.1 85.8 81.3 86.9 91.1 3.8 1.1 10.5 PURGE-H-NPO 88.3 87.2 85.4 88.1 93.1 2.9 0.9 6.7 PURGE-M-NPO 91.3 93.3 89.6 93.6 94.6 1.7 0.3 3.2 Qwen3-VL-8B-Instruct Baseline 81.8 83.2 78.3 84.4 84.7 3.5 1.2 7.4 PURGE-D-NPO 86.9 87.7 84.0 88.6 90.0 2.9 0.9 3.2 PURGE-H-NPO 91.1 93.3 89.7 93.5 93.6 1.4 0.2 1.9 PURGE-M-NPO 91.7 94.3 91.0 94.2 94.1 0.7 0.1 1.2 Qwen3.5-9B Baseline 78.2 76.6 73.1 78.3 82.3 5.1 1.7 11.6 PURGE-D-NPO 80.1 78.5 75.3 79.7 84.8 4.8 1.2 9.7 PURGE-H-NPO 82.3 81.4 78.3 82.4 86.4 4.0 1.0 6.8 PURGE-M-NPO 87.5 84.8 84.3 85.4 90.4 3.2 0.6 4.3 Category-wise MMHal analysis. Fig. 5 reports category-wise results on MMHal [29]. Hallucination and shortcut reliance are more pronounced in categories such as counting and adversarial examples, where models must distinguish object evidence from misleading context. Across models, PURGE variants consistently improve MMHal scores across categories, with PURGE-M achieving the largest overall gains. Since higher MMHal scores indicate better performance, these improvements demonstrate stronger object grounding and robustness across diverse reasoning tasks. We further evaluate the inpainting-based PURGE-D variant on MMHal, showing consistent category-wise improvements; detailed results are provided in Appendix C. (a) (b) (c) Figure 5: Category-wise MMHal [29] evaluation across different models. Object-existence evaluation on POPE. Table 4 reports results on the POPE Random split, which evaluates object-presence prediction. The vanilla baselines achieve lower accuracy and F1, indicating that models often fail to reliably determine whether an object is present. All PURGE variants improve performance, suggesting stronger object grounding and reduced hallucination. PURGE-M-NPO is generally the strongest variant across models, while PURGE-D-NPO and PURGE-H-NPO achieve the best performance in several model-split combinations. Table 4: POPE [27] results on the Random split. Method Obj. LLaVA-1.6-7B Qwen3-VL-8B-Instruct Qwen3.5-9B Acc Rec F1 Yes Acc Rec F1 Yes Acc Rec F1 Yes Baseline – 67.2 54.5 62.4 37.3 79.1 80.7 79.4 51.6 68.1 55.2 63.4 37.1 PURGE-D GA 87.5 88.0 87.6 50.5 90.1 89.2 90.0 49.1 88.2 86.1 88.0 47.9 KL 92.5 89.5 92.3 47.0 97.2 94.4 97.1 47.2 90.1 90.1 90.1 50.0 NPO 93.7 89.9 93.5 46.2 91.1 92.8 91.2 51.7 92.3 91.6 92.2 49.3 PURGE-H GA 89.5 88.5 89.4 49.0 88.7 89.1 88.7 50.4 86.2 85.4 86.1 49.2 KL 90.3 89.6 90.2 49.3 90.5 91.1 90.6 50.6 89.4 88.3 89.3 48.9 NPO 97.3 95.1 97.2 47.8 98.1 96.2 98.1 48.1 92.1 91.7 92.1 49.6 PURGE-M GA 91.6 92.1 91.6 50.5 92.3 93.0 92.4 50.7 88.1 88.3 88.1 50.2 KL 92.2 92.4 92.2 50.2 98.2 96.4 98.2 48.2 89.8 91.2 89.9 51.4 NPO 98.1 96.2 98.1 48.1 99.6 99.2 99.6 49.6 91.6 92.1 91.6 50.5 Ablation study. We compare LoRA-only updates, applied to the language model attention and MLP layers, with joint LoRA+projector training. As shown in Tables 6 and 6, LoRA-only updates already improve M-SpuBench accuracy and reduce hallucination on CHAIR. Updating the multimodal projector in addition to LoRA further improves performance, especially on CHAIR, indicating that adapting the vision-language interface is important for removing object-background shortcuts. For Qwen3.5-9B, we keep the vision encoder and multimodal fusion module fixed because they are tightly integrated, and apply LoRA only to the language model. Additional CHAIR results and the inpainting-based PURGE-D variant are reported in App. N and App. C. Table 5: Accuracy (%) on M-SpuBench. Method Setting LLaVA-1.6-7B Qwen3-VL- 8B-Instruct Qwen3.5-9B Baseline – 59.7 73.8 60.1 PURGE-D-NPO LLM only 61.4 74.7 61.2 LLM+Proj. 67.8 78.3 – PURGE-H-NPO LLM only 68.1 78.8 61.8 LLM+Proj. 70.3 82.4 – PURGE-M-NPO LLM only 72.3 83.7 64.3 LLM+Proj. 78.3 88.7 – Table 6: Ablation on CHAIR (lower is better). Method Setting LLaVA-1.6-7B Qwen3-VL- 8B-Instruct Qwen3.5-9B CHAIRs CHAIRi CHAIRs CHAIRi CHAIRs CHAIRi Baseline – 41.2 23.6 32.4 19.3 36.2 21.2 PURGE-D-NPO LLM only 38.6 18.3 24.6 17.8 24.8 18.9 LLM+Proj. 20.3 12.7 17.4 11.9 – – PURGE-H-NPO LLM only 30.1 17.2 22.6 16.8 23.3 17.0 LLM+Proj. 18.3 9.3 11.6 7.4 – – PURGE-M-NPO LLM only 29.1 15.5 20.2 13.7 20.7 14.2 LLM+Proj. 11.3 6.6 9.6 4.3 – – Qualitative analysis. Fig. 6 provides qualitative examples illustrating how spurious correlations contribute to hallucination. Baseline models often rely on misleading contextual cues, leading to incorrect object predictions or counting errors. On MMHal-Bench, this appears as undercounting or overcounting object instances; on M-SpuBench, predictions are influenced by spurious visual features; and on AMBER, models produce ungrounded answers in binary attribute-grounding queries. In contrast, PURGE produces responses that are more consistent with object-relevant evidence, improving counting, feature attribution, and object-presence judgments. Quantitative AMBER results are provided in App. P. Figure 6: Evaluation across hallucination and spurious correlation benchmarks. Left: MMHal-Bench, where baseline models undercount objects. Top-right: M-SpuBench example showing reliance on spurious features. Bottom-right: AMBER example illustrating reduced hallucination in a binary attribute-grounding query. 6 Conclusion and limitations We studied spurious object-background correlations as an important source of hallucination and shortcut reliance in vision-language models. We introduced PURGE, a framework that constructs structured retain/forget partitions and uses them for both controlled evaluation and partition-aware unlearning. Across multiple LVLMs, vision-language encoders, and benchmarks, PURGE reduces hallucination and improves robustness to spurious correlations while preserving competitive overall utility. Limitations. PURGE depends on the quality of masks, counterfactual images, cue proposals, and detector outputs. Its effectiveness may decrease when object and background cues are highly entangled, and it adds data processing and fine-tuning costs. Future work will explore more scalable partitioning and stronger counterfactual generation. References [1] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §1. [2] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: §1. [3] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny (2023) Minigpt-4: enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592. Cited by: §1. [4] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi (2023) Instructblip: towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, p. 49250–49267. Cited by: §1. [5] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al. (2024) Cogvlm: visual expert for pretrained language models. Advances in Neural Information Processing Systems 37, p. 121475–121499. Cited by: §1. [6] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. CoRR abs/2511.21631. External Links: Link Cited by: §1, §1, §5. [7] A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko (2018) Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 4035–4045. Cited by: Table 17, Table 17, Table 9, Table 10, Table 10, §1, §1, §5. [8] J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, et al. (2023) Amber: an llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397. Cited by: Table 9, §1, §1, §5. [9] A. F. Biten, L. Gómez, and D. Karatzas (2022) Let there be a clock on the beach: reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 1381–1390. Cited by: §1. [10] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann (2020) Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), p. 665–673. Cited by: §1. [11] A. Torralba and A. A. Efros (2011) Unbiased look at dataset bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1521–1528. Cited by: §1. [12] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang (2019) Distributionally robust neural networks for group shifts: on the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731. Cited by: Appendix A, §1, §2, §3, Table 1, §5, §5, §5. [13] Y. Yang, B. Nushi, H. Palangi, and B. Mirzasoleiman (2023) Mitigating spurious correlations in multi-modal models during fine-tuning. In International Conference on Machine Learning, p. 39365–39379. Cited by: §1, §2, Table 1, Table 1, §5, §5. [14] S. Lu, J. Chai, and X. Wang (2025) Mitigating spurious correlations in zero-shot multimodal models. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §2. [15] H. Jung, T. Jang, and X. Wang (2024) A unified debiasing approach for vision-language models across modalities and tasks. Advances in Neural Information Processing Systems 37, p. 21034–21058. Cited by: §1, §2. [16] M. Varma, J. Delbrouck, Z. Chen, A. Chaudhari, and C. Langlotz (2024) Ravl: discovering and mitigating spurious correlations in fine-tuned vision-language models. Advances in Neural Information Processing Systems 37, p. 82235–82264. Cited by: §1, §2, Table 2, §5. [17] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz (2019) Invariant risk minimization. arXiv preprint arXiv:1907.02893. Cited by: §1, §2. [18] Z. Song, S. Zhao, X. Zhang, J. Li, C. Zheng, and W. Qiang (2024) Learning invariant causal mechanism from vision-language models. arXiv preprint arXiv:2405.15289. Cited by: §1. [19] X. Hu, C. Wang, R. An, C. Shao, X. Ye, S. Zhou, and L. Li (2025) Causal-llava: causal disentanglement for mitigating hallucination in multimodal large language models. arXiv preprint arXiv:2505.19474. Cited by: §1, §2. [20] X. Wang, J. Pan, L. Ding, and C. Biemann (2024) Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In Findings of the Association for Computational Linguistics: ACL 2024, p. 15840–15853. Cited by: §1, §2. [21] S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2024) Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13872–13882. Cited by: §1, §2. [22] Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu (2024) Opera: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13418–13427. Cited by: §1, §2. [23] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, p. 740–755. Cited by: §1, §3. [24] R. Zhang, L. Lin, Y. Bai, and S. Mei (2024) Negative preference optimization: from catastrophic collapse to effective unlearning. In First Conference on Language Modeling, External Links: Link Cited by: §1, §4. [25] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: §1, §5. [26] Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §1, §5. [27] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 292–305. Cited by: Table 16, Table 16, Table 9, §1, Table 4, Table 4, §5. [28] W. Ye, B. Liu, G. Zheng, D. Wang, X. Cao, Y. Ma, B. Lai, J. M. Rehg, and A. Zhang (2026) Mm-spubench: towards better understanding of spurious biases in multimodal llms. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 2854–2865. Cited by: Appendix O, Table 9, §1, §5. [29] Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, et al. (2024) Aligning large multimodal models with factually augmented rlhf. In Findings of the Association for Computational Linguistics: ACL 2024, p. 13088–13110. Cited by: Table 9, Figure 7, Figure 7, §1, Figure 5, Figure 5, §5. [30] C. Udomcharoenchaikit, W. Ponwitayarat, P. Payoungkhamdee, K. Masuk, W. Buaphet, E. Chuangsuwanich, and S. Nutanong (2022) Mitigating spurious correlation in natural language understanding with counterfactual inference. In Proceedings of the 2022 conference on empirical methods in natural language processing, p. 11308–11321. Cited by: §2. [31] P. Hosseini, S. Nawathe, M. Moayeri, S. Balasubramanian, and S. Feizi (2026) SpurLens: automatic detection of spurious cues in multimodal LLMs. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: Appendix E, §2, §3.3. [32] A. Vo, K. Nguyen, M. R. Taesiri, V. T. Dang, A. T. Nguyen, and D. Kim (2026) Vision language models are biased. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. [33] C. Chuang, V. Jampani, Y. Li, A. Torralba, and S. Jegelka (2023) Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070. Cited by: §2. [34] C. Jiang, Y. Yang, R. Hu, and J. Sang (2025) Debiased prompt tuning for vision-language models without annotations. In 2025 International Joint Conference on Neural Networks (IJCNN), p. 1–8. Cited by: §2. [35] Y. Yang, C. P. Lee, S. Feng, D. Zhao, B. Wen, A. Z. Liu, Y. Tsvetkov, and B. Howe (2025) Escaping the spuriverse: can large vision-language models generalize beyond seen spurious correlations?. arXiv preprint arXiv:2506.18322. Cited by: §2. [36] A. Seth, M. Hemani, and C. Agarwal (2023) Dear: debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6820–6829. Cited by: §2. [37] N. M. An, Y. Jang, Y. Hirota, R. Hachiuma, I. Augenstein, and H. Shim (2026) Interpretable debiasing of vision-language models for social fairness. arXiv preprint arXiv:2602.24014. Cited by: §2. [38] W. Gerych, H. Zhang, K. Hamidieh, E. Pan, M. Sharma, T. Hartvigsen, and M. Ghassemi (2024) Bendvlm: test-time debiasing of vision-language embeddings. Advances in Neural Information Processing Systems 37, p. 62480–62502. Cited by: §2. [39] T. Lian, G. Hu, Y. Ren, D. Kollias, and O. Celiktutan (2026) A closed-form solution for debiasing vision-language models with utility guarantees across modalities and tasks. arXiv preprint arXiv:2603.12998. Cited by: §2. [40] Y. Chen, Y. Yao, Y. Zhang, B. Shen, G. Liu, and S. Liu (2026) Safety mirage: how spurious correlations undermine VLM safety fine-tuning and can be mitigated by machine unlearning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. [41] J. Chen, Z. Deng, K. Zheng, Y. Yan, S. Liu, P. Wu, P. Jiang, J. Liu, and X. Hu (2025) Safeeraser: enhancing safety in multimodal large language models through multimodal machine unlearning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 14194–14224. Cited by: §2. [42] J. Li, Q. Wei, C. Zhang, G. Qi, M. Du, Y. Chen, S. Bi, and F. Liu (2024) Single image unlearning: efficient machine unlearning in multimodal large language models. Advances in Neural Information Processing Systems 37, p. 35414–35453. Cited by: §2. [43] T. Chakraborty, E. Shayegani, Z. Cai, N. B. Abu-Ghazaleh, M. S. Asif, Y. Dong, A. Roy-Chowdhury, and C. Song (2024) Can textual unlearning solve cross-modality safety alignment?. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 9830–9844. Cited by: §2. [44] K. Kawamura, Y. Goto, R. Yanagi, H. Kataoka, and G. Irie (2025) Approximate domain unlearning for vision-language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. [45] J. Geng and Q. Li (2025) SAUCE: selective concept unlearning in vision-language models with sparse autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3023–3033. Cited by: §2. [46] K. Bhaila, A. Komanduri, M. Van, and X. Wu (2025) Cross-modal attention guided unlearning in vision-language models. arXiv preprint arXiv:2510.07567. Cited by: §2. [47] J. Huo, Y. Yan, X. Zheng, Y. Lyu, X. Zou, Z. Wei, and X. Hu (2025) Mmunlearner: reformulating multimodal machine unlearning in the era of multimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 7190–7206. Cited by: §2. [48] Y. Wang, X. Tao, X. Qi, X. Shen, and J. Jia (2018) Image inpainting via generative multi-column convolutional neural networks. Advances in neural information processing systems 31. Cited by: §C.2, §3.1. [49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §5. [50] Z. Xu, Z. Wang, J. Wu, J. Lu, and X. Wang (2026) Causal-halbench: uncovering lvlms object hallucinations through causal intervention. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 34169–34177. Cited by: Table 9, §5. [51] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: Appendix A. [52] I. Loshchilov and F. Hutter (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: Appendix A. [53] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky (2022) Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 2149–2159. Cited by: §C.1. [54] M. Minderer, A. Gritsenko, and N. Houlsby (2023) Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems 36, p. 72983–73007. Cited by: Appendix E. [55] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. (2024) Grounding dino: marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, p. 38–55. Cited by: Appendix H. Appendix Appendix A Training details Implementation overview. We provide detailed implementation and training configurations for the proposed PURGE framework to ensure full reproducibility. All experiments are conducted using parameter-efficient fine-tuning, where only a subset of model parameters is updated while the majority of the pretrained backbone remains frozen. Specifically, we update the language model using LoRA adapters [51] and, where applicable, jointly optimize the multimodal projector while keeping the visual encoder and remaining backbone parameters frozen. This design enables targeted modification of both reasoning and visual grounding components while preserving the general capabilities of the pretrained model. The language model captures high-level semantic reasoning and is responsible for suppressing context-driven priors, while the projector ensures correct alignment between visual features and language representations. Dataset construction, splits, and size. We construct structured retain and forget datasets using the three partitioning strategies described in Sec. 3. For all COCO-based LVLM experiments, the candidate samples used to construct the retain and forget sets for parameter updates are drawn exclusively from the MSCOCO train2017 split. The COCO val2017 split is held out from partition-aware training and is used only for evaluation. Thus, no image from COCO val2017 enters the retain or forget sets used for model parameter updates. The partition-construction procedure on COCO train2017 produces a pool of approximately 40,000 samples spanning all 80 COCO object categories. From this pool, we select approximately 8,000 retain samples and 8,000 forget samples across the eligible object categories for each method. Thus, the approximately 40,000 samples constitute the candidate partition pool, whereas approximately 16,000 samples in total are used for parameter updates for each method. Waterbirds and other non-COCO benchmarks follow their respective dataset-specific class definitions and experimental protocols. For PURGE-D, the retain and forget sets are constructed from the object-presence and object-removed counterfactual partitions described in Sec. 3. For PURGE-H, retain samples correspond to consistent predictions (1,1)(1,1) and (0,0)(0,0), whereas forget samples correspond to failure cases (1,0)(1,0) and (0,1)(0,1). For PURGE-M, we partition samples according to object and contextual-cue presence, Table 7: Training configuration. Parameter Value Number of classes 80 Retain set size ∼ 8,000 Forget set size ∼ 8,000 Total samples ∼ 16,000 Epochs 5 Precision bf16 Optimizer AdamW LR (LoRA) 2×10−52× 10^-5 LR (projector) 1×10−51× 10^-5 LoRA rank 8 λ 1.0 β 0.1 γ 1.0 using C∩ScC∩ S^c as the retain set and Cc∩SC^c∩ S as the forget set. The same sampling procedure and overall training budget are used across PURGE variants to ensure a consistent comparison. Training configuration. We train all models for five epochs using bf16 precision with gradient checkpointing for memory efficiency. Optimization is performed using the AdamW optimizer [52]. To ensure stable optimization across components, we use separate learning rates: 2×10−52× 10^-5 for LoRA parameters and 1×10−51× 10^-5 for the multimodal projector. We set the LoRA rank to 8, providing a good balance between parameter efficiency and adaptation capability. For multimodal models (e.g., LLaVA and Qwen3-VL), we consider two training settings: – ❶ updating only LoRA adapters in the language model, and – ❷ jointly training LoRA together with the multimodal projector (MLP) to improve vision-language alignment. In contrast, Qwen3.5-9B does not expose a separate multimodal projector in our training setup and is therefore adapted using LoRA only. The unlearning objective is controlled by hyperparameters λ, β, and γ. The parameter λ determines the strength of unlearning by scaling the contribution of the forget set, and we explore different values of λ to balance forgetting and retention. The parameter β regularizes the KL and NPO objectives to prevent excessive deviation from the pretrained model. For PURGE-D, we additionally use γ to enforce negative supervision on object-absent samples. Unless otherwise specified, hyperparameters are kept consistent across methods and models to ensure fair comparison. Backbone-specific LoRA configuration. For all PURGE variants, we freeze the visual encoder and perform parameter-efficient adaptation of the language model using LoRA. LoRA adapters are inserted into the attention and MLP projection modules of the language model, with rank r=8r=8, scaling factor α=16α=16, and dropout 0.050.05. For LLaVA-1.6-7B and Qwen3-VL-8B-Instruct, the multimodal projector/fusion component is additionally optimized, whereas Qwen3.5-9B is adapted using LoRA only because it does not use a separate multimodal projector. All remaining backbone parameters, including the visual encoder and non-LoRA language model parameters, remain frozen. The same trainable-component configuration is used across PURGE -D, PURGE -H, and PURGE -M and for the GA, KL, and NPO objectives. Table 8: Backbone-specific training configuration for PURGE. The visual encoder is frozen, while the multimodal projector/fusion component, when present, and LLM LoRA adapters are optimized. Backbone Vision Projector/Fusion LLM LoRA r α Dropout LLaVA-1.6-7B Frozen Trainable Trainable 8 16 0.05 Qwen3-VL-8B-Instruct Frozen Trainable Trainable 8 16 0.05 Qwen3.5-9B Frozen N/A Trainable 8 16 0.05 CLIP Waterbirds unlearning protocol. For Waterbirds [12], we formulate the task as binary classification between landbird and waterbird. Importantly, the bird class serves as the prediction target, whereas the background attribute is used only to characterize spurious-correlation groups and to construct the corresponding PURGE partitions. Thus, each sample retains its original Waterbirds class label y∈landbird,waterbirdy∈\landbird,waterbird\ throughout training. The two bird classes crossed with the two background types (land and water) define four bird-class-background groups. These group assignments characterize whether the bird class and background are aligned or conflicting and are used for partition construction and worst-group evaluation; they are not treated as classification labels. This distinction allows PURGE to suppress background-dependent predictions while preserving bird-class information. For CLIP-RN50, given an image x, the visual encoder and classification head produce class logits θ(x)z_θ(x), with pθ(y∣x)=softmax(θ(x))y,y∈landbird,waterbird.p_θ(y x)=softmax\! (z_θ(x) )_y, y∈\landbird,waterbird\. We instantiate the GA, KL, and NPO objectives defined in the main text using these class probabilities and the retain/forget partitions constructed by the corresponding PURGE strategy. In particular, the unlearning objective acts on the bird-class probability associated with each sample rather than predicting the four bird-class-background groups themselves. For KL and NPO, θ0 _0 denotes a fixed copy of the pre-unlearning CLIP classifier and serves as the reference model. All CLIP variants use the same CLIP-RN50 initialization. The text encoder is frozen, while the visual encoder and classification head are optimized using AdamW. We evaluate on the standard Waterbirds test split using average classification accuracy and worst-group accuracy (WGA), where WGA is the minimum classification accuracy across the four bird-class-background groups. Remark. We observe that the choice of sampling size per class provides a trade-off between computational efficiency and performance. Using 100 samples per class provides sufficient coverage while maintaining feasible training time. In preliminary experiments with smaller subsets (e.g., 30–60 samples per class), we observe consistent trends, indicating that the proposed method is robust to dataset size variations. Appendix B Evaluation benchmarks We evaluate our approach across multiple benchmarks that capture different aspects of hallucination and robustness. CHAIR and AMBER measure object hallucination and grounding quality in generative settings, while MMHal-Bench evaluates fine-grained multimodal reasoning. POPE focuses on object existence prediction to assess visual grounding. Causal-HalBench further examines robustness under counterfactual interventions, and M-SpuBench evaluates sensitivity to spurious object-context correlations. Together, these benchmarks provide a comprehensive evaluation of hallucination reduction, grounding accuracy, and robustness to spurious dependencies. Table 9: Evaluation benchmarks and metrics. We report benchmark-specific metrics to assess hallucination, grounding, and robustness to spurious correlations. Benchmark Task Metrics CHAIR [7] Object hallucination (generation) CHAIRs ↓ , CHAIRi ↓ AMBER [8] Generative hallucination and grounding CHAIR ↓ , Cover ↑ , Hal ↓ , Cog ↓ MMHal-Bench [29] Fine-grained multimodal reasoning Score ↑ , Hallucination rate ↓ POPE [27] Object existence prediction Accuracy ↑ , Recall ↑ , F1 ↑ , Yes Ratio Causal-HalBench [50] Counterfactual robustness Accuracy ↑ , CHR ↓ , ΔQ Q ↓ M-SpuBench [28] Spurious correlation robustness Accuracy ↑ Appendix C PURGE-D (Inpainting-based partitioning) PURGE-D-IP. We construct the dataset using an inpainting-based strategy that explicitly disentangles object evidence from contextual cues by generating counterfactual samples (see Fig. 4)(a). Given an input image x containing a target object o and background context z, we first obtain a binary object mask M(x)M(x) that localizes the object region. Using this mask, we generate two complementary views: xr=x⊙M(x),xf=Inp(x,M(x)),x_r=x M(x), x_f=Inp (x,M(x) ), where xrx_r denotes the object-centric view obtained by removing the background and retaining only the object region, while xfx_f denotes the context-only view obtained by removing the object and reconstructing the background via an inpainting operator Inp(⋅)Inp(·). Here, ⊙ represents element-wise masking, and Inp(⋅)Inp(·) fills the masked object region using the surrounding visual context. Based on these views, we construct the retain set Dr=(xr,yes)D_r=\(x_r,yes)\ and the forget set Df=(xf,yes)D_f=\(x_f,yes)\. For the background-only forget samples, the unlearning objective suppresses the model’s confidence in the yes response. A separate auxiliary set Df−D_f^- of object-absent samples provides explicit no supervision. The same GA, KL, and NPO objectives described in Sec. 4 are used for PURGE-D-IP. This construction enforces an explicit separation between object and context, enabling the model to learn object-grounded representations while suppressing spurious object-context correlations. As illustrated in Fig. 4a, the retain and forget partitions provide complementary signals that guide the model toward robust and reliable visual grounding. C.1 PURGE-D object removal and inpainting Mask extraction and preprocessing. For PURGE-D, we construct object-removed counterfactual images using the instance-level annotations provided by COCO. For each target object, the corresponding polygon/RLE segmentation annotation is decoded into a binary object mask. We consider only non-crowd instances and filter difficult removal cases based on object size, proximity to the image boundary, overlap with other objects, scene complexity, and mask solidity. Specifically, the target-object mask must occupy 1.5%1.5\%-20%20\% of the image, have a maximum bounding-box IoU of 0.080.08 with other annotated objects, and have a minimum mask solidity of 0.400.40. We additionally restrict images to at most 1010 annotated objects and exclude target instances lying within 3.5%3.5\% of the image boundary. Before inpainting, we dilate the binary object mask using a 13×1313× 13 elliptical kernel to remove residual pixels around the object boundary. The mask is subsequently smoothed using a 5×55× 5 Gaussian filter and re-binarized. Rather than inpainting the entire image, we extract a square local crop centered on the target-object bounding box with 40%40\% contextual padding. The crop size is constrained to 224224-640640 pixels and rounded to a multiple of eight. Object removal. We perform object removal using LaMa [53] through the SimpleLama implementation. The processed segmentation mask and corresponding local image crop are supplied to the inpainter, which reconstructs the masked target-object region from its surrounding visual context. If the generated crop differs from the original crop dimensions, it is resized using Lanczos interpolation before being pasted back at the original image location. Pixels outside the local crop remain unchanged. Quality control and failure handling. To suppress visually disruptive edits, we measure the mean absolute RGB change in a local ring surrounding the inpainted mask boundary and discard an edited sample when this boundary-change score exceeds 4545. Samples with missing or empty segmentation masks, corrupted or unreadable source images, and samples for which LaMa inpainting fails are skipped and are not included in the constructed dataset. C.2 Experimental result CHAIR evaluation of PURGE-D-IP. Table 10 reports CHAIR results on COCO val. All variants of PURGE-D significantly reduce hallucination compared to the baseline across all models, demonstrating the effectiveness of inpainting-based [48] data construction. We observe a consistent improvement from GA to KL and further to NPO, indicating that stronger unlearning objectives lead to better suppression of spurious object-context dependencies. Notably, PURGE-D-NPO achieves the lowest CHAIR scores, confirming improved object grounding and reduced hallucinated predictions. Table 10: CHAIR [7] evaluation on COCO val. Method LLaVA-1.6-7B Qwen3-VL-8B-Instruct Qwen3.5-9B CHAIRs ↓ CHAIRi ↓ CHAIRs ↓ CHAIRi ↓ CHAIRs ↓ CHAIRi ↓ Baseline 41.2 23.6 32.4 19.3 36.2 21.2 PURGE-D-IP-GA 25.9 14.1 19.4 13.7 20.5 14.7 PURGE-D-IP-KL 21.9 13.3 19.3 12.1 19.7 13.1 PURGE-D-IP-NPO 18.3 12.1 16.2 11.4 16.6 11.9 MMHal evaluation on PURGE-D-IP. Category-wise evaluation is performed on the MMHal benchmark for LLaVA-1.6 and Qwen3-VL-8B-Instruct using the inpainting-based PURGE-D dataset, as shown in Fig. 7. PURGE-D variants consistently improve MMHal scores across all categories compared with the baseline, indicating improved visual grounding and reduced hallucination. The largest improvements are observed for PURGE-D-NPO, particularly in challenging categories such as adversarial reasoning and counting, where predictions are often influenced by contextual bias. The spider plots exhibit consistent expansion toward higher scores across categories, reflecting stronger multimodal reasoning and reduced reliance on context-only signals. These results demonstrate that constructing PURGE-D using inpainting effectively reduces spurious correlations and improves robustness across diverse reasoning tasks. (a) LLaVA-1.6 (b) Qwen3-VL Figure 7: MMHal [29] category-wise performance under PURGE-D-IP. Appendix D Effect of forget-set ratio under NPO To study the impact of unlearning strength, we vary the proportion of forget-set samples during training while keeping the retain set fixed. Specifically, we consider forget-set ratios of 25%, 50%, 75%, and 100%, and evaluate performance on M-SpuBench. Figure 8: Effect of forget-set ratio under NPO. Increasing the proportion of forget samples improves robustness to spurious correlations across both LLaVA-1.6-7B (left) and Qwen3-VL-8B-Instruct (right). PURGE-M consistently achieves the best performance, followed by PURGE-H and PURGE-D. As shown in Fig. 8, increasing the forget-set ratio consistently improves performance across all methods, indicating that stronger exposure to spurious examples helps suppress shortcut-based reasoning. However, the magnitude of improvement varies across partition strategies. PURGE-D shows limited gains and saturates early, suggesting that pixel-level masking provides relatively weak supervision for disentangling object and context. In contrast, PURGE-H yields steady improvements, benefiting from model-driven identification of failure cases. PURGE-M consistently achieves the highest performance across all ratios, demonstrating that semantically grounded partitions targeting spurious regimes provide the most effective supervision for unlearning. We further observe that performance gains begin to plateau at higher ratios (75%-100%), indicating diminishing returns once sufficient spurious exposure is achieved. This trend is more pronounced for LLaVA-1.6-7B, while Qwen3-VL-8B-Instruct continues to benefit slightly, suggesting that stronger models can better leverage structured unlearning signals. Overall, these results highlight that both the quantity and quality of forget supervision are crucial, with partition-aware design playing a central role in effective spurious correlation removal. Appendix E PURGE-M cue discovery and partition construction Contextual-cue discovery. We adopt the automatic spurious-cue discovery pipeline of [31] and follow its released implementation and default cue-discovery settings. For each target object, GPT-4 is used to propose 12 candidate contextual cues that commonly co-occur with the object. The candidate cues are normalized and deduplicated, while unsuitable or object-overlapping cues are removed following the SpurLens filtering procedure. We retain the prompting, filtering, detector configuration, confidence-scoring, and cue-ranking procedure provided by the released SpurLens implementation, without introducing an additional manually designed cue-selection stage. We use OWLv2 [54] as the open-vocabulary detector to quantify the evidence for each candidate cue in an image. For an image v and candidate cue c, we define the cue score as s(v,c)=max(0∪q:(b,c,q)∈(v)),s(v,c)= (\0\∪\q:(b,c,q) (v)\ ), where (v)O(v) denotes the set of OWLv2 detections, b is a detected bounding box, and q is its confidence score. Thus, s(v,c)s(v,c) corresponds to the maximum OWLv2 confidence associated with cue c in image v. Images are ranked according to this score for each candidate cue. Following the [31] procedure, we compare model behavior between images exhibiting stronger and weaker evidence for each candidate cue and select the cue showing the strongest spurious association with the target object. No additional manual cue selection is performed. Object-cue partition construction. After identifying the contextual cue, we determine target-object and cue presence for each image and construct four object-cue partitions, as visualized in Fig 9. Partition membership is determined jointly by whether the target object and the selected contextual cue are present in the image. For partition-aware unlearning, PURGE-M uses images in which the target object is present but the contextual cue is absent as the retain set. These samples preserve target-object evidence while removing the identified contextual shortcut. Conversely, images in which the target object is absent but the contextual cue is present form the forget set, as they isolate contextual evidence that can spuriously support the target-object prediction. The remaining two object-cue combinations are reserved for evaluation and are not used as retain or forget samples during unlearning. Appendix F Group-wise analysis and visualization of PURGE-M We perform a group-wise diagnostic analysis of PURGE-M using four structured object-context partitions that explicitly separate target-object presence from contextual-cue presence. The first subscript denotes whether the target object is present, while the second denotes whether the associated contextual cue is present. Specifically, AyyA_y contains samples in which both the target object and contextual cue are present; AynA_yn contains samples in which the target object is present, but the contextual cue is absent; AnyA_ny contains samples in which the target object is absent but the contextual cue is present; and AnnA_n contains samples in which neither the target object nor the contextual cue is present. For each partition, we report object-existence prediction accuracy. Accordingly, the correct prediction is yes for AyyA_y and AynA_yn, where the target object is present, and no for AnyA_ny and AnnA_n, where the target object is absent. Among these partitions, AnyA_ny provides a direct diagnostic of contextual shortcut reliance because the target object is absent while its associated contextual cue remains present. An incorrect yes prediction in this setting indicates that the model predicts the target object from contextual evidence alone. Table 11 reports this group-wise diagnostic analysis on 6,332 samples from the PURGE-M training split. This analysis is distinct from the held-out test-set evaluation reported in Table 2 in the main text. Consequently, the Overall and WGA values in Table 2 are computed on the separate held-out test split and are not derived from the group-wise accuracies reported here. The baseline models obtain relatively low accuracy on AnyA_ny, with scores of 32%32\%, 46%46\%, and 47%47\% for LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, respectively. This behavior indicates sensitivity to contextual cues when object-specific evidence is absent. PURGE-M improves performance across all four partitions. In particular, PURGE-M-NPO improves AnyA_ny accuracy from 32%32\% to 59%59\% for LLaVA-1.6-7B, from 46%46\% to 63%63\% for Qwen3-VL-8B-Instruct, and from 47%47\% to 58%58\% for Qwen3.5-9B. Improvements are also observed on AynA_yn, where the model must recognize the target object without support from the associated contextual cue: accuracy increases from 52%52\% to 77%77\%, from 65%65\% to 80%80\%, and from 62%62\% to 75%75\% for the three backbones, respectively. These complementary improvements indicate both reduced context-driven false-positive predictions and stronger object-grounded recognition. Across all evaluated backbones and partitions, PURGE-M-NPO also outperforms the corresponding PURGE-M-KL variant. The consistent gains across AyyA_y, AynA_yn, AnyA_ny, and AnnA_n suggest that the NPO objective provides a stronger forgetting signal while maintaining performance across different object-context configurations. Table 11: Group-wise diagnostic analysis of PURGE-M on the four object-context partitions (AyyA_y, AynA_yn, AnyA_ny, and AnnA_n) using 6,332 samples from the PURGE-M training split. This analysis is separate from the held-out test-set evaluation reported in Table 2. Model Method ↑A_y ↑A_yn ↑A_ny ↑A_n LLaVA-1.6-7B Baseline 60 52 32 45 PURGE-M-KL 78 70 46 64 PURGE-M-NPO 83 77 59 70 Qwen3-VL-8B-Instruct Baseline 70 65 46 58 PURGE-M-KL 82 75 54 68 PURGE-M-NPO 87 80 63 75 Qwen3.5-9B Baseline 61 62 47 58 PURGE-M-KL 64 68 51 62 PURGE-M-NPO 72 75 58 68 The four PURGE-M training partitions analyzed here contain 1,469 AyyA_y samples, 1,578 AynA_yn samples, 1,455 AnyA_ny samples, and 1,830 AnnA_n samples, for a total of 6,332 object-context samples. These samples are used here to characterize model behavior across the four structured object-context groups. The held-out test samples used to compute the Overall and WGA results in Table 2 are separate from these training-partition samples. Figure 9 illustrates the partitioning strategy using boat as the target object and water as its associated contextual cue. In this example, AyyA_y contains images in which both the boat and water are present, while AynA_yn contains images in which the boat is present without water. Conversely, AnyA_ny contains images in which water is present, but the boat is absent, and AnnA_n contains images in which neither the boat nor water is present. For partition-aware unlearning, PURGE-M uses AynA_yn as the retain set because these samples contain target-object evidence without the associated contextual cue. In contrast, AnyA_ny forms the forget set because it contains the contextual cue in the absence of the target object and therefore isolates the shortcut signal to be suppressed. The AyyA_y and AnnA_n partitions are excluded from unlearning. They are retained for group-wise diagnostic analysis to characterize model behavior across the remaining object-context configurations. This construction separates object-specific evidence from contextual evidence and enables PURGE-M to suppress context-driven predictions while preserving object-grounded recognition. Figure 9: Additional visualization of the PURGE-M partitioning strategy. Example with boat as the target object and water as the contextual cue. YN (object present, cue absent) forms the retain set, while NY (object absent, cue present) forms the forget set. Y and N are not used for unlearning and are retained for evaluation. The construction explicitly separates object evidence from contextual evidence, encouraging preservation of object-grounded recognition while suppressing spurious object-context associations. Appendix G Qualitative failure analysis To better characterize the remaining limitations of PURGE, we examine representative cases in which both the baseline and PURGE-trained models produce incorrect object-existence predictions. As shown in Fig. 10, these failures primarily occur when object-specific visual evidence is weak. The target object may be very small, distant, partially occluded, or difficult to distinguish from the surrounding scene. For example, the spoon is small and partially occluded by surrounding objects, the bird occupies only a very small image region, the snowboard is partially occluded and blends with the snowy background, and the baseball glove is small and far from the camera. These examples clarify the scope of PURGE. The proposed framework is designed to mitigate context-driven shortcut reliance rather than all sources of LVLM error. When object-specific visual evidence itself is insufficient or ambiguous, suppressing contextual shortcuts alone may not recover the correct prediction. Figure 10: Qualitative failure analysis of PURGE. Representative failure cases involving small, distant, partially occluded, or visually ambiguous target objects. Both the baseline and PURGE-trained models fail in these visually challenging cases, illustrating that PURGE mitigates context-driven shortcut reliance but does not directly address failures caused by insufficient object-level visual evidence. Appendix H Quality verification of generated partitions To independently assess the quality of the automatically constructed PURGE partitions, we randomly sample 5,000 examples from each larger candidate partition produced during PURGE dataset construction and verify whether they satisfy their expected object-context configurations using Grounding DINO [55], an open-vocabulary detector that was not used during dataset construction. For PURGE-D and PURGE-H, we verify whether the target object is present in the retain/original samples and absent from the forget/edited samples. For PURGE-M, we additionally verify the presence or absence of the associated contextual cue. We report 95% Wilson confidence intervals to quantify the statistical uncertainty of the estimated verification rates. Partition-verification sampling. The verification experiment in Table 12 uses random samples drawn from the larger PURGE candidate partitions constructed before the final training and evaluation subsets are formed. This experiment is used solely as an independent quality check of the partition-construction procedure. Table 12: Independent verification of PURGE partition quality. For each PURGE partition, 5,000 examples are randomly sampled from the larger constructed candidate partition and independently verified using Grounding DINO. Dataset Partition Expected Configuration Success Rate (%) ↑ 95% Wilson CI PURGE-D Retain Object present 98.82 [98.48, 99.08] Forget Object absent 93.04 [92.30, 93.71] PURGE-H Original Object present 97.06 [96.55, 97.49] Edited Object absent 80.08 [78.95, 81.16] PURGE-M Y Object present, cue present 96.24 [95.68, 96.73] YN Object present, cue absent 90.70 [89.86, 91.47] NY Object absent, cue present 92.20 [91.42, 92.91] N Object absent, cue absent 92.60 [91.84, 93.29] Overall, the verification results indicate that the automatically constructed partitions largely satisfy their intended semantic configurations. The comparatively lower verification rate for the PURGE-H edited partition reflects the additional difficulty introduced by generative inpainting and motivates treating partition quality as an explicit limitation of the framework. Appendix I Prediction transition and regression analysis While PURGE is designed to suppress spurious object-background correlations, contextual information can occasionally provide useful inductive bias for object recognition. We therefore quantify whether PURGE introduces new errors by comparing the baseline LLaVA-1.6-7B model and the corresponding PURGE-M-NPO model on exactly the same evaluation samples. Each sample is categorized as Correct → Correct (C→C→ C), Incorrect → Correct (I→CI→ C), Correct → Incorrect (C→IC→ I), or Incorrect → Incorrect (I→I→ I). Table 13: Sample-wise prediction transition analysis between the baseline LLaVA-1.6-7B and PURGE-M-NPO. Regression denotes a correct baseline prediction that becomes incorrect after PURGE. POPE and AMBER transition results are computed on fixed 4,000-sample evaluation subsets. Benchmark # Samples Base Subset Acc. PURGE Subset Acc. C→C→ C I→CI→ C C→IC→ I I→I→ I Regression (%) ↓ M-SpuBench 2400 59.7 78.3 1415 464 18 503 0.75 POPE (4K subset) 4000 67.2 98.1 2676 1248 12 64 0.30 AMBER (4K subset) 4000 48.6 81.8 1930 1342 14 714 0.35 For POPE and AMBER, the transition analysis is performed on fixed 4,000-sample subsets evaluated identically before and after PURGE. Accordingly, the Base Subset Acc. and PURGE Subset Acc. columns denote accuracy computed only over these transition-analysis subsets. In particular, the AMBER baseline subset accuracy of 48.6% is obtained directly from the transition counts (1930+14)/4000=48.6%(1930+14)/4000=48.6\% and should not be confused with the numerically identical baseline hallucination rate reported in the full-set AMBER evaluation. Tables 19. report results on the full AMBER evaluation set and include discriminative F1; for LLaVA-1.6-7B, the corresponding baseline and PURGE-M-NPO F1 scores are 87.0 and 93.2, respectively. Therefore, the subset-level accuracies reported in Table 13 are not directly comparable with the full-set AMBER metrics in Table 19. Correct → Incorrect regressions remain below 1% across all three benchmarks, whereas Incorrect → Correct transitions are substantially more frequent. Thus, PURGE corrects substantially more errors than it introduces. We further observe that the remaining regressions primarily involve visually challenging examples in which the target object is small, distant, heavily occluded, or ambiguous. Representative examples are provided in Appendix G. Appendix J Robustness across random seeds To assess whether the observed improvements are robust to stochastic training variation, we repeat the PURGE training procedure using three independent random seeds (42, 123, and 3407). For computational efficiency, the seed-robustness experiments are conducted using a fixed reduced training subset consisting of approximately 4,000 retain and 4,000 forget samples for each run. Across seeds, we keep the sampled-data budget, training configuration, and hyperparameters fixed and vary only the random seed. We conduct this analysis for both LLaVA-1.6-7B and Qwen3-VL-8B-Instruct on M-SpuBench, POPE, and AMBER. The resulting standard deviations remain small, indicating low sensitivity to stochastic training variation. Because this experiment uses a reduced training subset, the absolute scores in Table 14 are intended for assessing seed stability rather than direct numerical comparison with the main experimental results. Table 14: Robustness across random seeds. We report accuracy (%) across three independent runs with seeds 42, 123, and 3407. Mean and standard deviation are computed over the three runs. Higher is better (↑ ). Model Benchmark Method Seed 42 Seed 123 Seed 3407 Mean ± Std LLaVA-1.6-7B M-SpuBench PURGE-D-NPO 67.54 68.08 67.79 67.80±0.2767.80± 0.27 PURGE-H-NPO 69.98 70.41 70.19 70.19±0.2270.19± 0.22 PURGE-M-NPO 77.96 78.75 78.21 78.31±0.4078.31± 0.40 POPE PURGE-D-NPO 96.71 96.55 96.83 96.70±0.1496.70± 0.14 PURGE-H-NPO 96.54 96.68 96.60 96.61±0.0796.61± 0.07 PURGE-M-NPO 95.92 96.08 95.97 95.99±0.0895.99± 0.08 AMBER PURGE-D-NPO 82.24 82.81 82.47 82.51±0.2982.51± 0.29 PURGE-H-NPO 80.74 81.02 80.91 80.89±0.1480.89± 0.14 PURGE-M-NPO 81.63 82.02 81.78 81.81±0.2081.81± 0.20 Qwen3-VL-8B M-SpuBench PURGE-D-NPO 73.08 74.25 74.12 73.82±0.6473.82± 0.64 PURGE-H-NPO 78.02 78.51 78.29 78.27±0.2578.27± 0.25 PURGE-M-NPO 79.12 78.92 78.12 78.72±0.5378.72± 0.53 POPE PURGE-D-NPO 98.01 98.36 99.12 98.50±0.5798.50± 0.57 PURGE-H-NPO 97.94 97.13 97.28 97.45±0.4397.45± 0.43 PURGE-M-NPO 97.28 97.51 97.69 97.49±0.2197.49± 0.21 AMBER PURGE-D-NPO 87.23 87.60 87.83 87.55±0.3087.55± 0.30 PURGE-H-NPO 88.43 88.21 88.81 88.48±0.3088.48± 0.30 PURGE-M-NPO 88.90 89.01 88.72 88.88±0.1588.88± 0.15 Appendix K Controlled object-absent analysis To further characterize the behavioral mechanism targeted by PURGE, we perform a controlled evaluation in which the target object is absent while contextual cues associated with that object remain available. Under this setting, predicting the absent object provides a direct behavioral measure of reliance on contextual shortcuts. Table 15: Hallucination rate (%) on controlled object-absent examples. Lower is better. Test Set Image Type Base PURGE-D-NPO PURGE-H-NPO PURGE-M-NPO PURGE-D Background-only 61.2 12.3 18.1 21.4 PURGE-H Inpainted object-absent 64.8 15.8 19.2 22.5 PURGE-M Cue-only object-absent 56.0 14.9 17.3 19.8 The baseline produces context-driven false-positive predictions in 56-65% of these controlled object-absent cases. All PURGE variants substantially reduce this behavior, lowering the hallucination rate to approximately 12-23%. These findings provide behavioral evidence that PURGE reduces reliance on contextual shortcuts when object-specific evidence is absent. We do not interpret this experiment as establishing a unique causal origin for the learned associations. Rather, biased object-context co-occurrence provides a statistical signal during training, while visual-semantic representation learning determines whether this regularity becomes encoded as a predictive shortcut. PURGE targets such shortcuts once they become behaviorally observable. Appendix L Evaluation prompts and answer normalization For completeness and reproducibility, we report the exact prompt formats used for the major evaluation benchmarks. POPE and AMBER discriminative evaluation. For binary object-existence questions, we use <image> Is there a <object> in the image? where <object> denotes the queried object category. For POPE, we use the official evaluation protocol to parse model-generated responses into binary yes/no predictions. The evaluator handles differences in capitalization, punctuation, and additional generated text according to the official parsing procedure. The same parser is applied to all baseline and PURGE variants across the Random, Popular, and Adversarial splits, with no method-specific manual post-processing. Accuracy, Precision, Recall, F1, and Yes ratio are then computed from the parsed predictions using the official evaluator. CHAIR and AMBER generative evaluation. For caption generation, we use <image> Describe the image in detail. MMHal-Bench. We directly use the official benchmark-provided image-question pairs without modification: <image> <official benchmark question> The official questions cover object recognition, attribute recognition, counting, spatial reasoning, and commonsense reasoning. M-SpuBench. We use the official multiple-choice evaluation format: <image> Question: <official benchmark question> A. <option A> B. <option B> C. <option C> D. <option D> The model is required to select the correct option from A-D. Causal-HalBench. We directly use the official binary object-existence questions for both original and counterfactual images: <image> Is there a <object> in the image? Answer normalization. For binary evaluations, model outputs are normalized before scoring. Responses such as Yes, yes, and yes. are mapped to Yes, while No, no, and no. are mapped to No. Case differences and terminal punctuation are ignored during normalization. Appendix M Object existence evaluation (POPE) across all splits Table 16 presents full results on the POPE across Random, Popular, and Adversarial splits, which progressively increase the strength of spurious correlations. The vanilla baseline consistently underperforms across all metrics, confirming its dependence on biased object-context associations. All PURGE variants substantially improve performance, with PURGE-M-NPO achieving the best overall results and PURGE-H-NPO often ranking second. Notably, gains are more pronounced in the Popular and Adversarial splits, highlighting improved robustness under stronger bias conditions. Overall, these results demonstrate that PURGE effectively mitigates spurious correlations and enhances reliable visual grounding. Table 16: POPE [27] results across Random, Popular, and Adversarial splits. Best and second-best values are highlighted in green and red, respectively. Split Method Obj. LLaVA-1.6-7B Qwen3-VL-8B-Instruct Qwen3.5-9B Acc Rec F1 Yes Acc Rec F1 Yes Acc Rec F1 Yes Random Baseline – 67.2 54.5 62.4 37.3 79.1 80.7 79.4 51.6 68.1 55.2 63.4 37.1 PURGE-D GA 87.5 88.0 87.6 50.5 90.1 89.2 90.0 49.1 88.2 86.1 88.0 47.9 KL 92.5 89.5 92.3 47.0 97.2 94.4 97.1 47.2 90.1 90.1 90.1 50.0 NPO 93.7 89.9 93.5 46.2 91.1 92.8 91.2 51.7 92.3 91.6 92.2 49.3 PURGE-H GA 89.5 88.5 89.4 49.0 88.7 89.1 88.7 50.4 86.2 85.4 86.1 49.2 KL 90.3 89.6 90.2 49.3 90.5 91.1 90.6 50.6 89.4 88.3 89.3 48.9 NPO 97.3 95.1 97.2 47.8 98.1 96.2 98.1 48.1 92.1 91.7 92.1 49.6 PURGE-M GA 91.6 92.1 91.6 50.5 92.3 93.0 92.4 50.7 88.1 88.3 88.1 50.2 KL 92.2 92.4 92.2 50.2 98.2 96.4 98.2 48.2 89.8 91.2 89.9 51.4 NPO 98.1 96.2 98.1 48.1 99.6 99.2 99.6 49.6 91.6 92.1 91.6 50.5 Popular Baseline – 70.0 54.4 64.5 34.4 80.1 81.0 80.3 50.9 69.9 78.3 72.2 58.4 PURGE-D GA 87.9 86.5 87.7 48.6 91.0 89.3 90.8 48.3 86.1 85.3 86.0 49.2 KL 91.1 92.0 91.2 50.9 96.1 92.1 95.9 46.0 90.2 89.8 90.2 49.6 NPO 92.6 92.0 92.6 49.4 96.2 92.5 96.1 46.3 92.0 90.4 91.9 48.4 PURGE-H GA 88.7 88.6 88.7 49.9 88.5 89.3 88.6 50.8 85.3 84.7 85.2 49.4 KL 89.7 90.0 89.7 50.3 92.0 91.4 92.0 49.4 89.2 88.3 89.1 49.1 NPO 95.2 94.0 95.1 48.8 96.3 92.6 96.2 46.3 92.7 90.1 92.5 47.4 PURGE-M GA 92.7 92.3 92.7 49.6 93.0 92.7 93.0 49.7 85.2 87.1 85.5 51.9 KL 92.5 93.1 92.5 50.6 92.9 90.6 92.7 47.7 88.4 89.1 88.5 50.7 NPO 96.0 94.6 95.9 48.6 97.5 95.0 97.4 47.5 90.1 91.3 90.2 51.2 Adversarial Baseline – 77.6 55.2 71.1 27.6 87.9 79.0 86.7 41.1 89.7 79.3 88.5 39.6 PURGE-D GA 90.4 84.2 89.8 43.8 95.1 90.3 94.9 45.2 88.9 82.1 88.1 43.2 KL 89.6 87.3 89.4 47.7 92.3 89.0 92.0 46.7 93.1 88.7 92.8 45.6 NPO 92.9 90.8 92.7 47.9 94.6 93.1 94.5 48.5 93.9 90.1 93.7 46.2 PURGE-H GA 91.9 88.2 91.6 46.3 93.3 89.1 93.0 45.8 91.0 85.2 90.4 44.2 KL 91.8 89.7 91.6 47.9 94.1 91.1 93.9 47.0 93.0 88.2 92.6 45.2 NPO 96.2 95.1 96.2 48.9 94.3 88.7 94.0 44.4 95.3 91.6 95.1 46.3 PURGE-M GA 94.6 89.3 94.3 44.7 94.1 92.1 94.0 48.0 91.6 84.1 90.9 42.5 KL 92.8 90.6 92.6 47.8 96.9 93.8 96.8 46.9 93.1 86.2 92.6 43.1 NPO 93.0 91.7 92.9 48.7 97.1 94.2 97.0 47.1 94.7 89.4 94.4 44.7 Appendix N Object hallucination reduction on CHAIR Table 17 evaluates object hallucination using the CHAIR metric on the COCO validation set. Baseline models exhibit high hallucination rates, as reflected by large CHAIRs and CHAIRi values, indicating frequent generation of non-existent objects. Our proposed PURGE framework consistently reduces hallucinations across all models and settings. In particular, the PURGE-M-NPO variant achieves the best performance, significantly lowering CHAIR scores compared to both baseline and other variants. For example, on LLaVA-1.6-7B, CHAIRs is reduced from 41.2 to 11.3, demonstrating Figure 11: Evaluation on M-SpuBench. PURGE consistently improves accuracy across models, with the largest gains under PURGE-M-NPO. a substantial improvement in object grounding. Moreover, the consistent gains across different architectures highlight the effectiveness of partition-aware unlearning in mitigating spurious correlations, which are an important contributor to hallucination. Table 17: CHAIR [7] evaluation on COCO validation set. Lower is better. Method LLaVA-1.6-7B Qwen3-VL-8B-Instruct Qwen3.5-9B CHAIRs↓_s CHAIRi↓_i CHAIRs↓_s CHAIRi↓_i CHAIRs↓_s CHAIRi↓_i Baseline 41.2 23.6 32.4 19.3 36.2 21.2 PURGE-D PURGE-D-GA 26.9 15.3 21.2 13.5 30.6 19.6 PURGE-D-KL 24.5 14.1 20.3 13.2 27.1 18.5 PURGE-D-NPO 20.3 12.7 17.4 11.9 24.8 18.9 PURGE-H PURGE-H-GA 22.5 17.6 19.3 12.4 25.8 19.1 PURGE-H-KL 19.9 15.7 17.4 11.1 24.6 18.3 PURGE-H-NPO 18.3 9.3 11.6 7.4 23.3 17.0 PURGE-M PURGE-M-GA 20.4 15.1 18.2 11.7 22.6 16.3 PURGE-M-KL 17.5 11.2 16.1 9.2 21.6 16.0 PURGE-M-NPO 11.3 6.6 9.6 4.3 20.7 14.2 Appendix O Fine-grained evaluation on M-SpuBench Fig. 11 evaluates performance on M-SpuBench [28]. All PURGE variants consistently improve accuracy across models, with PURGE-M achieving the largest gains, improving LLaVA-1.6-7B from 59.7 to 78.3 and Qwen3-VL-8B-Instruct from 73.8 to 88.7, demonstrating effective removal of spurious correlations. Table 18 presents category-wise performance across multiple spurious correlation dimensions. Baseline models exhibit strong bias, particularly in categories such as BG and RS, indicating over-reliance on contextual cues. Our PURGE framework consistently improves performance across all categories, with the largest gains observed in bias-sensitive metrics. Among variants, PURGE-M-NPO achieves the best results across all models, significantly improving both individual categories and overall accuracy. These improvements highlight the effectiveness of partition-aware unlearning in mitigating spurious correlations and enhancing generalization. Table 18: Category-wise accuracy (%) on M-SpuBench. All category metrics are higher-is-better (↑ ). Macro Avg denotes the unweighted mean accuracy across the nine spurious-correlation categories. Overall M-SpuBench accuracy is reported separately in Table 6. Model Method BG↑ TN↑ CO↑ RS↑ Col↑ Ori↑ LS↑ PA↑ Sha↑ Macro Avg↑ LLaVA-1.6-7B Baseline 67.5 52.7 61.7 73.8 43.0 70.1 50.9 44.5 59.5 58.2 PURGE-D-NPO 70.5 56.8 65.4 75.3 46.4 72.3 54.2 48.3 63.1 61.4 PURGE-H-NPO 74.1 60.2 68.2 78.3 50.6 75.5 58.7 52.7 67.3 65.1 PURGE-M-NPO 79.4 66.2 73.6 82.1 56.3 80.2 64.3 58.3 73.1 70.4 Qwen3-VL-8B-Instruct Baseline 82.9 51.3 70.6 84.7 59.4 74.8 72.4 66.0 66.2 69.8 PURGE-D-NPO 85.2 59.0 73.0 86.0 63.3 77.5 75.2 70.2 70.8 73.4 PURGE-H-NPO 87.0 64.0 77.0 88.0 67.4 81.0 79.8 75.1 75.3 77.2 PURGE-M-NPO 90.0 71.5 82.6 91.0 73.8 85.0 84.3 81.6 81.2 82.3 Qwen3.5-9B Baseline 60.5 50.3 61.2 72.4 42.0 68.0 52.1 46.5 60.2 57.0 PURGE-D-NPO 61.6 51.4 62.3 73.3 43.1 69.3 53.3 47.8 61.1 58.1 PURGE-H-NPO 62.4 52.8 63.8 74.0 44.8 70.0 54.1 48.5 62.4 59.2 PURGE-M-NPO 64.3 55.6 65.1 75.0 47.5 72.4 56.0 50.7 64.8 61.3 Appendix P Robustness and hallucination evaluation AMBER benchmark: hallucination and grounding. Table 19 evaluates hallucination and grounding performance across generative and discriminative settings. Baseline models exhibit high hallucination (CH, Hal) and lower coverage, indicating poor object grounding and reliance on spurious correlations. PURGE consistently improves performance across all models, with gains observed in both hallucination reduction and classification metrics. In particular, PURGE-M-NPO achieves the best results, significantly reducing hallucination while improving coverage and discriminative scores. For example, on Qwen3-VL-8B-Instruct, CH reduces from 5.9 to 2.8 and Hal from 26.5 to 8.3, demonstrating strong improvements in factual consistency and object grounding. Ablation study on AMBER. Table 20 analyzes the effect of different components of our method. We observe that partition-aware unlearning alone (LLM) already reduces hallucination compared to the baseline, confirming the effectiveness of removing spurious dependencies. Further improvements are achieved by incorporating the projector (LLM + Proj), which enhances visual alignment and leads to lower hallucination and higher coverage. These results indicate that both unlearning and improved vision-language alignment are complementary and essential for achieving robust and reliable predictions. Table 19: AMBER evaluation. Model Method Generative Discriminative CH↓ Cov↑ Hal↓ Cog↓ F1↑ F1E↑ F1A↑ F1R↑ LLaVA-1.6-7B Baseline 8.3 61.0 48.6 4.2 87.0 95.1 81.5 69.6 PURGE-D-NPO 6.3 65.6 23.7 3.8 90.2 93.5 87.4 57.2 PURGE-H-NPO 4.3 68.3 16.3 3.0 90.8 95.5 88.7 72.1 PURGE-M-NPO 3.1 74.1 11.3 2.7 93.2 96.8 90.0 78.4 Qwen3-VL-8B- Instruct Baseline 5.9 62.3 26.5 3.1 85.4 94.2 82.5 70.8 PURGE-D-NPO 5.0 72.5 21.6 2.8 87.2 94.5 85.2 77.6 PURGE-H-NPO 4.1 79.6 16.3 2.1 92.5 95.3 87.8 80.0 PURGE-M-NPO 2.8 85.2 8.3 1.1 94.6 96.7 90.1 85.2 Qwen3.5-9B Baseline 7.8 55.8 47.3 4.5 81.2 90.6 85.4 69.2 PURGE-D-NPO 6.6 58.3 29.4 3.9 84.1 91.3 87.6 72.7 PURGE-H-NPO 5.3 65.4 22.4 3.7 85.1 91.8 88.3 74.2 PURGE-M-NPO 4.4 68.2 21.6 3.2 87.2 92.4 88.6 78.5 Table 20: Full ablation on AMBER benchmark. Method LLaVA-1.6-7B Qwen3-VL-8B-Instruct Qwen3.5-9B CH↓ Cov↑ Hal↓ Cog↓ CH↓ Cov↑ Hal↓ Cog↓ CH↓ Cov↑ Hal↓ Cog↓ Baseline 8.3 61.0 48.6 4.2 5.9 62.3 26.5 3.1 7.8 55.8 47.3 4.5 PURGE-D-NPO LLM 7.3 62.1 39.3 4.0 5.1 63.7 25.7 3.0 6.6 58.3 29.4 3.9 LLM + Proj 6.3 65.6 23.7 3.8 5.0 72.5 21.6 2.8 - - - - PURGE-H-NPO LLM 5.8 63.0 30.0 3.8 4.6 73.1 20.2 2.5 5.3 65.4 22.4 3.7 LLM + Proj 4.3 68.3 16.3 3.0 4.1 79.6 16.3 2.1 - - - - PURGE-M-NPO LLM 4.0 66.8 26.3 3.0 4.2 75.1 19.4 2.3 4.4 68.2 21.6 3.2 LLM + Proj 3.1 74.1 11.3 2.7 2.8 85.2 8.3 1.1 - - - -