Paper deep dive
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Jiawei Feng, Jiancan Wu, Xingyu Zhu, Junkang Wu, Xiang Wang, Xiangnan He
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/21/2026, 3:21:26 AM
Summary
The paper introduces PEA-DPO, a framework for aligning Multimodal Large Language Models (MLLMs) that addresses 'visual insensitivity'—the failure of models to distinguish between images with and without critical visual context. PEA-DPO employs a dual preference learning approach, optimizing both response quality and visual context preferences. It constructs perception-enhanced preference data by masking critical visual elements using CLIP embeddings and jointly optimizes these signals to reduce hallucinations and improve multimodal alignment.
Entities (11)
Relation Signals (9)
PEA-DPO → addresses → Visual Insensitivity
confidence 95% · To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO)... to overcome visual insensitivity.
PEA-DPO → evaluatedon → MMHal-Bench
confidence 90% · Evaluations on MMHal Bench... demonstrate that PEA-DPO significantly outperforms
PEA-DPO → evaluatedon → Object HalBench
confidence 90% · Evaluations on... Object HalBench... demonstrate that PEA-DPO significantly outperforms
PEA-DPO → evaluatedon → AMBER
confidence 90% · Evaluations on... AMBER... demonstrate that PEA-DPO significantly outperforms
PEA-DPO → mitigates → Within-Image Insensitivity
confidence 90% · PEA-DPO provably mitigates both failure modes... Within-Image Insensitivity
PEA-DPO → mitigates → Across-Image Insensitivity
confidence 90% · PEA-DPO provably mitigates both failure modes... Across-Image Insensitivity
PEA-DPO → uses → CLIP
confidence 90% · leverage CLIP embeddings to identify masked variants that eliminate the most critical visual context
LLaVA → basemodelfor → PEA-DPO
confidence 85% · experiments using two sizes of LLaVA-v1.5 models... PEA-DPO
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.
Tags
Links
- Source: https://arxiv.org/abs/2608.19598v1
- Canonical: https://arxiv.org/abs/2608.19598v1
Trouble viewing inline? Open PDF directly →
Full Text
53,831 characters extracted from source content.
Expand or collapse full text
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs AlignmentConference: Proceedings of the 34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, BrazilProceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, BrazilDOI: 10.1145/3767308.3836113ISBN: 979-8-4007-2213-4/2026/11CCS: Computing methodologies Artificial intelligence Jiawei Feng email: jwf3ng@mail.ustc.edu.cn Affiliation: University of Science and Technology of China , Hefei , China , Jiancan Wu Note: Corresponding Author. email: wujcan@gmail.cn Affiliation: University of Science and Technology of China , Hefei , China , Xingyu Zhu email: xingyu.zhu@nus.edu.sg Affiliation: National University of Singapore , Kent Ridge , Singapore , Junkang Wu email: jkwu0909@mail.ustc.edu.cm Affiliation: University of Science and Technology of China , Hefei , China , Xiang Wang email: xiangwang@ustc.edu.cn Affiliation: University of Science and Technology of China , Hefei , China and Xiangnan He email: hexn@ustc.edu.cn Affiliation: University of Science and Technology of China , Hefei , China 2026; © c Abstract. Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations. Keywords: Multimodal LLMs, Direct Preference Optimization, Alignment †c-license: by 1. Introduction (a) LLaVA (b) LLaVA+DPO (c) LLaVA+PEA-DPO Figure 1. Comparison of representation distributions for different models. Representations are constructed from 200 samples (original images and images with removed key visual context), using the embedding of the last token from the LLM to represent image semantics.Three two-dimensional representation plots compare LLaVA, LLaVA with DPO, and LLaVA with PEA-DPO. Original and context-reduced images overlap for the first two models and are more clearly separated by PEA-DPO. Aligning multimodal large language models (MLLMs) (49; 48; 15) with human values is crucial for building reliable AI systems that can understand and reason about visual-textual content while generating helpful responses (36; 3; 18). The prevailing approaches in this field follow the advances established by text-only language model alignment (24; 28; 45; 47), applying Direct Preference Optimization (DPO) (26) and its variants (8; 21; 35; 20) to multimodal scenarios. They typically first construct multimodal preference datasets by pairing images with corresponding preferred and dispreferred textual responses, then apply standard DPO loss to align model outputs with human judgments (16; 33; 42; 43; 37; 7; 46; 50). (a) Across-Image Insensitivity (b) Within-Image Insensitivity Figure 2. Illustration of two manifestations of visual insensitivity in MLLM alignment. (a) Across-image insensitivity: the model assigns nearly identical preference across distinct images (e.g., outdoor reading vs. picnic), failing to capture discriminative visual context. (b) Within-image insensitivity: the model fails to discriminate semantically critical cues from irrelevant ones within the same image. (e.g., confusing a screwdriver with a toilet brush or a plunger), leading to visually ungrounded responses.Two examples of visual insensitivity. The first contrasts different scenes that receive nearly identical model preferences. The second highlights critical objects within one scene that the model confuses with visually unrelated objects. However, directly applying DPO to MLLMs implicitly treats the image as conditioning rather than a target of preference. As a result, the model can produce responses that are weakly grounded in the visual evidence. We term this failure visual insensitivity. Empirically, Figure 1 shows that the representation distributions of original images (i.e., chosen images) and their context-reduced counterparts (i.e., rejected images, with critical visual evidence removed) largely overlap for both a base LLaVA model and its DPO-tuned version, indicating poor separation between informative and uninformative visual inputs. To provide a theoretical foundation for these observations, we formalize visual insensitivity by decomposing the model’s preference into textual and visual components. The analysis exposes two failure modes: (i) Across-Image Insensitivity, where the model assigns nearly identical preference across distinct images; and (i) Within-Image Insensitivity, where the model fails to discriminate semantically critical cues from irrelevant ones within the same image. Figure 2 illustrates both phenomena: near-identical responses across different scenes (e.g., outdoor reading vs. picnic) and misrecognition of key objects (e.g., toilet brush vs. generic tool). To address these limitations, we propose PEA-DPO, which fundamentally shifts from treating images as static conditioning context to jointly optimizing response quality preferences and visual context preferences. At its core is a dual preference learning framework that optimizes two complementary signals simultaneously: (1) response quality preference, where given the same text-image input, the model learns to distinguish high-quality responses from worse ones (standard multimodal DPO), and (2) visual context preferences, where given the same text but different visual contexts (original vs. context-reduced images), the model should favor responses that properly utilize complete visual evidence. We implement this dual optimization through two key components: (1) Construction of Perception-enhanced Preference Data, wherein we generate candidate images by applying random masks to original images, then leverage CLIP embeddings to identify masked variants that eliminate the most critical visual context, creating meaningful visual grounding preference pairs; and (2) Joint Optimization Objective, which combines both preference learning terms to simultaneously enhance response quality and visual sensitivity, promoting differentiation between different visual contexts (cross-image sensitivity) while directing attention toward semantically critical visual elements within images (within-image sensitivity). To evaluate the effectiveness of PEA-DPO, we conduct experiments using two sizes of LLaVA-v1.5 models (18), with 7B and 13B parameters. Evaluations on MMHal Bench (29), Object HalBench (27), and AMBER (31) demonstrate that PEA-DPO significantly outperforms strong commercial multimodal models such as GPT-4V (1) in multimodal scenarios. Furthermore, for both 7B and 13B models, PEA-DPO achieves strong performance on all benchmarks, highlighting its effectiveness and scalability. 2. Related Work In this section, we review prior work on multimodal LLMs preference optimization from two perspectives: loss function design and preference data construction. Loss Function Design. Direct Preference Optimization (DPO) (26) was originally proposed for text-only LLMs and has since been extended to multimodal settings. SymPO (19) enforces theoretical consistency across modalities to ensure robustness under perturbations. AdPO (17) applies adversarial preference signals to strengthen multimodal models against input perturbations (51; 52). DAMA (20) jointly considers data and model characteristics to adjust the optimization objective adaptively. These methods primarily optimize over response-level preferences without explicitly modeling visual preference signals. mDPO (30) and V-DPO (34) take a step further by incorporating visual signals into the optimization objective. Preference Data Construction. Another line of research focuses on how to construct high-quality preference data for multimodal alignment. From the text response perspective, LLaVA-RLHF (29) collects human-annotated preference labels on model responses, while RLAIF-V (38) replaces human annotators with open-source MLLMs to generate AI feedback at scale. RLHF-V (37) further introduces fine-grained correctional feedback to improve factual alignment. From the visual input perspective, several recent works construct rejected images to form image-level preference pairs. MFPO (12) and LPOI (41) generate rejected images through predefined transformations. mDPO (30) uses a completely uninformative image as the rejected counterpart. OPA-DPO (35) emphasizes the importance of on-policy data collection for preference pairs. CHiP (8) proposes a hierarchical cross-modal framework that constructs preference data capturing multi-level dependencies. However, these approaches either apply coarse-grained transformations that fail to precisely remove key visual context, or rely on heavyweight models for data construction. 3. Background 3.1. Background: Direct Preference Optimization in Multimodal Scenario To further improve the performance of MLLMs, RLHF/RLAIF requires a reward model r(x,y,m)r(x,y,m) that evaluates human preference over a response y given a prompt x and image m. The standard learning objective is: (1) maxπθy[r(x,y,m)]−βKL[πθ(⋅∣x,m)∣πref(⋅∣x,m)], _ _θE_D_y[r(x,y,m)]- _KL[ _θ(· x,m) _ref(· x,m)], where D denotes the dataset, with prompts x and images m sampled from the reference policy πref _ref. The term KLD_KL is the KL divergence, and β controls the strength of regularization. DPO derives a closed-form solution to Eq. 1, revealing that the reward function can be expressed as: (2) r(x,y,m)=βlogπθ(y∣x,m)πref(y∣x,m)+βlogZ(x,m),r(x,y,m)=β _θ(y x,m) _ref(y x,m)+β Z(x,m), where Z(x,m)Z(x,m) is a partition function depending only on the prompt x and image m. Incorporating this into the Bradley–Terry model (4), and given a dataset of preference pairs (yw≻yl)(y_w y_l) under the same (x,m)(x,m), the optimization objective becomes: (3) ℒDPO _DPO =−[logσ(r(x,yw,m)−r(x,yl,m))] =-E_D[ σ(r(x,y_w,m)-r(x,y_l,m))] =−[logσ(βlogπθ(yw∣x,m)πref(yw∣x,m)−βlogπθ(yl∣x,m)πref(yl∣x,m))], =-E_D [ σ (β _θ(y_w x,m) _ref(y_w x,m)-β _θ(y_l x,m) _ref(y_l x,m) ) ], where σ(⋅)σ(·) denotes the sigmoid function. 3.2. Problem: Across-Image Insensitivity v.s. Within-Image Insensitivity In this section, we provide a theoretical analysis of visual insensitivity in multimodal DPO, which manifests in two forms: (1) across-image insensitivity and (2) within-image insensitivity. Definition 3.1 (image-based likelihood ratio). Inspired by (9; 22), we define the image-based likelihood ratio in log form: (4) ℓθ(x,m,y)≜logπθ(y∣x,m)πθ(y∣x), _θ(x,m;y) _θ(y x,m) _θ(y x), which quantifies the relative gain in the plausibility of response y when conditioning on the image m in addition to the prompt x. Definition 3.2 (response-level margin). Given a preference data (x,m,yw,yl)(x,m,y_w,y_l), the response-level margin is defined as follows: (5) Gr≜logπθ(yw∣x,m)−logπθ(yl∣x,m).G_r _θ(y_w\! x,m)- _θ(y_l\! x,m). GrG_r quantifies the relative preference of the policy πθ _θ between the preferred response ywy_w and the dispreferred response yly_l. GrG_r can be decomposed as: (6) Gr= G_r= [logπθ(yw∣x)−logπθ(yl∣x)]⏟Δt [ _θ(y_w x)- _θ(y_l x) ]_ _t +[ℓθ(x,m,yw)−ℓθ(x,m,yl)]⏟Δm, + [ _θ(x,m;y_w)- _θ(x,m;y_l) ]_ _m, where Δm _m measures the extent to which image m reinforces the preference for the preferred response. Given a pair of images mw,mlm_w,m_l, where mwm_w enables the prompt x to better align with the chosen response ywy_w than mlm_l. Theorem 3.3 (Across-Image Insensitivity). Suppose there exist samples (x,mw,ml,yw,yl)(x,m_w,m_l,y_w,y_l) such that (7) Gr(mw)−Gr(ml)≤δ(δ→0),G_r(m_w)-G_r(m_l)≤δ (δ→ 0), it implies the relation: (8) Δm(mw)−Δm(ml)≤δ(δ→0). _m(m_w)- _m(m_l)≤δ (δ→ 0). Intuitive explanation. The model cannot effectively distinguish the impact of mwm_w versus mlm_l on the responses, a phenomenon we term Across-Image Insensitivity. Figure 3. Overview of PEA-DPO. Left: Original text-based preference pairs. Top: Construction process of perception-enhanced preference pairs, where critical key visual context is removed from images using a CLIP-based approach. Right: The resulting perception-enhanced preference pairs. Bottom: Joint optimization text-based and perception-enhanced preferences. The PEA-DPO pipeline starts with an image, prompt, preferred response, and rejected response. It masks candidate image regions, uses CLIP similarity to select a context-reduced image, constructs visual-context preference pairs, and jointly optimizes textual response and visual-context preferences. Theorem 3.4 (Within-Image Insensitivity). Suppose there exist samples (x,ml,y)(x,m_l,y) such that (9) |ℓθ(x,ml,y)|≤ε(ε→0), | _θ(x,m_l;y) |≤ ( → 0), and that the model exhibits Across-Image Insensitivity. Then the following holds for mwm_w: (10) |ℓθ(x,mw;yw)−ℓθ(x,mw;yl)|≤δ+2ε(δ,ε→0). | _θ(x,m_w;y_w)- _θ(x,m_w;y_l) |≤δ+2 (δ, → 0). Intuitive explanation. This observation suggests that the model is unable to reliably generate the correct response even when conditioned on mwm_w. We refer to this phenomenon as Within-Image Insensitivity. Complete proofs are provided in Appendix . 4. Method In summary, both types of issues arise from visual insensitivity. To address these limitations, we introduce PEA-DPO, as illustrated in Figure 3. It consists of two key components: (1) Construction of Perception-Enhanced Preference Data, where rejected images mlm_l are generated by removing critical visual context using a CLIP-based approach; and (2) Joint Optimization Objective, which learns response quality preferences and visual sensitivity, simultaneously optimizing over response-level and image-level preferences, thereby aligning with human value preferences while enhancing visual sensitivity. 4.1. Construction of Perception-Enhanced Preference Data We begin with the chosen image mwm_w (i.e., the original image) and apply a random mask of fixed proportion to generate a perturbed image mpm_p: (11) mp=mw⊙(1−),m_p=m_w (1-P), where P denotes a random binary mask and ⊙ represents the Hadamard product. Repeating this process n times yields a candidate set of perturbed images: (12) ℳp=mpkk=1n,M_p=\m_p^k\_k=1^n, where mpkm^k_p denoting the k-th perturbed image. Since the perturbations are random, each perturbed image retains different portions of the original visual information. To quantify how much visual context is lost relative to the chosen image mwm_w, we compute semantic similarity using CLIP (25) embeddings. Let w=fCLIP(mw)v_w=f_CLIP(m_w) and pk=fCLIP(mpk)v^k_p=f_CLIP(m^k_p) denote the ℓ2 _2-normalized embeddings from the CLIP image encoder fCLIP(⋅)f_CLIP(·). The similarity score is defined as: (13) sk=cos(w,pk)=w⊤pk‖w‖2‖pk‖2,s_k= (v_w,v^k_p)= v_w v^k_p\|v_w\|_2\,\|v^k_p\|_2, where sk∈[−1,1]s_k∈[-1,1] measures semantic similarity. Intuitively, given that all perturbations have equal mask size, a lower similarity score indicates that critical visual context has been removed. Finally, we select the perturbed image with the lowest similarity as the rejected image: (14) ml=argminmpk∈ℳpsk,m_l= _m^k_p _ps_k, where mlm_l corresponds to the image with its key visual context removed. Optimizing over the perception-enhanced preference data (x,ml,mw,yw)(x,m_l,m_w,y_w) encourages the model to leverage critical visual cues, thereby improving its ability to generate preferred responses grounded in visual evidence. 4.2. Joint Optimization of Response quality and Visual context Preferences After constructing the Perception-Enhanced Preference Data, we obtain a dataset m=(x,mw,ml,yw)D_m=(x,m_w,m_l,y_w). Building upon the standard multimodal DPO, we replace the response quality preference data =(x,mw,yw,yl)D=(x,m_w,y_w,y_l) with mD_m in Eq. 3. Let h(x,m,y)=βlogπθ(y∣x,m)πref(y∣x,m)h(x,m,y)=β _θ(y x,m) _ref(y x,m). This yields Visual context Preference Optimization (VPO) Objective, formulated as: (15) ℒVPO _VPO =−m[logσ(h(x,mw,yw)−h(x,ml,yw))]. =-E_D_m [ σ (h(x,m_w,y_w)-h(x,m_l,y_w) ) ]. Combining this with the standard multimodal DPO (i.e., Response quality Preference Optimization, RPO) in Eq. 3, we obtain the PEA-DPO objective: (16) ℒPEA-DPO _PEA-DPO =ℒRPO+α⋅ℒVPO =L_RPO+α·L_VPO =−[logσ(h(x,mw,yw)−h(x,mw,yl))] =-E_D [ σ (h(x,m_w,y_w)-h(x,m_w,y_l) ) ] −m[logσ(h(x,mw,yw)−h(x,ml,yw))], -E_D_m [ σ (h(x,m_w,y_w)-h(x,m_l,y_w) ) ], where α is a weighting hyperparameter. This joint objective enables MLLMs to align with human preferences by simultaneously leveraging textual and critical visual modalities. To further reduce computational overhead and enable controllable preference optimization, inspired by the design of RePO (32), we introduce three modifications to ℒPEA-DPOL_PEA-DPO: (1) Replace log-probability ratios in Eq. 16 with length-normalized log-probabilities, thereby eliminating the need for a reference model; (2) Replace the sigmoid function with a ReLU activation, which filters out trivial data points and prevents overfitting; (3) Introduce target margin γr,γm\ _r, _m\ and remove the temperature parameter β, enabling a controllable optimization process. Let hm(x,m,y)=logπθ(y|x,m)|y|h_m(x,m,y)= _θ(y|x,m)|y|. The modified PEA-DPO loss function is then given as: (17) ℒmPEA-DPO=ℒmRPO+α⋅ℒmVPO _mPEA-DPO=L_mRPO+α·L_mVPO =ReLU[−(hm(x,mw,yw)−hm(x,mw,yl)−γr)] =E_D \ReLU [- (h_m(x,m_w,y_w)-h_m(x,m_w,y_l)- _r ) ] \ +αmReLU[−(hm(x,mw,yw)−hm(x,ml,yw)−γm)], + _D_m \ReLU [- (h_m(x,m_w,y_w)-h_m(x,m_l,y_w)- _m ) ] \, where |y||y| denotes the number of tokens in response y, and γr,γm\ _r, _m\ are the target reward margins, enforcing a minimum separation between preferred and rejected responses in both response-based preferences yw,yl\y_w,y_l\ and image-based preferences mw,ml\m_w,m_l\. 5. Theoretical Analysis: How mPEA-DPO Mitigates Visual Insensitivity We now establish the theoretical connection between the mPEA-DPO objective and the two forms of visual insensitivity identified in Section 3.2. Starting from definitions in Eq. 4 and Eq. 5, we show that each failure mode corresponds to a specific margin being small, and that mPEA-DPO is designed to directly enlarge both margins. 5.1. Image-Level Margin and Across-Image Insensitivity. We define the image-level margin as: (18) Gm=logπθ(yw∣x,mw)−logπθ(yw∣x,ml),G_m= _θ(y_w x,m_w)- _θ(y_w x,m_l), which measures how much the model’s confidence in the preferred response ywy_w increases when replacing the context-reduced image mlm_l with the correct image mwm_w. According to the definition of Δm _m in Eq. 6 and the image-based likelihood ratio in Eq. 4, we can derive (see Appendix for full details): Δm(x,mw)−Δm(x,ml) _m(x,m_w)- _m(x,m_l) = = logπθ(yw∣x,mw)−logπθ(yw∣x,ml)⏟image-level margin Gm _θ(y_w x,m_w)- _θ(y_w x,m_l)_image-level margin G_m (19) +logπθ(yl∣x,ml)−logπθ(yl∣x,mw)⏟secondary term. + _θ(y_l x,m_l)- _θ(y_l x,m_w)_secondary term. When the model exhibits Across-Image Insensitivity (Theorem 3.3), both sides are bounded by δ→0δ→ 0, implying that GmG_m is near zero. That is, the model’s confidence in the preferred response barely changes regardless of whether the correct or context-reduced image is provided. The secondary term captures how the dispreferred response shifts across images, but improving GmG_m has a more direct impact on performance as it explicitly boosts the probability of the correct output under the chosen image. 5.2. Response-Level Margin and Within-Image Insensitivity. Similarly, expanding using the image-based likelihood ratio yields: ℓθ(x,mw,yw)−ℓθ(x,mw,yl) _θ(x,m_w;y_w)- _θ(x,m_w;y_l) = = logπθ(yw∣x,mw)−logπθ(yl∣x,mw)⏟response-level margin Gr _θ(y_w x,m_w)- _θ(y_l x,m_w)_response-level margin G_r (20) +logπθ(yl∣x)−logπθ(yw∣x)⏟text-only bias. + _θ(y_l x)- _θ(y_w x)_text-only bias. When the model exhibits Within-Image Insensitivity (Theorem 3.4), this quantity is bounded by η=δ+2ε→0η=δ+2 → 0, implying that GrG_r is near zero. The model fails to distinguish the preferred from the dispreferred response even with the correct visual evidence. The text-only bias term reflects inherent language priors without visual input; improving GrG_r is more critical as it directly enhances visual grounding. 5.3. mPEA-DPO Directly Enlarges Both Margins. With the two failure modes characterized by small GmG_m and GrG_r respectively, we show that the mPEA-DPO objective is designed to directly enlarge both. Revisiting Eq. 17: ℒmPEA-DPO=ReLU[−(h(x,mw,yw)−h(x,mw,yl)−γr)]⏟ℒmRPO: increases Gr, mitigates Within-Image Insensitivity+α⋅mReLU[−(h(x,mw,yw)−h(x,ml,yw)−γm)]⏟ℒmVPO: increases Gm, mitigates Across-Image Insensitivity. split&L_mPEA-DPO\\ =& E_D \ReLU [- (h(x,m_w,y_w)-h(x,m_w,y_l)- _r ) ] \_L_mRPO: increases G_r, mitigates Within-Image Insensitivity\\ &+α· E_D_m \ReLU [- (h(x,m_w,y_w)-h(x,m_l,y_w)- _m ) ] \_L_mVPO: increases G_m, mitigates Across-Image Insensitivity. split The ℒmVPOL_mVPO term enforces h(x,mw,yw)−h(x,ml,yw)>γmh(x,m_w,y_w)-h(x,m_l,y_w)> _m, directly increasing the image-level margin GmG_m and enhancing the model’s discrimination between the correct and context-reduced images. The ℒmRPOL_mRPO term enforces h(x,mw,yw)−h(x,mw,yl)>γrh(x,m_w,y_w)-h(x,m_w,y_l)> _r, directly increasing the response-level margin GrG_r and enhancing the model’s ability to favor the preferred response under the correct image. Since ℒmPEA-DPOL_mPEA-DPO is a linear combination of both terms, it provably addresses Across-Image and Within-Image Insensitivity simultaneously. Following SimPO (21), we normalize log-probabilities by response length to prevent length bias. Full derivations and an empirical validation showing how GmG_m and GrG_r shift before and after training are provided in Appendix . 6. Experiments Table 1. Main results of LLaVA-v1.5-7B and LLaVA-v1.5-13B trained with different preference optimization objectives. We report overall score (Score) and hallucination rate (Hal.) on MMHalBench, CHAIR scores at both response and object levels on Object HalBench, along with CHAIR scores (C.), object coverage (cover.), hallucination rate (Hal.), and cognition (Cog.) on AMBER. The best result for each metric in each group is in bold. We have carefully followed the publicly available code and released checkpoints to reproduce these results, aiming to provide a fair comparison. † indicates results obtained using the official API, ‡ indicates results reproduced using the authors’ released code, and ♯ indicates results produced from the authors’ provided checkpoints. MMHalBench Object HalBench AMBER Score ↑ Hal. ↓ CHAIRs ↓ CHAIRi ↓ C. ↓ Cover. ↑ Hal. ↓ Cog. ↓ GPT-4V (1)† 3.49 0.28 13.6 7.3 4.6 67.1 30.7 2.6 Gemini-2.5-pro (6)† 3.55 0.28 12.0 8.2 9.5 78.0 75.1 5.2 7B MLLMs LLaVA-v1.5-7B (18) 2.11 0.54 53.6 25.2 7.8 51.0 36.4 4.2 + HACL (11)‡ 2.13 0.50 - - - - - - + OPERA (10)‡ 2.15 0.54 45.1 22.3 - - - - + VCD (14)‡ 2.12 0.54 48.8 24.3 - - - - + EOS (40)‡ 2.03 0.59 40.3 17.8 5.1 49.1 22.7 2.0 + POVID (44)‡ 2.08 0.56 48.1 24.4 - - - - + LLaVA-RLHF (29)♯ 1.88 0.71 58.0 15.6 9.7 53.2 46.6 5.3 + HA-DPO (42)♯ 1.97 0.60 39.9 19.9 6.7 49.8 30.9 3.3 + HALVA (28)♯ 2.25 0.54 - - 6.6 53.0 32.2 3.4 + mDPO (30)‡ 2.39 0.54 35.7 9.8 4.4 52.4 24.5 2.4 + RLAIF-V (38)♯ 3.00 0.38 16.0 3.7 3.0 50.4 16.2 1.0 + OPA-DPO (35)♯ 2.83 0.45 13.0 4.3 2.2 47.9 11.6 0.9 + DAMA (20)♯ 2.76 0.41 10.3 5.9 3.0 48.3 14.8 1.2 + mPEA-DPO 3.02 0.36 4.3 3.2 1.9 46.7 10.3 0.6 13B MLLMs LLaVA-v1.5-13B (18) 2.42 - 46.3 22.6 7.8 51.0 36.4 4.2 + LLaVA-RLHF (29)♯ 2.27 0.64 44.7 11.8 7.7 52.3 38.6 4.0 + RLHF-V (37)♯ 2.81 0.49 12.2 7.5 6.3 46.1 25.1 2.1 + HALVA (28)♯ 2.58 0.45 - - 6.4 52.6 30.4 3.2 + OPA-DPO (35)♯ 3.07 0.39 16.33 5.5 2.4 48.3 12.8 0.8 + DAMA (20)♯ 2.89 0.43 7.7 4.9 3.0 50.5 14.1 0.9 + mPEA-DPO 3.16 0.31 4.3 2.7 2.9 50.0 12.8 0.8 6.1. Experimental Setup 6.1.1. Models. We evaluate mPEA-DPO on two MLLMs with different parameter scales: LLaVA-v1.5-7B and LLaVA-v1.5-13B (18), both equipped with a CLIP ViT-L-336px vision encoder. The 7B model is built upon Vicuna-7B as its LLM backbone, while the 13B model utilizes Vicuna-13B as its LLM backbone. Both models are first pretrained on 558K image–text pairs datasets and then fine-tuned on 665K instruction-following instances. 6.1.2. Training Data. Following (20), we utilize the preference data of the LLaVA-1.5 model released by (38), where the language-based preference is annotated by the open-source LLaVA-NeXT-34B model. Specifically, the dataset comprises 22K preference instances, with 13K images used for training. To construct perception-enhanced preference data, we generate rejected images by masking informative regions in these 13K images. 6.1.3. Baselines. We report results across three categories of multimodal alignment approaches, while noting that direct comparison is non-trivial due to differences in base models, preference data, and alignment strategies. Specifically: (1) Hallucination-specific baselines: including VCD (14), OPERA (10), HALC (11), and EOS (40), (2) RLHF/RLAIF-based baselines: including POVID (44), LLaVA-RLHF (29), HALVA (28), RLHF-V (37), HA-DPO (42), HSA-DPO (33), RLAIF-V (38), mDPO (30), OPA-DPO (35), and DAMA (20). (3) Proprietary baseline: GPT-4V (1), which we use as a robust reference to compare the performance between open-source and proprietary commercial models. Gemini-2.5-Pro (6), which is also included in our comparison, given its status as a strong and up-to-date commercial baseline. 6.1.4. Benchmarks. (1) Object HalBench (27) is a widely adopted benchmark for evaluating object hallucination, focusing on detailed image descriptions of visual content. Following the protocol in (38), we evaluate on 300 instances and report hallucination rates at both the response-level (CHAIRSCHAIR_S) and object-level(CHAIRICHAIR_I). (2) AMBER (31) provides a multi-dimensional evaluation of hallucination in MLLMs. Using its generative task with 1K samples, we report CHAIR scores, object coverage, hallucination rates, and alignment with human cognition. (3) MMHal-Bench (29) is a question-answering benchmark comprising 96 image–question pairs across 12 object categories and 8 question types. Following the setup of (38), we assess both overall response quality (scored from zero to six) and hallucination rate, as judged by GPT-4. 6.1.5. Implementation Details. In our experiments, we construct perception-enhanced preference data by masking 9%9\% of each image, repeated n=30n=30 times. We employ LLaVA-1.5-7B and LLaVA-1.5-13B as backbone models, and perform full-parameter fine-tuning for 5,000 steps. Specifically, the batch size is set to 32 for the 7B model and 16 for the 13B model. In our objective Eq. 17, we set the hyperparameters as γr=1.5 _r=1.5, γm=4.5 _m=4.5 and α=0.2α=0.2. All experiments are conducted using 4 NVIDIA A100 80GB GPUs. 6.2. Main Results The experimental results of applying mPEA-DPO to LLaVA-v1.5-7B and LLaVA-v1.5-13B across various hallucination benchmarks are presented in Table 1. The main findings are summarized as follows: (1) mPEA-DPO significantly reduces the hallucinations of the 7B and 13B models. Compared with the 7B (13B) base model, mPEA-DPO reduces the hallucination rate in MMHalBench by 33%, the response-level and object-level hallucination rates in Object HalBench by 92% and 87% respectively, and the CHAIR, hallucination rate, and human cognition in AMBER by 76%, 72%, and 86% respectively. (2) Compared with existing MLLMs alignment methods, for LLaVA-v1.5-13B, mPEA-DPO achieves the best performance on 75.0% of the hallucination metrics, and for LLaVA-v1.5-7B, it increases to 87.5%. (3) However, these enhancements lead to a slight compromise in coverage metrics. This indicates that models trained with mPEA-DPO tend to adopt a slightly conservative strategy, avoiding uncertain assertions. Such a strategy enhances the credibility of the responses but may overlook some ambiguous details, which requires a trade-off. 6.3. Ablation 6.3.1. Impact of Component Combination. To evaluate the contribution of each component in mPEA-DPO and the effect of their combinations, we conducted a comprehensive ablation study on mPEA-DPO based on LLaVA. The experimental results are shown in Table 2. The main observations are as follows: (1) Both Response quality Preference Optimization (mRPO) and Visual context Preference Optimization (mVPO) are effective. On the MMHalBench and AMBER datasets, both mRPO (mPEA-DPO-ℒmRPOL_ mRPO) and mVPO (mPEA-DPO-ℒmVPOL_ mVPO) outperform the base model. This suggests that: (a) during optimization process, Response quality Preference Optimization (RPO) enables the model to concentrate on more challenging data points and mitigates overfitting in DPO; (b) the introduction of Visual context Preference Optimization (mVPO) enhances the model’s alignment between the image and text. (2) The combination of visual context preference optimization and response quality preference optimization strategies makes preference optimization the most powerful. Table 2. The ablation results of mPEA-DPO based on LLaVA. Values in bold denote the best performance. Model MMHalBench AMBER Score↑ Hal.↓ C.↓ Cog.↓ LLaVA-v1.5-7B 2.11 0.54 7.8 4.2 mPEA-DPO 3.02 0.36 1.9 0.6 -ℒmRPOL_ mRPO 2.44 0.45 4.2 0.9 -ℒmVPOL_ mVPO 2.81 0.38 2.5 0.9 6.3.2. Impact of Visual context Preference Optimization. Visual context Preference Optimization (mVPO) forces models to make preference judgements based on critical visual context. Here, we discuss the impact of its weight. We fully consider Response quality Preference Optimization (mRPO) since its global textual semantics by setting its parameter to 1 in Eq. 17. As for Visual context Preference Optimization, given its crucial role in enhancing MLLM’s attention to critical visual context, we fully explore the range of its weight α (as shown in Eq. 17). For the results of Figure 4, we observe that the best performance was achieved when α=0.2α=0.2 for LLaVA frameworks. (a) MMHal Score (b) MMHal Hal. (c) AMBER C. (d) AMBER Cog. Figure 4. Results of mPEA-DPO evaluated on the MMHalBench and AMBER dataset with different choices of weight α to control the strength of Visual context Preference Optimization. Findings: when α=0.2α=0.2, the best performance of the Score, CHAIR and Hallucination Rate metric is achieved on MMHal-Bench and AMBER based on LLaVA.Four line charts show MMHal score, MMHal hallucination rate, AMBER CHAIR, and AMBER cognitive hallucination as the visual-context preference weight changes. The best overall tradeoff occurs at alpha equal to 0.2. 6.4. Impact of rejected image construction strategy The quality of visual preference data depends on the rejected image quality and its gap from the chosen images. In this section, we compare the impact of our proposed strategy for constructing rejected images and various existing strategies on preference optimization. (a) Chosen (b) Blockwise (c) Rotation (d) Crop (e) Black (f) Ours Figure 5. Examples of rejected images constructed by different strategies. (a) is the chosen image.Six versions of the same image compare the chosen image with block masking, rotation, cropping, complete blackening, and the proposed context-removal strategy. The proposed strategy selectively removes key visual evidence while retaining the rest of the scene. 6.4.1. Strategies. The existing rejected image construction strategies are listed below: (1) Blockwise: The chosen image is divided into blocks, with 30% of blocks randomly masked. (2) Rotation: The chosen image is randomly rotated between 10 to 80 degrees. (3) Crop: Random cropping strategy is applied to the chosen image. (4) Blackness: All RGB values in the chosen image are set to 0. Table 3. Impact of rejected image construction strategy on mPEA-DPO. Bold indicates best. Strategy MMHal Bench AMBER Score↑ Hal.↓ CHAIR↓ HalRate↓ Ours 3.03 0.36 1.9 10.3 Blockwise 2.64 0.42 2.8 14.1 Rotation 2.47 0.46 2.6 12.5 Crop 2.69 0.42 2.1 11.3 Blackness 2.90 0.41 2.4 12.3 Figure 6. Impact of data scale on the performance of mPEA-DPO, using LLaVA as the base model. We assess: (1) the overall score and hallucination rate on MMHalBench, and (2) CHAIR and hallucination on AMBER. 6.4.2. Results. The experimental results of mPEA-DPO under different construction strategies of rejected images are shown in Table 3. We observe the rejected with removed key visual context can lead to better optimization results. The blockwise, rotation, and crop strategies retain a significant amount of the chosen image’s visual context, failing to effectively remove key visual context and thus leading to poorer performance. The blackness strategy, by completely masking the chosen image, virtually eliminates information about the chosen image, resulting in poorer performance. However, Perception-Enhanced Preference Data enhances the sensitivity of MLLMs to key visual information by removing the key visual context from the chosen images, thereby achieving the best performance. 6.5. Impact of data scale To investigate the impact of data scale on mPEA-DPO, we present its performance under varying amounts of training data in Figure 6. Even with only 2800 training instances, mPEA-DPO remains effective and outperforms the baseline. Additionally, we observe that the performance of mPEA-DPO consistently improves with increasing data scale, demonstrating significant performance gain. 6.6. Attribution of Hallucination Reduction As discussed in (2), the CHAIR metric has a known limitation in that it does not penalize shorter responses, which may trivially reduce hallucination scores. To demonstrate that the reduction in CHAIR metrics is indeed driven by reduced hallucination rather than shorter or less informative outputs, we conducted a comparison on Object HalBench between our method and two of the strongest existing baselines, OPA-DPO (35) and DAMA (20). We report three metrics: CHAIRs, CHAIRi, and Recall. Table 4. Comparison of hallucination and recall performance on Object HalBench. Bold denote the best performance. Model CHAIRs CHAIRi Recall OPA-DPO 13.3 4.3 43.29 DAMA 10.3 5.9 54.50 mPEA-DPO 4.3 3.2 52.30 As shown in Table 4, mPEA-DPO achieves clearly superior hallucination metrics compared with both OPA-DPO and DAMA. Importantly, mPEA-DPO attains recall performance comparable to DAMA while substantially outperforming OPA-DPO, indicating that the improvement in CHAIR metrics is not simply due to shorter responses but rather reflects a genuine reduction in hallucination. 6.7. Analysis of General Capability Preference optimization may negatively affect the model’s generalization ability. In this section, we evaluate the general capabilities of MLLM enhanced with our mPEA-DPO on several widely used benchmarks, including MMStar (5), AI2D (13), LLaVA-Bench (18), and MMMU (39). For LLaVA-Bench, we report the relative score judged by GPT-4o (23). The results are presented in Table 5. Table 5. The general capability evaluation results. Values in bold denote the best performance. Model MMStar AI2D LLaVA-Bench MMMU(val) MMMU(test) LLaVA-v1.5-7B 30.3 49.1 82.3 32.1 35.3 +mPEA-DPO 32.8 51.9 86.5 30.8 33.4 We observe that LLaVA+mPEA-DPO outperforms LLaVA on MMStar, AI2D, and LLaVA-Bench, while maintaining comparable performance on MMMU. The results suggest that visual preference optimization not only reduces hallucinations but also enhances the model’s instruction-following capability. These results indicate that mPEA-DPO preserves and even slightly improves the general capabilities of the base model. 7. Limitation and Conclusion Limitation. While mPEA-DPO yields notable gains in multimodal alignment and hallucination mitigation, several limitations remain. First, constructing perception-enhanced preference data incurs additional computation for generating and evaluating perturbed images. This overhead is manageable in our experiments. Second, due to limited computational capacity, we have not tested on the latest MLLMs (e.g., Muffin). Third, mPEA-DPO slightly reduces coverage metrics, likely reflecting a conservative generation strategy that avoids uncertain outputs. Conclusion. In summary, our study uncovers two fundamental issues of Direct Preference Optimization (DPO) in multimodal settings: (1) Across-Image Insensitivity and (2) Within-Image Insensitivity. Through both theoretical analysis and empirical evaluation, we systematically characterize the inherent limitations of existing multimodal DPO methods in exhibiting visual insensitivity. To address these limitations, we propose Perception-Enhanced Alignment (PEA)-DPO, a framework for MLLM alignment that explicitly leverages visual preference signals in conjunction with standard multimodal DPO. Experiments on three widely-used benchmarks demonstrate that PEA-DPO substantially improves the performance of LLaVA-v1.5-7B and LLaVA-v1.5-13B, achieving strong results and surpassing other RLHF/RLAIF-based methods. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1, §6.1.3, Table 1. Amirloo et al. (2024) E. Amirloo, J. Fauconnier, C. Roesmann, C. Kerl, R. Boney, Y. Qian, Z. Wang, A. Dehghan, Y. Yang, Z. Gan, et al. Understanding alignment in multimodal llms: a comprehensive study. arXiv preprint arXiv:2407.02477. Cited by: §6.6. Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Cited by: §1. Bradley and Terry (1952) R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §3.1. Chen et al. (2024) L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al. Are we on the right way for evaluating large vision-language models?. Advances in Neural Information Processing Systems 37, p. 27056–27087. Cited by: §6.7. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §6.1.3, Table 1. Deng et al. (2024) Y. Deng, P. Lu, F. Yin, Z. Hu, S. Shen, Q. Gu, J. Y. Zou, K. Chang, and W. Wang Enhancing large vision language models with self-training on image comprehension. Advances in Neural Information Processing Systems 37, p. 131369–131397. Cited by: §1. Fu et al. (2025) J. Fu, S. Huangfu, H. Fei, X. Shen, B. Hooi, X. Qiu, and S. Ng CHiP: cross-modal hierarchical direct preference optimization for multimodal llms. arXiv preprint arXiv:2501.16629. Cited by: §1, §2. Gutmann and Hyvärinen (2010) M. Gutmann and A. Hyvärinen Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, p. 297–304. Cited by: Definition 3.1. Huang et al. (2024) Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu Opera: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13418–13427. Cited by: §6.1.3, Table 1. Jiang et al. (2024) C. Jiang, H. Xu, M. Dong, J. Chen, W. Ye, M. Yan, Q. Ye, J. Zhang, F. Huang, and S. Zhang Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 27036–27046. Cited by: §6.1.3, Table 1. Jiang et al. (2025) S. Jiang, Y. Zhang, R. Chen, T. Hu, Y. Jin, Q. He, Y. Feng, J. Wu, and Z. Liu Modality-fair preference optimization for trustworthy MLLM alignment. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2025, Montreal, Canada, August 16-22, 2025, p. 403–411. Cited by: §2. Kembhavi et al. (2016) A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi A diagram is worth a dozen images. In European conference on computer vision, p. 235–251. Cited by: §6.7. Leng et al. (2024) S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13872–13882. Cited by: §6.1.3, Table 1. Li et al. (2025) C. Li, Z. Wang, Y. Sheng, X. Zhu, Y. Hao, and X. Wang Res-bench: benchmarking the robustness of multimodal large language models to dynamic resolution input. External Links: 2510.16926, Link Cited by: §1. Li et al. (2023) L. Li, Z. Xie, M. Li, S. Chen, P. Wang, L. Chen, Y. Yang, B. Wang, and L. Kong Silkie: preference distillation for large visual language models. arXiv preprint arXiv:2312.10665. Cited by: §1. Liu et al. (2025a) C. Liu, T. Gui, Y. Liu, and L. Xu AdPO: enhancing the adversarial robustness of large vision-language models with preference optimization. arXiv preprint arXiv:2504.01735. Cited by: §2. Liu et al. (2024) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 26296–26306. Cited by: §1, §1, §6.1.1, §6.7, Table 1, Table 1. Liu et al. (2025b) W. Liu, X. Song, J. Li, Y. Wei, N. Zheng, J. Yin, and L. Nie Mitigating hallucination through theory-consistent symmetric multimodal preference optimization. arXiv preprint arXiv:2506.11712. Cited by: §2. Lu et al. (2025) J. Lu, J. Wu, J. Li, X. Jia, S. Wang, Y. Zhang, J. Fang, X. Wang, and X. He DAMO: data-and model-aware alignment of multi-modal llms. arXiv preprint arXiv:2502.01943. Cited by: §1, §2, §6.1.2, §6.1.3, §6.6, Table 1, Table 1. Meng et al. (2024) Y. Meng, M. Xia, and D. Chen Simpo: simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37, p. 124198–124235. Cited by: §1, §5.3. Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: Definition 3.1. OpenAI (2024) OpenAI GPT-4o system card. CoRR abs/2410.21276. External Links: Link Cited by: §6.7. Pi et al. (2024) R. Pi, T. Han, W. Xiong, J. Zhang, R. Liu, R. Pan, and T. Zhang Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision, p. 382–398. Cited by: §1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §4.1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1, §2. Rohrbach et al. (2018) A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko Object hallucination in image captioning. arXiv preprint arXiv:1809.02156. Cited by: §1, §6.1.4. Sarkar et al. (2024) P. Sarkar, S. Ebrahimi, A. Etemad, A. Beirami, S. Ö. Arik, and T. Pfister Mitigating object hallucination via data augmented contrastive tuning. CoRR. Cited by: §1, §6.1.3, Table 1, Table 1. Sun et al. (2023) Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L. Gui, Y. Wang, Y. Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525. Cited by: §1, §2, §6.1.3, §6.1.4, Table 1, Table 1. Wang et al. (2024) F. Wang, W. Zhou, J. Y. Huang, N. Xu, S. Zhang, H. Poon, and M. Chen Mdpo: conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.11839. Cited by: §2, §2, §6.1.3, Table 1. Wang et al. (2023) J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, et al. Amber: an llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397. Cited by: §1, §6.1.4. Wu et al. (2025) J. Wu, K. Huang, X. Wang, J. Gao, B. Ding, J. Wu, X. He, and X. Wang RePO: relu-based preference optimization. arXiv preprint arXiv:2503.07426. Cited by: §4.2. Xiao et al. (2024) W. Xiao, Z. Huang, L. Gan, W. He, H. Li, Z. Yu, H. Jiang, F. Wu, and L. Zhu Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. arXiv preprint arXiv:2404.14233. Cited by: §1, §6.1.3. Xie et al. (2024) Y. Xie, G. Li, X. Xu, and M. Kan V-dpo: mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712. Cited by: §2. Yang et al. (2025) Z. Yang, X. Luo, D. Han, Y. Xu, and D. Li Mitigating hallucinations in large vision-language models via dpo: on-policy data hold the key. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 10610–10620. Cited by: §1, §2, §6.1.3, §6.6, Table 1, Table 1. Yin et al. (2024) S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen A survey on multimodal large language models. National Science Review 11 (12), p. nwae403. Cited by: §1. Yu et al. (2024a) T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H. Zheng, M. Sun, et al. Rlhf-v: towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13807–13816. Cited by: §1, §2, §6.1.3, Table 1. Yu et al. (2024b) T. Yu, H. Zhang, Y. Yao, Y. Dang, D. Chen, X. Lu, G. Cui, T. He, Z. Liu, T. Chua, et al. Rlaif-v: aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220. Cited by: §2, §6.1.2, §6.1.3, §6.1.4, Table 1. Yue et al. (2024a) X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9556–9567. Cited by: §6.7. Yue et al. (2024b) Z. Yue, L. Zhang, and Q. Jin Less is more: mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.14545. Cited by: §6.1.3, Table 1. Zadeh et al. (2025) F. P. Zadeh, Y. Oh, and G. Kim LPOI: listwise preference optimization for vision language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, p. 26830–26844. Cited by: §2. Zhao et al. (2023) Z. Zhao, B. Wang, L. Ouyang, X. Dong, J. Wang, and C. He Beyond hallucinations: enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839. Cited by: §1, §6.1.3, Table 1. Zhou et al. (2024a) W. Zhou, R. Agrawal, S. Zhang, S. R. Indurthi, S. Zhao, K. Song, S. Xu, and C. Zhu Wpo: enhancing rlhf with weighted preference optimization. arXiv preprint arXiv:2406.11827. Cited by: §1. Zhou et al. (2024b) Y. Zhou, C. Cui, R. Rafailov, C. Finn, and H. Yao Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411. Cited by: §6.1.3, Table 1. Zhu et al. (2026a) X. Zhu, J. Fang, S. Wang, B. Zhu, Z. Wang, Y. Yang, and X. He Mitigating hallucinations in large vision-language models without performance degradation. In ACL (1), p. 1995–2009. Cited by: §1. Zhu et al. (2025) X. Zhu, S. Wang, B. Zhu, M. Li, Y. Li, J. Fang, Z. Wang, D. Wang, and H. Zhang Dynamic multimodal prototype learning in vision-language models. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), p. 2501–2511. Cited by: §1. Zhu et al. (2026b) X. Zhu, H. Wu, S. Wang, B. Zhu, J. Ge, J. Zhang, and L. Chen Robustifying vision-language models via test-time prompt adaptation. arXiv preprint arXiv:2607.09450. Cited by: §1. Zhu et al. (2026c) X. Zhu, K. Zhao, L. Yi, S. Wang, Z. Wang, B. Zhu, H. Zhang, and X. He Look carefully: adaptive visual reinforcements in multimodal large language models for hallucination mitigation. In International Conference on Learning Representations, Vol. 2026, p. 81235–81256. Cited by: §1. Zhu et al. (2026d) X. Zhu, B. Zhu, J. Fang, S. Wang, Y. Zhang, X. Wang, and X. He Guardalign: test-time safety alignment in multimodal large language models. arXiv preprint arXiv:2602.24027. Cited by: §1. Zhu et al. (2024) X. Zhu, B. Zhu, Y. Tan, S. Wang, Y. Hao, and H. Zhang Enhancing zero-shot vision models by label-free prompt distribution learning and bias correcting. Advances in Neural Information Processing Systems 37, p. 2001–2025. Cited by: §1. Zhu et al. (2026e) X. Zhu, B. Zhu, S. Wang, J. Fang, K. Zhao, H. Zhang, and X. He Principled steering via null-space projection for jailbreak defense in vision-language models. arXiv preprint arXiv:2603.22094. Cited by: §2. Zhu et al. (2026f) X. Zhu, B. Zhu, S. Wang, K. Zhao, and H. Zhang Enhancing clip robustness via cross-modality alignment. Advances in Neural Information Processing Systems 38, p. 17553–17576. Cited by: §2.