Paper deep dive
Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, Zhendong Mao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/16/2026, 8:25:45 AM
Summary
The paper introduces Groc-PO (Grounded Context Preference Optimization), a novel framework designed to enhance the truthfulness and reliability of Multimodal Large Language Models (MLLMs). It addresses error propagation from early grounding stages to final reasoning by implementing explicit, stage-specific preference supervision. The authors construct the Grounded Context Preference Dataset (GCPD), which structures preference samples across three progressive stages: Object Grounding, Contextual Grounding, and Grounded Reasoning. Experiments demonstrate that Groc-PO significantly reduces hallucinations and improves faithful reasoning compared to standard Direct Preference Optimization (DPO) and other baselines.
Entities (8)
Relation Signals (5)
Groc-PO → appliesto → MLLMs
confidence 95% · we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs.
MLLMs → suffersfrom → Visual Hallucination
confidence 95% · they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility.
Groc-PO → utilizes → GCPD
confidence 95% · Built upon the newly constructed Grounded Context Preference Dataset (GCPD), Groc-PO leverages preference signals from three contextual stages...
GCPD → contains → Object Grounding
confidence 92% · GCPD organizes multi-stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning.
Groc-PO → outperforms → DPO
confidence 90% · compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite the rapid progress of Multimodal Large Language Models (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors often propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multi-stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reasoning and mitigates cross-stage error propagation. Extensive experiments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning.
Tags
Links
- Source: https://arxiv.org/abs/2607.13712v1
- Canonical: https://arxiv.org/abs/2607.13712v1
Trouble viewing inline? Open PDF directly →
Full Text
49,982 characters extracted from source content.
Expand or collapse full text
Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs Zhixiao Zheng zhixiao.zheng@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Zheren Fu ∗ fzr@ustc.edu.cn University of Science and Technology of China Hefei, China Zhiyuan Yao yaozhiyuan@mail.ustc.edu.cn University of Science and Technology of China Hefei, China Chunxiao Liu chunxiao6liu@gmail.com Xiaomi Corporation Beijing, China Dongming Zhang zhangdongming@people.cn State Key Laboratory of Communication Content Cognition, People’s Daily Online Beijing, China Zhendong Mao zdmao@ustc.edu.cn University of Science and Technology of China Hefei, China Abstract Despite the rapid progress of Multimodal Large Language Mod- els (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reason- ing, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors of- ten propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded prefer- ence optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multi- stage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reason- ing and mitigates cross-stage error propagation. Extensive experi- ments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning. ∗ Corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n Later- stage Early- stage (s) What is the man on the right holding, and what does this suggest about his intention ? He is holding a black remote control and using it to control the electric skateboard, helping the boy complete the jump. Wrong Grounding Wrong Reasoning Misread: camera → remote control (in the right man’s hands) (a) (b) (↑) Figure 1: Motivating example of error propagation across stages in MLLMs. (a) A case where an early grounding er- ror propagates to the later reasoning stage and leads to an incorrect answer. (b) Statistical experiments with LLaVA-v1.5- 7B [20] on GCPD dataset (constructed from RLHF-V [35]), showing that introducing errors into 0, 1, or 2 grounding stages is associated with progressively lower final reasoning accuracy, consistent with error propagation in MLLMs. CCS Concepts • Computing methodologies→ Machine learning. Keywords Multimodal LLMs, Preference Optimization, Truthful Models 1 Introduction Multimodal Large Language Models (MLLMs), by integrating pow- erful vision and language capabilities, are transforming human- computer interaction [2,3,19]. MLLMs also demonstrate astonish- ing potential in tasks like Visual Question Answering (VQA) and arXiv:2607.13712v1 [cs.CV] 15 Jul 2026 Conference’17, July 2017, Washington, DC, USAZhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao medical analysis [24,32,36]. However, despite these advancements, MLLMs suffer from the unfaithfulness problems, such as generat- ing content that contradicts visual facts (e.g., fabricated objects, attributes, or relationships) [5,10]. This significantly hinders their reliability and utility in real-world applications [18, 22]. To mitigate the above issues, preference learning based on human feedback has become a mainstream alignment paradigm. Among these methods, Direct Preference Optimization (DPO) [25] has been widely adopted because it directly optimizes the policy from prefer- ence data without requiring an explicit reward model. Existing DPO methods apply preference supervision at the final-answer-level to guide the model in generating more accurate and reliable outputs. However, multimodal reasoning often involves several stages, in- cluding early grounding and later reasoning, and the quality of the final answer depends on the robustness of the full process. As illus- trated in Fig. 1a, an error arising in an early grounding stage can propagate to later reasoning stages and lead to reasoning failure. Therefore, it remains unclear whether standard DPO, which mainly relies on holistic final-answer-level preferences, can adequately address reasoning errors rooted in early grounding stages. To this end, we conduct further controlled explorations. As shown in Fig. 1b, introducing errors into more grounding stages is associated with lower final reasoning accuracy, indicating that errors can propagate and accumulate across stages along the rea- soning process. This observation reveals a key structural limitation of final-answer-level preference optimization, namely the propa- gation of MLLM errors: later reasoning failures often do not arise solely at the answer-generation stage, but instead originate from early grounding stages and then accumulate along the reasoning process, affecting the final reasoning outcome. Yet standard DPO typically supervises only at the final-answer-level, offering holistic guidance while providing little stage-specific supervision for these early grounding stages. Consequently, an important question for MLLMs preference alignment is how to provide more targeted preference supervi- sion for early grounding stages, so as to reduce error propagation and improve faithful multimodal reasoning. To address these prob- lems, we propose Grounded Context Preference Optimization (Groc- PO), a grounded preference optimization framework for MLLMs. Built upon the newly constructed Grounded Context Preference Dataset (GCPD), Groc-PO leverages preference signals from three contextual stages: Object Grounding, Contextual Grounding, and Grounded Reasoning. This design provides more explicit supervi- sion for upstream grounded stages, thereby enhancing the model’s ability to robustly construct and faithfully utilize the grounded context. The framework incorporates the full context and adopts an adaptive stage-aware optimization strategy, improving complex reasoning while mitigating the propagation of upstream errors. Ex- tensive experiments demonstrate that Groc-PO yields notable gains in hallucination mitigation, contextual understanding, and faithful reasoning. The main contributions of this paper are as follows: •We propose the Groc-PO framework. By introducing explicit supervision over grounded pre-final stages, it improves faith- ful multimodal reasoning under multi-round contexts and effectively mitigates error propagation in MLLMs. •We construct the Grounded Context Preference Dataset (GCPD). GCPD organizes multi-stage preference data around Visual Grounding, Context Grounding, and Faithful Complex Rea- soning, providing support for explicit stage-wise supervision of grounded context. •Built on GCPD, Groc-PO employs an adaptive grounded preference optimization mechanism that dynamically allo- cates learning emphasis across different stages and sample complexities, enabling more targeted alignment. •We conduct systematic experiments across multiple datasets and benchmarks. The results show that Groc-PO consistently outperforms standard DPO and several strong baselines on hallucination and complex capability evaluations, validat- ing the effectiveness of explicit supervision for pre-final grounded stages. 2 Related Works 2.1 Unfaithfulness in MLLMs Unfaithfulness (or hallucination) in MLLMs refers to the generation of content inconsistent with the visual input, typically manifested as fabricated objects, incorrect attributes, or misinterpreted relation- ships [5]. Hallucinations stem from training data flaws [16]; module biases [9]; suboptimal training paradigms [6]; and inference-stage defects [12]. To address hallucinations, approaches fall into two categories: training-free methods (e.g., Opera [12], VCD [14]) and training-based techniques (e.g., RLHF, PPO) [26, 33]. 2.2 Preference Learning for faithful MLLMs Preference learning initially applied to LLM alignment via RLHF, but DPO has recently gained widespread adoption as a simpler and stable alternative. V-DPO [34] extends DPO by incorporating vi- sual context learning. POVID [38] creates a fine-grained dataset by injecting noise to texts and images. RLHF-V [35] collects segment- level human preference data and performs dense DPO. In addition, SPO-Task Planning [17] constructs preference pairs using curricu- lum learning to improve long-horizon planning. M-RLHF [37] constructed a preference dataset and proposed a novel reward model to achieve MLLM alignment. SPO [27] treats questioning and answering jointly as a policy trajectory, co-optimizing them via a structured reward function to enhance the model’s consideration of visual dependency in dialogue. The mrDPO [29] utilizes multi- round DPO and Rebirth Tuning to optimize audio-visual LLMs. 3 Methodology The overview of our Groc-PO framework is illustrated in Fig. 2. We first introduce the DPO, followed by the novel Grounded Context Preference Dataset (GCPD) and its generation pipeline, and the adaptive Groc-PO Loss. 3.1 Preliminaries: Direct Preference Optimization DPO directly optimizes the model through a contrastive learning objective, making it more inclined to generate human-preferred responses while reducing the probability of generating dispreferred responses. DPO learns from preference data(푥,푦 + ,푦 − ) ∼D, where Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMsConference’17, July 2017, Washington, DC, USA Stage 3: Grounded Reasoning Stage 2: Contextual Grounding Stage 1: Object Grounding Prompt: + Query: List each entity and its key attributes. Chosen: a boy, a skateboard , a man, a dog (yellow)... Rejected: a boy, a scooter, a man, a cat (yellow) ... Prompt: + Stage 1 (Query +Chosen) + Query: What sport is the boy doing? Chosen: He is playing skateboarding. Rejected: He is performing roller blasting. Prompt: + Stage 1 (Query + Chosen) + Stage 2 (Query + Chosen) + Query: If he loses balance, who is most likely to help first and why? Chosen: The man is closest and watching him, so he can react fastest. Rejected: The dog will jump to help him because it is trained to assist. Stage-1 Chosen Stage-1 Rejected Hardness- aware Loss 1 2 3 Pipeline Grounded Context Preference Dataset (GCPD) Groc-PO Training Stage-2 Chosen Stage-3 Chosen Stage-2 Rejected Stage-3 Rejected Teacher MLLM-Self + Query Model-Centric Verify Human (1) Random 10% of All (2) Flagged by Self-Correction Iterative Correction Human-in-the-Loop Verification Stage- aware Loss Focus: + Focus: + + Focus Stage Teacher’s Drafting + Model-Centric Sampling Rejected Sample Chosen Sample (Draft) Chosen Sample (Draft) Chosen Sample (Revised) Implict Reward: �(�,�)= �log � � (�|�) � � (�|�) Figure 2: Overview of our Grounded Context Preference Optimization (Groc-PO) framework, including GCPD dataset construc- tion. The left panel shows dataset construction, where multi-stage preference pairs are generated through teacher-assisted drafting, model-centric sampling, iterative correction, and human verification. The middle panel presents three stages of grounded preference supervision: Stage 1 for object grounding, Stage 2 for contextual grounding, and Stage 3 for grounded reasoning. The right panel shows Groc-PO training, which jointly uses preference pairs from all three stages with a stage-aware, hardness-aware loss to improve context-dependent reasoning and mitigate cross-stage error propagation. 푥is the input prompt,푦 + is the human-preferred /chosen response, 푦 − is the dispreferred /rejected response, andD is the dataset. The DPO objective function assumes that the human preference probability푝 ∗ (푦 + ≻ 푦 − | 푥)can be modeled via a latent reward function푟 ∗ (푥,푦):푝 ∗ (푦 + ≻ 푦 − | 푥)= 휎(푟 ∗ (푥,푦 + )−푟 ∗ (푥,푦 − )). DPO further relates the reward function to the model’s policy휋 휃 and a reference policy휋 ref :푟 ∗ (푥,푦)= 훽(log(휋 휃 (푦 | 푥))−log(휋 ref (푦 | 푥))). where훽is a hyperparameter controlling the ratio between re- ward function and policy deviation. DPO’s loss can directly optimize MLLM to maximize the probability of generating푦 + and minimize generating푦 − . Let us define the log-likelihood ratio for the pre- ferred response as푟 + = log ( 휋 휃 (푦 + | 푥)/휋 ref (푦 + | 푥) ) and for the dispreferred response as푟 − = log ( 휋 휃 (푦 − | 푥)/휋 ref (푦 − | 푥) ) . Then the DPO loss function is defined as: L DPO =− log휎 훽(푟 + −푟 − ) .(1) By minimizing this loss function, the model휋 휃 is trained to increase the difference between the log-probabilities of푦 + and푦 − , It makes DPO simpler and demonstrates comparable or superior performance to RLHF. 3.2 Grounded Context Preference Dataset (GCPD) Failures in multimodal reasoning often do not emerge only at the final response, but can originate from imperfect grounding and accumulated inconsistencies in pre-final stages. To explicitly super- vise these upstream stages, we construct the Grounded Context Preference Dataset (GCPD), a structured preference dataset or- ganized around progressively accumulated grounded context. Specifically, GCPD is built as a 3-stage context-dependent pref- erence dataset. Its three stages move from basic visual grounding, to context-grounded understanding, and finally to faithful complex reasoning. This formulation places preference supervision not only on the final answer, but also on the grounded context that supports it, providing a more direct signal for reducing error propagation and improving multimodal faithfulness. Cumulative Multi-stage Context: For any stage푠, the prompt includes all historical context from stage 1 to푠−1, ensuring con- tinuous information flow. Let I denote the image, Q denote the question, A denote the answer, A + 푠 means the chosen response (human-preferred). Then, we will have the following prompt struc- ture for every stage: • Stage 1 (S1): Prompt=퐼,푄 1 • Stage 2 (S2): Prompt=퐼,푄 1 ,퐴 + 푠 1 ,푄 2 • Stage 3 (S3): Prompt=퐼,푄 1 ,퐴 + 푠 1 ,푄 2 ,퐴 + 푠 2 ,푄 3 Then, we detail the progressive stages design: •S1: Object Grounding. S1 marks the starting point of CoT process—identifying basic facts. We present the model with a standardized question (e.g., List each entity and its key attributes) to extract core visual elements. •S2: Contextual Grounding. S2 simulates the intermediate steps of CoT. Building upon S1, S2 focuses on tasks such as relationship description, comprehensive captioning, or visual question answering. It requires the model not only to identify individual entities but also to understand how they form a meaningful whole. Conference’17, July 2017, Washington, DC, USAZhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao •S3: Grounded Reasoning. S3 is the culmination of the CoT simulation. We pose complex questions that require integrat- ing the image with context from S1 and S2, and performing logical inference, intent prediction, or reasoning tasks. This compels the model to perform high-level cognition based on established, reliable context. In this way, our GCPD dataset is no longer fragmented question- answer pairs but offers a progressive learning process aligned with human cognitive laws. 3.3Pipeline of GCPD’s Generation and Features 3.3.1 Basic Workflow. Our pipeline begins with a widely recog- nized RLHF-V dataset [35], which contains 5,733 images along with human-annotated, high-quality preference pair for tasks like cap- tioning, relational description. Our goal is to generate a structured 3-stage context for each image. For any stage푠, the core tasks is to generate a specific question푄 푠 and a high-quality preference pair(푦 + 푠 ,푦 − 푠 ) , consisting of a chosen and a rejected response. This three-stage structure is designed with progressively increasing com- plexity, following a perception-understanding-reasoning path. For the Stage 1, we define a universal base questionQ 1 : “List each entity and its key attributes,” to establish a factual foundation. The chosen response,푦 + 푠 1 , is initially generated by an advanced teacher model; the rejected response,푦 − 푠 1 , is derived from푦 + 푠 1 via introduced rule-based deficiencies, retaining structural soundness while including factual inaccuracies. Subsequently,푦 + 푠 1 undergoes rigorous verification and refinement to ensure its accuracy. For the Stage 2, aiming for both efficiency and quality, we adopt the existing data from the RLHF-V dataset corresponding to each image, as its design philosophy aligns perfectly with our goals for this stage. This means that the question푄 2 , the chosen response푦 + 푠 2 , and the rejected response푦 − 푠 2 are all sourced from this high-quality, human-validated dataset. It guarantees the superiority of the data for relational description and understanding. In the Stage 3, our objective is to enhance model’s ability of complex thinking. We leverage an advanced teacher model to generate a new, more profound, and complex question,푄 3 , based on the context of the original image and the preceding dialogue (푄 1 ,푦 + 푠 1 ,푄 2 ,푦 + 푠 2 ). Following the pattern of the 1st stage, we then generate a high-quality chosen response,푦 + 푠 3 , via the teacher model combined with rigorous verification and refinement, and a rejected response,푦 − 푠 3 , from our LLaVA base model to complete the final preference pair. 3.3.2 Iterative Self-Correction for Chosen Samples (푦 + 푠 ). To maxi- mize the chosen response quality and factual accuracy, we introduce an iterative self-correction mechanism. •In Stage 1, the initial list of entities in the chosen response, generated by the advanced teacher model, is fed back into the teacher model with a detailed verification prompt. The teacher model is instructed to comprehensively check and provide a score. If the response contains hallucinations or the average score is too low, it triggers a rewrite by the advanced teacher model. •Similarly, in Stage 3, the complex reasoning answer in the chosen response is sent back for a second review. In this step, the teacher model acts as a"Critic", inspecting the reasoning chain for logical fallacies and ensuring it is fully grounded in the provided visual and textual context. This"generate-and-refine"closed-loop process significantly en- hances the quality of our chosen responses, providing the model with a clear and reliable learning target. 3.3.3 Model-Centric Sampling. To further improve data quality and make training more efficient, we draw inspiration from the on-policy concept in RLHF [7] and adopt a Model-Centric Sampling strategy. The core idea is to ensure that the distribution of training data, aligns as closely as possible with the generation distribution of our targeted fine-tuned model. This approach enables the model to directly confront and rectify its own predominant error patterns, making the fine-tuning process highly targeted. This strategy is reflected in two key aspects: •The rejected responses in Stage 3 (푦 − 푠 3 ) are generated by our target LLaVA model. Consequently, these samples are representative of the model’s intrinsic failure modes, par- ticularly in areas like long-range dependency, contextual understanding, and complex reasoning, which manifest as logical fallacies or cumulative hallucinations. In contrast to negative samples from a more capable, external model teacher model, which often suffer from a distribution mis- match in푦 − 푟 . These ’model-centric’ samples provide a highly targeted and valuable learning signal for DPO. •In Stage 2, a key characteristic of the RLHF-V dataset lies in its preference pair construction: the rejected responses (푦 − 푠 2 ) are generated by the MLLM family, while the chosen responses (푦 + 푠 2 ) are human-revised versions of these same rejected samples. This approach ensures high distributional and stylistic alignment with our target model, effectively forming a tailored"problem-solution"paradigm for its spe- cific weaknesses. This mechanism provides the highly valu- able and targeted learning signal that is the core rationale for our adoption of this dataset. It is worth noting that the Stage 1 responses are highly uni- form, making generation variance across models minimal, and thus Model-Centric Sampling has little impact. 3.3.4 Human-in-the-Loop Verification. To ensure the rigor and quality of our GCPD dataset, we introduced Human-in-the-Loop verification. The audit team consisted of three MLLM-familiar PhD students who adhered to a guideline for all checks and corrections. The samples reviewed included: first, a 10% random sample of the entire dataset; second, a targeted review of푦 + 푠 samples flagged by the teacher model Critic as having major issues during the"Iterative Self-Correction" process. Overall, approximately 12% (∼2k) of the푦 + 푠 samples were man- ually audited, leading to the revision or rewriting of nearly 2% of severely problematic samples (primarily in R3). The total time cost was approximately 57 hours per reviewer (30h for auditing, 27h for revision). This mechanism ensures the reliability and faithfulness of the dataset. 3.4 Customized Groc-PO Loss The standard DPO loss (Equation 1) treats all samples in the dataset equally. This uniform approach overlooks the inherent gradient of Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMsConference’17, July 2017, Washington, DC, USA Table 1: Performance comparison with leading methods on various hallucination and general benchmarks. On LLaVA-v1.5-7B, LLaVA-v1.5-13B [20], Groc-PO achieves significant leads on key faithfulness metrics (e.g., AMBER, M-Hal) and simultaneously enhances general abilities (e.g., LLAVA-Bench, SEED). Here, AMBER-Gene. refers to AMBER-Generation, and AMBER-Discri. denotes the AMBER-Discrimination. Methods M-Hal [28]AMBER-Gene. [31]AMBER-Discri. [31] LLaVA [21] SEED [15] Score↑Hal-Rate↓CHAIR↓Hal-Rate↓Cog↓Acc↑F1↑ LLaVA-1.5-7B [20]2.0161.47.836.44.271.774.365.666.1 + DPO [25]2.1458.35.727.32.671.382.169.166.4 + CSR [39]2.0560.45.425.52.673.276.168.965.9 + POVID [38]2.2655.25.726.93.071.974.768.266.1 + V-DPO [34]2.1656.05.627.32.7-81.6-- + RLHF-V [35]2.0260.45.526.32.574.878.568.066.1 + mDPO [30]2.3954.04.424.52.4---- + Groc-PO (Ours)2.7647.04.225.21.578.085.072.967.1 LLaVA-1.5-13B [20]2.3853.17.033.13.371.473.173.168.2 + DPO [25]2.4751.06.126.32.771.982.172.868.6 + RLHF-V [35]2.5052.16.325.12.179.282.376.768.2 + Groc-PO (Ours)2.8545.03.824.41.383.588.276.568.8 cognitive depth and sample difficulty within our GCPD dataset. To better leverage this rich and structured information, we propose a Groc-PO Loss, with a sample-level adaptive weight,푤 푖 , enabling the model to dynamically focus on samples that are more informative and have higher learning value. The Groc-PO loss is defined as: L Groc-PO =−E (푥 푖 ,푦 푤,푖 ,푦 푙,푖 )∼D [ 푤 푖 · log휎(푟 푖 (휃)) ] (2) where푟 푖 (휃)= 훽 log 휋 휃 (푦 + 푖 |푥 푖 ) 휋 ref (푦 + 푖 |푥 푖 ) − 훽 log 휋 휃 (푦 − 푖 |푥 푖 ) 휋 ref (푦 − 푖 |푥 푖 ) is the implicit re- ward difference for preference pair푖. The core innovation lies in the design of adaptive weight푤 푖 , which is composed of two multi- plicative components: 푤 푖 = 휆 푟(푖) ·훾 푖 (3) 3.4.1 Stage-aware Importance Weight (휆 푟 ). This weight is designed to reflect the learning value of different rounds. Later rounds repre- sent more complex tasks that demand stronger long-range depen- dency and comprehensive abilities. Thus, for any stage푠, we design 휆 푠 as a monotonically increasing function of푠to encourage model to focus on these advanced knowledge: 휆 푟 = 1+ 훼(푟 − 1)(4) where푟 ∈ 1,2,3is the dataset stage, and훼 ≥0 is a hyperparame- ter that controls the growth rate of stage importance. When훼>0, samples from the latter 2 stages are assigned a higher loss weight. 3.4.2 Hardness-aware Focusing Weight (훾 푖 ). This weight aims to make model focus more on"hard samples"that are difficult to distinguish. When the model can easily distinguish between푦 + and 푦 − , the sample is"easy"and has low learning value. Conversely, when model perceives two responses as having similar quality, the sample is"hard" and should be prioritized. We define훾 푖 as: 훾 푖 =(1− 휎(푟 푖 (휃))) 휂 (5) where휂 ≥0 is a focusing parameter. For휂>0, this term signif- icantly decreases the loss for well-distinguished samples (where 휎(푟 푖 ) →1), thereby directing the optimization process toward the most challenging pairs. Through this dual-weighting mechanism, our Groc-PO Loss adaptively evaluates the importance of each training sample, con- sidering both its role in the progressive rounds (via휆 푡 ) and sample’s learning value (via훾 푖 ). 4 Experiments 4.1 Datasets, Metrics and Implementation Details Training Data: Based on our GCPD dataset generation pipeline described in Section 3.2, we constructed the 3-stages preference dataset comprising 5,733 diverse images and 17,199 high-quality preference pairs. Evaluation Benchmarks: To comprehensively evaluate mod- els’ performance, we employ several widely used benchmarks: For faithfulness evaluation, AMBER [31] is a LLM-free benchmark for evaluating hallucinations, which has two components: (a) Dis- crimination: deciding whether a statement is correct; (b) Generation: describing for an image. M-Hal [28] evaluates response-level hal- lucination rate and informativeness. For general capability evalu- ation, LLaVA-Bench [21] is a benchmark spanning diverse scenar- ios. SEED-Bench [15] is a large-scale benchmark to assess model abilities likes visual understanding and reasoning. For complex tasks, some sub-tasks of benchmarks are adopted, such as LLaVA- Bench-complex reasoning [21], LLaVA-Bench-conversation [21], SEED- visual reasoning [15], MME-commonsense-reasoning [8], and multi- turn dialogue benchmark M-MT [1]. Implementation Details: Our experiments leverage the widely adopted LLaVA-v1.5-7B and 13B [20] and Qwen2.5-VL-7B [4] mod- els to evaluate scalability and effectiveness of our method. We employed LoRA [11] and AdamW optimizer [23]. Training was Conference’17, July 2017, Washington, DC, USAZhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao Table 2: Comparison with different preference optimization methods on Qwen2.5-VL-7B [4]. MethodMM-Hal Score (↑)M-Hal Rate (↓) Qwen2.5-VL-7B [4]3.550.40 + DPO [25]3.590.38 + POVID [38]3.730.37 + CSR [39]3.710.41 + Groc-PO (Ours)3.830.32 Table 3: Comparison of Groc-PO and DPO on complex tasks. Tasks cover reasoning (LLaVA-Bench-complex reason- ing, SEED-visual reasoning, MME-commonsense-reasoning), single-turn conversation (LLaVA-Bench-conversation), and multi-turn dialogue (M-MT). Groc-PO shows superiority. Models LLaVA- complex reason [21] LLaVA- conversation [21] SEED- reason [15] MME- reason(%) [8] M -MT [1] DPO56.962.074.338.31.88 Groc-PO82.567.276.551.82.55 (+45%)(+8%)(+3%)(+35%)(+36%) performed over 2 epochs with an effective batch size of 32. The teacher model we used is GPT-4o [13]. 4.2 Main Results On multiple mainstream faithfulness and general benchmarks, we conduct a comprehensive comparison of Groc-PO with a series of representative baselines, such as llava-v1.5-7B [20], DPO [25], CSR [39], POVID [38], RLHF-V [35], V-DPO [34], and mDPO [30]. On Faithfulness and General Abilities Evaluation. Table 1 shows that Groc-PO achieves leading performance across almost all key evaluation metrics. These evaluations cover different types of faithfulnesss and general ability tests. These demonstrate that through our progressive preference data and adaptive training framework, the model can not only significantly suppress halluci- nations but also improve comprehensive abilities Furthermore, Groc-PO shows its scalability by delivering consis- tent and substantial gains across models from 7B to 13B, validating its efficiency and broad applicability. In addition, to validate the scalability and generalizability of our framework, we applied Groc-PO to Qwen2.5VL-7B [4]. As detailed in Table 2, the model achieved improvements on the faithfulness test. On Complex Understanding and Reasoning. Table 3 demon- strates the particularly prominent superiority of Groc-PO in com- plex reasoning tasks. Across demanding benchmarks, including LLaVA-bench-complex, LLaVA-conversation, SEED-reasoning, and MME-Commonsense-reasoning, our model consistently outper- forms the baseline, achieving up to a 45% relative improvement. Furthermore, Groc-PO demonstrates superior conversational ca- pabilities on M-MT benchmark (multi-round dialogue), validating the effectiveness of its structured context. Table 4: Ablation study of the loss components and hyperpa- rameters (훼,휂) on M-Hal [28], as described in Section 3.4. Loss Setting훼 휂 M-Hal Score (↑) M-Hal Rate (↓) (A) Component Ablation DPO Loss002.2460.0 + Stage-aware0.2502.4655.0 + Difficulty-aware022.4257.0 + Groc-PO Loss0.2522.7647.0 (B) Sensitivity to 훼 w/ 훼= 0022.4257.0 w/ 훼= 0.250.2522.7647.0 w/ 훼= 0.50.522.4055.5 (C) Sensitivity to 휂 w/ 휂= 00.2502.4655.0 w/ 휂= 10.2512.5751.0 w/ 휂= 20.2522.7647.0 4.3 Ablation Study 4.3.1 Contribution of Groc-PO Loss Components. To verify the effectiveness of our Groc-PO Loss, we compared it against 3 variants: (1) DPO Loss: uses GCPD data with standard DPO loss; (2) Stage-aware Only; and (3) Difficulty-aware Only. Table 4 shows that the full Groc-PO Loss achieves the best performance. The individual components each provide significant gains. In addition, We also performed ablation studies on the key hy- perparameters of the loss,훼(Stage-aware) and휂(Hardness-aware). 4.3.2Effect of Model-Centric Sampling. We conduct an abla- tion of the data construction strategy, comparing a model trained solely on Teacher-generated preference data with one trained on our final dataset using"model-centric sampling". Figure 3a shows that the latter performs better, suggesting that learning from its own imperfect responses provides a closer data distribution and more targeted alignment signals, thereby improving self-alignment. 4.3.3Contribution of Each Stage. To quantify the contribution of each context stage, we established three independent training settings:S1,S2, andS3. These models were exclusively trained on data from their respective stages, lacking historical context inS2 and S3. Figure 3b revealed thatS2outperformedS1andS3. BecauseS3 relies on preceding information, the absence of context leads to misalignment and performance decline. This affirms the necessity of collaboration among three context rounds, asserting that optimal performance requires structured integration. 4.3.4Impact of Multi-stage Context Depth. To investigate the influence of contextual learning depth, we used three training settings: (1) using the 1st-round data (S1-Only); (2) using the data of 1st and 2nd rounds (S1+S2); and (3) using full data of 3 stages. Figure 3c shows the model performance monotonically improves with increasing context depth and complexity. This demonstrates that our designed"perception→understanding→reasoning" progressive learning path is indispensable for building model’s com- pressive capabilities and faithfulness. Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMsConference’17, July 2017, Washington, DC, USA BaseNot -Sampling Sampling 1.5 2.0 2.5 3.0 3.5 M-Hal Score (a) Model-Centric Sampling or Not M-Hal Score () Hal-Rate () S1S2S3 1.5 2.0 2.5 3.0 3.5 M-Hal Score (b) Contribution of each Round M-Hal Score () Hal-Rate () S1S1+ S2 S1+ S2+S3 1.5 2.0 2.5 3.0 3.5 M-Hal Score (c) Impact of Context Depth M-Hal Score () Hal-Rate () S3S1+ S3 S1+ S2+S3 1.5 2.0 2.5 3.0 3.5 M-Hal Score (d) Sensitivity to Context History M-Hal Score () Hal-Rate () 40 50 60 70 Hal-Rate(%) 40 50 60 70 Hal-Rate(%) 40 50 60 70 Hal-Rate(%) 40 50 60 70 Hal-Rate(%) Figure 3: Ablations and Analysis. (a) Effectiveness of Model-Centric Sampling or not. (b) Contribution of individual stages: S2 peaks while S3 degrades from misalignment without history. (c) Impact of context depth: monotonic improvement with progressive history. (d) Sensitivity to history length: performance on a fixed stage-3 query improves as context is added. Complex Question: What is the setting or environment in which the image takes place? Token log-prob of DPO & Groc-PO Diff. of log-prob Figure 4: Token-level log-probability (log-prob) comparison on chosen responses푦 + 푟 for complex tasks: Groc-PO vs. DPO. The curves (green:Groc-PO, brown:DPO) show log-prob as- signed to each token. The top yellow bar chart illustrates the difference (Groc-PO - DPO), which is almost entirely positive. This suggests Groc-PO exhibits higher internal confidence on complex tasks. Table 5: Comparison of training with and without grounded context. The"Flattened DPO"(the same 3-stage 17k dataset but without grounded context) shows lower faithfulness than Groc-PO (with grounded context). SettingMM-Hal Score(↑)Hal-Rate(↓) Flattened DPO2.1761.0 Full Structured Context2.7647.0 4.4 Analysis and Discussion 4.4.1Dependence on Structured Context. To verify the impor- tance of structured context history, we designed a baseline named Table 6: Training Overhead Comparison. The average per- sample processing time and peak memory usage in DPO and Groc-PO, indicating that Groc-PO adds only minor overhead. ModelAvg. time (s) Peak Memory Usage (MB) DPO1.9940420 Groc-PO (Ours)2.05 (+3%)40432 (+0.03%) Player 10's kicking motion suggest that he intends to pass the ball to a teammate, and his action may be influenced by player 7's passive stance, which could potentially block the passing lane. Player 10's kicking motion suggest that he intends to attack aggressively, while player 7 remains without active running or offering a passing option, potentially affecting the success of player 10's action. Complex Question: What does player 10's raised arm and kicking motion reveal about his intended play, and how might player 7's passive stance influence the success of that action? The cows' calm and relaxed state, along with the lush green field, suggest that the farm's management practices are likely responsible , which can positively impact the herd's future well-being on the farm. The cows' calm and relaxed state, along with the lush field, suggest that the farm's management practices prioritize the well- being of the herd, which can bring the farm healthier and higher-yielding products. Complex Question: What does the cows' calm, relaxed state and the lush field suggest about the farm's management practices, and how might this impact the farm’s future? With a certain degree of errors Evidence-based, well-founded, and coherent With a certain degree of errors Evidence-based, well-founded, and coherent DPO Groc- PO DPO Groc- PO Figure 5: Comparison on complex tasks: Groc-PO vs. DPO. While DPO shows certain degree of errors, Groc-PO demon- strates robust, evidence-based, and coherent reasoning. "Flattened DPO", which uses the same 17k preference pairs but removes all context history, degrading all training data to single- stage question-answer pairs. Table 5 shows the"Flattened DPO" is far below structured context Groc-PO. A likely reason is that many Stage 3 require information from the first two stages, and removing history has left the model misaligned. This result under- scores the necessity of incorporating structured context to cultivate contextual coherence and enhance faithfulness. 4.4.2Sensitivity to History Length. We evaluated the model’s ability to leverage contextual history of varying lengths by testing Conference’17, July 2017, Washington, DC, USAZhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao the same third-stage question (S3) under three conditions: Zero- shot (image + S3), 1-stage context (image + S1 + S3), and 2-stage context (full history: image + S1 + S2 + S3). Figure 3d shows that performance exhibited a clear improvement: 2-stage>1-stage> Zero-shot . This empirically validates the critical role of multi-stage context in multimodal dialogue, demonstrating that more complete history enables the model to better localize the question, perform logical reasoning, and generate accurate responses. 4.4.3Training Overhead. We compared the training overhead of Groc-PO with DPO. Table 6 reports the average processing time per sample and peak memory usage. Results show that Groc-PO introduces only marginal overhead. 4.4.4Discussion: Effectiveness of Grounded Context Super- vision. Our analysis highlights the effectiveness of grounded con- text supervision. As shown in Figure 3c, model performance consis- tently improves as more complete grounded context is incorporated during training. This suggests that more targeted and stage-specific supervision on early grounding stages may improve the reliability of later-stage reasoning. Notably, the improvement is most evident on complex reasoning tasks. As reported in Table 3, the full Groc-PO achieves clear gains on these tasks. At the same time, it maintains strong performance on general and faithfulness benchmarks. These results indicate that grounded context supervision improves rea- soning quality, is beneficial for challenging multimodal tasks, and maintains competitive performance on general and faithfulness benchmarks. 4.5 Case Study and Visualization 4.5.1 Token-level Log-Probabilities on Chosen Responses of Complex Tasks: Groc-PO vs. DPO. To further investigate Groc-PO’s generation confidence for complex tasks, we conducted a case study for token-level log-probability (log-prob) comparison. In Figure 4, we observe that Groc-PO (green curve) assigns a higher log-prob to the majority of tokens in the chosen responses compared to DPO (brown curve), where the difference (yellow bars) is almost entirely positive. This indicates that Groc-PO exhibits higher internal confidence when handling complex tasks. 4.5.2Visualization of Qualitative Comparison. Figure 5 vi- sually confirms that while the DPO relatively fails on complex queries, Groc-PO generates logically accurate responses that are well-supported by explicit visual evidence. 5 Conclusion In this paper, we proposed Grounded Context Preference Optimiza- tion (Groc-PO), a framework that improves MLLM faithfulness through explicit preference supervision over grounded pre-final stages. To support this objective, we introduced the Grounded Con- text Preference Dataset (GCPD) and novel adaptive loss function. Extensive evaluations demonstrate that Groc-PO comprehensively enhances multiple capabilities, contributing to developing more faithful MLLMs. 6 Acknowledgements This work was supported by the Artificial Intelligence-National Science and Technology Major Project (2023ZD0121200) and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM103). References [1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al.2024. Pixtral 12B. arXiv preprint arXiv:2410.07073 (2024). [2]Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736. [3]Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al.2024. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477 (2024). [4]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025). [5] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024). [6]Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch- Elor. 2023. Mocha: Multi-objective reinforcement mitigating caption hallucina- tions. arXiv preprint arXiv:2312.03631 2 (2023). [7] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017). [8] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al.2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023). [9] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al.2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14375–14385. [10] Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing halluci- nations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18135–18143. [11]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3. [12]Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hal- lucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13418–13427. [13]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). [14]Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13872–13882. [15]Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). [16]Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023). [17]Xiwen Liang, Min Lin, Weiqi Ruan, Rongtao Xu, Yuecheng Liu, Jiaqi Chen, Bingqian Lin, Yuzheng Zhuang, and Xiaodan Liang. 2025. Structured preference optimization for vision-language long-horizon task planning. arXiv preprint arXiv:2502.20742 (2025). [18]Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. 2024. A Survey of Multimodel Large Language Models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering. 405–409. [19]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306. Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMsConference’17, July 2017, Washington, DC, USA [20]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruc- tion Tuning. [21]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [22]Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024). [23] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). [24]Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. 2024. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision. Springer, 382–398. [25]Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741. [26]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [27] Guohao Sun, Can Qin, Yihao Feng, Zeyuan Chen, Ran Xu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. 2025. Structured Policy Optimiza- tion: Enhance Large Vision-Language Model via Self-referenced Dialogue. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 741–751. [28]Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 (2023). [29] Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zujun Ma, and Chao Zhang. 2024. Enhancing multimodal LLM for detailed and accurate video captioning using multi-round preference optimization. arXiv preprint arXiv:2410.06682 (2024). [30]Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406.11839 (2024). [31]Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation. arXiv preprint arXiv:2311.07397 (2023). [32]Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2024. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. arXiv preprint arXiv:2404.14233 (2024). [33] Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2025. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25543–25551. [34]Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. 2024. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712 (2024). [35]Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13807–13816. [36]Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al.2024. Automated multi-level preference for mllms. Advances in Neural Information Processing Systems 37 (2024), 26171–26194. [37]Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al.2025. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391 (2025). [38]Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411 (2024). [39] Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chen- hang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 (2024).