Paper deep dive
Training-Free VLM Personalization via Calibrated Residual Decoding
Jiaao Yu, Yujian Ma, Xianming Hu, Pengran Wang, Ang Li
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model's generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model's default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile , a counterfactual profile , and an empty profile . Our method keeps the prediction under as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain.
Tags
Links
- Source: https://arxiv.org/abs/2608.22263v1
- Canonical: https://arxiv.org/abs/2608.22263v1
Trouble viewing inline? Open PDF directly →
Full Text
41,729 characters extracted from source content.
Expand or collapse full text
Training-Free VLM Personalization via Calibrated Residual Decoding Jiaao Yu Yujian Ma Xianming Hu Pengran Wang Ang Li Abstract Vision-language models can be personalized in a training-free manner by directly providing user profiles, preferences, or visual references at inference time, without updating model parameters. However, direct personalized prompting does not guarantee that the model will reliably exploit such evidence. The predictive distribution under the positive user profile often mixes two sources: personalized signals genuinely supported by the current profile, and the model’s generic visual or linguistic priors. As a result, from the positive-profile response alone, it is difficult to determine whether a high-confidence answer is supported by the user profile or merely reflects the model’s default preference. To address this problem, we propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive profile E+E^+, a counterfactual profile E−E^-, and an empty profile E0E^0. Our method keeps the prediction under E+E^+ as the anchored base, and explicitly estimates the marginal contribution of personalization from score differences across the three conditions. We further introduce normalized-entropy-based uncertainty calibration, allowing the strength of personalized enhancement to adapt to the reliability of the residual signal. Experiments on MMPB, YoLLaVA, and MyVLM show that the proposed method improves personalized multimodal understanding without fine-tuning, with consistent gains on identity-sensitive visual personalization tasks. Additional analysis shows that entropy calibration stabilizes residual decoding when the contrastive personalization signal is uncertain. Introduction Vision-language models (VLMs) have achieved remarkable progress in image understanding, visual question answering, and multimodal reasoning, enabling models to generate plausible responses conditioned on visual content and natural language instructions. However, real-world human-AI interaction is rarely fully generic: different users often have different preferences, background knowledge, long-term habits, and visual experiences. For the same image and the same question, the answer that matters to one user may depend on their personal preferences, identity cues, or existing profile. In scenarios such as personalized recommendation, companion assistants, adaptive shopping, personal object recognition, and long-term multimodal interaction, a model must not only understand what is present in the image, but also determine what is relevant to a particular user. Personalized multimodal understanding is therefore becoming an important capability for VLMs to evolve toward human-centered intelligent systems. Figure 1: Ambiguity of positive-profile prompting. The output distribution under the positive profile E+E^+ mixes profile-supported signals with the VLM’s generic prior. From E+E^+ alone, a high-scoring response may be difficult to attribute to either source. Recent studies have begun to explore personalized VLMs (34; 30; 2; 19; 6). Existing approaches typically rely on additional training, user memory modules, or curated personalized data to adapt models to user-specific preferences and concepts. For example, MyVLM (1), Yo’LLaVA (20), and PersonaVLM (21) inject personalization by learning user-specific concept representations, personalized latent tokens, or long-term memory alignment. Although these methods improve adaptation to specific users, they also introduce training cost, privacy concerns, and deployment complexity. In contrast, a lightweight alternative is to directly provide user profiles, historical preferences, or reference images as context to an existing multimodal model. Such prompting-based personalization does not require parameter updates and is therefore easier to deploy on top of frozen VLMs. However, directly supplying personalized context does not guarantee that the model will use it stably or appropriately. As shown in Figure 1, under the positive-profile condition, the resulting predictive distribution mixes two sources: personalized signals genuinely supported by the current user profile, and the model’s generic visual or linguistic priors. From the positive-profile response alone, we cannot determine whether a high-scoring answer is preferred because it is truly supported by the current user profile, or simply because the model tends to favor it regardless of personalization. As a result, absolute scores, confidence, or margins under the positive profile alone are insufficient to decide when personalized enhancement should be trusted. To reliably exploit personalization in a training-free setting, we need additional reference conditions that explicitly measure the marginal effect of user-specific evidence. This observation makes our setting related to, yet fundamentally different from, prior contrastive decoding methods such as Context-Aware Decoding (CAD) (26), Visual Contrastive Decoding (VCD) (16), and DoLa (5). These methods demonstrate that comparing outputs across different conditions can improve model behavior by suppressing hallucinations, reducing visual bias, or exposing shallow priors. However, they typically treat the removed condition as a noise source or bias source, with the goal of weakening unreliable evidence. Personalization presents a different scenario: the positive user profile is not a bias to be removed, but legitimate evidence that should be preserved. Our goal is therefore not to suppress personalization, but to calibrate how its effect should be incorporated into the final prediction while retaining the positive-profile response as the base. In other words, what we need is not de-personalization, but calibrated personalized enhancement. Motivated by this perspective, we revisit personalized multimodal decoding and propose a training-free calibrated residual decoding framework. Given the same image and question, we construct three evidence conditions: a positive user profile E+E^+, a counterfactual user profile E−E^-, and an empty profile E0E^0. Here, E+E^+ represents the current user’s true preferences or identity cues, E−E^- represents the preferences or identity cues of an alternative user, and E0E^0 denotes a generic condition without personalized evidence. By comparing the scores assigned to the same candidate response across these three evidence conditions, we construct a personalized residual signal while preserving the prediction under the positive profile as the anchored base. Furthermore, we introduce an uncertainty calibration mechanism based on normalized entropy, allowing the model to adaptively modulate the strength of personalization according to the reliability of the residual signal. This yields a calibrated response score that can be directly used for personalized decoding in VLMs. The main contributions of this work are as follows: • We propose a training-free personalized multimodal decoding framework that anchors on the positive user profile and explicitly estimates personalized residual signals from counterfactual and empty profiles. • We introduce an entropy-based uncertainty calibration mechanism that adaptively controls the strength of personalized enhancement according to the reliability of the residual signal. • We validate the proposed framework on multiple personalized multimodal benchmarks, showing that it improves personalized understanding without model fine-tuning and provides a unified inference-time view across text-profile and image-reference personalization. Figure 2: Our decoding framework estimates the marginal contribution of personalization via three evidence conditions (E+E^+,E−E^-,E0E^0), with entropy-based calibration controlling the residual strength. Related Work Personalized vision-language models. Personalization has been widely explored in generative and language models (15; 22; 33; 10; 8). Recent work has explored how to equip VLMs and MLLMs with user-specific knowledge. MyVLM (1) and Yo’LLaVA (20) learn personalized concept representations or latent visual tokens from user-provided examples, while PVIT (25) improves personalization through visual instruction tuning. To reduce the need for model adaptation, PLVM (24) studies tuning-free referential concept alignment, and R2P (7) retrieves user-specific fingerprints for personalized concept reasoning. RAP (11) further introduces retrieval-augmented personalization for MLLMs, and PersonaVLM (21) builds long-term multimodal memories for persistent user profiles. Among existing benchmarks, MMPB (13) evaluates profile-dependent preference and identity reasoning, while MyVLM and Yo’LLaVA focus on personalized visual recognition from user-provided reference images. Our work is complementary to these studies: while prior methods mainly address how personalized evidence is learned, retrieved, or maintained, we study how a frozen VLM should use already-provided evidence during decoding. Inference-time Decoding. Inference-time adaptation broadly includes prompting (31; 27), retrieval (17), and decoding-time control (3). Inference-time methods have been widely used to adapt frozen language and multimodal models without parameter updates, including in-context learning, retrieval augmentation, decoding-time control, and contrastive decoding. Our work is also related to contrastive decoding, which improves generation by comparing output distributions from different models, contexts, inputs, or internal states. Early methods such as Contrastive Decoding (17), DExperts (18), and related contrastive generation methods (28; 23) control generation by contrasting expert, weaker, or alternative distributions, while DoLa (5) contrasts predictions from different model layers. Closer to our setting, Context-Aware Decoding (26) and adaptive variants (29; 14) compare predictions with and without external context to reduce reliance on parametric priors. In LVLMs, Visual Contrastive Decoding (16) and hallucination-aware decoding methods (12; 4) mitigate hallucinations through input-level contrast or decoding-time calibration. Unlike these methods, which mainly use contrastive signals to suppress undesirable sources, personalized decoding must preserve the positive user profile as legitimate evidence. We therefore anchor decoding on the positive-profile prediction and use counterfactual and empty-profile conditions as references for estimating and calibrating the marginal contribution of personalized evidence. Method Motivating Observation Before introducing our decoding rule, we first examine a limitation of direct positive-profile prompting. A prediction made under the positive profile E+E^+ is not necessarily purely driven by personalized evidence; it may still follow the model’s non-personalized tendency observed under the empty profile E0E^0. To quantify this effect, we analyze two profile-dependent MMPB sub-tasks, preference inconsistency and recognition awareness. We define a prior-aligned error as an example where positive prompting is wrong and agrees with the empty-profile prediction, i.e., y^+=y^0≠y∗ y^+= y^0≠ y^*. As shown in Table 1, positive prompting makes errors on 24.09% of the combined subset. Among these errors, 84.26% are prior-aligned, suggesting that many positive-profile failures remain consistent with the model’s non-personalized preference. This motivates our use of reference evidence conditions: by comparing the positive profile E+E^+ with the empty profile E0E^0 and a counterfactual profile E−E^-, we can estimate the residual contribution of personalized evidence and use it to calibrate decoding. Subset #Samples E+E^+ Err. Prior-Aligned Pref. Incon. 1250 25.92 78.09 Rec. Aware 1598 22.65 89.78 Combined 2848 24.09 84.26 Table 1: Prior-aligned errors under positive-profile prompting on MMPB. All values except #Samples are percentages. Prior-Aligned denotes the share of E+E^+ errors where y^+=y^0≠y∗ y^+= y^0≠ y^*. Problem Formulation We study training-free personalization for vision-language models. Given an image x, a question q, and a candidate response space Y, the goal is to select the response y∈y that best matches both the visual input and the user’s personalized evidence. The candidate space can correspond to answer options in multiple-choice QA, or token continuations in open-ended decoding. In prompting-based personalization, the model is typically conditioned on a positive user profile E+E^+, which may contain textual preferences, identity cues, or visual references. A straightforward baseline scores each candidate response only under this positive profile. However, as discussed above, the positive-profile score alone mixes profile-supported signals with the model’s generic visual and linguistic priors. We therefore introduce two additional evidence conditions: a counterfactual profile E−E^- and an empty profile E0E^0. For each candidate response y, we compute: s+(y) s^+(y) =logpθ(y∣x,q,E+), = p_θ(y x,q,E^+), (1) s−(y) s^-(y) =logpθ(y∣x,q,E−), = p_θ(y x,q,E^-), s0(y) s^0(y) =logpθ(y∣x,q,E0), = p_θ(y x,q,E^0), where pθp_θ denotes the frozen VLM. Personalized Residual from Reference Conditions The reference conditions make the marginal effect of personalization observable. The counterfactual profile E−E^- asks whether the same response would also be supported under an alternative user’s evidence. The empty profile E0E^0 asks whether the same response would remain likely without any personalized evidence. If a candidate receives high support under E+E^+ but not under E−E^- or E0E^0, the difference indicates a profile-specific signal. We define a personalized residual: C(y)=α(s+(y)−s−(y))+β(s+(y)−s0(y)),C(y)=α (s^+(y)-s^-(y) )+β (s^+(y)-s^0(y) ), (2) where α and β control the relative strength of the counterfactual and empty-profile comparisons. The first term captures how much the target profile supports y relative to an alternative profile, while the second term captures how much the target profile supports y beyond the model’s non-personalized prior. This residual is not used to replace the positive-profile prediction. Instead, it estimates how the positive-profile score should be adjusted after comparing it with controlled reference conditions, keeping the decoding rule positive-anchored: E+E^+ remains the base evidence, while E−E^- and E0E^0 provide contrastive measurements of profile-specific support. Entropy-Calibrated Residual Decoding The residual signal can vary in reliability across instances. On some examples, the residual is sharply concentrated on one candidate, indicating a clear personalized preference. On others, it is diffuse, suggesting that the reference conditions do not provide a stable direction for personalization. Applying the same residual strength in both cases may over-amplify noisy contrasts. To calibrate the residual, we convert residual scores into a distribution over candidates: pC(y)=exp(C(y)/T)∑y′∈exp(C(y′)/T),p_C(y)= (C(y)/T) _y (C(y )/T), (3) where T is a temperature parameter. We then compute a normalized confidence coefficient: γ=1−H(pC)log||,γ=1- H(p_C) |Y|, (4) where H(⋅)H(·) denotes entropy. By construction, γ∈[0,1]γ∈[0,1]: a concentrated residual distribution yields low entropy and high confidence (γ→1γ→ 1), while a flat residual distribution yields high entropy and low confidence (γ→0γ→ 0). The final calibrated score is: S(y)=s+(y)+γC(y).S(y)=s^+(y)+γ\,C(y). (5) This formulation preserves the positive-profile score as the anchored base and adds a reliability-weighted residual enhancement. When the residual is confident, the model strengthens the profile-specific signal. When the residual is uncertain, the entropy coefficient reduces its influence and the method falls back toward positive-profile prompting. Instantiations The proposed decoding rule is instantiated differently for multiple-choice and open-ended QA. Multiple-choice QA. The candidate space Y is the set of answer options. We compute s+(y)s^+(y), s−(y)s^-(y), and s0(y)s^0(y) for each option y, construct the residual C(y)C(y), and obtain the calibrated score S(y)S(y). The final answer is y^=argmaxy∈S(y) y= _y S(y). Open-ended QA. We apply the same principle at the token level. At each decoding step t, let y<ty_<t denote the generated prefix and v a candidate next token. We first compute the per-token scores st(v)=logpθ(v∣x,q,E,y<t)s_t^~(v)= p_θ(v x,q,E^~,y_<t) under each evidence condition, and define the token-level residual Ct(v)C_t(v) analogously to Eq.(2). To compute the token-level confidence coefficient γt _t, we restrict the candidate token space to the top-K tokens under the positive-profile distribution, denoted VtV_t. This avoids diluting the entropy signal with the overwhelming noise of the full vocabulary. Concretely: γt=1−H(pC,t)logK, _t=1- H(p_C,t) K, (6) where pC,tp_C,t is the residual-induced distribution over VtV_t (Eq.(3) applied at token level) and K=24K=24 in all experiments. The calibrated next-token score is St(v)=st+(v)+γtCt(v)S_t(v)=s_t^+(v)+ _t\,C_t(v), and the next token is selected by argmaxv∈VtSt(v) _v∈ V_tS_t(v). Note that token-level residuals are inherently noisier than response-level ones, since a single token reflects far weaker personalization signal than a complete response. The entropy coefficient is therefore particularly important in this setting: it suppresses residual corrections at uncertain decoding steps, allowing generation to remain anchored to the positive-profile baseline when the token-level evidence is unreliable. Experiments Experimental Setup Benchmarks. We evaluate on three personalized multimodal benchmarks: MMPB (13) for text-profile-driven preference and identity reasoning, and YoLLaVA (20) and MyVLM (1) for image-reference-based personalized visual recognition. We report macro accuracy on MMPB and weighted accuracy on YoLLaVA and MyVLM. Baselines. We compare with four decoding baselines: No Profile, which uses the frozen model without personalized evidence (E0E^0); E+ Prompting, which directly prompts with the positive profile; and two single-reference residual variants, E0 Contrast with C(y)=s+(y)−s0(y)C(y)=s^+(y)-s^0(y) and E- Contrast with C(y)=s+(y)−s−(y)C(y)=s^+(y)-s^-(y). Prior personalized VLM systems often differ in how evidence is acquired, retrieved, or learned, so they are not direct decoding baselines under our setting; we provide a contextual comparison in Appendix D. Implementation details. We construct E−E^- from the same dataset: for text profiles, we invert preference semantics where possible and otherwise sample profiles from other users; for visual references, we select same-category images of different subjects. We use α=1.25α=1.25, β=1.25β=1.25, and T=1.25T=1.25 by default, and compute token-level confidence over the top-K=24K=24 tokens under E+E^+ for open-ended decoding. Prompt templates, counterfactual construction details, and hyperparameter sensitivity are provided in Appendix B & C. Model Method MMPB (Macro) YoLLaVA MyVLM Avg. Qwen2.5-VL-3B No Profile 49.82 42.05 50.00 47.29 E+E^+ Prompting 68.98 86.67 95.86 83.84 E0E^0 Contrast 69.68+0.70 89.23+2.56 96.13+0.27 85.01+1.17 E−E^- Contrast 70.77+1.79 95.90+9.23 98.62+2.76 88.43+4.59 Ours 70.76+1.78 97.44+10.77 99.38+3.52 89.19+5.35 Qwen2.5-VL-7B No Profile 48.80 52.31 50.69 50.60 E+E^+ Prompting 72.30 91.28 96.87 86.82 E0E^0 Contrast 72.71+0.41 91.79+0.51 96.91+0.04 87.14+0.32 E−E^- Contrast 73.89+1.59 97.95+6.67 98.18+1.31 90.01+3.19 Ours 72.91+0.61 98.97+7.69 98.83+1.96 90.24+3.42 Qwen3-VL-2B No Profile 48.93 35.90 51.12 45.32 E+E^+ Prompting 70.99 86.15 96.25 84.46 E0E^0 Contrast 71.25+0.26 89.23+3.08 97.12+0.87 85.87+1.41 E−E^- Contrast 72.72+1.73 97.95+11.80 96.36+0.11 89.01+4.55 Ours 73.03+2.04 98.46+12.31 97.04+0.79 89.51+5.05 InternVL3-2B No Profile 49.94 41.03 52.37 47.78 E+E^+ Prompting 71.02 51.28 79.15 67.15 E0E^0 Contrast 72.15+1.13 73.33+22.05 82.44+3.29 75.97+8.82 E−E^- Contrast 72.47+1.45 76.41+25.13 94.52+15.37 81.13+13.98 Ours 72.03+1.01 87.18+35.90 92.57+13.42 83.93+16.78 InternVL3-8B No Profile 50.50 47.18 49.86 49.18 E+E^+ Prompting 69.04 94.87 95.60 86.50 E0E^0 Contrast 72.32+3.28 94.36-0.51 98.95+3.35 88.54+2.04 E−E^- Contrast 70.91+1.87 98.97+4.10 97.94+2.34 89.27+2.77 Ours 71.28+2.24 98.46+3.59 99.44+3.84 89.73+3.23 Table 2: Overall performance across models and personalized multimodal benchmarks. “No Profile” denotes the no-personalization lower bound using the empty-profile condition. Green subscripts indicate absolute gains over E+E^+ Prompting. “E0E^0 Contrast” and “E−E^- Contrast” are simple single-reference residual baselines without entropy calibration. We report macro accuracy on MMPB and weighted accuracy on YoLLaVA and MyVLM. Avg. denotes the arithmetic mean over the three benchmarks. Main Results Table 2 reports the overall performance across three benchmarks and five backbones. Positive prompting provides a strong but incomplete baseline. Direct E+E^+ prompting already yields large gains over the no-profile model, confirming that current VLMs can exploit user profiles when explicitly provided. However, E+E^+ scores conflate personalized evidence with generic model priors, and the resulting improvement is uneven: on preference-oriented tasks (Table 3), E+E^+ prompting remains weak (e.g., only 32.85 on Pref. Aware), indicating that profile-supported and prior-driven responses are not yet well separated. Counterfactual contrast is a stronger reference than empty-profile contrast. Comparing the two single-reference baselines, E−E^- Contrast consistently and substantially outperforms E0E^0 Contrast across nearly all model–benchmark pairs. For instance, on YoLLaVA with Qwen2.5-VL-3B, E−E^- Contrast achieves 95.90 while E0E^0 Contrast achieves only 89.23 (+6.67). This confirms that contrasting against a concrete alternative user (E−E^-) exposes personalized evidence more effectively than contrasting against the mere absence of evidence (E0E^0), supporting our design of jointly using both references. The two references provide complementary signals. While E0E^0 exposes the model’s generic prior in the absence of personalization, E−E^- tests whether the same response would remain preferred for an alternative user. The stronger performance of E−E^- Contrast suggests that counterfactual evidence is particularly useful for isolating user-specific signals, and we further analyze this effect in Figure 3. Entropy-calibrated decoding provides the most reliable performance. Our full method achieves competitive performance on identity-sensitive visual tasks: on YoLLaVA, it outperforms E+E^+ prompting by +3.59 to +35.90 across models; on MyVLM, it reaches 99% on two of the five models. On the text-profile benchmark MMPB, our method closely tracks or matches E−E^- Contrast, with the entropy coefficient providing conservative modulation when the residual is diffuse. Fine-grained analysis (Table 3) shows the largest gains on preference-oriented sub-tasks: Pref. Aware improves from 32.85 to 43.00 (+10.15), and Pref. Incon. from 73.90 to 90.00 (+16.10). On recognition-oriented sub-tasks, gains are more moderate but consistent. Taken together, the results confirm that training-free personalization benefits from two complementary ingredients: anchoring on the positive-profile response, and adaptively calibrating the residual through contrastive evidence and uncertainty-aware modulation. Method P.A. P.I. R.A. R.I. E+E^+ Prompting 32.85 73.90 76.80 81.95 E0E^0 Contrast 40.90 83.50 81.08 79.40 E−E^- Contrast 39.65 88.00 94.29 80.90 Ours 43.00 90.00 94.22 79.55 Table 3: Fine-grained results on representative MMPB sub-tasks. Pref.Aware(P.A.) and Pref.Incon.(P.I.) evaluate preference-oriented personalization, while Rec.Aware(R.A.) and Rec.Incon.(R.I.) evaluate personalized visual recognition under user-specific context. Method Accuracy Δ vs Base Base 84.84 0.00 w/o Entropy 75.23 -9.61 Ours 81.59 -3.25 Table 4: Robustness on the public MME benchmark under irrelevant textual personalization context. Base denotes the original model without any user profile. Ours w/o Entropy removes entropy-based confidence calibration, while Ours uses the full calibrated residual decoding rule. Method MMPB (Macro) YoLLaVA E+E^+ Prompting 68.98 86.67 w/o Entropy 70.35 95.39 Ours 70.76 97.44 Table 5: Ablation of entropy-based confidence calibration on personalized benchmarks. Ours w/o Entropy removes confidence calibration and directly applies residual personalization, while Ours uses the full entropy-calibrated decoding rule. Figure 3: Residual evidence analysis. Each point corresponds to an error made by direct E+E^+ prompting; blue points are corrected by our method, while red points remain incorrect. Panel (a) compares the residual margins provided by the empty-profile reference E0E^0 and the counterfactual reference E−E^-, where larger values indicate stronger support for the gold answer over the original E+E^+ prediction. Panel (b) relates entropy confidence γ to the residual margin for the gold answer, showing whether the residual is both reliable and directed toward the correct response. Point size in panel (a) indicates γ. Ablation Study We ablate the reference evidence conditions and entropy-based confidence calibration. As shown in Table 2, E−E^- Contrast consistently outperforms E0E^0 Contrast across most model–benchmark pairs, suggesting that a concrete counterfactual profile provides a stronger discriminative signal than the absence of personalization, while E0E^0 still anchors the residual against generic priors. For entropy calibration, we evaluate two complementary settings. On the public MME benchmark with an irrelevant textual profile prepended to each sample (9), uncalibrated residual decoding degrades the base model by −9.60-9.60 (Table 4), showing that noisy profile evidence can be over-amplified. The full method reduces this drop to −3.25-3.25. On personalized benchmarks (Table 5), entropy calibration further improves or preserves the gains of residual decoding. Together, these results indicate that entropy calibration makes the residual reliability-aware rather than uniformly amplified. Component Analysis Beyond aggregate ablations, we further analyze how the reference conditions and entropy confidence affect individual predictions. For each error made by direct E+E^+ prompting, we compute reference residual margins Δ0=C0(y∗)−C0(y^+) _0=C_0(y^*)-C_0( y^+) and Δ−=C−(y∗)−C−(y^+) _-=C_-(y^*)-C_-( y^+), where C0=s+−s0C_0=s^+-s^0 and C−=s+−s−C_-=s^+-s^-. Here, E0E^0 measures how much the model favors a response without personalized evidence, while E−E^- tests whether the same response remains preferred under an alternative user profile. As shown in Figure 3(a), corrected errors often have large positive Δ− _-, suggesting that the counterfactual reference provides discriminative user-specific evidence beyond the generic prior exposed by E0E^0. The cluster of corrected samples in the upper-right region further indicates that the two references can provide complementary support, yielding a more stable residual signal when they agree on the correction direction. Figure 3(b) further shows that corrected errors are concentrated in the upper-right region, where high entropy confidence coincides with a positive residual margin for the gold answer. This suggests that the entropy coefficient provides a useful proxy for the reliability of residual-based correction. Open-Ended Personalized QA Subset Win Tie Avg. Score (↑ ) Overall 63.0 18.0 3.54 → 4.70 Text-profile QA 45.0 30.0 3.80 → 4.38 Image-profile QA 75.0 10.0 3.37 → 4.92 Table 6: LLM-judge evaluation on the 100-example open-ended personalized QA benchmark using Qwen2.5-VL-3B. We compare standard positive prompting against our token-level calibrated residual decoding. Win/Tie denotes the percentage of cases where the judge prefers our method, considers the two responses tied. Avg. Score reports the mean overall judge score on a 0–6 scale. Extension to Open-ended Personalized Generation Although our main experiments focus on closed-set personalized reasoning, the proposed decoding framework can also be applied to open-ended personalized generation. To examine this extension, we evaluate token-level calibrated residual decoding on an open-ended personalized QA benchmark. Evaluation setup. We construct a 100-example open-ended personalized QA benchmark from public datasets, with 40 text-profile examples from MMPB and 60 image-profile examples from YoLLaVA and MyVLM. We compare direct Positive Prompting with Ours (Token-level), which applies calibrated residual decoding at each generation step. An external LLM judge (32) compares response pairs along factuality, personalization, and counterfactual robustness, each scored from 0 to 2. We report win/tie rates and average overall scores. Open-ended results. As shown in Table 6, our method achieves a 63.0% overall win rate and improves the average score from 3.54 to 4.70. The gain is strongest on image-profile QA (75.0% win rate, 3.37→ 4.92), where personalization is grounded in identity-sensitive visual evidence. Text-profile QA shows a smaller but positive gain (45.0% win rate, 3.80→ 4.38), likely because long textual preference descriptions introduce greater ambiguity in free-form generation. Figure 4: Qualitative examples of token-level personalized decoding. The calibrated residual shifts the first-token decision toward the profile-supported answer and assigns higher confidence to identity-specific tokens during full-answer generation. Qualitative analysis. Figure 4 illustrates how calibrated residual decoding affects open-ended generation at the token level. In the first-token decision, our method can shift the model away from the incorrect direct E+E^+ prompting response toward the profile-supported answer. During full-answer generation, the modulation is selective: low-confidence tokens such as punctuation are only weakly adjusted, while identity-specific tokens receive higher residual confidence and larger probability shifts. This suggests that the proposed decoding rule improves open-ended personalization by strengthening reliable user-specific evidence rather than uniformly amplifying all generated tokens. More details on the open-ended QA construction, judge prompt, dimension-level scores, and additional qualitative cases are included in Appendix D & E. Conclusion We proposed a training-free calibrated residual decoding framework for personalized VLMs, addressing the ambiguity that direct positive-profile prompting can mix genuine user-specific evidence with generic model priors. Our method anchors decoding on the positive-profile prediction, estimates the marginal contribution of personalized evidence using counterfactual and empty-profile references, and calibrates the residual strength with entropy-based confidence. Experiments across text-profile, image-reference, and open-ended personalized QA settings show consistent gains without model fine-tuning. A limitation is the additional inference cost from evaluating multiple evidence conditions, especially in token-level open-ended generation. Future work may reduce this overhead through caching, candidate pruning, or selectively applying residual decoding at key decision steps. References Alaluf et al. (2024) Y. Alaluf, E. Richardson, S. Tulyakov, K. Aberman, and D. Cohen-Or Myvlm: personalizing vlms for user-specific queries. In European Conference on Computer Vision, p. 73–91. Cited by: Introduction, Personalized vision-language models., Benchmarks.. Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: Introduction. Chen et al. (2023) C. Chen, S. Borgeaud, G. Irving, J. Lespiau, L. Sifre, and J. Jumper Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: Inference-time Decoding.. Chen et al. (2024) Z. Chen, Z. Zhao, H. Luo, H. Yao, B. Li, and J. Zhou Halc: object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425. Cited by: Inference-time Decoding.. Chuang et al. (2024) Y. Chuang, Y. Xie, H. Luo, Y. Kim, J. R. Glass, and P. He Dola: decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, Vol. 2024, p. 54158–54183. Cited by: Introduction, Inference-time Decoding.. Dai et al. (2023) W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi Instructblip: towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, p. 49250–49267. Cited by: Introduction. Das et al. (2025) D. Das, D. Talon, Y. Wang, M. Mancini, and E. Ricci Training-free personalization via retrieval and reasoning on fingerprints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9683–9692. Cited by: Personalized vision-language models.. Ding et al. (2023) Z. Ding, X. Zhang, Z. Xia, L. Jebe, Z. Tu, and X. Zhang Diffusionrig: learning personalized priors for facial appearance editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12736–12746. Cited by: Personalized vision-language models.. Fu et al. (2026) C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme: a comprehensive evaluation benchmark for multimodal large language models. Advances in Neural Information Processing Systems 38. Cited by: Ablation Study. Guo et al. (2023) Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai Animatediff: animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725. Cited by: Personalized vision-language models.. Hao et al. (2025) H. Hao, J. Han, C. Li, Y. Li, and X. Yue Rap: retrieval-augmented personalization for multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 14538–14548. Cited by: Personalized vision-language models.. Huang et al. (2024) Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu Opera: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13418–13427. Cited by: Inference-time Decoding.. Kim et al. (2026) J. Kim, W. Kim, W. Park, and J. Do Mmpb: it’s time for multi-modal personalization. Advances in Neural Information Processing Systems 38. Cited by: Personalized vision-language models., Benchmarks.. Kim et al. (2024) Y. Kim, H. J. Kim, C. Park, C. Park, H. Cho, J. Kim, K. M. Yoo, S. Lee, and T. Kim Adaptive contrastive decoding in retrieval-augmented generation for handling noisy contexts. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 2421–2431. Cited by: Inference-time Decoding.. Kumari et al. (2023) N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J. Zhu Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1931–1941. Cited by: Personalized vision-language models.. Leng et al. (2024) S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13872–13882. Cited by: Introduction, Inference-time Decoding.. Li et al. (2023) X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. B. Hashimoto, L. Zettlemoyer, and M. Lewis Contrastive decoding: open-ended text generation as optimization. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), p. 12286–12312. Cited by: Inference-time Decoding.. Liu et al. (2021) A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi DExperts: decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 6691–6706. Cited by: Inference-time Decoding.. Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning. Advances in neural information processing systems 36, p. 34892–34916. Cited by: Introduction. Nguyen et al. (2024) T. Nguyen, H. Liu, Y. Li, M. Cai, U. Ojha, and Y. J. Lee Yo’llava: your personalized language and vision assistant. Advances in Neural Information Processing Systems 37, p. 40913–40951. Cited by: Introduction, Personalized vision-language models., Benchmarks.. Nie et al. (2026) C. Nie, C. Fu, Y. Zhang, H. Yang, and C. Shan Personavlm: long-term personalized multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15000–15009. Cited by: Introduction, Personalized vision-language models.. Ning et al. (2025) L. Ning, L. Liu, J. Wu, N. Wu, D. Berlowitz, S. Prakash, B. Green, S. O’Banion, and J. Xie User-llm: efficient llm contextualization with user embeddings. In Companion Proceedings of the ACM on Web Conference 2025, p. 1219–1223. Cited by: Personalized vision-language models.. O’Brien and Lewis (2023) S. O’Brien and M. Lewis Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117. Cited by: Inference-time Decoding.. Pham et al. (2025) C. Pham, H. Phan, D. Doermann, and Y. Tian PLVM: a tuning-free approach for personalized large vision-language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 3632–3641. Cited by: Personalized vision-language models.. Pi et al. (2025) R. Pi, J. Zhang, T. Han, J. Zhang, R. Pan, and T. Zhang Personalized visual instruction tuning. In International Conference on Learning Representations, Vol. 2025, p. 43247–43266. Cited by: Personalized vision-language models.. Shi et al. (2024) W. Shi, X. Han, M. Lewis, Y. Tsvetkov, L. Zettlemoyer, and W. Yih Trusting your evidence: hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), p. 783–791. Cited by: Introduction, Inference-time Decoding.. Song et al. (2023) C. H. Song, J. Wu, C. Washington, B. M. Sadler, W. Chao, and Y. Su Llm-planner: few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2998–3009. Cited by: Inference-time Decoding.. Su et al. (2022) Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, and N. Collier A contrastive framework for neural text generation. Advances in Neural Information Processing Systems 35, p. 21548–21561. Cited by: Inference-time Decoding.. Wang et al. (2025) H. Wang, A. Prasad, E. Stengel-Eskin, and M. Bansal Adacad: adaptively decoding to balance conflicts between contextual and parametric knowledge. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 11636–11652. Cited by: Inference-time Decoding.. Wang et al. (2024) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: Introduction. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, p. 24824–24837. Cited by: Inference-time Decoding.. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Evaluation setup.. Zhang et al. (2024) K. Zhang, Y. Kim, and X. Liu Personalized llm response generation with parameterized memory injection. arXiv preprint arXiv:2404.03565. Cited by: Personalized vision-language models.. Zhu et al. (2025) J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. Internvl3: exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479. Cited by: Introduction.