Paper deep dive

Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking

Timothee Mickus, Claudio Savelli, Eduardo Calò, Emilio Raimond, Stella Frank, Hengyu Luo, Flavio Giobergia, Vincent Segonne, Chuyuan Li, Aman Sinha, Lorenzo Vaiani, Jörg Tiedemann, Raúl Vázquez

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 104

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In an age of rapid model turnover, how do we make hallucination evaluation more perennial? We explore whether human-written hallucination samples could take the place of model-generated hallucinations, in order to make benchmarking detection independent of particular models. To this end, we construct a dataset of 1,600 human-written samples, spanning four languages (Chinese, English, French, Italian), and 18,400 samples from five vision-and-language models, all annotated for hallucinations using a fine-grained span-level labeling scheme. We find that human-written samples result in higher agreement and allow greater control of dataset contents, while remaining distributionally similar to samples derived from vision-and-language samples and providing a reasonable portrayal of detection capabilities - suggesting that human data is a viable substitute for model-based hallucination benchmarks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

102,884 characters extracted from source content.

Expand or collapse full text

Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking Timothee Mickus Claudio Savelli Eduardo Calò Emilio Raimond Stella Frank Hengyu Luo Flavio Giobergia Vincent Segonne Chuyuan Li Aman Sinha Lorenzo Vaiani Jörg Tiedemann Raúl Vázquez University of Helsinki Politecnico di Torino Universiteit Utrecht Université Bretagne Sud University of Copenhagen University Grenoble Alpes University of Lorraine Abstract In an age of rapid model turnover, how do we make hallucination evaluation more perennial? We explore whether human-written hallucination samples could take the place of model-generated hallucinations, in order to make benchmarking detection independent of particular models. To this end, we construct SHEEP, a dataset of 1,600 human-written samples, spanning four languages (Chinese, English, French, Italian), and 18,400 samples from five vision-and-language models, all annotated for hallucinations using a fine-grained span-level labeling scheme. We find that human-written samples result in higher agreement and allow greater control of dataset contents, while remaining distributionally similar to samples derived from vision-and-language samples and providing a reasonable portrayal of detection capabilities — suggesting that human data is a viable substitute for model-based hallucination benchmarks. Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking Timothee Mickus Claudio Savelli Eduardo Calò Emilio Raimond Stella Frank Hengyu Luo Flavio Giobergia Vincent Segonne Chuyuan Li Aman Sinha Lorenzo Vaiani Jörg Tiedemann Raúl Vázquez University of Helsinki Politecnico di Torino Universiteit Utrecht Université Bretagne Sud University of Copenhagen University Grenoble Alpes University of Lorraine 1 Introduction Reliably detecting LLM hallucinations — outputs that seem plausible, look fluent, but are ultimately factually incorrect (e.g., Rawte et al., 2023; Huang et al., 2025) — is an urgent problem for NLP. This is even more problematic for visually-grounded language models: they have specific hallucination propensities related to image misunderstanding, along with the pathologies inherited from their language backbones (Liu et al., 2024a). A key part of the solution is developing accurate and robust benchmarks and evaluation practices (e.g., Vazquez et al., 2025; Niu et al., 2024). Here, a fundamental challenge is the difficulty of developing hallucination detection benchmarks that are independent of the specific models used to generate the benchmark data. Indeed, most hallucination benchmarks are deeply dependent on specific models, either as sources of faulty outputs to be annotated (e.g., Ravichander et al., 2025; Gamba et al., 2026), or as tools to produce synthetic data that emulates the phenomenon (e.g., Muhlgay et al., 2024). As a result, these benchmarks make it difficult to discern whether good detection performance is due to correct understanding of hallucinations or simply to overfitting to idiosyncrasies of a specific model. The problem is compounded by the rapid pace of development of modern NLP: a dataset compiled using a popular model may fail to represent contemporary systems by the time of publication, rendering its findings potentially obsolete (e.g., Laskar et al., 2023; Liu et al., 2025). Beyond this first set of issues, the construction of hallucination benchmarks itself is rife with practical hurdles. Randomly sampling outputs from a foundation model is representative of model behavior, but results in sparse representation of rare hallucination types, which may cause problems for evaluating a detection method’s ability to identify these hallucinations. More targeted sampling using a LLM-judge for preselecting items, while it may allow us to balance hallucination types somewhat, ultimately depends on the judge’s accuracy, making the effectiveness of such setups hard to predict. In this work, we ask whether human-generated hallucinations could be a solution to these two sets of problems. Human-written samples could not only allow us to precisely control the prevalence of hallucinations of any particular type, but also bypass the reliance on specific models and shelter benchmarks from obsolescence. We test this hypothesis in the context of large vision-and-language models (LVLMs), and propose the SHEEP dataset: a Set for Human-written and Electronic Erroneous Productions. We define a set of hallucination types that are caused by mistakes in visual grounding, such as miscounting objects or misreading text in the image, and compare hallucination rates and hallucination type distributions in randomly sampled data, items preselected by a LVLM-judge, as well as hallucinations manually invented by humans. SHEEP spans a total of 20,000 samples equally balanced over four languages (Chinese, English, French, Italian) labeled at a span-level by three annotators. Through careful analysis of the collected data, we find that human-written samples yield higher inter-annotator agreement and more controllable dataset contents, while maintaining a distributional profile similar to LLM-derived samples and providing a reasonable assessment of the performances of detection tools on LVLM data. These observations validate that human-written items are a viable substitute for LVLM-based hallucination benchmarks. Our contributions are: 1. SHEEP, a multilingual dataset containing 20,000 model and human outputs; 2. Fine-grained, multi-annotator annotations at the character-level, differentiating between five possible hallucination labels; 3. Analyses demonstrating the viability of human-written hallucinations for hallucination-detection evaluation. SHEEP is available under a C-BY-NC license at helsinki-nlp.github.io/shroom/2026. 2 Related works Hallucinations in LVLMs. Hallucinations in vision-and-language models are generally defined as content that is not supported or contradicts the visual input (Liu et al., 2024a; Bai et al., 2025). Early works organized the phenomenon on a coarse triad of object, attribute, and relation errors (Liu et al., 2024a). Later proposals refine this scheme in multiple directions: Jiang et al. (2024) add event hallucinations to account for entirely fabricated scenes, while Rani et al. (2024) distinguish up to eight fine-grained classes, including miscounting, misreading of visible text, and identity incongruity. Our own classification (Section˜3) draws from this line of work, but is deliberately kept compact so that it can be applied reliably at the span level. LVLM hallucination benchmarks. Existing benchmarks for LVLM hallucination broadly fall into three paradigms (Liu et al., 2024a). 1. Caption-centric approaches assess the factual accuracy of model-generated descriptions or answers. Metrics such as CHAIR (Rohrbach et al., 2018) and its open-vocabulary extension (Petryk et al., 2024) remain restricted to object-level errors. 2. Discriminative approaches reframe evaluation as a classification task to detect preexisting hallucinations in image-text pairs. Shekhar et al.’s (2017) seminal work inserted single-word substitutions into the gold captions to produce a benchmark. Li et al. (2023) established the standard for object hallucination detection via yes/no probing about object presence. Recent variants have improved reliability by addressing dataset biases and devising novel consensus strategies (Lovenia et al., 2024; Pham and Schott, 2024). 3. Hybrid approaches integrate both generative and verification tasks. As such, this category concentrates on works that attempt to holistically measure model reliability from different standpoints. Some works target entangled visual illusions and language hallucinations (Guan et al., 2024), others focus on perturbing input data for hallucination detection (Ding et al., 2024; Saito et al., 2026), and yet others automate benchmark generation (Wu et al., 2024). Another line of work focuses on fine-grained classifications of failures, e.g., Ye-Bin et al. (2024) introduce image edits for true understanding, ignorance, stubbornness, and indecision. Wang et al. (2024a) offer an LLM-agnostic evaluation pipeline constructing a dataset where answers are easily verifiable with perhaps the closest taxonomy to the one we propose. Also closely related to our setup are M-HalDetect (Gunjal et al., 2024), providing 16k fine-grained annotations over model responses, and HalLoc (Park et al., 2025) offering token-level hallucination localization. Open problems. The vast majority of these resources are built around English outputs from one or a handful of LVLMs. This raises two concerns that directly motivate our work. First, benchmarks based on LVLM generations inherit the characteristics of their source models, and their diagnostic value degrades as the underlying systems are replaced (Laskar et al., 2023; Liu et al., 2025), a recurring theme also observed in LLM-centric resources (Ravichander et al., 2025; Vazquez et al., 2025). Efforts to sidestep this via synthetic generation (Muhlgay et al., 2024) still rely on an LLM as a generator. Our work addresses both gaps, with span-level annotations over five recent LVLMs and a human-written subset across four languages. Label Description A Invention An entity, object, property, or event that is not present in the image. B Mischaracterization An entity, object, property, or event present in the image but described incorrectly. C OCR Problem Misreading text that is visible in the image. D Miscounting Incorrectly reporting the quantity of visible items. E Other Does not correspond to any of the previous classes (use sparingly). F None (Unused by annotators when marking spans; inferred from the lack of annotation for an item.) Table 1: Hallucination classification labels. 3 Theoretical framework We focus on devising a classification of hallucinations that can serve as a theoretical basis for dataset creation and that annotators can reliably follow during annotation. The main issues we consider in designing this classification are (1) error coverage and (2) category granularity. We aim to be as comprehensive as possible by covering a wide range of errors, while avoiding overly fine-grained distinctions that would introduce unnecessary confusion. We define hallucination as content that is either unsupported by or contradictory to the semantic reference, which is the input image in our case. Therefore, we focus on output correctness rather than coverage: is the model generation true with regard to the image, not whether everything in the image has been described. We only consider single-turn interactions, focusing on open-ended tasks such as visual question answering or image captioning. We develop the classification scheme outlined in Table˜1, drawing inspiration from previous benchmarks. We reorganize previously proposed taxonomies, merging categories that we consider similar (e.g., contextual guessing in Rani et al., 2024 and existence in Yan et al., 2026), and removing categories that are not relevant to our purposes (e.g., gender anomaly in Rani et al.). We encountered several challenges during the development of our classification, particularly regarding grounding criteria (i.e., whether outputs should rely solely on visual evidence or may also incorporate external knowledge), and interpretative subjectivity (i.e., cases where the output may or may not be inferable depending on the annotator’s background). To address these ambiguities, rather than creating separate categories, we instruct annotators to adhere to our definition of hallucination. Thus, factual claims unverifiable from the image are treated as a hallucination; expressions of uncertainty (e.g., “it looks like”) are not to be marked as hallucinations as this is a desirable behavior. 4 Data collection We generate outputs using two existing datasets. HaloQuest (Wang et al., 2024b) is a VQA dataset designed to provoke hallucinations, for example with visually challenging images or false-premise questions not matching the image. We use only the non-synthetic images. VISaGE (Frank and Allaway, 2025) contains non-synthetic images of objects with unusual characteristics (e.g., three-legged cats, boat ambulances), along with questions about these characteristics. To obtain LVLM outputs which may include hallucinations, we use five models: Gemma3 (Gemma Team et al., 2025), InternVL3 (Chen et al., 2024b), MiniCPM V 4.5 (Yu et al., 2025), Llava-NeXT (Liu et al., 2024b), and Qwen3-VL (Qwen Team, 2025b). All models have 8B parameters except Gemma3 with 27B. We use a temperature of τ=0.7τ=0.7 and a maximum length of 512 tokens, across 5 random seeds. To encourage Gemma3 to generate responses in the same language as the input query, we localize the system prompt. We translate the two source datasets using NLLB-200 3.3B (NLLB Team et al., 2022) for Italian and French, and using Qwen3 8B (Qwen Team, 2025a) for Chinese; see Section˜B.1 for details. Sampling strategies. We select the data for human annotation using three sampling strategies: (1) random, (2) model-assisted pre-selection, and (3) human-written. For the first two, we select items from the LVLM-generated outputs. For the third, we collect human-invented responses that include hallucinations to create a model-independent benchmark for hallucination detection. 1. Random. From the generations for HaloQuest and VISaGE, we randomly select 460 samples for each language and model pair, for 2,300 samples per language in total. 2. Model-assisted pre-selection (MAP). As a comparison point for our intended human-created benchmark, we test whether an LVLM-guided sampling process can lead to a high quality evaluation benchmark. We use Gemma3 27B IT to pre-select LVLM-generated outputs for annotation. We run Gemma3 on all our outputs to assess which outputs contain hallucinations, using our classification in Table˜1. For each datapoint, the judge receives the input image, the question, and the complete LVLM response, together with a prompt containing a role description, label definitions, and formatting instructions.111See Figure 6 in Section B.3 for prompt instructions. The instructions closely match the human guidelines we provide to annotators. We then use these preliminary labels to construct a more label-balanced annotation set. In practice, the judge predicts OCR problems and miscounting much less frequently than invention and mischaracterization. We therefore include all datapoints predicted as OCR problems or miscounting, and sample from the remaining predicted labels so as to balance invention and mischaracterization as much as possible. This yields 2,300 MAP samples per language. 3. Human-written samples. Finally, we collect human-written samples as a model-independent source of hallucination examples. For each language, we (i) randomly select 100 image-question inputs from the data we used to generate LVLM outputs; (i) have writers fluent in the language manually create outputs that deliberately contain mistakes akin to hallucinations; (i) machine-translate these 100 samples into the other three languages;222Using the pipeline described in Section B.1. (iv) manually check translated samples and post-edit them when relevant. This results in a total of 1,600 human-written samples, out of which 400 are directly written by humans and 1,200 are machine-translated and manually curated. This manual curation step proved necessary as we observed that the NLLB backbone used for FR and IT had a tendency to ignore anything but the first sentence. We provide a brief overview of the improvement in quality, as measured by COMETKiwi scores (Rei et al., 2022333Unbabel/wmt22-cometkiwi-da model in Table˜2: while absolute values are hard to interpret, we confirm that post-editing systematically improves quality estimation scores. to EN to FR to IT to ZH from EN – 0.79910.7991 0.81740.8174 0.77970.7797 from FR 0.84420.8442 – 0.8080.808 0.78710.7871 from IT 0.82040.8204 0.81530.8153 – 0.8150.815 from ZH 0.82820.8282 0.77150.7715 0.80290.8029 – (a) Before post-editing to EN to FR to IT to ZH from EN – 0.84920.8492 0.85110.8511 0.82890.8289 from FR 0.84590.8459 – 0.81110.8111 0.79810.7981 from IT 0.84850.8485 0.85530.8553 – 0.81770.8177 from ZH 0.83060.8306 0.78810.7881 0.80760.8076 – (b) After post-editing Table 2: Quality improvement from post-editing Writers were familiar with the annotation task: they participated in pilot experiments and had access to the same 10 throwaway training items we provide to annotators. They were asked to generate a single type of hallucination for each sample (see Section˜3), and label each sample with the hallucination type. The collection interface tracked the distribution of intended hallucination types to encourage an even distribution across labels. Annotation. We recruited 12 C1-to-native proficient annotators per language, mostly European university students.444Due to one dropout, French has 11 annotators. To obtain three judgments per sample, each annotator processed ∼ 1,250 items over one month, compensated at ∼ €15/h.55529 items have 4 annotations due to assignment collisions. The workflow included autonomous training on throwaway samples, a briefing session, and access to support via a collaborative FAQ and direct means of contact. Annotators could self-pace and edit previous entries. See annotation and writing guidelines and interface details in Section˜B.3. (a) Associated image from HaloQuest (Wang et al., 2024b). "id": "train-fr-4186", "split": "train", "language": "fr", "prompt": "Que fait le mouton noir?", "image_name": "1035_1474161379_308597d6ca_o.jpg", "response": "Le mouton noir est en train de brouter l’herbe. On peut voir sa tête inclinée vers le bas, ce qui indique qu’il est occupé à pâturer dans la pelouse verte environnante.", "comments": "y8pM-y1Ki7DUj7nR8ebGs_DFOPj2oSHKSXzuit9IiaY": null, "RBa6IqYMnvjmMNq94k7zKqqgIXmnQi12XIlcbbAIH-g": null, "Hb-KgRqHQTYOcU5cFq_i6Kmgx2jEbDmIdpz1vmx1Tjs": null , "raw_annots": "y8pM-y1Ki7DUj7nR8ebGs_DFOPj2oSHKSXzuit9IiaY": "3:14;31:46;48:89;109:132|A;A;A;A", "RBa6IqYMnvjmMNq94k7zKqqgIXmnQi12XIlcbbAIH-g": "3:46;64:132|A;A", "Hb-KgRqHQTYOcU5cFq_i6Kmgx2jEbDmIdpz1vmx1Tjs": "0:168|A" , "labels":["start":0, "prob":0.3333, "label":"A", "end":3, "start":3, "prob":1.0, "label":"A", "end":14, "start":14, "prob":0.6667, "label":"A", "end":31, "start":31, "prob":1.0, "label":"A", "end":46, "start":46, "prob":0.6667, "label":"A", "end":64,"start":64, "prob":1.0, "label":"A", "end" :89, "start":89, "prob":0.6667, "label":"A", "end":109, "start":109, "prob":1.0, "label":"A", "end":132, "start":132, "prob":0.3333, "label":"A", "end":167], "metadata": "prelabel": "B", "orig_dataset": "haloquest", "model": "minicpm", "strategy": "random" (b) Annotated datapoint; the model is prompted to describe what the black sheep is doing, the response claims it is grazing. Figure 1: Example SHEEP datapoint. Example datapoint. We include an example datapoint in Figure˜1 for reference. We provide both individual and aggregated annotations, along with comments and relevant metadata. 5 Analyses We assess whether our human-written samples are a viable substitution for an LVLM-generated benchmark in two ways: we look at characteristics of the collected data (inter-annotator agreement, ability to surface specific hallucination types, distributional gaps) in Section˜5.1, and then compare evaluations of hallucination detectors in Section˜5.2. 5.1 Data characteristics Figure 2: Top: Samples per strategy, language, and split. Bottom: Proportion of annotations grouped by number of hallucination spans marked in an item. Descriptive statistics. In Figure˜2 (top), we show the number of samples for each language grouped by sampling strategy.666Images used in human-written data are all in the test split. Figure˜2 (bottom) presents the distribution of the number of hallucination spans annotated per item, again by sampling strategy. Annotators rarely mark more than one hallucination span per sample: the distributions for the random and MAP strategies are approximately Zipfian, with most annotations marking no spans, while annotations with five or more spans are rare (∼ 3.5%). In contrast, the human-written samples contain a larger amount of annotations with exactly one span, whereas the MAP data comprises more annotations with three or more spans. This lines up with our expectations: whereas an uninformed random sample is very ineffective at surfacing hallucinations, human writers are quite capable of matching our classification; the success of the MAP is contingent on the accuracy of the LVLM-judge. Inter-annotator agreement. We consider three metrics to assess inter-annotator agreement (IAA). Figure 3: Annotator agreement grouped by sampling process (left), generating LVLM (center), and output language (right). Center and right plots show only LVLM-generated data; left also includes human-generated data. Error bars represent 95% bootstrap confidence, sampling with replacement. See Section˜C.1 for table versions. 1. Presence agreement. We measure whether annotators agree on which samples contain hallucinations, regardless of hallucination type and specific span of hallucinated text. We adopt a free-marginal version of Krippendorff’s alpha: αfree=1−(k/n)∑ovv′v,v′Vδ(v,v′) _free=1-(k/n)Σ_v,v ^Vo_v δ(v,v ) where n is the total number of paired ratings, k is the number of labels (i.e., 2), δ is the nominal metric, and o is the matrix of observed coincidences. We use the free-marginal formulation because it is less sensitive to differences in label distributions arising from varying rates of hallucinations (Byrt et al., 1993; Randolph, 2005). This property is particularly important as the three sampling strategies are designed to yield different hallucination rates. 2. Label agreement. We quantify to what extent annotators agree on the hallucination type present in a sample, using a standard (fixed-marginal) Krippendorff’s alpha. We contrast agreement across the full dataset (Unconditional) vs. what we observe only on data where all annotators agree on the presence of a hallucination (Given a hallucination).777To handle annotations with more than one span, we ignore all spans beyond the first. See Section C.1 for discussion. 3. Span agreement. The overlap between spans of text marked as containing hallucinations. We use an Intersection over Union (IoU), as per prior work (Vazquez et al., 2025). We compute the average pairwise IoU across all possible pairs of annotations of the same item. We compute span agreement when factoring in hallucination type in the intersection (Requiring label match) and regardless of assigned label (Ignoring labels). IAA scores per sampling strategy are reported in Figure˜3 (left). Unsurprisingly, controlling for similarity on other annotation levels (presence or absence of hallucination, choice of specific labels) improves agreement rates. More interestingly, are the clear distinctions across strategies: annotators are most consistent on the human-written samples and least consistent on the randomly sampled datapoints, suggesting that sampling strategies very clearly influence the ability of annotators to reach a consensus, and that agreement improves as the selection becomes more controlled. A key question to assess whether human-written data can serve as a proxy for model-dependent hallucination benchmarks is how IAA varies across generating LVLMs and languages. Such a substitution is only meaningful if the resulting annotations exhibit levels of agreement comparable to the variation naturally induced by changes in model or language. To this end, Figure˜3 reports IAA on the random and MAP subsets across LVLM (center) and language (right). (a) MAP judge labels vs. annotator labels, MAP data. (b) MAP judge labels vs. annotator labels, random data. (c) Writer labels vs. annotator labels, human data. Figure 4: Distribution of expected labels (row) vs. annotator labels (columns) for all available annotations. None of the generating LVLMs in Figure˜3 (center) yields agreement levels higher than those observed for the human-written samples in Figure˜3 (left); and variation across generating models is often as large as, or larger than, the variation induced by the sampling strategies themselves. Among the evaluated models, annotators tend to agree most on samples generated by the largest model (Gemma3), whereas QwenVL consistently produces lower agreement scores. Similarly, across the four languages in Figure˜3 (right), we observe a similar pattern: agreement varies substantially, with French consistently exhibiting lower IAA and Chinese yielding the highest. Overall, these results suggest that the increase in agreement observed for human-written samples falls within the range of variation induced by changes in the generating LVLM or the language. Agreement with expected labels. Do annotators agree with the MAP-judge assigned hallucination labels or the human writers of hallucinations? 3-way classification Origin EN FR IT ZH LVLM-visage 0.9580.958 0.8620.862 0.8590.859 0.8570.857 LVLM-haloq. 0.8780.878 0.8470.847 0.8690.869 0.7570.757 human 0.8670.867 0.7430.743 0.7380.738 0.7700.770 6-way classification Origin EN FR IT ZH gemma3 0.8570.857 0.6960.696 0.5950.595 0.8240.824 internvl 0.4290.429 0.4770.477 0.4950.495 0.4740.474 llava 0.7180.718 0.6150.615 0.6540.654 0.7030.703 minicpm0123456 0.7210.721 0.6330.633 0.6550.655 0.6140.614 qwenvl 0.8770.877 0.8320.832 0.7390.739 0.8640.864 human 0.8760.876 0.7480.748 0.7600.760 0.7560.756 Table 3: Per-class F1 scores for 3-way and 6-way classification across languages. Figure˜4 quantifies the extent to which annotations agree with these expected labels. The main departures from agreement along the diagonal are due to two patterns: first, confusion between invention and mischaracterization, particularly in the MAP and human-written subsets, and secondly, human annotators disagreeing with the labels provided by the MAP and human writers (F columns).888Disagreement between writers and annotators is to be expected, given the IAAs reported in Figure 3: what counts as a hallucination is oftentimes subjective (Mickus et al., 2024). This also underscores there are limits in terms of subtlety in judgment we can expect from annotators. We also observe a high false positive rate in terms of binary detection for our LVLM-judge, much higher than what we observe for human-written samples. In contrast, the random data contains a large proportion of correctly identified non-hallucinations; however the MAP judge labels also miss many hallucinations found by annotators (F row). We see that the human-written sampling strategy is more effective at targeting specific hallucination types, since it avoids the high false negative and false positive rates observed when using the MAP strategy with LVLM-judge labels. Distributional shifts. To assess distributional shift between human-written and LVLM-generated data, we train a logistic regression classifier using character- and word-level n-gram features to predict the origin of each sample in multi-way settings, and report 5-fold cross-validation results in Table˜3. In the 3-way setting (human-written, LVLM responses on VISaGE and HaloQuest), human-written samples are usually distinguishable from generated data, suggesting a clear distributional shift. However, this separability is comparable to that observed between the two LVLM benchmarks themselves (VISaGE vs. HaloQuest). In the 6-way setting, where responses are grouped by generating LVLM, human-written data remains identifiable but is not uniquely distinct: Gemma3 and QwenVL are equally or even more easily separable based on textual features alone. Both results suggest that stylistic and lexical variation across LVLMs is already substantial, and that the distributional gap between human-written and LVLM-generated samples is comparable to variation across LVLMs. 5.2 Usefulness as a benchmark Thus far, we have discussed whether human-written samples have characteristics that would differ significantly from those of LVLM-generated items. We have yet to assess the usefulness of human-written samples when it comes to benchmarking performances on LVLM hallucination detection. We evaluate a modern hallucination detection model, HalluShift++ (Nath et al., 2025), as well as our Gemma3 LLM-judge on our data. human gemma3 internvl llava minicpm qwenvl all acc 0.664 3750.664\,375 0.542 7630.542\,763 0.531 9150.531\,915 0.574 4680.574\,468 0.485 4370.485\,437 0.459 4590.459\,459 F1 0.498 9130.498\,913 0.442 0320.442\,032 0.400 7340.400\,734 0.455 1470.455\,147 0.409 6730.409\,673 0.426 8780.426\,878 EN acc 0.695 0000.695\,000 0.521 7390.521\,739 0.516 1290.516\,129 0.567 9010.567\,901 0.448 7180.448\,718 0.402 1740.402\,174 F1 0.510 7810.510\,781 0.391 2590.391\,259 0.328 0040.328\,004 0.515 8050.515\,805 0.344 7030.344\,703 0.306 6160.306\,616 FR acc 0.677 5000.677\,500 0.539 4740.539\,474 0.463 4150.463\,415 0.538 4620.538\,462 0.480 5190.480\,519 0.426 6670.426\,667 F1 0.509 4760.509\,476 0.449 8390.449\,839 0.321 2230.321\,223 0.448 9900.448\,990 0.420 9880.420\,988 0.458 2230.458\,223 IT acc 0.650 0000.650\,000 0.539 4740.539\,474 0.618 4210.618\,421 0.676 0560.676\,056 0.518 5190.518\,519 0.433 3330.433\,333 F1 0.482 3030.482\,303 0.451 8610.451\,861 0.383 1050.383\,105 0.516 4840.516\,484 0.437 3020.437\,302 0.458 1200.458\,120 ZH acc 0.635 0000.635\,000 0.566 2650.566\,265 0.532 2580.532\,258 0.507 6920.507\,692 0.493 1510.493\,151 0.594 2030.594\,203 F1 0.491 0000.491\,000 0.505 6750.505\,675 0.505 2340.505\,234 0.390 3690.390\,369 0.393 3170.393\,317 0.424 6210.424\,621 Table 4: Gemma3 classification: F1 and accuracy. Label assessment with Gemma3 judge. Our first baseline is the Gemma3 LVLM we used earlier as part of the preselection MAP strategy. We now use this model to label items in the random and human-written samples with the six possible labels in Table˜1: note that the task here is to label items, not spans. For evaluation, we mark the item as correct if at least one annotator provided the same label as predicted by Gemma3. The results are shown Table˜4, broken down by item language and item generation type (model or human), with full results provided in Section˜C.2. Overall, we see that Gemma3 labels the human-written samples with the highest accuracy, compared to the models; somewhat surprisingly, this holds true even when Gemma3 is the source model. However, human-written samples are not completely outside the range of model samples, especially when looking at F1, precision, or precall rather than accuracy. Note that we expect accuracy to shift if detectors are not equally capable on all hallucination types. gemma3 internvl llava minicpm qwenvl human 0.7960.796 0.7700.770 0.9160.916 0.8670.867 0.7080.708 gemma3 0.7170.717 0.7710.771 0.7000.700 0.7690.769 internvl 0.6580.658 0.8000.800 0.6600.660 llava 0.8310.831 0.6830.683 minicpm 0.7660.766 Table 5: Comparison of performance rankings for human- and machine written data (Pearson’s r, n=40n=40). This illustrates that the error profile is roughly conserved on the human-written data, but it still leaves open one major point: whether performance rankings from different judges would be accurately captured by our human-written data. To that end, we select ten LVLMs to serve as judges (Gemma 3 4b, 12b and 27b, internVL 4b, 8b and 38b, QwenVL 2b, 4b, 8b and 32b) and compare their performances on human-written vs machine-written data. For each judge, we compute F1 scores per language (i.e., generated by a specific LVLM or by human writers), for a total of 40 setups per data source. We then measure whether performances are correlated across sources, using a Pearson correlation coefficient. This yields the scores displayed in Table˜5: crucially, we find that human-written data yields the highest correlations. In other words, human-written samples are as adequate as samples generated by any arbitrarily chosen LVLM. Labeled-span assessment with HalluShift++. Next, we consider a baseline inspired by HalluShift++, which detects hallucinations from internal LVLM signals, including representation shifts, attention patterns, and confidence-related features. We adapt this idea to our span-level localization setting by running a teacher-forced forward pass over each image–question–answer triple and extracting token-level features from an evaluator model. We extract these features from Llava-NeXT, yielding both a model-dependent self-probing setup as well as an external evaluation setup, depending on which LVLM generated the samples in our data. The features are used to train a token-level classifier in a multiclass setting where tokens are assigned to one of the labels A–F (Table˜1).999More details and experiments, including with EUQ (Huang et al., 2026), are provided in Sections C.3 and C.4. Strategy acc prec rec F1 random 0.440 3780.440\,378 0.225 7350.225\,735 0.506 0650.506\,065 0.203 5800.203\,580 MAP 0.403 3980.403\,398 0.240 1610.240\,161 0.517 3900.517\,390 0.223 8060.223\,806 human 0.332 0050.332\,005 0.240 0650.240\,065 0.442 5830.442\,583 0.203 3900.203\,390 (a) HalluShift++ classification metrics across strategies. random MAP all gemma3 0.156 8370.156\,837 0.227 1910.227\,191 0.188 7160.188\,716 internvl 0.163 2040.163\,204 0.202 7540.202\,754 0.186 0560.186\,056 llava 0.185 7310.185\,731 0.223 8220.223\,822 0.212 0490.212\,049 minicpm 0.182 0570.182\,057 0.204 8350.204\,835 0.195 3320.195\,332 qwenvl 0.225 4530.225\,453 0.209 0430.209\,043 0.220 9660.220\,966 (b) HalluShift++ F1 per generating LVLM. Table 6: HalluShift++ performance. As Table˜6 shows, usually, we obtain low F1 scores, driven primarily by a low precision, highlighting the challenging nature of our collected data. Importantly, performances on the human-written data (bottom row of Table˜6(a)) are comparable to what we observe from specific generating LVLMs. calibrate on human gemma3 internvl llava minicpm qwenvl test on human 0.063 9470.063\,947 0.056 3150.056\,315 0.053 4900.053\,490 −0.011 961-0.011\,961 0.050 4830.050\,483 0.069 6920.069\,692 gemma3 0.070 0690.070\,069 0.068 8980.068\,898 0.064 7920.064\,792 −0.009 049-0.009\,049 0.061 9380.061\,938 0.076 7070.076\,707 internvl 0.043 1260.043\,126 0.040 3660.040\,366 0.039 1850.039\,185 −0.007 594-0.007\,594 0.037 7120.037\,712 0.044 4830.044\,483 llava 0.046 1310.046\,131 0.047 4100.047\,410 0.047 4580.047\,458 −0.006 649-0.006\,649 0.046 5640.046\,564 0.041 2450.041\,245 minicpm 0.048 2300.048\,230 0.046 7690.046\,769 0.046 0500.046\,050 −0.007 259-0.007\,259 0.045 2160.045\,216 0.047 0750.047\,075 qwenvl 0.062 2880.062\,288 0.061 2210.061\,221 0.060 3610.060\,361 −0.007 612-0.007\,612 0.058 9960.058\,996 0.062 6970.062\,697 Table 7: F1 delta after recalibration (all data). One of the reasons for the modest performances we observe in Table˜6 is the high prevalence of non-hallucinated tokens. This prompts us to recalibrate model probabilities for non-hallucination labels by computing the optimal probability threshold in a one-versus-all setting, comparing F to all other labels. We calibrate on data derived from all possible origins (human data or any of the 5 LVLMs) and record the delta in F1 in Table˜7. This usually leads to improvements, with one exception: calibration on Llava systematically degrades performances. As this LVLM is also the backbone for the detection pipeline, this appears to illustrate the biases that model-dependent benchmarks can introduce. 6 Discussion The core question we address in this work is whether human-written samples can be used to evaluate hallucination detection models. To this end, we have constructed a large dataset of vision-language hallucination samples using three sampling strategies: random sampling, model-assisted pre-selection, and human-written items. Our analyses provide evidence that human-written samples are a viable basis for evaluating hallucination detection. First, they allow us to more carefully control the actual contents of our dataset (Figure˜4). Second, annotators reach a higher degree of consensus on these items (Figure˜3). Third, human-written samples exhibit a distinct but comparable distributional profile relative to LVLM responses (Table˜3), and the performance of hallucination detectors on human-written samples is broadly consistent with their performance on LVLM-generated data, while avoiding pitfalls that model-dependent benchmarks suffer from (Table˜7). It is worth stressing that the second of these points is not trivial: prior work has frequently highlighted that annotators express genuine disagreement as to what counts as a hallucination (Mickus et al., 2024) and where hallucination spans begin and end (Vazquez et al., 2025; Schmidtova et al., 2026). While in general, some of this discrepancy can be imputed to gaps in expertise (Calò et al., 2026), our experimental setup allows us to cleanly isolate the effects of data sampling since the same cohort of annotators and the same inputs were used throughout our dataset. Our work is not without its caveats. First, we observe that the data we collect is remarkably challenging, with some models performing at or close to random on specific subsets. Second, we cover a small number of hallucination detection models; establishing whether performance on human-written samples reliably predicts performance on true hallucinations requires broader empirical validation. Results in Table˜5 are encouraging, but stronger guarantees will require concerted research efforts. A shared task is currently underway to further validate this point.101010See helsinki-nlp.github.io/shroom/2026/ A similar comment holds for the representativeness of data derived from different LVLMs, as we do not systematically quantify variation across all possible generators. At the same time, human-written samples can always serve as a sanity check for hallucination detection systems: because they are constructed to contain the target phenomena, a large performance gap between human-written and LVLM-generated data would likely indicate reliance on spurious signals. Our approach alleviates some of the issues that plague modern NLP evaluation, yet it also frames hallucination detection as a black-box problem. Methods relying on internal model signals will require adaptation, as discussed in Section˜C.3 for Hallushift++. This also reflects a broader trend toward proprietary black-box models, where evaluation must proceed without access to internal states. It is worth acknowledging that there can be concerns as to the fitness of human-written data for evaluation purposes, as humans could introduce their own biases.111111Bias is a valence-heavy and ambiguous word that can refer to a wide variety of phenomena — ranging from judgments made on socionormative grounds, to distributional and structural differences between data sources (in this case, human-written vs machine-written). As far as the latter goes, we argue that some biases can be good: e.g., it is genuinely beneficial for us to produce more instances of OCR and miscounting, as long as we do not introduce trivial distributional patterns. Previous work has found genuine mismatches between human and system-based assessments, especially on disagreement modeling (e.g., Pavlick and Kwiatkowski, 2019; Mickus et al., 2025). However, such mismatches appear to exhibit a profile distinct from that observed in the present work: the gaps described in these studies appear to emerge between human and machine assessments, whereas what we target (and document in Table˜7) is bias specific to one particular model. More broadly, prior work has documented socionormative and cognitive biases of annotators (Mieleszczenko-Kowszewicz et al., 2023; Gautam and Srinath, 2024; Blodgett et al., 2020) and how they can impact performances (Geva et al., 2019). The biases observed in NLP systems tend to be exacerbated versions of human socionormative biases (Sheng et al., 2021; Chen et al., 2024a; Almatarneh et al., 2025), i.e., prior work suggests that human annotators present milder cases of socionormative biases. If socionormative biases can serve as a spurious, exploitable feature that can compromise evaluation, then we should therefore expect human data to be preferable to data generated by NLP systems. As far as SHEEP is concerned, our dataset construction sidesteps some of the issues pointed out in prior work. In particular, we keep track of annotators, hence future studies can follow Geva et al. and Gautam and Srinath’s recommendations. 7 Conclusion We present SHEEP, a dataset of 20,000 items across four languages (Chinese, English, French, Italian) for hallucination detection across 5 LVLMs. We assess three strategies for obtaining samples: randomly selecting outputs of LVLMs, using an LVLM-as-an-annotator pre-selection, and writing samples manually. The dataset is richly annotated with 5 possible hallucination labels at the span level, with each item assessed independently by three annotators. The performance of modern hallucination detectors underscores the challenging nature of SHEEP. We find that human-written samples exhibit several beneficial characteristics: they yield items that are more consensual among our annotators while maintaining a distributional profile similar to that observed in LVLM-generated samples. We argue that this strategy avoids some pitfalls common in hallucination evaluation, including a tendency toward rapid obsolescence as models quickly cease to represent the state of the field. Limitations We frame this study as a first outlook into whether human-written data could replace LVLM-generated data for hallucination detection benchmarking. As such, there are clear limitations due to the exploratory nature of this project: The tasks and setups considered here are not an exhaustive or representative subset of all applications and contexts where LVLM hallucination detection is relevant; we focus primarily on high-resource languages, although prior literature shows that hallucinations do not impact all languages equally (Vazquez et al., 2025; Gamba et al., 2026; Datta et al., 2026). An important consideration is that we propose a static dataset, hence it is in principle possible for leakage to occur. As of writing, our testset labels are kept private and accessible through a dedicated scoring interface at https://shroom.pythonanywhere.com/ so as to mitigate this risk. Ethical Considerations Data collection risks. The present work describes a data collection process, and therefore several ethical considerations apply. The data was deemed unlikely to present any potential harm to annotators, as inputs strictly target common objects with uncommon attributes (VISaGE, Frank and Allaway, 2025) or commonplace questions and images mismatched with one another (HaloQuest, Wang et al., 2024b). The dataset contains no offensive or harmful topics, nor any personally identifying information. Annotators’ usernames on the annotation platform are anonymized in the final distributed dataset. Potential risks of the research agenda. This work targets hallucination evaluation in LVLM technology. We believe that fostering a research ecosystem that actively combats the spread of misinformation and technologies that can produce misinformation at scale is of great importance. While part of this ecosystem has to include evaluation benchmarks such as the one we propose, we recognize that by design our dataset contains information that is false, incoherent, or misleading, and strictly emphasize that the intended use of this artifact is to promote research into hallucination detection and mitigation. Acknowledgments The construction of this dataset was made possible thanks to a grant from the Finnish Society of Sciences and Letters. This work has also received funding from the Digital Europe Programme under grant agreement No 101195233 (OpenEuroLLM). The contents of this publication are the sole responsibility of its authors and do not necessarily reflect the opinion of the EU. References S. Almatarneh, A. BaniMustafa, G. Samara, R. Alazaidah, and Q. Obeidat (2025) Bias and fairness in NLP: addressing social and cultural biases. In Advances in Computational Intelligence: 18th International Work-Conference on Artificial Neural Networks, IWANN 2025, A Coruña, Spain, June 16–18, 2025, Proceedings, Part I, Berlin, Heidelberg, p. 609–619. External Links: ISBN 978-3-032-02727-6, Link, Document Cited by: §6. Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou (2025) Hallucination of multimodal large language models: a survey. External Links: 2404.18930, Link Cited by: §2. S. L. Blodgett, S. Barocas, H. Daumé I, and H. Wallach (2020) Language (technology) is power: a critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 5454–5476. External Links: Link, Document Cited by: §6. T. Byrt, J. Bishop, and J. B. Carlin (1993) Bias, prevalence and kappa. J Clin Epidemiol 46 (5), p. 423–429 (en). Cited by: §5.1. E. Calò, S. Mahamood, A. Gatt, and K. Van Deemter (2026) A logic-based approach to hallucinations in data-to-text NLG: experiments with human and LLM annotators. In Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026), S. M. Mohammad and N. Ousidhoum (Eds.), San Diego, California, United States, p. 28–62. External Links: Link, Document, ISBN 979-8-89176-413-2 Cited by: §6. G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang (2024a) Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 8301–8327. External Links: Link, Document Cited by: §6. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024b) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24185–24198. Cited by: §4. D. Datta, M. K. Chilukuri, Y. Kumar, S. Ghosh, and M. B. Zafar (2026) Do LLM hallucination detectors suffer from low-resource effect?. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), V. Demberg, K. Inui, and L. Marquez (Eds.), Rabat, Morocco, p. 2959–2985. External Links: Link, Document, ISBN 979-8-89176-380-7 Cited by: Limitations. A. P. Dempster (1967) Upper and lower probability inferences based on a sample from a finite univariate population. Biometrika 54 (3-4), p. 515–528. Cited by: §C.4. P. Ding, J. Wu, J. Kuang, D. Ma, X. Cao, X. Cai, S. Chen, J. Chen, and S. Huang (2024) Hallu-pi: evaluating hallucination in multi-modal large language models within perturbed inputs. In Proceedings of the 32nd ACM International Conference on Multimedia, M ’24, New York, NY, USA, p. 10707–10715. External Links: ISBN 9798400706868, Link, Document Cited by: §2. S. Frank and E. Allaway (2025) VISaGE: understanding visual generics and exceptions. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 32549–32558. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §4, Data collection risks.. F. Gamba, A. Sinha, T. Mickus, R. Vazquez, P. Bhamidipati, C. Savelli, A. Chattopadhyay, L. A. Zanella, Y. Kankanampati, B. A. Remesh, A. A. Chandramania, R. Agarwal, C. Li, I. Buhnila, and R. Mamidi (2026) Confabulations from acl publications (cap): a dataset for scientific hallucination detection. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, and A. Toral (Eds.), Palma, Mallorca, Spain, p. 2512–2524. External Links: Document Cited by: §1, Limitations. S. Gautam and M. Srinath (2024) Blind spots and biases: exploring the role of annotator cognitive biases in NLP. In Proceedings of the Third Workshop on Bridging Human–Computer Interaction and Natural Language Processing, S. L. Blodgett, A. Cercas Curry, S. Dev, M. Madaio, A. Nenkova, D. Yang, and Z. Xiao (Eds.), Mexico City, Mexico, p. 82–88. External Links: Link, Document Cited by: §6. Gemma Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot (2025) Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §4. M. Geva, Y. Goldberg, and J. Berant (2019) Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 1161–1166. External Links: Link, Document Cited by: §6. T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou (2024) HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14375–14385. Cited by: §2. A. Gunjal, J. Yin, and E. Bas (2024) Detecting and preventing hallucinations in large vision language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. External Links: ISBN 978-1-57735-887-9, Link, Document Cited by: §2. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu (2025) A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans. Inf. Syst. 43 (2), p. 42:1–42:55. External Links: ISSN 1046-8188, Link, Document Cited by: §1. T. Huang, R. Wang, X. Liu, Y. Qin, L. Duan, and L. Jing (2026) Detecting misbehaviors of large vision-language models by evidential uncertainty quantification. arXiv preprint arXiv:2602.05535. Cited by: §C.4, footnote 9. C. Jiang, H. Jia, M. Dong, W. Ye, H. Xu, M. Yan, J. Zhang, and S. Zhang (2024) Hal-eval: a universal and fine-grained hallucination evaluation framework for large vision language models. In Proceedings of the 32nd ACM International Conference on Multimedia, M ’24, New York, NY, USA, p. 525–534. External Links: ISBN 9798400706868, Link, Document Cited by: §2. M. T. R. Laskar, M. S. Bari, M. Rahman, M. A. H. Bhuiyan, S. Joty, and J. X. Huang (2023) A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 431–469. External Links: Link, Document Cited by: §1, §2. Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J. Wen (2023) Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 292–305. External Links: Link, Document Cited by: §2. H. Liu, W. Xue, Y. Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng (2024a) A survey on hallucination in large vision-language models. External Links: 2402.00253, Link Cited by: §1, §2, §2. H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee (2024b) LLaVA-next: improved reasoning, ocr, and world knowledge. External Links: Link Cited by: §4. Y. Liu, J. Cao, C. Liu, K. Ding, and L. Jin (2025) Datasets for large language models: a comprehensive survey. Artif. Intell. Rev. 58 (12). Cited by: §1, §2. H. Lovenia, W. Dai, S. Cahyawijaya, Z. Ji, and P. Fung (2024) Negative object presence evaluation (NOPE) to measure object hallucination in vision-language models. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), J. Gu, T. (. Fu, D. Hudson, A. Celikyilmaz, and W. Wang (Eds.), Bangkok, Thailand, p. 37–58. External Links: Link, Document Cited by: §2. T. Mickus, A. Sinha, and R. Vázquez (2025) Your model is overconfident, and other lies we tell ourselves. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 5401–5417. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §6. T. Mickus, E. Zosa, R. Vazquez, T. Vahtola, J. Tiedemann, V. Segonne, A. Raganato, and M. Apidianaki (2024) SemEval-2024 task 6: SHROOM, a shared-task on hallucinations and related observable overgeneration mistakes. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), A. Kr. Ojha, A. S. Doğruöz, H. Tayyar Madabushi, G. Da San Martino, S. Rosenthal, and A. Rosá (Eds.), Mexico City, Mexico, p. 1979–1993. External Links: Link, Document Cited by: §6, footnote 8. W. Mieleszczenko-Kowszewicz, K. Kanclerz, J. Bielaniewicz, M. Oleksy, M. Gruza, S. Woźniak, E. Dzięcioł, P. Kazienko, and J. Kocoń (2023) Capturing human perspectives in NLP: questionnaires, annotations, and biases. In Proceedings of the Second Workshop on Perspectivist Approaches to NLP (NLPerspectives), External Links: Link Cited by: §6. D. Muhlgay, O. Ram, I. Magar, Y. Levine, N. Ratner, Y. Belinkov, O. Abend, K. Leyton-Brown, A. Shashua, and Y. Shoham (2024) Generating benchmarks for factuality evaluation of language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Y. Graham and M. Purver (Eds.), St. Julian’s, Malta, p. 49–66. External Links: Link, Document Cited by: §1, §2. S. Nath, A. Basu, S. Dasgupta, and S. Das (2025) HalluShift++: bridging language and vision through internal representation shifts for hierarchical hallucinations in mllms. arXiv preprint arXiv:2512.07687. Cited by: §C.3, §C.3, §5.2. C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, J. Song, and T. Zhang (2024) RAGTruth: a hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 10862–10878. External Links: Link, Document Cited by: §1. NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang (2022) No language left behind: scaling human-centered machine translation. External Links: 2207.04672, Link Cited by: §B.1, §4. E. Park, M. Kim, and G. Kim (2025) HalLoc: token-level localization of hallucinations for vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 29893–29903. External Links: Link Cited by: §2. E. Pavlick and T. Kwiatkowski (2019) Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics 7, p. 677–694. External Links: Link, Document Cited by: §6. S. Petryk, D. M. Chan, A. Kachinthaya, H. Zou, J. Canny, J. E. Gonzalez, and T. Darrell (2024) ALOHa: a new measure for hallucination in captioning models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, p. 342–357. External Links: Link, Document Cited by: §2. N. Pham and M. Schott (2024) H-pope: hierarchical polling-based probing evaluation of hallucinations in large vision-language models. External Links: 2411.04077, Link Cited by: §2. Qwen Team (2025a) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §B.1, §4. Qwen Team (2025b) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: §4. J. J. Randolph (2005) Free-marginal multirater kappa (multirater k[free]): an alternative to fleiss’ fixed-marginal multirater kappa. Note: Paper presented at the Joensuu Learning and Instruction Symposium External Links: Link Cited by: §5.1. A. Rani, V. Rawte, H. Sharma, N. Anand, K. Rajbangshi, A. Sheth, and A. Das (2024) Visual hallucination: definition, quantification, and prescriptive remediations. External Links: 2403.17306, Link Cited by: §2, §3. A. Ravichander, S. Ghela, D. Wadden, and Y. Choi (2025) HALoGEN: fantastic LLM hallucinations and where to find them. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 1402–1425. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §2. V. Rawte, S. Chakraborty, A. Pathak, A. Sarkar, S. T. I. Tonmoy, A. Chadha, A. Sheth, and A. Das (2023) The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 2541–2573. External Links: Link, Document Cited by: §1. R. Rei, M. Treviso, N. M. Guerreiro, C. Zerva, A. C. Farinha, C. Maroti, J. G. C. de Souza, T. Glushkova, D. Alves, L. Coheur, A. Lavie, and A. F. T. Martins (2022) CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), P. Koehn, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. Jimeno Yepes, T. Kocmi, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, M. Negri, A. Névéol, M. Neves, M. Popel, M. Turchi, and M. Zampieri (Eds.), Abu Dhabi, United Arab Emirates (Hybrid), p. 634–645. External Links: Link, Document Cited by: §4. A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko (2018) Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 4035–4045. Cited by: §2. K. Saito, R. Shinoda, S. Tanaka, T. Hirasawa, F. Okura, and Y. Ushiku (2026) HalDec-bench: benchmarking hallucination detector in image captioning. External Links: 2511.20515, Link Cited by: §2. P. Schmidtova, O. Dusek, and S. Mahamood (2026) HotelCheckSpan: a benchmark dataset for llm faithfulness. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), S. Piperidis, N. Bel, H. van den Heuvel, N. Ide, S. Krek, and A. Toral (Eds.), Palma, Mallorca, Spain, p. 9973–9987. External Links: Document Cited by: §6. R. Shekhar, S. Pezzelle, Y. Klimovich, A. Herbelot, M. Nabi, E. Sangineto, and R. Bernardi (2017) FOIL it! find one mismatch between image and language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M. Kan (Eds.), Vancouver, Canada, p. 255–265. External Links: Link, Document Cited by: §2. E. Sheng, K. Chang, P. Natarajan, and N. Peng (2021) Societal biases in language generation: progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, p. 4275–4293. External Links: Link, Document Cited by: §6. R. Vazquez, T. Mickus, E. Zosa, T. Vahtola, J. Tiedemann, A. Sinha, V. Segonne, F. Sanchez - Vega, A. Raganato, J. Libovický, J. Karlgren, S. Ji, J. Helcl, L. Guillou, O. De Gibert, J. Bengoetxea, J. Attieh, and M. Apidianaki (2025) SemEval-2025 task 3: mu-SHROOM, the multilingual shared-task on hallucinations and related observable overgeneration mistakes. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), S. Rosenthal, A. Rosá, D. Ghosh, and M. Zampieri (Eds.), Vienna, Austria, p. 2472–2497. External Links: Link, ISBN 979-8-89176-273-2 Cited by: §1, §2, §5.1, §6, Limitations. J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, J. Wang, H. Xu, M. Yan, J. Zhang, and J. Sang (2024a) AMBER: an llm-free multi-dimensional benchmark for mllms hallucination evaluation. External Links: 2311.07397, Link Cited by: §2. Z. Wang, G. Bingham, A. W. Yu, Q. V. Le, T. Luong, and G. Ghiasi (2024b) HaloQuest: a visual hallucination dataset for advancing multimodal reasoning. In Computer Vision – ECCV 2024, A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.), Cham, p. 288–304. External Links: ISBN 978-3-031-72980-5 Cited by: 1(a), §4, Data collection risks.. X. Wu, T. Guan, D. Li, S. Huang, X. Liu, X. Wang, R. Xian, A. Shrivastava, F. Huang, J. L. Boyd-Graber, et al. (2024) Autohallusion: automatic generation of hallucination benchmarks for vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 8395–8419. Cited by: §2. B. Yan, J. Zhang, Z. Yuan, S. Shan, and X. Chen (2026) Measuring the measurers: quality evaluation of hallucination benchmarks for large vision-language models. External Links: 2406.17115, Link Cited by: §3. M. Ye-Bin, N. Hyeon-Woo, W. Choi, and T. Oh (2024) Beaf: observing before-after changes to evaluate hallucination in vision-language models. In European Conference on Computer Vision, p. 232–248. Cited by: §2. T. Yu, Z. Wang, C. Wang, F. Huang, W. Ma, Z. He, T. Cai, W. Chen, Y. Huang, Y. Zhao, B. Xu, J. Cui, Y. Xu, L. Ruan, L. Zhang, H. Liu, J. Tang, H. Liu, Q. Guo, W. Hu, B. He, J. Zhou, J. Cai, J. Qi, Z. Guo, C. Chen, G. Zeng, Y. Li, G. Cui, N. Ding, X. Han, Y. Yao, Z. Liu, and M. Sun (2025) MiniCPM-v 4.5: cooking efficient mllms via architecture, data, and training recipe. External Links: 2509.18154, Link Cited by: §4. Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi (2025) Siren’s song in the AI ocean: a survey on hallucination in large language models. Computational Linguistics 51 (4), p. 1373–1418. External Links: Link, Document Cited by: §C.1. Appendix A Contributions of individual authors Timothee Mickus: overall organization, experimental designs, annotator recruitment, inter-annotator agreement (free-marginal for presence agreement), span annotation platform development, guideline development, data selection, data creation (English). Claudio Savelli: HalluShift++ implementation, translation pipeline development, code development, guideline development, data creation (Italian). Eduardo Calò: Theoretical framework, state of the art surveying, guideline development, data creation (Italian). Emilio Raimond: LLM-as-a-judge implementation, code development, state of the art surveying, data creation (French). Stella Frank: Guideline development, data selection, data creation (English), experimental design. Hengyu Luo: Data creation (Chinese), translation pipeline development, code development, Flavio Giobergia: Inter-annotator agreement, data quality checking. Vincent Segonne: Data creation (French), translation pipeline development, sample-writing platform development. Chuyuan Li: Distributional shift analysis. Aman Sinha: EUQ implementation. Lorenzo Vaiani: Code development. Jörg Tiedemann: Organizational support. Raúl Vázquez: overall organization, experimental designs, annotator recruitment, annotator briefing, span annotation platform development, guideline development, data selection. Appendix B Reproducibility details B.1 Translation pipeline Model selection. We evaluated candidate translation models by manually assessing 100 randomly sampled instances per target language (Italian, French, and Chinese) to measure adequacy and fluency. Based on this evaluation, we adopt a hybrid strategy. NLLB-200 (3.3B) (NLLB Team et al., 2022) is used for translations between English and Romance languages (Italian and French), where it performs reliably. For Chinese, we instead use Qwen3 8B (Qwen Team, 2025a), which we found produces more fluent, semantically accurate outputs, particularly for short prompts and question-like inputs. You are a translation engine. Translate only. Never explain, never reason, never add notes. Output only the final translation text. (a) System prompt. You are a translation engine. Translate SOURCE_TEXT into TARGET_LANGUAGE. Rules: 1) Translate exactly what is written. 2) If SOURCE_TEXT is a question, translate the question; do NOT answer it. 3) Keep named entities, numbers, URLs, and facts unchanged unless translation requires adaptation. 4) Output only the translated text. 5) Do not add explanations, notes, or extra lines. SOURCE_TEXT: <input text> (b) User prompt template. Figure 5: Prompt template used for Qwen3 translation. TARGET_LANGUAGE is replaced with the desired language (e.g., Italian, French, Chinese), and <input text> with the source string. Translation settings. We apply the translation pipeline in two scenarios: (i) English prompt translation, where all prompts and references are translated from English into the other three languages, as both datasets are originally in English solely. (i) multilingual augmentation, used for human-written samples, where inputs may originate in any of the four languages and are translated into the remaining three. All these samples are manually checked and post-edited when necessary, ensuring high-quality multilingual consistency. In the second setting, Qwen3 is used for all translation directions involving Chinese (↔ ), while NLLB is used for directions that do not involve Chinese (e.g., Italian ↔ French or English). NLLB pipeline. For Italian and French (and more generally for translations not involving Chinese), we use a sequence-to-sequence pipeline based on NLLB-200. We explicitly specify source and target language codes (e.g., eng_Latn → ita_Latn) and enforce the target language via a forced beginning-of-sequence token. Decoding is performed using beam search with 4 beams and no sampling. Qwen pipeline. For Chinese and all translation directions involving Chinese, we use Qwen3 8B in an instruction-following setup. We design a structured prompt that enforces strict translation behavior (i.e., no explanations, no additional text, and no question answering). The full prompt template is reported in Figure˜5. We use the tokenizer chat template and perform decoding with beam search (4 beams) and no sampling to ensure deterministic outputs. B.2 Gemma3 model-assisted pre-selection prompt We provide the exact prompts used in our experiments in Figure˜6. Because the purpose of the judge is to support large-scale pre-selection rather than to provide final annotations, we ask it to output only the predicted label, without a reasoning trace. B.3 Guidelines & Annotation interfaces Guidelines for annotators and writers. In this appendix, we include a copy of the guidelines provided to human writers (Figure˜7) and annotators (Figures˜8 and 9). The annotators had access to their guideline as a single document (split across two pages in the present reproductions for pagination purposes); minor details such as a table of contents have been omitted. Annotators also had access to a user guide for the annotation platform they were required to use, which is not reproduced here. Your role is to assign a label to the provided text and image based on whether or not the text contains a *hallucination*. Hallucinations correspond to cases where the provided text states information that contradicts the image. LABELING INSTRUCTIONS: You must assign one of the following labels (A, B, C, D, E or F): A. **Invention.** The text refers to an entity, object, property, or event that is not present in the image at all. This includes but is not limited to: fabricated objects, invented people, imagined actions unsupported by the image Examples: - "red bus" (no bus visible in the image) - "a dog nearby" (no dog visible in the image) - "an umbrella" (not present in the image) B. **Mischaracterization.** The text refers to something that is present in the image but describes it incorrectly. This includes but is not limited to: wrong object type (e.g., a taxi described as a bus), wrong colors (e.g., a blue shirt described as green), an incorrect identity (e.g., a person standing described as sitting). Examples: - Calling a van a "bus" - Describing a blue shirt as "green" - Describing a dog running when the animal is standing or laying down. C. **Wrong Reading.** The hallucination arises from misreading text that is visible in the image. This applies whenever there is something readable in the image itself, but the provided text misquotes it, alters it, or misunderstands it. Examples: - the image includes a "STOP" sign but the provided text mentions a "SHOP" - the image includes a "50% OFF" sign but the provided text mentions "30% OFF" D. **Miscounting.** The text expresses an incorrect quantity of visible items. This includes but is not limited to: wrong counts of objects shown in the image (e.g., people, cars, animals...); incorrect quantities; explicit numeric misstatements.... Examples: - the text refers to "three elephants" when only two elephants are visible in the image. - the text refers to "many people" when only two persons are visible in the image E. **Other.** A hallucination that does not fit categories A, B, C or D. F. **No hallucination.** The provided text does not contain any hallucination: the text does not contradict the image and is perfectly coherent with the information shown in the image. Do not explain your reasoning. Only reply with the letter corresponding to the label you assign, i.e., your answer must contain only the letter A, B, C, D, E or F. PROVIDED TEXT: Figure 6: Prompt used for the LLM-judge (Gemma3 27B IT). This document outlines guidelines for creating samples for the human-written section of DATASET. Motivation One of the things we want to verify with DATASET is whether we can evaluate the quality of hallucination detectors independently from any specific LLM output. To that end we intend to include a human-written subset of outputs in the test set, so as to be able to compare performances on LLM-generated and human-written hallucinations. Overview of the task You will be writing answers to a series of provided questions grounded on images. You will need to imitate the style of LLMs and make sure that the answer you provide contains a hallucination. Read these guidelines carefully and in their entirety before you start You are not allowed to use an LLM to generate your answers. Answers should be written from scratch. All other rules are provided as guidance: use your best judgment, and leave comments on this document whenever relevant. Detailed instructions • Before you start, read the guidelines for annotators to keep in mind how the data will be presented to annotators and what task they will need to perform: <URL> this document also contains the following items, which you will need to be familiar with: – a definition of what hallucinations are – a taxonomy of possible types of hallucinations (labels A, B, C, D or E) • Written samples are to be collected through the dedicated annotation platform, available from our private github repo: <URL> • Your written samples should be plausible answers to the questions as provided. Do not modify the questions. • Each answer you write should contain a mistake corresponding to one of the four specific labels A, B, C or D (but not E). For instance, if your answer contains a mistake that pertains to an OCR problem (label C), it should not also include a made up entity (label A) • Your answer must otherwise be coherent, grammatical, and self-consistent. • Across all answers that you produce, you should target a balanced proportion of all four labels (25% of A, 25% of B, 25% of C, and 25% of D) Your mistakes must be discoverable given the image. The annotator should not have to perform an external google search to spot the mistake you included. In terms of style: • Before you start writing samples, familiarize yourself with LLM style and the type of hallucinations they make. You can use the self-hosted pilot interface available at: <URL> • Do not look at samples written by other organizers, as much as possible. • Produce diverse samples. – The mistakes you include do not have to target the most salient part of the question or image. For instance, consider including mistakes that target * objects in the background, * spatial relationships between objects * superfluous claims not directly/immediately relevant to the question prompt – Hallucinations can often be triggered by surprising images that depict non-standard or otherwise exceptional situations (such as three-legged cats, red bananas, cows wearing Christmas trees on their heads…), in which case LLMs can either fabricate completely ungrounded information or generate text corresponding to the most probable situation (e.g., “this cat has four legs”). Consider whether the image provided could confuse an LLM. – Consider imitating different styles of LLMs, such as * samples with or without markdown * samples with or without a sycophantic opening line (“excellent question!”) * samples with or without explicit markers of reasoning (“To answer this question, I will carefully analyze all components of the image”). – Your answers can be of varying length, from a single sentence to a few paragraphs. They should not be overly short, since this would indicate that there is not much in the answer aside from the hallucination. Figure 7: Guidelines for human-written samples production. Actual URLs are replaced with the placeholder <URL>. Introduction In this annotation project you will be shown a series of question-answer pairs, plus a relevant image. The answer will be a passage of text produced by a Large Language Model (LLM) in response to the question. You will be asked to identify, with respect to the image, which spans of text in the answer constitute a “hallucination”, i.e., which parts of the answer are not supported by the image, given the question. Each datapoint consists of: • An image • A question related to the image (i.e., a prompt) • A response produced by a model General Instructions 1. Carefully examine the image, and read the prompt, and model response. 2. Highlight each span of text in the model response that is not supported by the information in the image (i.e., contains a hallucination or overgeneration). Base this exclusively on the definition of hallucination we are using (see below). Important: • Annotate conservatively: annotations must include only the minimum number of characters that would need to be edited or deleted in order to make the answer correct. • Prefer highlighting content words over function words. • Avoid selecting full sentences unless absolutely necessary. These are not rigid rules, i.e., use your best judgment when needed. 3. Classify each marked span using the proposed taxonomy (see below). Each span must be marked with only one class label (A, B, C, D, or E). 4. If a response contains multiple hallucinations: • Highlight each span separately. • Assign one label (A–E) per span. Do not merge unrelated errors into a single span. • Do not overlap or embed spans. 5. If a response contains no hallucination: • write “No hallucination” (in English) in the comment box. You can also use the comment box to mark anything else that you want to raise to our attention, such as: • model answers that are ungrammatical, • outputs that are generated in the wrong language, • cases where you are especially uncertain, • anything else you deem relevant. Please write comments in English, so that the whole team can read them. What counts as a hallucination? Definition. Content that contains or describes facts that are not supported by the image. In other words: hallucinations are cases where the answer text is inaccurate, or more specific than it should be, given the provided image. A hallucination is any content that: • Refers to something not present in the image • Misdescribes something visible in the image • Incorrectly counts visible entities • Misreads visible text • Or otherwise goes beyond what can be visually grounded What to do with unverifiable statements & expressions of uncertainty in model outputs? Note that some model responses introduce information that cannot be verified from the image alone. These are statements that add external facts, background explanations, or narrative details that cannot be inferred directly from the image. If the statement introduces new factual information that cannot be verified from the image, it should generally be treated as a hallucination. For example: • Hallucination to annotate: statements of factual information not visible in the image (“This is a beach in Thailand” for a generic-looking tropical beach). • Plausible visual inference: do not annotate: “This is a street in Paris” when the Eiffel Tower is in the background. Model responses may also include uncertainty-marking language (e.g., “it looks like”, “it seems”, “possibly”, “it might be”). In such cases, do not annotate the uncertainty phrase itself. Instead, evaluate whether the underlying claim is visually supported. Examples: • "The man is a teacher." -> highlight "teacher" (unless the image very clearly indicates this is the case, for example if he is standing in front of a classroom of students.) • "This restaurant serves Mexican food." -> highlight "Mexican" (when the restaurant in the image could serve other types of food, there is no recognizable food in sight nor any other sign of the restaurant being Mexican) • "He looks sad because he lost his job" in an image with clear visual cues connecting sadness to job loss. -> DO NOT HIGHLIGHT (Not a hallucination: need to consider "sadness" being supported by facial expression and posture, Job loss is visually grounded by: a termination notice, packing office belongings into a box, etc.) Figure 8: Annotator instructions, part 1 of 2: overview and general definitions. Proposed taxonomy of hallucinations This classification consists of 5 classes of hallucinations. A. Invention The highlighted span refers to an entity, object, property, or event that is not present in the image at all. Includes: fabricated objects, invented people, imagined actions unsupported by the image Examples: • "red bus" (no bus visible) • "a dog nearby" (no dog visible) • "an umbrella" (not present) Decision rule: If the entity does not exist in the image choose A B. Identity incongruity (Mischaracterization) The span refers to something that is present in the image but describes it incorrectly. Includes: • Wrong object type (taxi -> bus) • Wrong color (blue shirt -> green) • Wrong identity (man -> woman) • Wrong activity (standing -> sitting) • Incorrect spatial relations between objects or entities (below -> above, right -> left) Examples: • Calling a van a "bus" • Describing a blue shirt as "green" • Describing a dog running when the animal is standing or laying down • Saying a cup is on the table when it is actually in someone’s hand • Saying a person is behind the car when they are standing next to it Decision rule: If the object exists but is misidentified or misdescribed choose B C. Wrong Reading / OCR Problem The hallucination arises from misreading text that is actually visible in the image. Applies only if there is readable text in the image and the model misreads or alters it. Examples: • Image: "STOP" but response has to do with "SHOP" • Image: "50% OFF" but response is about "30% OFF" Do NOT use C in case: • The text does not exist in the image. Use A in this case • The error is numerical counting (not text reading). Use D in this case D. Numeric Discrepancy / Miscounting The span expresses an incorrect quantity of visible items. Includes: • Wrong count of people, cars, animals • Incorrect quantities • Explicit numeric misstatements Examples: • "three people" (only two visible) • "two dogs" (only one visible) • "N something" (the thing is uncountable) Do NOT use D in case: If a number is misread from visible written text. Use C instead. E. Other (Use Sparingly) A hallucination that does not fit categories A to D. Try to use as minimally as possible. When using this tag, please leave a comment indicating why you have used it. Figure 9: Annotator instructions, part 2 of 2: classification of hallucinations. Annotation interface We developed a bespoke annotation interface using a stack composed of Django 5, jQuery and Bootstrap, deployed with NGINX and Gunicorn. The platform was hosted on a server and used by annotators both for the training phase and for the actual annotation experiment. Figure˜10 provides a screenshot of the annotation interface for an experimental item, including the image, question, and answer. Annotators could highlight spans directly in the answer text. Upon highlighting a span, a box appeared asking them to assign one label to the selected span (Figure˜11). Annotated spans were color-coded according to their assigned labels (Figure˜12). Annotators were also free to leave additional remarks in a dedicated comment box. A similar platform with minor modifications was used to collect human-written samples, as shown in Figure˜13. Figure 10: Screenshot of the annotation interface used (experimental item). Figure 11: Screenshot of the annotation interface used (hallucination category selection box). Figure 12: Screenshot of the annotation interface used (hallucination span color-coding). Figure 13: Screenshot of the collection interface used for human-written samples. The pie chart on top of the page provides the distribution of intended labels. Appendix C Supplementary results Figure 14: Distribution of labels assigned by an LLM judge (Gemma3). Label agreement (longest) Label agreement (first) All labels Hallu. only All labels Hallu. only Random 0.33460.3346 ± 0.00670.0067 0.34530.3453 ± 0.00880.0088 0.37160.3716 ± 0.01500.0150 0.43280.4328 ± 0.01480.0148 MAP 0.39250.3925 ± 0.01020.0102 0.41260.4126 ± 0.00890.0089 0.44750.4475 ± 0.01410.0141 0.50200.5020 ± 0.01130.0113 Human 0.43950.4395 ± 0.01490.0149 0.47130.4713 ± 0.01630.0163 0.58810.5881 ± 0.01450.0145 0.63180.6318 ± 0.01890.0189 model=gemma3 0.44270.4427 ± 0.01420.0142 0.46110.4611 ± 0.01340.0134 0.50890.5089 ± 0.01980.0198 0.53850.5385 ± 0.01640.0164 model=internvl 0.41140.4114 ± 0.01430.0143 0.42820.4282 ± 0.01420.0142 0.45950.4595 ± 0.02260.0226 0.48660.4866 ± 0.02220.0222 model=llava 0.34190.3419 ± 0.01220.0122 0.37300.3730 ± 0.01270.0127 0.38100.3810 ± 0.01490.0149 0.44630.4463 ± 0.01880.0188 model=minicpm 0.39140.3914 ± 0.01130.0113 0.40490.4049 ± 0.01160.0116 0.40360.4036 ± 0.02010.0201 0.47460.4746 ± 0.02530.0253 model=qwenvl 0.26260.2626 ± 0.01090.0109 0.27860.2786 ± 0.00860.0086 0.36280.3628 ± 0.02050.0205 0.42890.4289 ± 0.01830.0183 language=EN 0.36980.3698 ± 0.01030.0103 0.40330.4033 ± 0.00680.0068 0.38710.3871 ± 0.01110.0111 0.45010.4501 ± 0.01430.0143 language=FR 0.31040.3104 ± 0.00910.0091 0.32040.3204 ± 0.00630.0063 0.41190.4119 ± 0.02520.0252 0.45920.4592 ± 0.02730.0273 language=IT 0.39630.3963 ± 0.01260.0126 0.40900.4090 ± 0.01460.0146 0.41230.4123 ± 0.02050.0205 0.45990.4599 ± 0.01940.0194 language=ZH 0.42210.4221 ± 0.00920.0092 0.43800.4380 ± 0.01160.0116 0.47850.4785 ± 0.02400.0240 0.53600.5360 ± 0.01700.0170 Table 8: Effects of agglomeration policy for multi-span annotations on label annotator agreement for Model-assisted pre-selected (MAP), random and human data, as well as separately for generating model, and output language (on MAP + random data). C.1 Detailed results for IAA assessments In Table˜9, we present IAA measurements per strategy; in Table˜10, per language and model. These correspond to Figure˜3 (left) and Figure˜3 (center, right) respectively. Effect of label aggregation process on IAA. In Section˜5.1, we mention that, when computing label agreement, we aggregate observations from annotators by simply selecting the label of the first span provided by an annotator. An alternative strategy consists of selecting the label of the longest span instead, in terms of character coverage. A comparison of these two strategies is provided in Table˜8: as is apparent, the first-span approach yields higher IAA scores than the longest-span strategy; differences are statistically significant under our bootstrapping. It is not entirely clear to us why this is the case, we conjecture that it has to do with hallucination ‘snowballing’ (Zhang et al., 2025), which annotators might react to in different ways. Agreement Random MAP Human Type Metric Pres. 0.40950.4095 ± 0.00600.0060 0.46160.4616 ± 0.01060.0106 0.56680.5668 ± 0.01470.0147 Label Unconditional 0.34910.3491 ± 0.00470.0047 0.41400.4140 ± 0.01110.0111 0.47100.4710 ± 0.01770.0177 Given a hallu. 0.42700.4270 ± 0.01320.0132 0.49710.4971 ± 0.00710.0071 0.63420.6342 ± 0.02510.0251 Span Requiring label match 0.25810.2581 ± 0.00560.0056 0.31330.3133 ± 0.00430.0043 0.44470.4447 ± 0.01700.0170 Ignoring labels 0.35960.3596 ± 0.00690.0069 0.42100.4210 ± 0.00600.0060 0.53870.5387 ± 0.00660.0066 Table 9: Annotator agreement for Random, MAP, and Human data. Values reported as mean ± 95% bootstrap confidence interval, sampling items with replacement. Generating LVLM Language Metric gemma3 internvl llava minicpm qwenvl EN FR IT ZH Presence 0.50130.5013 ± 0.02410.0241 0.48850.4885 ± 0.01700.0170 0.42600.4260 ± 0.00980.0098 0.47180.4718 ± 0.01210.0121 0.30090.3009 ± 0.01150.0115 0.46640.4664 ± 0.01650.0165 0.32360.3236 ± 0.01650.0165 0.46160.4616 ± 0.00660.0066 0.49500.4950 ± 0.01780.0178 Label (longest/all) 0.44050.4405 ± 0.01010.0101 0.42420.4242 ± 0.01470.0147 0.34170.3417 ± 0.00750.0075 0.38540.3854 ± 0.01240.0124 0.26340.2634 ± 0.01450.0145 0.37140.3714 ± 0.00930.0093 0.30510.3051 ± 0.00830.0083 0.38910.3891 ± 0.00930.0093 0.42750.4275 ± 0.01650.0165 Label (first/all) 0.45960.4596 ± 0.01400.0140 0.42620.4262 ± 0.01190.0119 0.37580.3758 ± 0.01860.0186 0.41260.4126 ± 0.01130.0113 0.27610.2761 ± 0.01980.0198 0.39850.3985 ± 0.00690.0069 0.32610.3261 ± 0.01130.0113 0.40810.4081 ± 0.01160.0116 0.43900.4390 ± 0.00910.0091 Label (longest/hallu.) 0.49190.4919 ± 0.02610.0261 0.46910.4691 ± 0.02120.0212 0.38500.3850 ± 0.01830.0183 0.41760.4176 ± 0.01340.0134 0.36730.3673 ± 0.01230.0123 0.37940.3794 ± 0.01320.0132 0.41650.4165 ± 0.01080.0108 0.42360.4236 ± 0.02200.0220 0.48770.4877 ± 0.01570.0157 Label (first/hallu.) 0.53910.5391 ± 0.02990.0299 0.49580.4958 ± 0.02010.0201 0.45170.4517 ± 0.01530.0153 0.47650.4765 ± 0.01190.0119 0.42150.4215 ± 0.02770.0277 0.45980.4598 ± 0.01820.0182 0.44490.4449 ± 0.01780.0178 0.46970.4697 ± 0.01390.0139 0.53680.5368 ± 0.01140.0114 Span (ignoring labels) 0.43640.4364 ± 0.01860.0186 0.44860.4486 ± 0.01070.0107 0.38840.3884 ± 0.00860.0086 0.37990.3799 ± 0.00540.0054 0.32200.3220 ± 0.01030.0103 0.35410.3541 ± 0.00670.0067 0.38650.3865 ± 0.01340.0134 0.42570.4257 ± 0.01000.0100 0.42510.4251 ± 0.01100.0110 Span (Requiring label match) 0.31850.3185 ± 0.01270.0127 0.32770.3277 ± 0.01470.0147 0.28620.2862 ± 0.00670.0067 0.28970.2897 ± 0.01080.0108 0.23620.2362 ± 0.01480.0148 0.25280.2528 ± 0.00660.0066 0.28660.2866 ± 0.00610.0061 0.31170.3117 ± 0.01340.0134 0.32420.3242 ± 0.01360.0136 Table 10: Annotator agreement grouped by generating LVLM and output language on the random and MAP subsets. See Table˜9 for an overview of the metrics. IAA scores are reported as mean ± 95% bootstrap confidence interval. C.2 Supplementary results on Gemma3-assisted labeling acc prec rec F1 overall human 0.664 3750.664\,375 0.575 7070.575\,707 0.508 2430.508\,243 0.498 9130.498\,913 gemma3 0.542 7630.542\,763 0.504 9820.504\,982 0.444 4660.444\,466 0.442 0320.442\,032 internvl 0.531 9150.531\,915 0.460 3290.460\,329 0.383 3150.383\,315 0.400 7340.400\,734 llava 0.574 4680.574\,468 0.464 6560.464\,656 0.489 8720.489\,872 0.455 1470.455\,147 minicpm 0.485 4370.485\,437 0.491 5280.491\,528 0.393 2000.393\,200 0.409 6730.409\,673 qwenvl 0.459 4590.459\,459 0.503 4130.503\,413 0.412 1790.412\,179 0.426 8780.426\,878 English human 0.695 0000.695\,000 0.582 5020.582\,502 0.543 9110.543\,911 0.510 7810.510\,781 gemma3 0.521 7390.521\,739 0.685 5160.685\,516 0.396 5310.396\,531 0.391 2590.391\,259 internvl 0.516 1290.516\,129 0.462 5370.462\,537 0.303 1950.303\,195 0.328 0040.328\,004 llava 0.567 9010.567\,901 0.590 8570.590\,857 0.544 4740.544\,474 0.515 8050.515\,805 minicpm 0.448 7180.448\,718 0.457 9370.457\,937 0.344 3440.344\,344 0.344 7030.344\,703 qwenvl 0.402 1740.402\,174 0.405 2420.405\,242 0.321 8460.321\,846 0.306 6160.306\,616 French human 0.677 5000.677\,500 0.576 9330.576\,933 0.517 6470.517\,647 0.509 4760.509\,476 gemma3 0.539 4740.539\,474 0.442 2980.442\,298 0.532 8280.532\,828 0.449 8390.449\,839 internvl 0.463 4150.463\,415 0.324 6030.324\,603 0.358 9610.358\,961 0.321 2230.321\,223 llava 0.538 4620.538\,462 0.441 2820.441\,282 0.545 6900.545\,690 0.448 9900.448\,990 minicpm 0.480 5190.480\,519 0.468 0300.468\,030 0.415 2780.415\,278 0.420 9880.420\,988 qwenvl 0.426 6670.426\,667 0.513 1730.513\,173 0.481 4010.481\,401 0.458 2230.458\,223 Italian human 0.650 0000.650\,000 0.582 4110.582\,411 0.486 7220.486\,722 0.482 3030.482\,303 gemma3 0.539 4740.539\,474 0.504 6500.504\,650 0.475 4390.475\,439 0.451 8610.451\,861 internvl 0.618 4210.618\,421 0.407 3430.407\,343 0.380 4350.380\,435 0.383 1050.383\,105 llava 0.676 0560.676\,056 0.554 0820.554\,082 0.515 7020.515\,702 0.516 4840.516\,484 minicpm 0.518 5190.518\,519 0.609 3050.609\,305 0.412 8600.412\,860 0.437 3020.437\,302 qwenvl 0.433 3330.433\,333 0.450 0000.450\,000 0.523 0160.523\,016 0.458 1200.458\,120 Chinese human 0.635 0000.635\,000 0.562 9490.562\,949 0.495 0670.495\,067 0.491 0000.491\,000 gemma3 0.566 2650.566\,265 0.520 2660.520\,266 0.518 0650.518\,065 0.505 6750.505\,675 internvl 0.532 2580.532\,258 0.596 9230.596\,923 0.457 9550.457\,955 0.505 2340.505\,234 llava 0.507 6920.507\,692 0.365 2470.365\,247 0.449 1510.449\,151 0.390 3690.390\,369 minicpm 0.493 1510.493\,151 0.422 1130.422\,113 0.385 7490.385\,749 0.393 3170.393\,317 qwenvl 0.594 2030.594\,203 0.572 1210.572\,121 0.371 8430.371\,843 0.424 6210.424\,621 Table 11: Gemma3 classification full results. Full classification results. In Table˜11, we provide a complete overview of classification metrics for the label assessment using the Gemma3 LLM-judge. Similar observations as mentioned in the main text hold. Gemma3 assessment across languages and generating LVLM. For LLM-based external labels, we additionally study the distribution of labels assigned by the Gemma3 judge across models and languages, as shown in Figure 14. While the distribution of LLM-based assessment across languages is stable, noteworthy patterns emerge when comparing models: Gemma3 rates its output as less likely to contain hallucinations, whereas Internvl and Llava are rated as containing more mischaracterizations. C.3 Supplementary results on HalluShift++ HalluShift++ implementation details. Our HalluShift++ baseline follows the intuition of Nath et al. (2025): hallucinated content can be detected from internal model signals, including uncertainty, representation shifts, and attention behavior. Given an image, a question, and a generated answer, we perform a teacher-forced forward pass through an evaluating VLM. This evaluation model can either be the same LVLM that produced the answer, corresponding to a model-dependent self-probing setup, or a separate LVLM used to evaluate outputs generated by other models. In all cases, the original generation process remains unchanged; the forward pass is used only post hoc to extract token-level features. For each response token, we extract uncertainty-, representation-, and attention-based features. The uncertainty features include the maximum softmax probability, its reciprocal as a confidence proxy, and the probability assigned to the observed token under teacher forcing. From the latter, we compute token-level negative log-likelihood and perplexity. The representation features include layer-wise hidden-state norms and layer-consistency scores, computed as the normalized cosine similarity between an early and a late decoder layer, together with their complements. The attention features include layer-wise attention means and entropies, as well as attention-concentration statistics based on the mean and standard deviation of Gini coefficients over the last attention layers. We additionally include sequence-level confidence and token-pattern features. These comprise the mean and standard deviation of the inverse maximum probability, the confidence trend across the answer, the mean confidence, the fraction of low-confidence tokens, repetition ratios, and normalized lexical diversity. Since our prediction target is token-level, sequence-level features are attached to every token of the corresponding answer. To obtain token-level supervision, we align each decoded token with its character span in the generated answer. Tokens whose character span overlaps an annotated hallucination span are labeled as hallucinated; tokens with no overlap are labeled as non-hallucinated. In the multiclass setting, hallucinated tokens are assigned the corresponding hallucination type. We then train a lightweight multilayer perceptron over the resulting token-level feature vectors, using the same classifier configuration as Nath et al. (2025). Our adaptation differs from the original HalluShift++ formulation in one main respect: while HalluShift++ decomposes generated descriptions into semantic chunks, such as object, attribute, and relation units, our annotations are span-based. We therefore align internal signals directly to decoded tokens and train a token-level classifier. Strategy acc prec rec F1 random 0.658 5660.658\,566 0.179 7250.179\,725 0.671 5800.671\,580 0.283 5640.283\,564 MAP 0.633 2100.633\,210 0.267 7260.267\,726 0.700 2560.700\,256 0.387 3550.387\,355 human-written 0.565 4020.565\,402 0.262 8360.262\,836 0.846 1480.846\,148 0.401 0850.401\,085 (a) HalluShift++ performance (accuracy, recall, precision, F1) across strategies. random MAP all gemma3 0.256 9880.256\,988 0.418 3440.418\,344 0.324 4600.324\,460 internvl 0.280 2900.280\,290 0.382 3420.382\,342 0.335 7590.335\,759 llava 0.398 2390.398\,239 0.401 8630.401\,863 0.400 5870.400\,587 minicpm 0.276 3980.276\,398 0.345 3220.345\,322 0.312 2470.312\,247 qwenvl 0.247 7870.247\,787 0.391 0950.391\,095 0.326 9510.326\,951 (b) HalluShift++ F1 per generating LVLM. Table 12: HalluShift++ performance (binary label-classification setup). Performance in binary setting. In Table˜12, we provide an overview of performances for the HalluShift++ method based on span detection, framing the problem as a binary token-labeling task instead (i.e., F vs. all). Similar conclusions emerge: a very low precision severely curtails the performances of these models. In Table˜12(b), we see a significant downgrade in performance for randomly selected items. C.4 Span assessment with EUQ Another baseline we consider is based on Evidential Uncertainty Quantification (EUQ; Huang et al., 2026). It provides a training-free framework that explicitly decomposes token-level epistemic uncertainty into different types of misbehaviors. The framework applies basic belief assignment Dempster (1967) to the token-level pre-logits feature obtained from the output head of LVLM to compute evidence weights. These weights are then decomposed into positive and negative components, which are fused to estimate the final uncertainties that can detect conflict and ignorance. Conflict refers to error caused by the difference in the image and text modality; whereas, ignorance refers to error caused by missing information. Using the optimal threshold obtained from a calibration set for conflict and ignorance, we filter spans for containing hallucinations. acc prec rec F1 strategy random 0.794 4250.794\,425 0.195 6300.195\,630 0.198 1890.198\,189 0.196 9010.196\,901 MAP 0.733 1110.733\,111 0.269 0080.269\,008 0.167 3430.167\,343 0.206 3320.206\,332 human-written 0.719 3560.719\,356 0.266 7730.266\,773 0.214 8560.214\,856 0.238 0170.238\,017 random gemma3 0.816 7690.816\,769 0.179 0500.179\,050 0.230 0090.230\,009 0.201 3550.201\,355 internvl 0.804 3120.804\,312 0.201 1250.201\,125 0.219 3250.219\,325 0.209 8310.209\,831 llava 0.708 0150.708\,015 0.330 6880.330\,688 0.176 7320.176\,732 0.230 3540.230\,354 minicpm 0.799 4200.799\,420 0.190 8150.190\,815 0.213 2860.213\,286 0.201 4260.201\,426 qwenvl 0.809 3960.809\,396 0.155 7390.155\,739 0.187 9010.187\,901 0.170 3150.170\,315 MAP gemma3 0.753 4490.753\,449 0.322 9380.322\,938 0.173 3630.173\,363 0.225 6110.225\,611 internvl 0.756 9900.756\,990 0.311 3340.311\,334 0.186 6930.186\,693 0.233 4170.233\,417 llava 0.685 9570.685\,957 0.320 4060.320\,406 0.146 4950.146\,495 0.201 0610.201\,061 minicpm 0.764 1960.764\,196 0.274 0890.274\,089 0.223 8530.223\,853 0.246 4370.246\,437 qwenvl 0.735 3670.735\,367 0.213 7530.213\,753 0.152 2830.152\,283 0.177 8570.177\,857 all gemma3 0.794 0340.794\,034 0.225 9080.225\,908 0.199 6410.199\,641 0.211 9640.211\,964 internvl 0.779 8990.779\,899 0.255 6820.255\,682 0.198 4200.198\,420 0.223 4410.223\,441 llava 0.693 8470.693\,847 0.324 2480.324\,248 0.156 7130.156\,713 0.211 3010.211\,301 minicpm 0.782 4740.782\,474 0.232 1120.232\,112 0.219 3500.219\,350 0.225 5500.225\,550 qwenvl 0.773 6020.773\,602 0.184 7170.184\,717 0.165 5230.165\,523 0.174 5940.174\,594 Table 13: Span detection with EUQ, token-level classification results. Results are shown in Table˜13. The high label imbalance between hallucinated and non-hallucinated tokens leads to poor performances out-of-the-box, lower than what we observed with HalluShift++ in Table˜12. For accuracy, recall, and precision, we can always find specific LVLM-based setups that either under- or over-perform with respect to the human-written samples subset, although the human-written data does yield the highest F1 score. Appendix D Disclaimers Use of existing artifacts. We rely on preexisting models and datasets to construct our own dataset; the use is compatible with explicit licenses. Use of AI tools. AI technology was used to jump-start coding. Authors take full responsibility for the contents of this paper. Hardware and compute. Our experiments were run on a variety of compute clusters, including V100, A100, RTX A6000, MI250x, H1000. LVLM answer generation for all models except Gemma3 was carried out on a node equipped with two NVIDIA RTX A6000 GPUs. Generation required approximately 4.5 seconds per sample across almost all models, whereas Qwen3-VL required approximately 9 seconds per sample on average. LVLM answer generation for Gemma3 was carried out on a node equipped with one AMD MI250x GPU. Generation required approximately 6 seconds per sample on average.

Instant research discovery