Paper deep dive

GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensics

Yan Zhang, Simiao Ren, Ankit Raj, En Wei, Dennis Ng, Alex Shen, Jiayue Xu, Yuxin Zhang, Evelyn Marotta

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 64

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/22/2026, 6:23:40 AM

Summary

The paper introduces GPT4o-Receipt, a benchmark dataset of 1,235 receipt images (935 AI-generated, 300 authentic) designed to evaluate AI document forensics. It reveals a 'visual-arithmetic asymmetry' where humans excel at identifying visual artifacts but fail to detect AI-generated documents, while multimodal LLMs outperform humans by identifying systematic arithmetic errors that are invisible to visual inspection.

Entities (5)

GPT-4o · generative-ai-model · 100%GPT4o-Receipt · dataset · 100%Claude Sonnet 4 · multimodal-llm · 95%Gemini 2.5 Flash · multimodal-llm · 95%Arithmetic Errors · forensic-signal · 90%

Relation Signals (3)

GPT4o-Receipt → contains → AI-generated receipts

confidence 100% · We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones

Claude Sonnet 4 → outperforms → Human annotators

confidence 95% · their binary detection F1 falls well below Claude Sonnet 4

Arithmetic Errors → detectedby → Multimodal LLMs

confidence 90% · arithmetic errors—invisible to visual inspection but systematically verifiable by LLMs

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Can humans detect AI-generated financial documents better than machines? We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones from established datasets, evaluated by five state-of-the-art multimodal LLMs and a 30-annotator crowdsourced perceptual study. Our findings reveal a striking paradox: humans are better at seeing AI artifacts, yet worse at detecting AI documents. Human annotators exhibit the largest visual discrimination gap of any evaluator, yet their binary detection F1 falls well below Claude Sonnet 4 and below Gemini 2.5 Flash. This paradox resolves once the mechanism is understood: the dominant forensic signals in AI-generated receipts are arithmetic errors -- invisible to visual inspection but systematically verifiable by LLMs. Humans cannot perceive that a subtotal is incorrect; LLMs verify it in milliseconds. Beyond the human--LLM comparison, our five-model evaluation reveals dramatic performance disparities and calibration differences that render simple accuracy metrics insufficient for detector selection. GPT4o-Receipt, the evaluation framework, and all results are released publicly to support future research in AI document forensics.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

63,842 characters extracted from source content.

Expand or collapse full text

GPT4o-Receipt: A Dataset and human study for AI-Generated Document Forensics Yan Zhang ∗ , Simiao Ren ∗† , Ankit Raj, En Wei, Dennis Ng, Alex Shen, Jiayue Xu, Yuxin Zhang, Evelyn Marotta ∗ Equal contribution † Corresponding author: benren@scam.ai Abstract—Can humans detect AI-generated financial documents better than machines? We present GPT4o-Receipt, a benchmark of 1,235 receipt images pairing GPT-4o-generated receipts with authentic ones from established datasets, evaluated by five state- of-the-art multimodal LLMs and a 30-annotator crowdsourced perceptual study. Our findings reveal a striking paradox: humans are better at seeing AI artifacts, yet worse at detecting AI documents. Human annotators exhibit the largest visual discrimination gap of any evaluator, yet their binary detection F1 falls well below Claude Sonnet 4 and below Gemini 2.5 Flash. This paradox resolves once the mechanism is understood: the dominant forensic signals in AI-generated receipts are arithmetic errors—invisible to visual inspection but systematically verifiable by LLMs. Humans cannot perceive that a subtotal is incorrect; LLMs verify it in milliseconds. Beyond the human–LLM comparison, our five-model evalu- ation reveals dramatic performance disparities and calibration differences that render simple accuracy metrics insufficient for detector selection. GPT4o-Receipt, the evaluation framework, and all results are released publicly to support future research in AI document forensics. I. INTRODUCTION A. The Rise of AI-Generated Documents The emergence of large-scale generative artificial intelligence (AI) models has fundamentally altered the landscape of digital content creation. Text-to-image (T2I) models such as Stable Diffusion [1] and the native image generation capabilities of GPT-4o [2] can now synthesize photorealistic images from natural language descriptions, while large language models (LLMs) [3] can compose coherent text across virtually any domain. These capabilities, while transformative for creative and scientific applications, simultaneously lower the barrier to a particularly consequential form of misuse: the automated generation of fraudulent financial documents. Unlike traditional forgeries produced by manipulating authentic documents in image editing software, AI-generated documents are synthe- sized entirely from scratch, bypassing forensic approaches that rely on detecting pixel-level traces of manipulation [4], [5] and posing challenges for human reviewers whose intuitions are calibrated to natural photographic artifacts. B. Receipts as a Multi-Dimensional Benchmark Domain Financial receipts represent a compelling and practically important domain for studying AI document forensics, for several reasons. First, receipts are ubiquitous in everyday commercial life: they document retail transactions, expense claims, insurance submissions, and tax filings, making their authenticity consequential in a broad range of real-world 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate (Recall) Claude Sonnet 4 Gemini 2.5 Flash GPT-5 Nano Grok 4 LLaMA 4 Scout Human Better Detector Calibration: Recall vs. FPR Fig. 1. Recall vs. false positive rate for each detector. Upper-left is better. Claude Sonnet 4 achieves the highest overall detection (F1 = 0.975); Gemini 2.5 Flash exhibits the best calibration among effective detectors (lowest FPR = 0.023); Grok 4 reaches near-perfect recall at a 90.3% FPR; LLaMA 4 Scout has the lowest FPR but misses 89% of AI receipts. Human annotators (star) occupy a mid-tier position with moderate recall and low FPR. settings. Second, receipts are informationally rich in a way that makes them challenging to generate convincingly: a realistic receipt must simultaneously exhibit plausible visual properties (appropriate typography, merchant-specific layout, correct paper texture), plausible factual content (item names and prices consistent with the merchant type, a recognizable address format), and internal logical consistency (prices that sum correctly, taxes computed from the correct rate). This combination of visual, semantic, and arithmetic constraints makes receipts an unusually stringent multi-dimensional stress test for generative AI systems—a model that renders numbers as visual tokens, without performing or verifying computations, may produce documents that look superficially correct but fail the logical checks that forensic analysts and automated systems can apply. Third, and most importantly for the present work, receipts are a domain where the forensic capabilities of humans and automated systems are likely to diverge in interesting and practically important ways. Humans are skilled at visual arXiv:2603.11442v1 [cs.AI] 12 Mar 2026 anomaly detection but have limited capacity for rapid arithmetic verification at scale; LLMs can perform systematic cross-field verification but may or may not share humans’ perceptual sensitivity to visual generation artifacts. Understanding this divergence is essential for designing hybrid human-AI forensic workflows. C. The Human–LLM Detection Gap Existing work on synthetic media detection has largely focused on artifacts introduced by generative models at the pixel level, such as frequency-domain signatures [5] or classifier- based detection of GAN artifacts [4]. A newer generation of forensic approaches leverages multimodal LLMs to perform holistic, reasoning-based analysis of document images [6], [7]. However, no prior work has systematically compared the detection performance of humans and multimodal LLMs on a purpose-built dataset of AI-generated financial documents, nor has any existing benchmark been constructed to support this comparison. The relative strengths of human perceptual judgment and LLM-based logical verification remain an open and practically important question: understanding where each approach succeeds and fails is a prerequisite for building effective forensic systems. D. Contributions This paper makes three primary contributions. First, we in- troduce GPT4o-Receipt, a dataset of 1,235 receipt images with binary ground-truth labels, spanning 935 AI-generated receipts (GPT-4o, 159 merchant categories) and 300 authentic receipts sourced from publicly available receipt image collections [8], [9]. The dataset is designed to support both automated and human evaluation, with controlled coverage across merchant domains and documented provenance for all images. Second, we present a crowdsourced human perceptual study in which 30 annotators collectively evaluated all 1,235 images across structured visual quality dimensions (each annotator assessed approximately 100 images), providing the first quantitative characterization of human visual detection performance on fully AI-generated financial documents. Third, we benchmark five state-of-the-art multimodal LLMs as zero-shot forensic detectors on the same corpus, evaluating each across three forensic dimensions—visual realism, arithmetic integrity, and factual consistency—and comparing their performance directly to the human baseline. The central finding to emerge from this comparison is a visual–arithmetic asymmetry: humans are more sensitive visual discriminators than most LLMs, but the best LLMs outperform humans in binary detection by leveraging consistency signals that are imperceptible to visual inspection alone. The remainder of this paper is structured as follows. Sec- tion I reviews related work. Section I describes the GPT4o- Receipt dataset. Section IV details the evaluation framework. Sections V and VI present LLM and human results respectively. Section VII discusses implications, and Section VIII concludes. I. RELATED WORK A. Generative Models and Numerical Hallucination Diffusion models [1], [10] and autoregressive image genera- tors [2] can synthesize photorealistic documents from natural language descriptions, but treat all image content—including numerical text—as visual patterns rather than values to be computed. This produces a text hallucination failure mode in which characters are rendered plausibly but assembled into numerically incoherent strings; GeckoNum [11] confirms with 479,000 annotations that state-of-the-art T2I models fail system- atically on numerical content [12]. Large language models [3] similarly exhibit arithmetic hallucination, generating plausible- looking but incorrect calculations. The same limitation extends to perception: HallusionBench [13] and VisNumBench [14] show that frontier MLLMs fail systematically on tasks requiring precise numerical extraction from images. These findings provide mechanistic grounding for our core observation: GPT- 4o’s image generator produces visually convincing receipt layouts, but the numbers it renders are sampled from a distribution of receipt-like values rather than computed from line items. B. Image and Document Forgery Detection Traditional forensics detected pixel-level manipulation ar- tifacts such as printer noise inconsistencies and copy-paste traces [4], [5], [15]. TruFor [16] extends this to transformer- based architectures with pixel-level localization. In the docu- ment domain, DocTamper [17] provides 170,000 annotated document images with a Frequency Perception Head for robust detection under compression; the ICDAR 2023 competi- tion [18] advanced the state of the art with 11,385 community- benchmarked images. More recently, OSTF [19] benchmarks eight methods on diffusion-model inpainting attacks, and RealDTT [20] advances ecological validity with 304,000 real- world tampered text images. Fully AI-synthesized documents are free of the frequency signatures and copy-paste artifacts of traditional forgeries [21]. AIForge-Doc [22] evaluates TruFor and DocTamper on diffusion-model inpainting of financial documents, finding both degrade sharply out-of-distribution. Standard perceptual metrics—FID [23], LPIPS [24], CLIPScore [25]—cannot assess arithmetic correctness, and cross-generator generalization remains unsolved [26]. Complementary detection approaches include diffusion reconstruction error [27] and CLIP-feature- based detectors [28]. Real-world reliability studies confirm that high-performing benchmark models degrade in deploy- ment [29], [30], a pattern of distributional sensitivity also observed in facial analysis [31] and satellite segmentation [32]. C. Receipt and Financial Document Forensics Tornés et al. [33] introduce a 988-image receipt forgery dataset with image-level and transcription-level annotations covering copy-paste, text imitation, and pixel modification. Schmidberger et al. [34] use fine-tuned LLMs to generate executable plausibility-check rules for financial document consistency—anticipating our finding that arithmetic consis- tency is an efficient forensic filter. Wang et al. [35] propose CSIAD, which compares suspicious payment records against a reference corpus, achieving a 79.6% F1 improvement over prior state-of-the-art for financial fraud detection. On the human perception side, crowdsourced studies show AI detection accuracy near chance [36], with GAN-generated faces rated as more trustworthy [37] or more realistic [38] than real ones—a baseline against which our human study’s performance can be contextualized. Table I summarizes existing document forgery datasets alongside GPT4o-Receipt. GPT4o-Receipt is the first to contain receipts synthesized entirely from scratch by a generative AI model, rather than authentic documents subjected to pixel-level editing or regional inpainting. D. Multimodal LLMs as Forensic Detectors Jia et al. [6] demonstrate that GPT-4V achieves up to 83.4% accuracy on deepfake detection benchmarks through semantic inconsistency detection, establishing the viability of zero-shot MLLM reasoning as a forensic tool. Ren et al. [39] find that chain-of-thought reasoning provides mea- surable but inconsistent gains across forgery types. In the document domain, Liang et al. [40] identify key failure modes when MLLMs encounter subtle or semantically plausible tampering. FakeShield [7] integrates multi-modal detection with forgery localization and natural language explanation, outperforming specialized detectors while providing evidence- grounded forensic reasoning. He et al. [41] find that GPT- 4o often identifies the correct synthesis region but its stated reasoning is inconsistent across equivalent prompts—motivating the structured, dimension-specific prompting adopted in our evaluation framework. I. THE GPT4O-RECEIPT DATASET A. AI-Generated Receipt Collection The AI-generated subset of GPT4o-Receipt was produced using a two-stage pipeline leveraging the OpenAI GPT-4o and GPT-Image-1 APIs. To ensure broad coverage of real-world merchant types, we compiled a taxonomy of 159 merchant categories spanning seven domains: grocery and supermarket chains (e.g., Walmart, Kroger, Whole Foods), quick-service and casual dining restaurants (e.g., McDonald’s, Chipotle, Shake Shack), pharmacies and drug stores (e.g., CVS, Walgreens, Rite Aid), electronics and specialty retailers (e.g., Best Buy, Apple Store, Micro Center), apparel and lifestyle brands (e.g., H&M, Zara, Nike), automotive and hardware stores (e.g., AutoZone, Home Depot, Advance Auto Parts), and fuel and convenience retailers (e.g., Shell, ExxonMobil, Speedway). 1) Stage 1: Textual Receipt Generation: In the first stage, GPT-4o was prompted to generate structured textual receipt content for each merchant. The prompt specified realistic store names and addresses, dates and times of purchase, cashier identifiers, 8–10 line items with correct quantities, unit prices, and item totals, a subtotal with 8.25% tax and final total, and a barcode number. The model was instructed to format output as a monospaced printed receipt with proper spacing and alignment. Each generated text receipt contained a complete header (store name, address, phone number), an itemized body with arithmetic fields (line item totals, subtotal, tax, and grand total), and a footer. The textual outputs were saved as paired .txt files for traceability. 2) Stage 2: Photorealistic Image Rendering: In the second stage, the textual receipt content from Stage 1 was passed to GPT-Image-1 with instructions to render a photorealistic photograph of a physical receipt using the exact text content provided. The image generation prompt specified real-world visual cues including background context, natural lighting, cam- era angle variation, paper creases and texture, and substantial header and footer detail. This two-stage approach—separating content generation from visual rendering—was designed to produce receipts that are simultaneously textually coherent and visually plausible. 3) Generation Characteristics and Known Artifacts: The pipeline produces images with varying levels of visual realism. Observed generation artifacts include: (1) inconsistent sharpness within a single image, such as sudden blurring at the top of a receipt while the body remains sharp; (2) occasional placeholder addresses (e.g., “Anytown, USA”) rather than real merchant locations; and (3) variable environmental realism across generated images. Despite the prompt specifying arith- metically consistent line items, GPT-4o treats numerical content as visual tokens rather than computed values during the image rendering stage, introducing systematic arithmetic errors— subtotals that do not equal the sum of line items, tax amounts inconsistent with the stated rate—that are invisible to casual visual inspection but detectable by automated verification. This arithmetic incoherence proves to be the dominant forensic signal exploited by LLM-based detectors (Section V). For each merchant category, between 1 and 25 re- ceipt images were generated. This process yielded 935 AI- generated receipt images in PNG format (filename convention: receipt_[store]_[index].png), with paired text files, which form the positive (AI) class of the benchmark. B. Real Receipt Collection The authentic receipt subset consists of 300 images drawn from two publicly available sources. The first is the Expres- sExpense receipt image collection [8], a free dataset of real receipt photographs compiled for OCR and machine learning research. The second is the Receipt-or-Invoice dataset hosted on Roboflow Universe [9], which contains labeled receipt and invoice images contributed by the community. Together, these sources provide photographs of genuine printed receipts from a diverse range of merchants, geographic locations, and time periods. We selected images that represent a variety of receipt formats, paper types, and photographic conditions (including varying lighting, perspective, and image quality) to ensure that the real receipt subset reflects the natural distribution of authentic document images encountered in forensic scenarios. TABLE I COMPARISON OF PUBLICLY RELEASED DOCUMENT FORGERY DATASETS. AI-gen: NO = TRADITIONAL IMAGE EDITING (COPY-PASTE, PHOTOSHOP); INPAINTING = AI MODEL USED TO MODIFY REGIONS OF AUTHENTIC DOCUMENTS; FULL = ENTIRE DOCUMENT SYNTHESIZED FROM SCRATCH BY A GENERATIVE AI MODEL. DatasetDocument TypeForgery MethodAI-genForged Tornés et al. [33]ReceiptsCopy-paste, pixel edit, text imitationNo163 DocTamper [17]Mixed docs (receipts, invoices)Pixel-level text editingNo170,000 ICDAR 2023 DTT [18]Scene text / documentsPixel-level region editingNo11,385 OSTF [19]Scene text imagesTrad. editing + diffusion inpaintingInpainting1,980 RealDTT [20]Text images (real-world)Real-world digital editingNo304,000 AIForge-Doc [22]Financial documentsDiffusion-model inpaintingInpainting4,061 GPT4o-Receipt (ours)ReceiptsFull AI synthesis (GPT-4o)Full935 DocTamper, ICDAR 2023 DTT, OSTF, and RealDTT consist entirely of tampered images. AIForge-Doc contains 8,122 images total (4,061 real + 4,061 forged, 1:1 ratio). GPT4o-Receipt contains 935 AI-generated and 300 authentic receipts. AI-Generated (GPT-4o) AI Receipt (GPT-4o) AI Receipt (GPT-4o) AI Receipt (GPT-4o) AI Receipt (GPT-4o) Authentic (CORD/SROIE) Authentic Receipt Authentic Receipt Authentic Receipt Authentic Receipt Sample Receipts Replace with actual dataset images (run scripts/create_receipt_examples.py) Fig. 2. Representative samples from GPT4o-Receipt. Top row: AI-generated receipts produced by the two-stage pipeline (GPT-4o text→GPT-Image-1 rendering); Bottom row: authentic receipts from ExpressExpense and Roboflow. AI-generated receipts exhibit high visual plausibility—realistic fonts, plausible merchant layouts, paper texture—but contain systematic arithmetic errors invisible to casual inspection. C. Dataset Statistics and Splits The complete GPT4o-Receipt dataset comprises 1,235 re- ceipt images: 935 AI-generated (75.7%) and 300 real (24.3%). The AI-generated subset spans 159 distinct merchant categories, with a median of 5 images per category. Geographically, the AI- generated receipts predominantly reflect North American retail formats (United States and Canada), with some representation of international chains (e.g., SPAR in South Africa, Bunnings Warehouse in Australia, Boots in the United Kingdom). Real receipts cover a broader geographic range. All AI-generated images are in PNG format; real receipt images are in JPEG format. Ground-truth labels are determined by filename con- vention: files matching the patternreceipt_ * _ * .pngare AI-generated, while all others are authentic. All 1,235 images were used for both LLM-based forensic evaluation (Section IV) and the human perceptual study (Section VI). IV. FORENSIC EVALUATION FRAMEWORK A. LLM Detector Models We evaluate five state-of-the-art multimodal large language models as zero-shot forensic detectors. All models are accessed via the OpenRouter API and support vision input (i.e., they can analyze image content directly alongside text instructions). Table I summarizes the evaluated models. B. LLM Evaluation Dimensions Each model is prompted to analyze a receipt image across three forensic dimensions, returning a structured JSON re- sponse: Visual Realism (score 1–5): The model assigns an integer score reflecting the overall visual plausibility of the receipt, considering font consistency and rendering quality, paper texture and aging characteristics, layout adherence to real receipt formats, and the presence of visual artifacts indicative of AI generation such as unnaturally uniform typography or implausible backgrounds. Arithmetic Integrity (Pass/Fail with sub-checks): The model verifies three arithmetic properties: (1) sum_check — whether the sum of line-item prices equals the stated subtotal; (2) tax_check — whether the stated tax amount is consistent with the applicable tax rate and subtotal; and (3) rounding_check — whether monetary values follow standard rounding conventions. The overall arithmetic status is Pass only if all applicable sub-checks pass. Factual Consistency (Pass/Fail with sub-checks): The model checks three factual properties: (1) address_check — whether the merchant address, phone number, and store identifier are plausible and internally consistent; (2) items_check — whether the listed products and prices are consistent with the purported merchant type; and (3) dates_check — whether date and time formatting is plausible for the stated region and context. In addition to the dimension-specific assessments, each model produces a binary verdict (is_ai_generated: true/false) with a confidence score in [0, 1] and a natural-language summary explaining its reasoning. C. LLM Evaluation Protocol Each of the 1,235 images in GPT4o-Receipt was analyzed by all five detector models, yielding 6,175 individual forensic assessments. Evaluations were conducted using a LangGraph- based workflow with structured Pydantic output schemas to ensure consistent, machine-parseable responses. The system executed up to 10 API calls concurrently (subject to provider TABLE I MULTIMODAL LLMS EVALUATED AS FORENSIC DETECTORS IN THIS STUDY. ALL MODELS ARE ACCESSED VIA OPENROUTER AND SUPPORT DIRECT IMAGE INPUT. ALL MODELS ACCESSED BETWEEN FEBRUARY 15–25, 2026. AliasFull ModelProviderAPI Model Identifier claude-sonnetClaude Sonnet 4Anthropic claude-sonnet-4-6 gemini-flashGemini 2.5 FlashGoogle gemini-2.5-flash-preview gpt5-nanoGPT-5 NanoOpenAI gpt-5-nano grok4Grok 4xAI grok-4 llama-4-scoutLLaMA 4 ScoutMeta meta-llama/llama-4-scout rate limits), with automatic exponential-backoff retry for transient failures and JSON repair for malformed model responses. A smart-resume mechanism ensured that already- completed assessments were not repeated upon restarting the pipeline, enabling reliable batch processing of the full dataset. Of the 6,175 requested assessments, 6,173 completed successfully (99.97% success rate); 2 records failed due to persistent JSON parsing errors and are excluded from analysis. All evaluations were conducted in February 2026; results may not be reproducible with future model updates. Statistical analysis. All reported performance metrics (ac- curacy, F1, recall, FPR) are computed on the full evaluation set; 95% confidence intervals are obtained by non-parametric bootstrap (2,000 resampling iterations) with bias-corrected percentile intervals. Inter-rater agreement for the human study is reported as Cohen’sκcomputed on the 133 doubly-annotated images. Ground truth for model evaluation is determined by filename convention, not by any pixel-level annotation or manual review. This approach provides a clean, reproducible binary label: receipt images generated by GPT-4o follow the naming pat- ternreceipt_[store]_[index].png, while authentic receipt images do not match this pattern. D. Human Study Protocol A crowdsourced human perceptual study was conducted in parallel to establish a visual baseline for AI receipt detection. The study was administered via a Label Studio annotation platform. Thirty annotators participated, each evaluating a randomized set of approximately 100 receipt images drawn from the full 1,235-image dataset (935 AI-generated, 300 real). Each annotator evaluated the visual appearance of each receipt, ensuring the human study measures pure visual perceptual performance. For each image, annotators answered three structured ques- tions covering typography quality, layout consistency, and artifact presence (each with three ordered response options), and assigned an overall visual realism score from 1 to 5. Receipts were displayed at their original resolution without contextual information. For images with multiple annotations (133 images had 2–3 annotations), responses were aggregated by averaging numeric scores and taking the modal response for categorical questions. The median annotation time per image was 14 seconds. To derive a binary classification decision from the continuous visual realism scores, we apply a pre-specified threshold of ≤ 3 (scores 1–3 are classified as AI-generated; scores 4–5 as authentic). This threshold reflects the scale midpoint and is set a priori, not optimized on the data. The sensitivity of detection metrics to this threshold choice is reported in SectionVI-D. All human detection metrics derived in this way are labeled as inferred estimates; the study did not include a direct binary classification question, and this distinction from the LLM binary verdict is maintained throughout the analysis. V. RESULTS: LLM-BASED FORENSIC DETECTION A. Overall Detection Performance Figure 3 and Table V (reported alongside the human baseline in Section VI) summarize the detection performance of all five models on the full 1,235-image GPT4o-Receipt set. The results reveal a striking range of capability, with F1 scores spanning from 0.204 (LLaMA 4 Scout) to 0.975 (Claude Sonnet 4), a nearly five-fold difference. Importantly, Grok 4’s F1 of 0.873 is only marginally above the majority-class baseline (0.862), inflated by a near-universal positive-prediction strategy rather than genuine discrimination ability, as detailed in the calibration analysis (SectionV-B). Claude Sonnet 4 achieves the highest overall accuracy (0.962; 95% CI [0.950, 0.971]) and F1 score (0.975; [0.967, 0.982]), correctly detecting 909 of 935 AI-generated receipts (recall = 0.972; [0.961, 0.982]) while maintaining a low false positive rate (FPR = 0.070; [0.044, 0.100]). Gemini 2.5 Flash is the second-best detector overall (accuracy = 0.848; [0.826, 0.867]; F1 = 0.890; [0.874, 0.904]) and exhibits the best calibration among high-performing models, with an FPR of only 0.023 ([0.009, 0.044]) compared to Claude’s 0.070. The confidence intervals for Claude and Gemini F1 do not overlap, confirming a statistically meaningful performance difference. GPT-5 Nano takes a more conservative approach, achieving an accuracy of 0.631 with recall of only 0.530—it correctly identifies roughly half of AI-generated receipts—but with a low FPR (0.057), meaning it raises relatively few false alarms on authentic receipts. B. Calibration Analysis Figure 1 visualizes the recall-versus-FPR tradeoff for each model, revealing that the five detectors are not simply ordered by capability but exhibit qualitatively different calibration behaviors. Grok 4 occupies the upper-right extreme: it achieves near-perfect recall (0.999; [0.996, 1.000]), flagging all but one 0.000.250.500.751.00 Score Claude Sonnet 4 Gemini 2.5 Flash GPT-5 Nano Grok 4 LLaMA 4 Scout LLM Detector Performance on GPT4o-Receipt Accuracy F1 Recall FPR Fig. 3. Detection performance of five multimodal LLMs on GPT4o-Receipt. FPR (hatched, ↓ better); Accuracy, F1, Recall (↑ better). AI-generated receipt correctly, but does so by treating nearly every receipt—real or fake—as AI-generated, resulting in an FPR of 0.903 ([0.867, 0.935]). This behavior renders Grok 4 functionally unusable as a practical forensic tool, as it would flood investigators with false alarms. LLaMA 4 Scout occupies the opposite extreme: its FPR of 0.017 ([0.004, 0.034]) is the lowest of all models, but it detects only 11.4% of AI- generated receipts (recall = 0.114; [0.096, 0.136]), making it nearly blind to the forgeries it is tasked with identifying. The human detector (recall = 0.770, FPR = 0.120) occupies a well-calibrated mid-tier position, with low FPR and moderate recall (Figure 1). C. Error Taxonomy Table I and Figure 4 report the rates at which each detector identified specific error types in the AI-generated receipts. Across all models that demonstrate reasonable detection capa- bility, arithmetic errors—particularly sum errors—emerge as the dominant failure category. Claude Sonnet 4 identifies sum errors in 97.2% of AI-generated receipts and tax errors in 75.9%, consistent with its high recall. Gemini 2.5 Flash detects sum errors in 85.9% and tax errors in 78.2%. These rates suggest that the vast majority of AI-generated receipts in GPT4o-Receipt fail basic arithmetic verification when examined by a capable detector. Address-related factual errors are the second-most Sum Fail Tax Fail Round Fail Addr Fail Claude Sonnet 4 Gemini 2.5 Flash GPT-5 Nano Grok 4 LLaMA 4 Scout 97%76%45%60% 86%78%9%25% 63%37%20%23% 96%57%39%55% 10%5%8%4% Error Taxonomy: Failure Rates on AI-Generated Receipts 0% 25% 50% 75% 100% Failure Rate Fig. 4. Failure rates (%) for each error category across AI-generated receipts, as assessed by each detector model. Darker red indicates higher failure rates. LLaMA 4 Scout’s near-zero error detection rates are consistent with its overall failure to identify AI-generated receipts. common failure category: Claude identifies address errors in 59.5% of AI receipts, and Grok 4 in 55.0%. A noteworthy pattern in Table I is that arithmetic error detection rates are strongly correlated with overall detection accuracy. LLaMA 4 Scout, which fails to detect most AI- generated receipts, also reports arithmetic pass rates of 89.3%, consistent with a model that largely accepts receipts as genuine without scrutinizing arithmetic. Claude Sonnet 4 finds arithmetic problems in over 97% of AI receipts, consistent with its near-perfect recall. It is important to note that the error rates in Table I reflect each model’s assessments; these assessments have been confirmed to align with genuine arithmetic errors present in the AI-generated receipts, consistent with the known limitation of GPT-4o treating numbers as visual tokens rather than computed values. A model’s tendency to report high error rates may reflect both genuine error detection and that model’s calibration bias toward flagging AI receipts generally. D. Visual Realism Analysis Figure 5 (see Section VI) presents mean visual realism scores with statistical significance for the AI-vs-real gap; full numerical values with confidence intervals appear in Table VI (Appendix). Only Claude Sonnet 4 (gap = 1.38; p < 10 −220 ) and Gemini 2.5 Flash (gap = 1.24;p < 10 −70 ) exhibit substantial visual discrimination among LLMs. GPT-5 Nano exhibits a near-zero gap (−0.02;p = 0.50) that is not statistically significant. Grok 4 (gap =+0.17;p < 0.05) and LLaMA 4 Scout (gap =+0.09;p < 0.001) show small but statistically significant gaps. Human annotators achieve the largest gap of any evaluator (1.87;p < 10 −120 ; 95% CI [1.74, 1.99]). TABLE I FORENSIC ERROR RATES ON AI-GENERATED RECEIPTS (%). “ARITH. PASS” AND “FACTUAL PASS” ARE THE FRACTIONS WITH NO DETECTED ERRORS IN EACH CATEGORY; REMAINING COLUMNS SHOW SPECIFIC FAILURE RATES. THESE RATES REFLECT EACH MODEL’S assessments, NOT INDEPENDENTLY VERIFIED GROUND-TRUTH ERRORS. ModelArith. PassSum FailTax FailRound FailFactual PassAddr Fail Claude Sonnet 42.697.275.945.217.259.5 Gemini 2.5 Flash13.085.978.29.021.524.7 GPT-5 Nano32.363.237.120.251.623.0 Grok 41.696.056.939.07.855.0 LLaMA 4 Scout89.310.44.97.792.84.3 These findings confirm that visual appearance is an unreliable forensic signal for most LLM evaluators: three of five models assign nearly identical or only slightly different visual quality scores to AI-generated and authentic receipts. The notable find- ing is that human annotators discriminate visually better than any LLM evaluator in this study, yet still fail to match the best machine detector—a finding explained by the imperceptibility of arithmetic errors, developed in Section VI-E. VI. RESULTS: HUMAN PERCEPTUAL STUDY A. Study Design and Annotation Quality A crowdsourced perceptual study was conducted to establish a human visual baseline for AI-generated receipt detection. The study was administered through a Label Studio annotation platform. Thirty participants were recruited and each was assigned a randomized set of approximately 100 receipt images drawn from GPT4o-Receipt. Annotators were recruited through a crowdsourcing platform; demographic information (age, geographic location, professional background) was not systematically collected as part of this study. All 1,235 images were annotated: 935 AI-generated receipts and all 300 authentic receipts. The study protocol was conducted in accordance with the data provider’s terms of service; participants provided informed consent to participate in the annotation task. Ethical oversight was provided at the institutional level consistent with crowdsourced annotation research. For each image, annotators responded to three structured visual quality questions and assigned an overall score, as summarized in Table IV. Annotators were asked to evaluate visual appearance only. The median annotation time per image was 14 seconds (mean: 24.1 seconds, excluding 15 outlier sessions >300 seconds likely reflecting pauses). To assess annotation reliability, 133 images received 2–3 independent annotations (from different annotators; 124 with two, 9 with three). Binary agreement on the low/high score split (threshold≤ 3= low quality) was 77.4%, and Cohen’sκ was 0.523, indicating moderate inter-rater agreement consistent with established crowdsourced annotation benchmarks [42], [43]. The Pearson correlation between paired score ratings was r = 0.669(p < 0.001). For images with multiple annotations, scores were averaged and categorical responses were resolved by majority vote. Important note on human detection metrics. The human study was designed as a visual quality assessment, not a direct binary classification task. Detection performance metrics (accuracy, F1, recall, FPR) reported in this section are inferred by applying a score threshold: receipts with mean score≤ 3 are classified as AI-generated, and those with score> 3are classified as authentic. This is a post-hoc inference by the researchers, not an explicit judgment made by annotators. This methodological distinction must be borne in mind when comparing human and LLM detection performance: LLM metrics are derived from explicit ‘is_ai_generated‘ binary verdicts, while human metrics are derived from continuous quality ratings. A threshold of≤ 3was chosen as the pre- specified midpoint of the 1–5 scale; the sensitivity of results to this choice is reported in Section VI-D. B. Human Visual Discrimination The mean visual realism score assigned to AI-generated receipts by human annotators is 2.60 (SD = 1.17), compared to 4.47 (SD = 0.89) for authentic receipts—a gap of 1.87 points (95% CI: [1.74, 1.99];t-testp = 3.1× 10 −120 ). This gap is the largest of any evaluator in this study, exceeding the gaps produced by Claude Sonnet 4 (1.38; 95% CI [1.32, 1.44];p < 10 −220 ) and Gemini 2.5 Flash (1.24; 95% CI [1.16, 1.31];p < 10 −70 ). In contrast, GPT-5 Nano exhibits a near- zero gap (−0.02) that is not statistically significant (p = 0.50), while Grok 4 (+0.17;p < 0.05) and LLaMA 4 Scout (+0.09; p < 0.001) show small but significant gaps (Table VI, Figure 5). The per-question breakdown confirms that the human visual signal is distributed across multiple perceptual dimensions. Only 24.3% of AI-generated receipts were rated Yes (Realistic) on typography (Q1), versus 82.7% of authentic receipts; 29.0% of AI receipts received No (Obvious Fake), compared to 5.4% of real receipts. For artifact presence (Q3), 83.8% of authentic receipts were rated Yes (Clean), versus 41.5% of AI-generated receipts. These patterns confirm that human annotators perceive AI-generated receipts as exhibiting inferior typography quality and more visible generation artifacts. C. Human Binary Detection Performance Under the pre-specified threshold (≤ 3= AI-generated), human annotators achieve accuracy 0.797 (95% CI: [0.774, 0.818]), F1 = 0.852 [0.833, 0.869], recall = 0.770 [0.743, 0.797], and FPR = 0.120 [0.085, 0.160] (Table V). These confidence intervals confirm that the human F1 is statistically significantly below Claude Sonnet 4 (F1 = 0.975 [0.967, 0.982]); the TABLE IV ANNOTATION QUESTIONS ADMINISTERED TO EACH HUMAN EVALUATOR FOR EVERY RECEIPT IMAGE. RESPONSES ARE ORDERED FROM MOST AUTHENTIC (LEFT) TO LEAST AUTHENTIC (RIGHT). IDDimensionQuestionResponse options (authentic→ artificial) Q1TypographyDoes the typography look realistic?Yes (Realistic) / Somewhat / No (Obvious Fake) Q2LayoutIs the receipt layout consistent?Yes (Consistent) / Somewhat / No (Broken/Messy) Q3ArtifactsIs the image free of visible artifacts?Yes (Clean) / Somewhat / No (Noticeable Artifacts) VOverallVisual realism score (1–5)1 (clearly artificial)→ 5 (indistinguishable from real) TABLE V DETECTION PERFORMANCE ON GPT4O-RECEIPT. ALL EVALUATORS ASSESSED THE FULL 1,235-IMAGE SET (935 AI + 300 REAL). HUMAN METRICS ARE INFERRED VIA VISUAL SCORE THRESHOLD≤ 3RATHER THAN DIRECT BINARY JUDGMENT (SEE SECTION VI). VALUES ARE POINT ESTIMATES WITH 95% BOOTSTRAP CONFIDENCE INTERVALS. BEST VALUES IN BOLD. EvaluatorAccuracyF1RecallFPR Claude Sonnet 40.962 [.950–.971]0.975 [.967–.982]0.972 [.961–.982]0.070 [.044–.100] Gemini 2.5 Flash0.848 [.826–.867]0.890 [.874–.904]0.807 [.779–.831]0.023 [.009–.044] GPT-5 Nano0.631 [.605–.658]0.685 [.656–.712]0.530 [.497–.560]0.057 [.032–.083] Grok 40.780 [.756–.804]0.873 [.857–.887]0.999 [.996–1.00]0.903 [.867–.935] LLaMA 4 Scout0.326 [.300–.351]0.204 [.172–.240]0.114 [.096–.136]0.017 [.004–.034] Majority-class ‡ 0.757 [—]0.862 [—]1.000 [—]1.000 [—] Human † (n=30)0.797 [.774–.818]0.852 [.833–.869]0.770 [.743–.797]0.120 [.085–.160] † Human metrics are inferred via visual score threshold ≤3→ AI; not from direct binary judgment. Evaluated on all 1,235 images. ‡ Majority-class baseline: always predicts AI-generated. F1 = 2×935/(2×935+0+300) = 0.862. intervals do not overlap. The human F1 is comparable to—but slightly below—the majority-class baseline (0.862); however, the human FPR (0.120) is dramatically lower than the majority baseline (1.000), indicating that human annotators perform genuine visual discrimination rather than defaulting to a constant prediction. Human F1 significantly exceeds GPT-5 Nano (0.685 [0.656, 0.712]) and substantially outperforms LLaMA 4 Scout (0.204) in calibration terms. D. Threshold Sensitivity Because the binary detection decision is inferred via thresh- olding rather than measured directly, we report results across all four integer thresholds (Figure 6). Using threshold≤ 1: Acc = 0.390, F1 = 0.328, Recall = 0.197, FPR = 0.010. Threshold ≤ 2: Acc = 0.595, F1 = 0.644, Recall = 0.485, FPR = 0.060. Threshold≤ 3(paper default): Acc = 0.797, F1 = 0.852, Recall = 0.770, FPR = 0.120. Threshold≤ 4: Acc = 0.873, F1 = 0.917, Recall = 0.931, FPR = 0.307. F1 score peaks at threshold≤ 4(0.917) rather than at the selected threshold of≤ 3(0.852). However, threshold≤ 3was selected a priori as the scale midpoint, not optimized on the data. The threshold≤ 4result demonstrates that if annotators who rated receipts as “4 out of 5” are also included in the “AI-flagged” group, recall reaches 0.931—but at the cost of a higher FPR (0.307). The pre-specified threshold≤ 3provides better calibration for practical use. Importantly, the conclusion that the best LLM detector (Claude Sonnet 4, F1 = 0.975) outperforms human visual detection holds across all threshold choices. E. The Visual–Arithmetic Asymmetry The juxtaposition of two findings—humans have the largest visual discrimination gap (1.87 points) among all evaluators, yet their binary detection performance (F1 = 0.852) falls short of Claude Sonnet 4 (0.975) and Gemini 2.5 Flash (0.890)—reveals a fundamental asymmetry with important forensic implications. The key observation is that arithmetic errors in AI-generated receipts are visually imperceptible. A receipt where itemized prices sum to $24.74 but the stated subtotal reads $24.99 appears completely normal to visual inspection; there is no pixel-level cue distinguishing it from an arithmetically correct receipt. Human annotators, whose task was entirely visual, could not be expected to detect these errors regardless of their visual acuity—and our data confirm this: human annotators achieve high visual sensitivity (gap = 1.87) but can recover only a fraction of the forensic signal available to LLMs that perform explicit arithmetic verification. Arithmetic errors are, by their nature, expressed in the semantic domain rather than the pixel domain. Verifying arithmetic from an image—without OCR or calculator—would require extraordinary effort per image, making it impractical at the scale required for forensic deployment. The present study therefore reveals a genuine and practically important limitation of human visual forensics: visual inspection alone is structurally insufficient for AI receipt detection because the primary forensic signal is arithmetically encoded, not visually encoded. The performance advantage of the best LLM detectors (Claude, Gemini) over human annotators is attributable to their capacity for automated arithmetic cross-checking—a capability that should also be deployable programmatically from OCR 2345 Mean Visual Realism Score (1--5) ± SD LLaMA 4 Scout Grok 4 GPT-5 Nano Gemini 2.5 Flash Claude Sonnet 4 Human (n=30) *** * ns *** *** *** Visual Realism Scores: AI vs. Real Receipts AI-GeneratedReal Fig. 5. Mean visual realism scores (±SD) for AI-generated and real receipts across all evaluators. Significance:∗ p < 0.001;∗ p < 0.05; ns not significant. Humans exhibit the largest AI-vs-real gap (1.87 points, 95% CI [1.74, 1.99]). output, as discussed in Section VII. VII. DISCUSSION A. Human Perceptual Sensitivity vs. LLM Forensic Performance Human annotators exhibit the largest visual discrimination gap of any evaluator (1.87 points on a 1–5 scale), confirming that GPT-4o-generated receipts are perceptually distinguishable from authentic ones. Yet their binary detection F1 (0.852) falls well below Claude Sonnet 4 (0.975) and below Gemini 2.5 Flash (0.890). The reason is that the dominant forensic signal— arithmetic incoherence—is invisible to visual inspection but trivially verifiable by LLMs. The practical implication is that human review should be augmented by automated arithmetic and logical consistency checks rather than treated as sufficient alone; conversely, human review remains most valuable for detecting the visual artifacts that currently escape the weaker LLM detectors (GPT-5 Nano, Grok 4, LLaMA 4 Scout). B. Dataset Coverage and Representativeness GPT4o-Receipt spans 159 merchant categories across seven commercial domains, ensuring evaluation across diverse receipt layouts, item types, and pricing conventions. Authentic receipts <= 1<= 2<= 3<= 4 Threshold (score thresh. AI) 0.0 0.2 0.4 0.6 0.8 1.0 Score Paper thresh. Human Detection: Threshold Sensitivity Accuracy F1 Recall FPR Fig. 6. Human detection performance across visual score thresholds. Paper default (dashed line):≤ 3. F1 peaks at≤ 4but with high FPR;≤ 3gives the best recall–FPR balance. from ExpressExpense [8] and Roboflow Universe [9] provide geographic and stylistic diversity. The class imbalance (75.7% AI, 24.3% real) reflects a forensic evaluation scenario; a majority-class baseline (always predict AI) achieves F1 = 0.862, providing context for interpreting model performance. One boundary condition is that the AI-generated subset uses GPT-4o exclusively, limiting applicability to GPT-4o-specific artifacts; future work should incorporate Stable Diffusion, DALL-E 3, and FLUX, and extend to non-Latin numeral systems and non-Western receipt formats. C. Calibration and Practical Detector Selection The five LLM detectors exhibit qualitatively different cali- bration behaviors that aggregate F1 scores mask. Grok 4’s near- perfect recall comes at the cost of a 90.3% FPR—operationally unusable where false positives impose costs. LLaMA 4 Scout has the lowest FPR (1.7%) but detects only 11.4% of AI receipts. Only Claude Sonnet 4 and Gemini 2.5 Flash achieve both high recall and reasonable FPR, with Claude providing the best overall F1 (0.975) and Gemini offering the lowest FPR among effective detectors (0.023). The practical lesson is that model selection should treat recall and FPR as joint constraints: a high-recall/high-FPR detector suits a first-stage filter; a low-FPR detector suits settings where false accusations carry high cost. No single model provides a universally satisfactory operating point. D. The Role of Forensic Signal Dimensions Only Claude Sonnet 4 and Gemini 2.5 Flash exhibit genuine visual discrimination; the remaining three models produce near-identical or only slightly different visual scores for AI- generated and real receipts, consistent with multimodal LLM benchmarks showing poor fine-grained visual engagement [13], [14]. Arithmetic integrity provides the dominant forensic signal: sum errors are detected in over 85% of AI receipts by both capable detectors (though these are model assessments, not ground truth). This signal reflects a current GPT-4o limitation whose forensic value will diminish as generation improves. The multi-dimensional framework’s lasting value is diagnostic: by decomposing detection into visual, arithmetic, and factual dimensions, it makes explicit which aspects of AI-generated documents remain forensically exploitable across successive model generations. E. Adversarial Robustness and the Forensic Arms Race Because arithmetic failure is near-universal among AI- generated receipts in GPT4o-Receipt—sum errors are detected in 97.2% of AI receipts by Claude and 85.9% by Gemini—it is worth asking how detection performance would change if an adversary closed this gap by pre-validating arithmetic correct- ness before deploying forgeries. To simulate this scenario, we compute a “no-arithmetic” F1 for each model by restricting its positive predictions to those receipts where either the visual realism score is at or below 2 (on the model’s own scale) or the factual consistency check fails, treating arithmetic-only detections as missed. This provides a conservative estimate of the forensic signal each model could maintain without arithmetic. Simulating this adversarial scenario reveals that the top three detectors are surprisingly resilient (Figure 7). Grok 4 retains 98.6% of its baseline F1 (0.873→0.861), Claude Sonnet 4 retains 94.2% (0.975→0.918), and Gemini 2.5 Flash retains 94.5% (0.890→0.840)—all three leverage factual consistency signals (address errors, item–store mismatches) that are independent of arithmetic. In contrast, GPT-5 Nano drops to 75.6% retention (0.685→0.518), and LLaMA 4 Scout collapses to 56.9% (0.204 → 0.116). These results indicate that for the best detectors, arithmetic errors are not the sole forensic channel: factual consistency and visual realism provide complementary signals that survive arithmetic hardening. However, a sophisticated adversary who post-processes generated images to ensure both numerical and factual consistency would pose a substantially greater challenge. The longer-term challenge for the forensics community is to develop classifiers that integrate visual, semantic, and structural signals in ways that are jointly difficult to satisfy, rather than relying on any single dimension of forensic signal. This challenge parallels adversarial robustness problems in other visual AI domains: Shen et al. [44] demonstrate that even low-cost cosmetic interventions can fool age estimation systems, illustrating how minor targeted modifications can defeat detectors that rely on a narrow set of signals. F. Limitations Several limitations should be noted. The AI-generated subset uses GPT-4o exclusively; failure modes may not generalize to Stable Diffusion [1], DALL-E 3, or FLUX. The dataset covers primarily English-language North American and UK receipts, limiting applicability to other numeral systems and receipt formats. LLM forensic assessments are not ground truth—substantial inter-model disagreement in arithmetic pass rates (LLaMA 4 Scout: 89.3% vs. Claude: 2.6%) reflects model calibration differences as much as genuine error rates, and independently verified arithmetic annotations are absent. Human detection metrics are inferred via score thresholding rather than direct binary judgment, and LLM results were obtained in February 2026 via the OpenRouter API and may not be reproducible with future model updates. VIII. CONCLUSION We presented GPT4o-Receipt, a benchmark of 1,235 receipt images for evaluating forensic detection of AI-generated financial documents, pairing 935 GPT-4o-generated receipts with 300 authentic images from established benchmarks. Our five-model LLM evaluation and 30-annotator perceptual study reveal a visual–arithmetic asymmetry: humans exhibit the largest visual discrimination gap of any evaluator (1.87 points on a 1–5 scale) yet achieve lower binary detection F1 (0.852; threshold≤ 3, range 0.328–0.917 across thresholds) than Claude Sonnet 4 (0.975) and Gemini 2.5 Flash (0.890), because the dominant forensic signal—arithmetic incoherence— is imperceptible to visual inspection. The calibration analysis further shows that model selection for forensic deployment must consider recall and false positive rate jointly, not F1 alone; no single current model provides a universally satisfactory operating point. Adversarial robustness analysis reveals that the top detectors maintain>94% of their detection capability even without arithmetic signals, leveraging complementary factual and visual channels. Future work should extend GPT4o-Receipt to additional generative models and non-English receipt formats, and develop detection approaches resilient to adversarially- hardened forgeries that close both the arithmetic and factual gaps exploited by current detectors. REFERENCES [1]R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, p. 10 684–10 695. [2] OpenAI, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2024. [3]T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, p. 1877–1901, 2020. [4]S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “CNN- generated images are surprisingly easy to spot. . . for now,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, p. 8695–8704. [5]T. Dzanic, K. Shah, and F. Witherden, “Fourier spectrum discrepancies in deep network generated images,” Advances in Neural Information Processing Systems, vol. 33, p. 3022–3032, 2020. [6] S. Jia, R. Lyu, K. Zhao, Y. Chen, Z. Yan, Y. Ju, C. Hu, X. Li, B. Wu, and S. Lyu, “Can ChatGPT detect DeepFakes? a study of using multimodal large language models for media forensics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024, p. 4324–4333. [7]Z. Xu, X. Zhang, R. Li, Z. Tang, Q. Huang, and J. Zhang, “FakeShield: Explainable image forgery detection and localization via multi-modal large language models,” in International Conference on Learning Representations (ICLR), 2025. Claude Sonnet 4Gemini 2.5 FlashGrok 4GPT-5 NanoLLaMA 4 Scout 0.0 0.2 0.4 0.6 0.8 1.0 F1 Score -0.057 -0.050 -0.012 -0.167 -0.088 Detection F1: Baseline vs. Arithmetically Hardened Forgeries Baseline F1 Hardened F1 0102030405060 F1 Performance Lost (%) LLaMA 4 Scout GPT-5 Nano Grok 4 Gemini 2.5 Flash Claude Sonnet 4 43.1% 24.4% 1.4% 5.5% 5.8% Detection Capability Lost After Arithmetic Hardening Adversarial Robustness: Impact of Arithmetic Hardening on LLM Detection Fig. 7. Impact of arithmetic hardening on LLM detection performance. Left: Baseline F1 vs. estimated F1 when arithmetic verification is unavailable (i.e., forgeries have been pre-validated for arithmetic correctness). Right: Percentage of detection capability (F1) lost after arithmetic hardening. Claude Sonnet 4, Gemini 2.5 Flash, and Grok 4 retain>94% of their baseline F1, indicating reliance on multiple forensic channels beyond arithmetic; GPT-5 Nano and LLaMA 4 Scout are more dependent on arithmetic signals. [8]ExpressExpense,“FreereceiptimagesforOCRand machinelearningdataset,”https://expressexpense.com/blog/ free-receipt-images-ocr-machine-learning-dataset/,2024,accessed: February 2026. [9]Jakob, “Receipt or invoice dataset (v5),” https://universe.roboflow.com/ jakob-awn1e/receipt-or-invoice/dataset/5, 2024, accessed: February 2026. [10]J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems, vol. 33, p. 6840– 6851, 2020. [11]I. Kaji ́ c, O. Wiles, I. Albuquerque, M. Bauer, S. Wang, J. Pont-Tuset, and A. Nematzadeh, “Evaluating numerical reasoning in text-to-image models,” in Advances in Neural Information Processing Systems, vol. 37, 2024. [12] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232, 2023. [13]T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou, “HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, p. 14 375–14 385. [14]T. Weng et al., “VisNumBench: Evaluating number sense of multimodal large language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. [15]J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, and T. Holz, “Leveraging frequency analysis for deep fake image recognition,” in Proceedings of the 37th International Conference on Machine Learning (ICML), 2020, p. 3247–3258. [16]F. Guillaro, D. Cozzolino, A. Sud, N. Dufour, and L. Verdoliva, “TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p. 20 606–20 615. [17]C. Qu, C. Liu, Y. Liu, X. Chen, D. Peng, F. Guo, and L. Jin, “Towards robust tampered text detection in document image: New dataset and new solution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p. 4551–4561. [18]D. Luo, Y. Zhou, Y. Liu, E. Zhang, B. Yang, Z. Huang, and X. Bai, “ICDAR 2023 competition on detecting tampered text in images,” in Document Analysis and Recognition – ICDAR 2023, ser. Lecture Notes in Computer Science, vol. 14188. Springer, 2023. [19]C. Qu, Y. Zhong, F. Guo, and L. Jin, “Revisiting tampered scene text detection in the era of generative AI,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 1, 2025, p. 694–702. [20]J. Duan, H. Sun, F. Ji, K. Zhou, Z. Wang, H. Huang, and L. Jin, “RealDTT: Towards a comprehensive real-world dataset for tampered text detection,” International Journal of Computer Vision, vol. 133, no. 10, p. 6993– 7011, 2025. [21]R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, and L. Verdoliva, “On the detection of synthetic images generated by diffusion models,” arXiv preprint arXiv:2211.00680, 2023. [22] J. Wu, Y. Zhou, M. Xu, Z. Liang, S. Ren, J. Xue, M. Yang, S. Chen, and J. Huan, “AIForge-Doc: A benchmark for detecting AI-forged tampering in financial and form documents,” arXiv preprint arXiv:2602.20569, 2026. [23]M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local Nash equilibrium,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [24] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, p. 586–595. [25] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi, “CLIPScore: A reference-free evaluation metric for image captioning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, p. 7514–7528. [26]M. Zhu, H. Chen, Q. Yan, X. Huang, G. Lin, W. Li, Z. Tu, H. Hu, J. Hu, and Y. Wang, “GenImage: A million-scale benchmark for detecting AI-generated images,” in Advances in Neural Information Processing Systems, vol. 36, 2023. [27]Z. Wang, J. Bao, W. Zhou, W. Wang, H. Hu, H. Chen, and H. Li, “DIRE for diffusion-generated image detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [28]U. Ojha, Y. Li, and Y. J. Lee, “Towards universal fake image detection exploiting CLIP,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p. 24 205– 24 214. [29]S. Ren, H. Xu, T. Ng, K. Zewde, S. Jiang, R. Desai, D. Patil, N.-Y. Cheng, Y. Zhou, and R. Muthukrishnan, “Do deepfake detectors work in reality?” arXiv preprint arXiv:2502.10920, 2025. [30]S. Ren, Y. Zhou, X. Shen, K. Zewde, T. Duong, G. Huang, N. H. Tiangratanakul, D. T. Ng, E. Wei, and J. Xue, “How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study,” arXiv preprint arXiv:2602.07814, 2026. [31]S. Ren, X. Shen, A. Raj, A. Dai, C. Zhang, Y. Xu, Z. Chen, S. Wu, C. Gong, and Y. Zhang, “Out of the box age estimation through facial imagery: A comprehensive benchmark of vision-language models vs. out- of-the-box traditional architectures,” arXiv preprint arXiv:2602.07815, 2026. [32]S. Ren, F. Luzi, S. Lahrichi, K. Kassaw, L. M. Collins, K. Bradbury, and J. M. Malof, “Segment anything, from space?” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, p. 8355–8365. [33]B. M. Tornés, T. Taburet, E. Boros, K. Rouis, P. Gomez-Krämer, N. Sidère, A. Doucet, and V. Poulain d’Andecy, “Receipt dataset for document forgery detection,” in Document Analysis and Recognition – ICDAR 2023, ser. Lecture Notes in Computer Science, vol. 14187. Springer, 2023. [34]V. Schmidberger, M. Eberhardinger, S. Maghsudi, and J. Maucher, “Generation of programmatic rules for document forgery detection using large language models,” arXiv preprint arXiv:2512.19228, 2024. [35] Q. Wang, J. Yang, Z. Lin, Z. Ying, W. Wang, and C. Lin, “Innovative image fraud detection with cross-sample anomaly analysis: The power of LLMs,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025. [36] D. Cooke et al., “As good as a coin toss: Human detection of AI-generated images, videos, audio, and audiovisual stimuli,” Communications of the ACM, vol. 68, no. 10, 2025. [37] S. J. Nightingale and H. Farid, “AI-synthesized faces are indistinguishable from real faces and more trustworthy,” Proceedings of the National Academy of Sciences, vol. 119, no. 8, p. e2120481119, 2022. [38]E. J. Miller, B. A. Steward, Z. Witkower, C. A. M. Sutherland, E. G. Krumhuber, and A. Dawel, “AI hyperrealism: Why AI faces are perceived as more real than human ones,” Psychological Science, vol. 34, no. 12, p. 1390–1403, 2023. [39]S. Ren, Y. Yao, K. Zewde, Z. Liang, D. T. Ng, N.-Y. Cheng, X. Zhan, Q. Liu, Y. Chen, and H. Xu, “Can multi-modal (reasoning) LLMs work as deepfake detectors?” arXiv preprint arXiv:2503.20084, 2025. [40]Z. Liang, K. Zewde, R. P. Singh, D. Patil, Z. Chen, J. Xue, Y. Yao, Y. Chen, Q. Liu, and S. Ren, “Can multi-modal (reasoning) LLMs detect document manipulation?” arXiv preprint arXiv:2508.11021, 2025. [41]Y. He, Y. Cao, B. Yang, and Z. Zhang, “Can GPT tell us why these images are synthesized? Empowering multimodal large language models for forensics,” in Proceedings of the ACM Workshop on Information Hiding and Multimedia Security, 2025. [42]R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng, “Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks,” in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2008, p. 254–263. [43]R. J. Passonneau and B. Carpenter, “The benefits of a model of annotation,” Transactions of the Association for Computational Linguistics, vol. 2, p. 311–326, 2014. [44]X. Shen, T. Duong, X. An, Z. Zhao, Z. Hu, H. Hu, Z. Wang, F. Guo, and S. Ren, “Can a teenager fool an AI? Evaluating low-cost cosmetic attacks on age estimation systems,” arXiv preprint arXiv:2602.19539, 2026. APPENDIX TABLE VI MEAN VISUAL REALISM SCORES (1–5) ASSIGNED TO AI-GENERATED VS. REAL RECEIPTS BY EACH LLM AND BY HUMAN ANNOTATORS. GAP = REAL −AI (95% CI IN BRACKETS). SIGNIFICANCE:∗ p < 0.001;∗ p < 0.05; ns NOT SIGNIFICANT (TWO-SAMPLE t-TEST). EvaluatorAIRealGap [95% CI]Sig. Claude Sonnet 42.513.89+1.38 [1.32, 1.44] ∗ Gemini 2.5 Flash3.544.78+1.24 [1.16, 1.31] ∗ GPT-5 Nano4.013.99 −0.02 [−0.08, 0.03]ns Grok 43.283.45+0.17 [0.07, 0.27] ∗ LLaMA 4 Scout3.994.07+0.09 [0.04, 0.14] ∗ Human (n=30)2.604.47+1.87 [1.74, 1.99] ∗ The adversarial hardening simulation restricts positive pre- dictions to images whose visual realism score does not exceed a thresholdT, i.e., the model believes the image looks AI- generated. The main paper usesT ≤ 2as the primary visual realism threshold (combined with factual consistency). Table VII repeats the no-arithmetic F1 analysis at bothT ≤ 2 andT ≤ 3(using visual realism score alone, without factual consistency) to show that the conclusions are robust to this choice. No-Arith F1 is the F1 score computed only over the subset of images that fall at or below the threshold; images scored above the threshold are excluded (the model “abstains” on high- realism images under adversarial hardening). Recall reports the percentage of all 935 AI-generated receipts that fall within the selected subset, i.e., how many forgeries the model would even attempt to classify under this regime. TABLE VII NO-ARITHMETIC F1 AND AI-IMAGE COVERAGE (RECALL) FOR EACH LLM EVALUATOR AT VISUAL REALISM THRESHOLDST ≤ 2ANDT ≤ 3. RECALL = FRACTION OF ALL 935 AI RECEIPTS WHOSE VISUAL REALISM SCORE ≤ T FOR THAT MODEL; F1 IS COMPUTED OVER THAT SUBSET ONLY. POSITIVE CLASS = AI-GENERATED RECEIPT. T ≤2T ≤3 ModelF1RecallF1Recall Claude Sonnet 40.98751.8%0.98997.2% Gemini 2.5 Flash0.0000.0%0.99462.2% GPT-5 Nano0.8890.9%0.91712.8% Grok 40.86213.0%0.88558.4% LLaMA 4 Scout1.0001.2%0.97610.9% AtT ≤ 2, most models flag very few AI images (low recall), so the F1 score is computed over a narrow, highly confident subset. AtT ≤ 3, the subset broadens substantially; Claude Sonnet 4 captures 97.2% of AI receipts at near-perfect F1, confirming its strong visual discrimination. Gemini 2.5 Flash jumps from 0% coverage atT ≤ 2to 62.2% atT ≤ 3, reflecting its score distribution concentrated near 3–4. The pattern confirms that the paper’s choice ofT ≤ 2is conservative but not uniquely determinative: raising the threshold shifts the coverage–precision trade-off without reversing the qualitative ranking of models.

Instant research discovery